fix(detectors): skip HF detectors when models are unreachable - #2138
fix(detectors): skip HF detectors when models are unreachable#2138rwinkelman wants to merge 2 commits into
Conversation
…#1033) When graceful_fail is enabled (default for HFDetector), Hugging Face model load failures mark the detector skipped instead of aborting the run. Signed-off-by: Ray Winkelman <rwinkelman@users.noreply.github.com>
0b47b4e to
15561ab
Compare
…le (NVIDIA#1033) Signed-off-by: Ray Winkelman <rwinkelman@users.noreply.github.com>
|
Extended #1033: LocalHFTranslator now falls back to passthru when HF translation models cannot be loaded offline. |
jmartin-tech
left a comment
There was a problem hiding this comment.
The scope here changes behavior I am not sure it should.
Can you provide a scenario where an HFDetector load causes a crash on main today after inference for a probe has completed?
The change to the local LangProvider makes an explicitly invalid configuration suppress raising the issue early in the run and would execute inference without a clear indication to the user that the run did not execute in the requested way as the report would still contain probes and detector results as if translation had occurred which cannot be easily differentiated if only the html report were forwarded as proof of the run.
I suspect #1736, approaches the core ask in the issue more closely and accounts for places where a late loaded detector may terminate a run while still resulting in a clear indication in the output that the run deviated from the requested actions since it would suppress entire probes inference phase when no detectors can be instantiated to process the inference results.
There was a problem hiding this comment.
LangProvider is a core service the user must specifically enable via configuration. The change in behavior here is likely not appropriate.
In this case failure to load the model requested should continue to fail the run early before spending token budgets on inference that will not match what the user requested via explicit configuration parameters.
There was a problem hiding this comment.
This seems reasonable, since detectors are often lazy loaded terminating the run is not the preferred action, however I question if this is actually needed, probewise.py guards for detectors that fail to init as does pxd.py which loads all explicitly called detectors before starting inference.
Summary
graceful_failis enabled (default forHFDetector), Hugging Face model load failures no longer terminate the whole garak runskip=Trueand log a warning insteadFixes #1033 (scoped to HF detector init; other online-resource paths may follow separately).
Test plan
AutoConfig.from_pretrainedfailure and assertsskip+[None]detect scoresSigned-off-by: Ray Winkelman rwinkelman@users.noreply.github.com