Skip to content

fix(detectors): skip HF detectors when models are unreachable - #2138

Open
rwinkelman wants to merge 2 commits into
NVIDIA:mainfrom
rwinkelman:fix/hf-detector-offline-graceful-1033
Open

fix(detectors): skip HF detectors when models are unreachable#2138
rwinkelman wants to merge 2 commits into
NVIDIA:mainfrom
rwinkelman:fix/hf-detector-offline-graceful-1033

Conversation

@rwinkelman

Copy link
Copy Markdown

Summary

  • When graceful_fail is enabled (default for HFDetector), Hugging Face model load failures no longer terminate the whole garak run
  • Unreachable or offline environments mark the detector as skip=True and log a warning instead

Fixes #1033 (scoped to HF detector init; other online-resource paths may follow separately).

Test plan

  • Unit test mocks AutoConfig.from_pretrained failure and asserts skip + [None] detect scores
  • CI

Signed-off-by: Ray Winkelman rwinkelman@users.noreply.github.com

…#1033)

When graceful_fail is enabled (default for HFDetector), Hugging Face model
load failures mark the detector skipped instead of aborting the run.

Signed-off-by: Ray Winkelman <rwinkelman@users.noreply.github.com>
@rwinkelman
rwinkelman force-pushed the fix/hf-detector-offline-graceful-1033 branch from 0b47b4e to 15561ab Compare September 1, 2026 08:14
…le (NVIDIA#1033)

Signed-off-by: Ray Winkelman <rwinkelman@users.noreply.github.com>
@rwinkelman

Copy link
Copy Markdown
Author

Extended #1033: LocalHFTranslator now falls back to passthru when HF translation models cannot be loaded offline.

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scope here changes behavior I am not sure it should.

Can you provide a scenario where an HFDetector load causes a crash on main today after inference for a probe has completed?

The change to the local LangProvider makes an explicitly invalid configuration suppress raising the issue early in the run and would execute inference without a clear indication to the user that the run did not execute in the requested way as the report would still contain probes and detector results as if translation had occurred which cannot be easily differentiated if only the html report were forwarded as proof of the run.

I suspect #1736, approaches the core ask in the issue more closely and accounts for places where a late loaded detector may terminate a run while still resulting in a clear indication in the output that the run deviated from the requested actions since it would suppress entire probes inference phase when no detectors can be instantiated to process the inference results.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LangProvider is a core service the user must specifically enable via configuration. The change in behavior here is likely not appropriate.

In this case failure to load the model requested should continue to fail the run early before spending token budgets on inference that will not match what the user requested via explicit configuration parameters.

Comment thread garak/detectors/base.py

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable, since detectors are often lazy loaded terminating the run is not the preferred action, however I question if this is actually needed, probewise.py guards for detectors that fail to init as does pxd.py which loads all explicitly called detectors before starting inference.

@jmartin-tech jmartin-tech changed the title fix(detectors): skip HF detectors when models are unreachable (#1033) fix(detectors): skip HF detectors when models are unreachable Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fall back when online services can't be reached.

2 participants