feat(evaluators): add hit@k attack success rate metric - #2090
feat(evaluators): add hit@k attack success rate metric#2090stefanoamorelli wants to merge 2 commits into
Conversation
Pooled ASR reports the fraction of all generations that breach the target, which understates risk: an attacker only needs one success and can retry. pass@k reframes this per prompt -- "given k attempts, does at least one breach?" -- and averages across prompts, using the unbiased estimator from Chen et al. (2021, arXiv:2107.03374). The evaluator computes per-prompt (scoreable, hits) counts, estimates pass@k per (probe, detector), records a pass_at_k field in the eval report entry, and prints it on the CLI attack-success-rate line. The k values are set via reporting.pass_at_k (default [1, 5]; empty disables). Signed-off-by: Stefano Amorelli <stefano@amorelli.tech> Assisted-by: AI
cde5451 to
7f4cbd4
Compare
jmartin-tech
left a comment
jmartin-tech
left a comment
There was a problem hiding this comment.
Awesome contribution, the maintainer team took a bit of time working thru what this metric means and this scoring pattern seems like it would surface a better understanding of the run results.
One item I see missing is that this only incorporates these values for terminal output. The most common artifact reviewed is understood to be the report.html. This PR's scope is not required to expand to expose this in that space however it should likely expand to define how and where in the final digest object written to report.jsonl this might be able to provide the same values being presented on stdout.
Further confirmation of how the scores function is stil ongoing but the overall feedback should enable iteration while that review is completed.
|
|
||
| These intervals account for sampling uncertainty. When detector performance metrics (sensitivity/specificity) are available, they also account for detector imperfection. Otherwise, a perfect detector is assumed. | ||
|
|
||
| pass@k Attack Success Rate |
There was a problem hiding this comment.
Given that terminology in garak, while I understand this diverges from the paper the metric here is probably more clear as hit@k for the following reasons:
detector.always.Passrefers maps to scoring as0.0- the detector base class refers to a successful result as a
hitand apasshas opposite semantics - Detections for closer evaluation are logged in
hitlog.jsonlfurther aligning this product's terminology.
There was a problem hiding this comment.
makes sense 👍 i've renamed here (I still kept the reference in comments to the original paper as generally people might know the original term).
| bootstrap_confidence_level: 0.95 | ||
| bootstrap_min_sample_size: 30 No newline at end of file | ||
| bootstrap_min_sample_size: 30 | ||
| pass_at_k: [1, 5] |
There was a problem hiding this comment.
As noted in another comment garak's jargon would align better as hit_at_k.
These seem like reasonable values, and the 5 here matches to the default generations value. Do you think there is value in having this configuration be for expanded coverage instead of all values to be used?
Consider that a default behavior where the number of generations the run was launched for would always include the hit@k where k == generations or more simply the for an Attempt we could always report at least k = len(self.outputs). Then augment that set with any additional values in reporting.hit_at_k that are less than the number outputs in the Attempt.
I could see some probes this might not align to generations such as atkgen.Tox though if the upper bound on k is based on len(Attempt.outputs) that might turn out to be a non-issue.
With the expectation that this is the number of inference generations aligned to at least one hit identified this metric seem very helpful.
There was a problem hiding this comment.
Do you think there is value in having this configuration be for expanded coverage instead of all values to be used?
Consider that a default behavior where the number of generations the run was launched for would always include the hit@k where k == generations or more simply the for an Attempt we could always report at least k = len(self.outputs). Then augment that set with any additional values in reporting.hit_at_k that are less than the number outputs in the Attempt.
this is actually a very good point, I'm aligned with expanded coverage (drafted here lmk wdyt).
I could see some probes this might not align to generations such as atkgen.Tox though if the upper bound on k is based on len(Attempt.outputs) that might turn out to be a non-issue.
I had a look a bit deeper on atkgen.Tox and we should indeed be covered (in the sense that upper bound on k is based on the count of scoreable outputs for each attempt).
However, in this case I think the implications can become more nuanced, hence we could also introduce an additional score.
let's say prompt A has 1 hit in 5 generations, and prompt B has 0 hits in 3 generations. report could be:
"hit_at_k": {"1": {"score": 0.1, "prompts": 2}, "n": {"score": 0.5, "prompts": 2}}
so the "n" key shows that half the prompts were breached (at least) once, and 0.1 is calculated as follows:
prompt A1 hit among 5 generations,1/5=0.2Prompt B0 hits among 3 generations,0/3=0
And the average across prompts with equal weight is (0.2 + 0.0) / 2 = 0.1
IMOH the importance of n is that it shows how effective is "retrying" in increasing the odds of a successful attack.
lmk your thoughts, your feedback and ideas are really appreciated as always! 🙇
Following review, I renamed the metric to hit@k, since garak scores an attack success as a hit. It still uses the estimator from Chen et al. (2021), https://arxiv.org/abs/2107.03374, with the naming divergence noted. I also score every prompt at k equal to its own generation count, where the estimator collapses to "was this prompt ever breached", so a run reports hit@generations without configuration. reporting.hit_at_k (default [1]) adds coverage below that, null disables the metric, and prompts with unequal counts pool into a bucket keyed n. The scores now also land on each detector entry in the report digest. Signed-off-by: Stefano Amorelli <stefano@amorelli.tech> Assisted-by: AI
stefanoamorelli
changed the title
really appreciate it @jmartin-tech 🙇 aligned, I should have addressed all your initial comments. TIL about the HTML artifact being the most common, can link it easily in a follow up once we clarify the pending points. |
2 participants
Adds hit@k for attack success rate (implements #1911).
Pooled ASR reports the fraction of all generations that breach the target, which understates real risk: a jailbreak that lands on 1 reply in 5 is a working jailbreak, yet pooled ASR records it as 20%. hit@k instead asks per prompt "given k attempts, does at least one breach the target?" and averages across prompts, following the pass@k estimator of Chen et al. (2021). It is named hit@k because garak scores an attack success as a hit.
Example, 5 prompts, each breached on 1 of 5 generations:
Every prompt is always scored at k equal to the number of generations it actually got, where the estimator collapses to "was this prompt breached at least once" (prompts with unequal generation counts share an
"n"bucket).reporting.hit_at_k(default[1]) adds further k values below that, andnulldisables the metric entirely.A
hit_at_kfield is written in theevalreport entry, copied into each detector's entry in the report digest next tototal_evaluatedandpassed, and printed on the CLI.This is an additive change and the existing pooled ASR remains unchanged.
Config examples
Default
[1], on a run withrun.generations: 5(5 prompts, each breached on 1 of 5 generations):More k values, on a run with
run.generations: 10(5 prompts, each breached on 2 of 10 generations):Disabled: the CLI suffix is dropped and no
hit_at_kfield is written.Tests
I used claude code to help me with this. I reviewed every line and take responsibility for the change.