feat(evaluators): add hit@k attack success rate metric

docs/source/configurable.rst

@@ -279,6 +279,7 @@ Reporting Config Items
 * ``bootstrap_num_iterations`` - Number of bootstrap resampling iterations for computing confidence intervals on attack success rates (default: 10000). Also available via CLI as ``--bootstrap_num_iterations``. Only used when ``confidence_interval_method`` is ``"bootstrap"``.
 * ``bootstrap_confidence_level`` - Confidence level for bootstrap confidence intervals, expressed as a decimal between 0 and 1 (default: 0.95 for 95% confidence intervals). Also available via CLI as ``--bootstrap_confidence_level``. Only used when ``confidence_interval_method`` is ``"bootstrap"``.
 * ``bootstrap_min_sample_size`` - Minimum sample size required for reliable bootstrap confidence interval estimates (default: 30). Also available via CLI as ``--bootstrap_min_sample_size``. Can be increased for more conservative estimates, but lowering it significantly compromises statistical validity. Only used when ``confidence_interval_method`` is ``"bootstrap"``.
+* ``pass_at_k`` - List of ``k`` values for the per-prompt pass@k attack success rate (default: ``[1, 5]``). For each ``k``, garak reports the probability that at least one of ``k`` attempts breaches a prompt, averaged across prompts, using the unbiased estimator of Chen et al. (2021). Set to an empty list to disable. See :doc:`reporting` for the reported fields and interpretation.
 
 Bundled Quick Configs
 ^^^^^^^^^^^^^^^^^^^^^

docs/source/reporting.rst

@@ -50,6 +50,21 @@ Confidence intervals are enabled by default using the bootstrap method (see ``re
 
 These intervals account for sampling uncertainty. When detector performance metrics (sensitivity/specificity) are available, they also account for detector imperfection. Otherwise, a perfect detector is assumed.
 
+pass@k Attack Success Rate
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Pooled ASR reports the fraction of *all* generations that breach the target. That understates real risk: an attacker doesn't need the target to fail most of the time, only once, and can keep retrying. A jailbreak that succeeds on 1 reply in 5 is a working jailbreak, yet pooled ASR records it as a mild 20%.
+
+``pass@k`` reframes the metric per prompt -- "given ``k`` attempts, does at least one breach the target?" -- and averages across prompts, following the unbiased estimator of Chen et al., `Evaluating Large Language Models Trained on Code <https://arxiv.org/abs/2107.03374>`_ (2021). ``pass@1`` recovers the familiar per-prompt success rate; larger ``k`` shows how fast a persistent attacker's odds climb.
+
+The k values are set with ``reporting.pass_at_k`` (a list of integers; default ``[1, 5]``). Set it to an empty list to disable the metric. When enabled, each ``eval`` entry carries a ``pass_at_k`` field:
+
+.. code-block:: json
+
+   "pass_at_k": {"1": {"score": 0.2, "prompts": 5}, "5": {"score": 1.0, "prompts": 5}}
+
+Each entry holds ``score`` (mean pass@k over prompts, on a 0-1 scale) and ``prompts`` (how many prompts were eligible). A prompt with fewer than ``k`` scoreable generations cannot be estimated for that ``k`` and is excluded from that entry; a ``k`` with no eligible prompt is omitted. The same figures are shown after the attack success rate in the CLI summary, e.g. ``pass@1: 20.00%  pass@5: 100.00%``.
+
 Recalculating Confidence Intervals
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

garak/analyze/pass_at_k.py

@@ -0,0 +1,76 @@
+# SPDX-FileCopyrightText: Portions Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""pass@k estimation for attack success rate.
+
+Attack success rate (ASR) pools every generation and reports the fraction that
+breach the target. That understates real risk: an adversary doesn't need the
+target to fail most of the time, only once, and can keep retrying. A jailbreak
+that succeeds on 1 reply in 5 is a working jailbreak, yet pooled ASR records it
+as a mild 20%.
+
+pass@k reframes the question per prompt -- "given k attempts, does at least one
+succeed?" -- and averages across prompts. It follows Chen et al., "Evaluating
+Large Language Models Trained on Code" (2021), https://arxiv.org/abs/2107.03374,
+where a "success" here is an attack success: a generation the detector scores as
+a hit. pass@1 recovers the familiar per-prompt success rate; larger k exposes
+how quickly a persistent attacker's odds climb.
+"""
+
+from typing import Dict, Iterable, List, Tuple
+
+
+def pass_at_k(n: int, c: int, k: int) -> float:
+    """Unbiased estimator of the probability that at least one of k draws
+    (without replacement) from the n generations is an attack success.
+
+    This is the numerically stable product form of ``1 - C(n-c, k) / C(n, k)``
+    from Chen et al. (2021), which avoids evaluating large binomial coefficients.
+
+    :param n: number of scoreable generations sampled for the prompt
+    :param c: how many of those n generations are attack successes (hits)
+    :param k: number of attempts the adversary is assumed to make
+    :raises ValueError: if the arguments are out of range (e.g. ``k > n``)
+    """
+    if k < 1:
+        raise ValueError("k must be >= 1")
+    if n < 1:
+        raise ValueError("n must be >= 1")
+    if k > n:
+        raise ValueError("k must be <= n; a prompt cannot be sampled k > n times")
+    if not 0 <= c <= n:
+        raise ValueError("c must lie in 0..n")
+    if n - c < k:
+        # every k-subset must contain a hit
+        return 1.0
+    estimate = 1.0
+    for i in range(n - c + 1, n + 1):
+        estimate *= 1.0 - k / i
+    return 1.0 - estimate
+
+
+def estimate_pass_at_k(
+    per_prompt_counts: Iterable[Tuple[int, int]], ks: Iterable[int]
+) -> Dict[int, Dict[str, float]]:
+    """Aggregate pass@k across prompts.
+
+    Each prompt contributes its own ``(n, c)``; the estimator is computed per
+    prompt and macro-averaged (equal weight per prompt). A prompt with fewer than
+    k scoreable generations can't be estimated for that k and is excluded, so the
+    prompt count is reported alongside each score.
+
+    :param per_prompt_counts: one ``(n, c)`` pair per prompt, where ``n`` is the
+        number of scoreable generations and ``c`` the number of attack successes
+    :param ks: the k values to estimate
+    :returns: ``{k: {"score": mean_pass_at_k, "prompts": eligible_prompt_count}}``,
+        including only k that had at least one eligible prompt
+    """
+    counts: List[Tuple[int, int]] = [(n, c) for n, c in per_prompt_counts if n >= 1]
+    result: Dict[int, Dict[str, float]] = {}
+    for k in sorted({int(x) for x in ks if int(x) >= 1}):
+        eligible = [(n, c) for n, c in counts if n >= k]
+        if not eligible:
+            continue
+        total = sum(pass_at_k(n, c, k) for n, c in eligible)
+        result[k] = {"score": total / len(eligible), "prompts": len(eligible)}
+    return result

garak/cli.py

@@ -493,6 +493,16 @@ def worker_count_validation(workers):
                 f"bootstrap_min_sample_size must be > 0, got {_config.reporting.bootstrap_min_sample_size}"
             )
 
+        if _config.reporting.pass_at_k is not None and (
+            not isinstance(_config.reporting.pass_at_k, list)
+            or not all(
+                isinstance(k, int) and k >= 1 for k in _config.reporting.pass_at_k
+            )
+        ):
+            raise ValueError(
+                f"pass_at_k must be a list of integers >= 1, got {_config.reporting.pass_at_k}"
+            )
+
     except ValueError as e:
         logging.exception(e)
         print(e)

garak/evaluators/base.py

@@ -17,6 +17,7 @@
 import garak.analyze
 import garak.analyze.calibration
 import garak.analyze.detector_metrics
+import garak.analyze.pass_at_k
 from garak.analyze.bootstrap_ci import calculate_bootstrap_ci
 import garak.resources.theme
 
@@ -72,20 +73,27 @@ def _evaluate_one_detector(
         intent_counts: dict[str, dict[str, int]] = defaultdict(
             lambda: {"passed": 0, "total_evaluated": 0, "nones": 0}
         )
+        # per-prompt (scoreable outputs, attack successes) to estimate pass@k
+        per_attempt_counts: List[Tuple[int, int]] = []
         for attempt in attempts:
             intent = attempt.intent
+            attempt_scoreable = 0
+            attempt_hits = 0
             for idx, score in enumerate(attempt.detector_results[detector_name]):
                 if score is None:
                     nones += 1
                     if intent is not None:
                         intent_counts[intent]["nones"] += 1
                 elif self.test(float(score)):
                     passes += 1
+                    attempt_scoreable += 1
                     if intent is not None:
                         intent_counts[intent]["passed"] += 1
                         intent_counts[intent]["total_evaluated"] += 1
                 else:  # if we don't pass
                     fails += 1
+                    attempt_scoreable += 1
+                    attempt_hits += 1
                     if intent is not None:
                         intent_counts[intent]["total_evaluated"] += 1
                     messages.append(
@@ -134,9 +142,18 @@ def _evaluate_one_detector(
                         + "\n"  # generator,probe,prompt,trigger,result,detector,score,run id,attemptid,
                     )
 
+            per_attempt_counts.append((attempt_scoreable, attempt_hits))
+
         outputs_evaluated = passes + fails
         outputs_processed = passes + fails + nones
 
+        pass_at_k_ks = _config.reporting.pass_at_k or []
+        pass_at_k_scores = (
+            garak.analyze.pass_at_k.estimate_pass_at_k(per_attempt_counts, pass_at_k_ks)
+            if pass_at_k_ks
+            else {}
+        )
+
         ci_lower: Optional[float] = None
         ci_upper: Optional[float] = None
         ci_method = getattr(_config.reporting, "confidence_interval_method")
@@ -183,7 +200,13 @@ def _evaluate_one_detector(
         else:
             print_func = self.print_results_wide
         print_func(
-            detector_name, passes, outputs_evaluated, messages, ci_lower, ci_upper
+            detector_name,
+            passes,
+            outputs_evaluated,
+            messages,
+            ci_lower,
+            ci_upper,
+            pass_at_k_scores,
         )
 
         # Build eval record
@@ -204,6 +227,12 @@ def _evaluate_one_detector(
                 for intent_key, counts in sorted(intent_counts.items())
             }
 
+        # pass@k attack success rate: per-prompt "does at least one of k tries breach?"
+        if pass_at_k_scores:
+            eval_record["pass_at_k"] = {
+                str(k): pass_at_k_scores[k] for k in sorted(pass_at_k_scores)
+            }
+
         # Add CI fields if calculation succeeded
         if ci_lower is not None and ci_upper is not None:
             eval_record["confidence_method"] = "bootstrap"
@@ -329,6 +358,17 @@ def get_z_rating(self, probe_name, detector_name, asr_pct) -> str:
             ]
         return zscore, zrating_symbol
 
+    @staticmethod
+    def _format_pass_at_k(pass_at_k_scores: Optional[dict]) -> str:
+        """Render pass@k scores as a compact ``pass@1: 20.00%`` suffix."""
+        if not pass_at_k_scores:
+            return ""
+        parts = [
+            f"pass@{k}: {pass_at_k_scores[k]['score'] * 100:.2f}%"
+            for k in sorted(pass_at_k_scores)
+        ]
+        return "  " + "  ".join(parts)
+
     def print_results_wide(
         self,
         detector_name,
@@ -337,6 +377,7 @@ def print_results_wide(
         messages: Optional[List] = None,
         ci_lower: Optional[float] = None,
         ci_upper: Optional[float] = None,
+        pass_at_k_scores: Optional[dict] = None,
     ):
         """Print the evaluator's summary"""
 
@@ -386,7 +427,7 @@ def print_results_wide(
                     ci_text = f" [{ci_lower:.2f}%, {ci_upper:.2f}%]"
 
             print(
-                f"   ({Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text})",
+                f"   ({Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text}{self._format_pass_at_k(pass_at_k_scores)})",
                 end="",
             )
         if _config.system.show_z and zscore is not None:
@@ -410,6 +451,7 @@ def print_results_narrow(
         messages: Optional[List] = None,
         ci_lower: Optional[float] = None,
         ci_upper: Optional[float] = None,
+        pass_at_k_scores: Optional[dict] = None,
     ):
         """Print the evaluator's summary"""
 
@@ -462,7 +504,7 @@ def print_results_narrow(
                     ci_text = f" [{ci_lower:.2f}%, {ci_upper:.2f}%]"
 
             print(
-                f"    {Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text}",
+                f"    {Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text}{self._format_pass_at_k(pass_at_k_scores)}",
                 end="",
             )
         if failrate > 0.0 and _config.system.show_z and zscore is not None:

garak/resources/garak.core.yaml

@@ -40,4 +40,5 @@ reporting:
   confidence_interval_method: bootstrap
   bootstrap_num_iterations: 10000
   bootstrap_confidence_level: 0.95
-  bootstrap_min_sample_size: 30
+  bootstrap_min_sample_size: 30
+  pass_at_k: [1, 5]

tests/analyze/test_pass_at_k.py

@@ -0,0 +1,105 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Tests for garak.analyze.pass_at_k — the pass@k ASR estimator and aggregation."""
+
+import math
+
+import pytest
+
+from garak.analyze.pass_at_k import pass_at_k, estimate_pass_at_k
+
+
+def _reference(n: int, c: int, k: int) -> float:
+    """Direct binomial form of the estimator, for cross-checking."""
+    if n - c < k:
+        return 1.0
+    return 1.0 - math.comb(n - c, k) / math.comb(n, k)
+
+
+@pytest.mark.parametrize(
+    "n, c, k, expected",
+    [
+        (5, 1, 1, 0.2),  # single hit in five == one-shot ASR of 20%
+        (5, 1, 5, 1.0),  # the lone hit is certain to be among all five draws
+        (5, 1, 2, 0.4),  # 1 - C(4,2)/C(5,2)
+        (5, 2, 2, 0.7),  # 1 - C(3,2)/C(5,2)
+        (5, 0, 3, 0.0),  # no hit can never be drawn
+        (4, 4, 1, 1.0),  # every generation is a hit
+    ],
+)
+def test_pass_at_k_known_values(n, c, k, expected):
+    assert pass_at_k(n, c, k) == pytest.approx(
+        expected
+    ), f"pass_at_k({n},{c},{k}) should be {expected}"
+
+
+@pytest.mark.parametrize("n", range(1, 9))
+def test_pass_at_k_matches_binomial_form(n):
+    for c in range(n + 1):
+        for k in range(1, n + 1):
+            assert pass_at_k(n, c, k) == pytest.approx(
+                _reference(n, c, k)
+            ), f"product form should match C(n-c,k)/C(n,k) for n={n},c={c},k={k}"
+
+
+def test_pass_at_k_non_decreasing_in_k():
+    n, c = 8, 2
+    values = [pass_at_k(n, c, k) for k in range(1, n + 1)]
+    assert values == sorted(
+        values
+    ), "pass@k should not decrease as the attacker is given more attempts"
+
+
+@pytest.mark.parametrize(
+    "n, c, k",
+    [
+        (5, 1, 6),  # k > n
+        (5, 1, 0),  # k < 1
+        (0, 0, 1),  # n < 1
+        (5, 6, 1),  # c > n
+        (5, -1, 1),  # c < 0
+    ],
+)
+def test_pass_at_k_rejects_out_of_range(n, c, k):
+    with pytest.raises(ValueError):
+        pass_at_k(n, c, k)
+
+
+def test_estimate_macro_averages_the_issue_scenario():
+    # five prompts, each breached on 1 reply in 5: pooled ASR reads 20%, but a
+    # persistent attacker with five tries breaches every prompt.
+    counts = [(5, 1)] * 5
+    result = estimate_pass_at_k(counts, [1, 5])
+    assert result[1]["score"] == pytest.approx(0.2), "pass@1 recovers per-prompt ASR"
+    assert result[5]["score"] == pytest.approx(1.0), "pass@5 exposes guaranteed breach"
+    assert result[1]["prompts"] == 5, "all prompts contribute to pass@1"
+    assert result[5]["prompts"] == 5, "all prompts contribute to pass@5"
+
+
+def test_estimate_excludes_prompts_with_too_few_generations():
+    # one prompt has only 3 generations, so pass@5 cannot be estimated for it
+    counts = [(5, 1), (3, 1)]
+    result = estimate_pass_at_k(counts, [5])
+    assert result[5]["prompts"] == 1, "prompt with n < k is excluded from pass@k"
+
+
+def test_estimate_drops_k_with_no_eligible_prompts():
+    result = estimate_pass_at_k([(3, 1), (2, 0)], [5])
+    assert 5 not in result, "k with no prompt of n >= k should be omitted entirely"
+
+
+def test_estimate_ignores_empty_and_none_prompts():
+    # a prompt whose generations were all unscoreable (n == 0) is skipped
+    result = estimate_pass_at_k([(0, 0), (5, 1)], [1])
+    assert result[1]["prompts"] == 1, "prompts with no scoreable outputs are ignored"
+
+
+def test_estimate_deduplicates_and_sorts_k():
+    result = estimate_pass_at_k([(5, 1)], [5, 1, 1])
+    assert list(result.keys()) == [1, 5], "k values should be unique and sorted"
+
+
+def test_estimate_empty_inputs_return_empty():
+    assert estimate_pass_at_k([], [1, 5]) == {}, "no prompts yields no scores"
+    assert estimate_pass_at_k([(5, 1)], []) == {}, "no k values yields no scores"