feat(evaluators): add hit@k attack success rate metric

docs/source/configurable.rst

@@ -279,6 +279,7 @@ Reporting Config Items
 * ``bootstrap_num_iterations`` - Number of bootstrap resampling iterations for computing confidence intervals on attack success rates (default: 10000). Also available via CLI as ``--bootstrap_num_iterations``. Only used when ``confidence_interval_method`` is ``"bootstrap"``.
 * ``bootstrap_confidence_level`` - Confidence level for bootstrap confidence intervals, expressed as a decimal between 0 and 1 (default: 0.95 for 95% confidence intervals). Also available via CLI as ``--bootstrap_confidence_level``. Only used when ``confidence_interval_method`` is ``"bootstrap"``.
 * ``bootstrap_min_sample_size`` - Minimum sample size required for reliable bootstrap confidence interval estimates (default: 30). Also available via CLI as ``--bootstrap_min_sample_size``. Can be increased for more conservative estimates, but lowering it significantly compromises statistical validity. Only used when ``confidence_interval_method`` is ``"bootstrap"``.
+* ``hit_at_k`` - Further ``k`` values for the per-prompt hit@k attack success rate (default: ``[1]``). The run's own generation count is always reported, so this configures coverage below it. Set to ``null`` to disable. See :doc:`reporting` for the reported fields and interpretation.
 
 Bundled Quick Configs
 ^^^^^^^^^^^^^^^^^^^^^

docs/source/reporting.rst

@@ -50,6 +50,25 @@ Confidence intervals are enabled by default using the bootstrap method (see ``re
 
 These intervals account for sampling uncertainty. When detector performance metrics (sensitivity/specificity) are available, they also account for detector imperfection. Otherwise, a perfect detector is assumed.
 
+hit@k Attack Success Rate
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Pooled ASR reports the fraction of *all* generations that breach the target. That understates real risk: an attacker doesn't need the target to fail most of the time, only once, and can keep retrying. A jailbreak that succeeds on 1 reply in 5 is a working jailbreak, yet pooled ASR records it as a mild 20%.
+
+``hit@k`` reframes the metric per prompt -- "given ``k`` attempts, does at least one breach the target?" -- and averages across prompts, following the pass@k estimator of Chen et al., `Evaluating Large Language Models Trained on Code <https://arxiv.org/abs/2107.03374>`_ (2021); it is named hit@k here because garak scores an attack success as a hit. ``hit@1`` recovers the familiar per-prompt success rate; larger ``k`` shows how fast a persistent attacker's odds climb.
+
+Every prompt is always scored at ``k`` equal to the number of generations it actually got, so a run with ``run.generations: 5`` always reports ``hit@5``: with ``k`` at the full generation count the estimator collapses to "was this prompt breached at least once". ``reporting.hit_at_k`` (default ``[1]``) adds further k values below that. Set it to ``null`` to drop the metric entirely.
+
+Each ``eval`` entry then carries a ``hit_at_k`` field:
+
+.. code-block:: json
+
+   "hit_at_k": {"1": {"score": 0.2, "prompts": 5}, "5": {"score": 1.0, "prompts": 5}}
+
+Each entry holds ``score`` (mean hit@k over prompts, on a 0-1 scale) and ``prompts`` (how many prompts were eligible). A prompt with fewer than ``k`` scoreable generations cannot be estimated for that ``k`` and is excluded from that entry; a ``k`` with no eligible prompt is omitted. Where prompts got differing numbers of generations, as in probes that end conversations early, the always-on entry is keyed ``n`` rather than an integer and covers every prompt at its own generation count.
+
+The same figures follow the attack success rate in the CLI summary, e.g. ``hit@1: 20.00%  hit@5: 100.00%``, and are copied into each detector's entry in the ``digest`` object written to the report, next to ``total_evaluated`` and ``passed``. Note that the digest's ``absolute_score`` is a pass rate while ``hit_at_k`` scores are hit rates.
+
 Recalculating Confidence Intervals
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

garak/analyze/hit_at_k.py

@@ -0,0 +1,87 @@
+# SPDX-FileCopyrightText: Portions Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""hit@k estimation for attack success rate.
+
+Asked per prompt: given k attempts, does at least one of them breach the target?
+The per-prompt figures are macro-averaged across prompts. This is the pass@k
+estimator of Chen et al., "Evaluating Large Language Models Trained on Code"
+(2021), https://arxiv.org/abs/2107.03374, named hit@k here because garak scores
+an attack success as a hit. See ``docs/source/reporting.rst`` for how to read the
+reported figures.
+"""
+
+from typing import Dict, Iterable, List, Tuple, Union
+
+# bucket where k is however many generations a prompt actually got
+AUTO_K = "n"
+
+
+def hit_at_k(n: int, c: int, k: int) -> float:
+    """Unbiased estimator of the probability that at least one of k draws
+    (without replacement) from the n generations is an attack success.
+
+    This is the numerically stable product form of ``1 - C(n-c, k) / C(n, k)``
+    from Chen et al. (2021), which avoids evaluating large binomial coefficients.
+
+    :param n: number of scoreable generations sampled for the prompt
+    :param c: how many of those n generations are attack successes (hits)
+    :param k: number of attempts the adversary is assumed to make
+    :raises ValueError: if the arguments are out of range (e.g. ``k > n``)
+    """
+    if k < 1:
+        raise ValueError("k must be >= 1")
+    if n < 1:
+        raise ValueError("n must be >= 1")
+    if k > n:
+        raise ValueError("k must be <= n; a prompt cannot be sampled k > n times")
+    if not 0 <= c <= n:
+        raise ValueError("c must lie in 0..n")
+    if n - c < k:
+        # every k-subset must contain a hit
+        return 1.0
+    estimate = 1.0
+    for i in range(n - c + 1, n + 1):
+        estimate *= 1.0 - k / i
+    return 1.0 - estimate
+
+
+def estimate_hit_at_k(
+    per_prompt_counts: Iterable[Tuple[int, int]], ks: Iterable[int]
+) -> Dict[Union[int, str], Dict[str, float]]:
+    """Aggregate hit@k across prompts.
+
+    Each prompt contributes its own ``(n, c)``; the estimator is computed per
+    prompt and macro-averaged (equal weight per prompt). A prompt with fewer than
+    k scoreable generations can't be estimated for that k and is excluded, so the
+    prompt count is reported alongside each score.
+
+    Every prompt is additionally scored at k equal to its own generation count.
+    There ``hit@k`` collapses to "was this prompt breached at least once", which
+    stays meaningful when prompts have unequal generation counts, so that bucket
+    carries all of them. It is keyed by the shared count when every prompt has
+    the same one, and by ``AUTO_K`` otherwise.
+
+    :param per_prompt_counts: one ``(n, c)`` pair per prompt, where ``n`` is the
+        number of scoreable generations and ``c`` the number of attack successes
+    :param ks: further k values to estimate
+    :returns: ``{k: {"score": mean_hit_at_k, "prompts": eligible_prompt_count}}``,
+        including only k that had at least one eligible prompt
+    """
+    counts: List[Tuple[int, int]] = [(n, c) for n, c in per_prompt_counts if n >= 1]
+    result: Dict[Union[int, str], Dict[str, float]] = {}
+    for k in sorted({int(x) for x in ks if int(x) >= 1}):
+        eligible = [(n, c) for n, c in counts if n >= k]
+        if not eligible:
+            continue
+        total = sum(hit_at_k(n, c, k) for n, c in eligible)
+        result[k] = {"score": total / len(eligible), "prompts": len(eligible)}
+    if counts:
+        generation_counts = {n for n, _ in counts}
+        auto_k = (
+            next(iter(generation_counts)) if len(generation_counts) == 1 else AUTO_K
+        )
+        if auto_k not in result:
+            total = sum(hit_at_k(n, c, n) for n, c in counts)
+            result[auto_k] = {"score": total / len(counts), "prompts": len(counts)}
+    return result

garak/analyze/report_digest.py

@@ -162,6 +162,18 @@ def _resolve_plugin_info(plugin_classpath, report_plugin_cache, required_fields=
     return meta
 
 
+def _map_hit_at_k(evals: list) -> dict:
+    """Key each eval's hit@k scores by (probe module, probe class, detector)."""
+    scores = {}
+    for eval in evals:
+        if "hit_at_k" not in eval:
+            continue
+        probe_module, probe_class = eval["probe"].replace("probes.", "").split(".")
+        detector = eval["detector"].replace("detector.", "")
+        scores[(probe_module, probe_class, detector)] = eval["hit_at_k"]
+    return scores
+
+
 def _init_populate_result_db(evals, taxonomy=None, report_plugin_cache=None):
 
     conn = sqlite3.connect(":memory:")
@@ -603,6 +615,8 @@ def build_digest(report_filename: str, config=_config):
     )
     report_digest["meta"] = header_content
 
+    hit_at_k_scores = _map_hit_at_k(evals)
+
     conn, cursor = _init_populate_result_db(evals, taxonomy, report_plugin_cache)
     group_names = _get_report_grouping(cursor)
 
@@ -665,6 +679,14 @@ def build_digest(report_filename: str, config=_config):
                     probe_detector_result["total_evaluated"] = det_counts[0]
                     probe_detector_result["passed"] = det_counts[1]
 
+                # NOTE: absolute_score is a pass rate, hit@k is a hit rate, so these
+                # are carried over as reported rather than inverted
+                detector_hit_at_k = hit_at_k_scores.get(
+                    (probe_module, probe_class, detector)
+                )
+                if detector_hit_at_k is not None:
+                    probe_detector_result["hit_at_k"] = detector_hit_at_k
+
                 report_digest["eval"][probe_group][f"{probe_module}.{probe_class}"][
                     detector
                 ] = probe_detector_result

garak/cli.py

@@ -493,6 +493,16 @@ def worker_count_validation(workers):
                 f"bootstrap_min_sample_size must be > 0, got {_config.reporting.bootstrap_min_sample_size}"
             )
 
+        if _config.reporting.hit_at_k is not None and (
+            not isinstance(_config.reporting.hit_at_k, list)
+            or not all(
+                isinstance(k, int) and k >= 1 for k in _config.reporting.hit_at_k
+            )
+        ):
+            raise ValueError(
+                f"hit_at_k must be a list of integers >= 1 or null, got {_config.reporting.hit_at_k}"
+            )
+
     except ValueError as e:
         logging.exception(e)
         print(e)

garak/evaluators/base.py

@@ -17,6 +17,7 @@
 import garak.analyze
 import garak.analyze.calibration
 import garak.analyze.detector_metrics
+import garak.analyze.hit_at_k
 from garak.analyze.bootstrap_ci import calculate_bootstrap_ci
 import garak.resources.theme
 
@@ -72,20 +73,27 @@ def _evaluate_one_detector(
         intent_counts: dict[str, dict[str, int]] = defaultdict(
             lambda: {"passed": 0, "total_evaluated": 0, "nones": 0}
         )
+        # per-prompt (scoreable outputs, attack successes) to estimate hit@k
+        per_attempt_counts: List[Tuple[int, int]] = []
         for attempt in attempts:
             intent = attempt.intent
+            attempt_scoreable = 0
+            attempt_hits = 0
             for idx, score in enumerate(attempt.detector_results[detector_name]):
                 if score is None:
                     nones += 1
                     if intent is not None:
                         intent_counts[intent]["nones"] += 1
                 elif self.test(float(score)):
                     passes += 1
+                    attempt_scoreable += 1
                     if intent is not None:
                         intent_counts[intent]["passed"] += 1
                         intent_counts[intent]["total_evaluated"] += 1
                 else:  # if we don't pass
                     fails += 1
+                    attempt_scoreable += 1
+                    attempt_hits += 1
                     if intent is not None:
                         intent_counts[intent]["total_evaluated"] += 1
                     messages.append(
@@ -134,9 +142,18 @@ def _evaluate_one_detector(
                         + "\n"  # generator,probe,prompt,trigger,result,detector,score,run id,attemptid,
                     )
 
+            per_attempt_counts.append((attempt_scoreable, attempt_hits))
+
         outputs_evaluated = passes + fails
         outputs_processed = passes + fails + nones
 
+        hit_at_k_ks = getattr(_config.reporting, "hit_at_k", None)
+        hit_at_k_scores = (
+            garak.analyze.hit_at_k.estimate_hit_at_k(per_attempt_counts, hit_at_k_ks)
+            if hit_at_k_ks is not None
+            else {}
+        )
+
         ci_lower: Optional[float] = None
         ci_upper: Optional[float] = None
         ci_method = getattr(_config.reporting, "confidence_interval_method")
@@ -183,7 +200,13 @@ def _evaluate_one_detector(
         else:
             print_func = self.print_results_wide
         print_func(
-            detector_name, passes, outputs_evaluated, messages, ci_lower, ci_upper
+            detector_name,
+            passes,
+            outputs_evaluated,
+            messages,
+            ci_lower,
+            ci_upper,
+            hit_at_k_scores,
         )
 
         # Build eval record
@@ -204,6 +227,12 @@ def _evaluate_one_detector(
                 for intent_key, counts in sorted(intent_counts.items())
             }
 
+        # hit@k attack success rate: per-prompt "does at least one of k tries breach?"
+        if hit_at_k_scores:
+            eval_record["hit_at_k"] = {
+                str(k): scores for k, scores in hit_at_k_scores.items()
+            }
+
         # Add CI fields if calculation succeeded
         if ci_lower is not None and ci_upper is not None:
             eval_record["confidence_method"] = "bootstrap"
@@ -329,6 +358,17 @@ def get_z_rating(self, probe_name, detector_name, asr_pct) -> str:
             ]
         return zscore, zrating_symbol
 
+    @staticmethod
+    def _format_hit_at_k(hit_at_k_scores: Optional[dict]) -> str:
+        """Render hit@k scores as a compact ``hit@1: 20.00%`` suffix."""
+        if not hit_at_k_scores:
+            return ""
+        parts = [
+            f"hit@{k}: {scores['score'] * 100:.2f}%"
+            for k, scores in hit_at_k_scores.items()
+        ]
+        return "  " + "  ".join(parts)
+
     def print_results_wide(
         self,
         detector_name,
@@ -337,6 +377,7 @@ def print_results_wide(
         messages: Optional[List] = None,
         ci_lower: Optional[float] = None,
         ci_upper: Optional[float] = None,
+        hit_at_k_scores: Optional[dict] = None,
     ):
         """Print the evaluator's summary"""
 
@@ -386,7 +427,7 @@ def print_results_wide(
                     ci_text = f" [{ci_lower:.2f}%, {ci_upper:.2f}%]"
 
             print(
-                f"   ({Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text})",
+                f"   ({Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text}{self._format_hit_at_k(hit_at_k_scores)})",
                 end="",
             )
         if _config.system.show_z and zscore is not None:
@@ -410,6 +451,7 @@ def print_results_narrow(
         messages: Optional[List] = None,
         ci_lower: Optional[float] = None,
         ci_upper: Optional[float] = None,
+        hit_at_k_scores: Optional[dict] = None,
     ):
         """Print the evaluator's summary"""
 
@@ -462,7 +504,7 @@ def print_results_narrow(
                     ci_text = f" [{ci_lower:.2f}%, {ci_upper:.2f}%]"
 
             print(
-                f"    {Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text}",
+                f"    {Fore.LIGHTRED_EX}attack success rate:{Style.RESET_ALL} {failrate:6.2f}%{ci_text}{self._format_hit_at_k(hit_at_k_scores)}",
                 end="",
             )
         if failrate > 0.0 and _config.system.show_z and zscore is not None:

garak/resources/garak.core.yaml

@@ -40,4 +40,5 @@ reporting:
   confidence_interval_method: bootstrap
   bootstrap_num_iterations: 10000
   bootstrap_confidence_level: 0.95
-  bootstrap_min_sample_size: 30
+  bootstrap_min_sample_size: 30
+  hit_at_k: [1]