You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7c2af1d
Browse filesBrowse the repository at this point in the historyBrowse files
feat(eval): add an Improvement recommendations section to the eval report (#2472)
Implements the "Improvement recommendations" section from the parent epic
(#2087), per docs/evaluation/eval-framework.md §9.1 -- surfaces findings in
the per-cell eval report as concrete, targeted actions rather than a bare
statement of which metrics failed.
eval/recommendations.py deterministically gathers evidence for six finding
types (over-grant, under-grant, sensitivity, invariance, consistency,
Scale correctness mistake) from data the suites already record, each
EvidenceCase carrying the concrete scenario(s)/cell(s) that produced it. A
category with no supporting evidence in a given run is omitted entirely,
never printed as a vacuous pass statement. One batched structured-LLM call
clusters same-root-cause cases into distinct, case-specific recommendations
instead of one templated blurb per failing scenario, with a deterministic,
clearly-labeled fallback grouping when the LLM call fails or returns
nothing usable -- the section is never silently blank or fabricated. Also
hardened end-to-end: a failure anywhere in the recommendations pipeline
(including a settings/client-construction error, not just a transport
failure) can never prevent the rest of the per-cell report from being
written.
under_grant_cases also surfaces a pair that is simultaneously granted AND
denied (e.g. a coarse scope split by the auditor into one approved and one
best-effort-rejected sub-decision) -- previously such a pair landed in
incorrectly_denied without ever reaching under_grants, so the finding never
became an EvidenceCase.
eval/conftest.py wires this into pytest_sessionfinish via
_collect_recommendation_evidence/_render_recommendations_section, reusing
the existing nodeid-marker dicts to attribute each already-recorded
property dict to its scenario and finding type.
docs/evaluation/eval-framework.md §9.1 records the two implementation
decisions the code cites by number: no numeric degradation threshold for
the Scale row (reuses the suite's own zero-tolerance gate), and the
Consistency cluster-vs-random classification's deterministic-fraction
heuristic.
eval/test_recommendations.py -- 24 new unit tests covering each
evidence-gathering function, the LLM-draft/fallback path, and the
end-to-end build_recommendations assembly, plus one live-LLM test
(require_env_or_skip-gated) confirming the real batched call merges
same-root-cause cases.
Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Amitfre15 <amitfrework@gmail.com>
Copy file name to clipboardExpand all lines: docs/evaluation/eval-framework.md
+19-1Lines changed: 19 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -275,14 +275,32 @@ produced whenever the supporting evidence is present:
275
275
| Sensitivity family failures (output did not change when it should have) | Flag which policy-text edit types the PRB is insensitive to (e.g. negation words, exception clauses, restriction words like "only"/"just"); recommend adding those edit patterns to the Robustness corpus and reviewing PRB prompts for those constructs. |
276
276
| Invariance family failures (output changed when it shouldn't have) | Flag which surface-form changes destabilize the PRB; recommend prompt hardening or normalization pre-processing. |
277
277
| Consistency disagreements | Note whether disagreements cluster on specific scenarios (structural prompt sensitivity) or appear random (temperature/batching noise); recommend `temperature=0` enforcement or a retry-with-majority-vote strategy accordingly. |
278
-
| Scale correctness degradation above a threshold | Identify whether degradation is in per-decision scale (large candidate lists) or total-corpus scale; recommend context-window management changes (chunking, summarization) or candidate-list pruning strategies respectively. |
278
+
| Scale correctness mistake (over-grant, under-grant, or incorrect denial in a Scale-suite scenario) | Identify whether the mistake is in per-decision scale (large candidate lists) or total-corpus scale; recommend context-window management changes (chunking, summarization) or candidate-list pruning strategies respectively. |
279
279
280
280
Recommendations that have no supporting evidence in the current run are
281
281
omitted (not printed as vacuous "no issues found" items). Each
282
282
recommendation references the specific failing scenario(s) or metric
283
283
cell(s) that produced it, so the reader can verify the evidence directly
284
284
in the same report.
285
285
286
+
**Implementation decisions** (referenced as "decision 1"/"decision 4" in
287
+
`eval/recommendations.py`/`eval/conftest.py`):
288
+
289
+
-**Decision 1 — no numeric degradation threshold for the Scale row.** The Scale
290
+
suite's correctness check reuses the same zero-tolerance gate as the
0 commit comments