refactor(eval): move trend_log + dashboard into eval/dashboard/

Groups the trend-log writer/reader and the dashboard renderer (plus their
data/output files) under one package, a sibling of eval/reports/, instead of
loose at the eval/ top level. Fixes dashboard.py's reports_dir default (was
HERE / "reports", silently wrong once dashboard.py no longer sits beside
eval/reports/) and updates every import, path, doc reference, and
.gitignore entry accordingly. Also commits the real trend_log.jsonl row from
the live agent_delegation/unreachable_resources run (eval/reports/report_
06_10_11_21_16.md).

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Amitfre15 <amitfrework@gmail.com>

Author: Amitfre15

.gitignore

@@ -18,9 +18,10 @@ test/integration/rego_out/
 eval/rego_out/
 eval/reports/
 
-# eval results dashboard (eval/dashboard.py, issue #2542) — regenerated per invocation from
-# eval/trend_log.jsonl (committed) + eval/reports/ (gitignored, above), not source of truth itself.
-eval/dashboard.html
+# eval results dashboard (eval/dashboard/dashboard.py, issue #2542) — regenerated per invocation
+# from eval/dashboard/trend_log.jsonl (committed) + eval/reports/ (gitignored, above), not source
+# of truth itself.
+eval/dashboard/dashboard.html
 
 # uc1-onboarding demo's captured .rego snapshots — regenerated per run, not source of truth.
 demo/use-cases/uc1-onboarding/generated/

docs/evaluation/README.md

@@ -105,7 +105,7 @@ run.
 Every run of the six suites above writes a Markdown report to the gitignored `eval/reports/`
 (`report_<DD_MM_HH_MM_SS>.md`) — see `eval/conftest.py`. The two Correctness suites, the
 Robustness suite's mechanical tier, and the Scale suite (one row per dimension/level) additionally
-append one row each to the **committed** `eval/trend_log.jsonl` (spec:
+append one row each to the **committed** `eval/dashboard/trend_log.jsonl` (spec:
 [eval-framework.md §9](eval-framework.md#9-reporting-and-trend-persistence)) — see
 [policy-eval-correctness-prb.md § Trend log](policy-eval-correctness-prb.md#trend-log),
 [policy-eval-robustness-consistency.md § Trend log](policy-eval-robustness-consistency.md#trend-log),
@@ -114,16 +114,16 @@ log](policy-eval-scale.md#test-report-and-trend-log).
 
 ## Dashboard
 
-`eval/dashboard.py` renders a static, self-contained HTML view (inline SVG, no external
+`eval/dashboard/dashboard.py` renders a static, self-contained HTML view (inline SVG, no external
 dependency, no CI wiring) from the two artifacts above — no new data source. Regenerate it after a
 run:
 
 ```bash
-.venv/bin/python -m eval.dashboard
+.venv/bin/python -m eval.dashboard.dashboard
 ```
 
-Writes `eval/dashboard.html` (gitignored, regenerated each time) with one historical trend chart
-per suite present in `eval/trend_log.jsonl`, plus a collapsible scenario drill-down section per
+Writes `eval/dashboard/dashboard.html` (gitignored, regenerated each time) with one historical
+trend chart per suite present in `eval/dashboard/trend_log.jsonl`, plus a collapsible scenario drill-down section per
 report found under `eval/reports/`. A chart point links to its matching report's section when one
 is found on disk within a few hours of that trend-log row's timestamp; otherwise it still shows its
 exact values via a hover tooltip, just without a link.

docs/evaluation/policy-eval-correctness-e2e.md

@@ -197,9 +197,9 @@ render behavior itself.
 
 Same mechanism as
 [policy-eval-correctness-prb.md § Trend log](policy-eval-correctness-prb.md#trend-log) (#2091,
-`eval/trend_log.py`) — `test_e2e_correctness` `record_property`s the same `true_positives`/
+`eval/dashboard/trend_log.py`) — `test_e2e_correctness` `record_property`s the same `true_positives`/
 `denied_total` counts, and `eval/conftest.py` pools them into a `suite="correctness_e2e"` row in
-the same committed `eval/trend_log.jsonl`, distinct from the PRB-level suite's
+the same committed `eval/dashboard/trend_log.jsonl`, distinct from the PRB-level suite's
 `"correctness_prb"` row.
 
 ## Relationship to other integration tests

docs/evaluation/policy-eval-correctness-prb.md

@@ -284,12 +284,12 @@ This is **one** integration-test spec among several indexed by the master PRD
 
 ## Trend log
 
-`eval/trend_log.py`'s shared `append_row`/`pool_correctness_metrics` (#2091) pools this suite's
+`eval/dashboard/trend_log.py`'s shared `append_row`/`pool_correctness_metrics` (#2091) pools this suite's
 `true_positives`/`denied_total` counts (and `over_grants`/`under_grants`/`incorrectly_denied` pair
 dicts) — `record_property`'d by `test_prb_correctness` alongside the existing
 precision/recall/denial-precision floats — across every scenario that reached `score_scenario` in
 a run, and appends one row (`suite="correctness_prb"`) to the committed, append-only
-`eval/trend_log.jsonl` from `eval/conftest.py`'s `pytest_sessionfinish`. Pooled by summed count,
+`eval/dashboard/trend_log.jsonl` from `eval/conftest.py`'s `pytest_sessionfinish`. Pooled by summed count,
 not averaged per-scenario — same union-not-average philosophy as `score_scenario`'s own
 cross-gate aggregation. See `docs/evaluation/eval-framework.md` §9.
 

docs/evaluation/policy-eval-robustness-consistency.md

@@ -390,14 +390,14 @@ is also read back by `_write_trend_log` (see [Trend log](#trend-log)).
 `record_property("invariant"/"sensitive", bool)` values, via nodeid-substring matching
 (`_ROBUSTNESS_TEST_MARKERS`, since the single flat `eval` marker — shared by every suite under
 `eval/` — spans four test functions across two families × two tiers that must stay unblended per
-spec §4), and appends **four** rows to the committed, append-only `eval/trend_log.jsonl` from
+spec §4), and appends **four** rows to the committed, append-only `eval/dashboard/trend_log.jsonl` from
 `pytest_sessionfinish` — one per family/tier combination, never combined: `suite=
 "robustness_mechanical_invariance"`, `"robustness_semantic_invariance"` (#2467),
 `"robustness_mechanical_sensitivity"`, and `"robustness_semantic_sensitivity"` (#2467). Each row
 carries that row's own pass/fail rate (`invariance_rate`/`sensitivity_rate`, mean of that row's
 booleans) *and* that row's own `precision`/`recall`/`denial_precision`, pooled from the same
 `true_positives`/`denied_total`/`over_grants`/`under_grants`/`incorrectly_denied` shape
-`_record_scoring` records (`eval/trend_log.py`'s `pool_correctness_metrics` — the same pooling
+`_record_scoring` records (`eval/dashboard/trend_log.py`'s `pool_correctness_metrics` — the same pooling
 function the two Correctness suites use, reused as-is here) — so each row's chart is directly
 comparable in shape to a Correctness chart: two measuring the PRB against *original* inputs
 (invariance, mechanical vs. semantic tier), two against *deliberately edited/reworded* inputs
@@ -406,7 +406,7 @@ filter that only exercises one test function) simply gets no row at all, rather
 mislabeling one row's count against another's. See `docs/evaluation/eval-framework.md` §9.
 
 **Consistency (#2468)** gets its own single row, `suite="consistency"`, pooled by
-`eval/trend_log.py`'s `pool_consistency_metrics` — not `pool_correctness_metrics`, since there's no
+`eval/dashboard/trend_log.py`'s `pool_consistency_metrics` — not `pool_correctness_metrics`, since there's no
 truth table here (Consistency compares PRB runs to each other, not to ground truth), just each
 scenario's own `inconsistent` boolean rolled up into an `agreement_rate` (the fraction of the
 corpus's scenarios that agreed across all of their `PRB_CONSISTENCY_REPEATS` repeats). Framed as

docs/evaluation/policy-eval-scale.md

@@ -172,7 +172,7 @@ Reuses `eval.correctness_scorer.score_scenario`/`score_gate` (the same scorer
   "gates" the way a real scenario's three gates are — each candidate becomes a `(role_name,
   scope_name)` pair against the fixed focal entity (subject-side pairs for the scope direction,
   resource-side pairs for the role direction), so `score_scenario`'s existing per-gate aggregation,
-  `eval/conftest.py`'s report rendering, and `eval/trend_log.py`'s pooling all apply with **zero**
+  `eval/conftest.py`'s report rendering, and `eval/dashboard/trend_log.py`'s pooling all apply with **zero**
   special-casing.
 
 Same zero-tolerance-on-over-grants gate every sibling suite uses; under-grants and incorrect
@@ -187,7 +187,7 @@ dimension/level are checked by separate test functions but land on one row
 (`scale_total_corpus_prb`/`scale_total_corpus_e2e`/`scale_per_decision_prb`/`scale_per_decision_e2e`
 — total-corpus and per-decision, and PRB and e2e, always kept on **separate** rows, per spec §5's
 explicit warning that a system can pass one dimension/level while silently failing the other).
-`eval.trend_log.pool_scale_metrics` pools the structural half (`structural_pass`/
+`eval.dashboard.trend_log.pool_scale_metrics` pools the structural half (`structural_pass`/
 `structural_issue_count`/`wall_clock_seconds`/`total_tokens`/`token_coverage`, common fields both
 dimensions record regardless of their own concrete-check taxonomy) into `structural_pass_rate`/
 `structural_issue_count`/`total_tokens`/`mean_wall_clock_seconds`/`mean_token_coverage`; the
@@ -261,6 +261,6 @@ placement (Nightly, not per-PR).
 
 ## Blocked-by
 
-\#2091 (the committed trend log — already shipped; `eval.trend_log.append_row`/
+\#2091 (the committed trend log — already shipped; `eval.dashboard.trend_log.append_row`/
 `pool_correctness_metrics` reused here unmodified, `pool_scale_metrics` added alongside them). Same
 PRB/e2e prerequisites as every other suite in this family — no new production dependency introduced.

eval/conftest.py

@@ -63,7 +63,7 @@
 ``true_positives`` and ``denied_total`` — not rendered in the Markdown report (the floats and pair
 breakdowns already cover a human reader) but read back here by ``_write_trend_log`` to pool a
 run's precision/recall/denial_precision into one committed, append-only trend-log row per suite
-(``eval/trend_log.py``, spec: ``docs/evaluation/eval-framework.md`` §9) — pooled by summed count
+(``eval/dashboard/trend_log.py``, spec: ``docs/evaluation/eval-framework.md`` §9) — pooled by summed count
 across every scenario that reached ``score_scenario`` in the run, not averaged per-scenario, so a
 run with an uneven pair count per scenario isn't skewed by weighting every scenario equally. A
 scenario whose own setup failed never recorded these, so it contributes nothing to the pooled row
@@ -80,7 +80,7 @@
 committed trend-log row per family/tier combination -- ``suite="robustness_mechanical_invariance"``/
 ``"robustness_semantic_invariance"``/``"robustness_mechanical_sensitivity"``/
 ``"robustness_semantic_sensitivity"`` -- each carrying that row's own precision/recall/
-denial_precision (``eval/trend_log.py``'s ``pool_correctness_metrics``, the same pooling the two
+denial_precision (``eval/dashboard/trend_log.py``'s ``pool_correctness_metrics``, the same pooling the two
 Correctness suites use, so the resulting chart is directly comparable to theirs: two measuring
 performance against the *original* inputs, two against *deliberately edited/reworded* inputs) plus
 that row's own pass/fail rate (``invariance_rate``/``sensitivity_rate``).
@@ -91,7 +91,7 @@
 mismatch detail when ``True``) via ``_render_consistency_block``, on every entry, pass or fail,
 unlike every other test in this suite besides the correctness/robustness metrics blocks above -- and
 pooled by nodeid substring (``_CONSISTENCY_TEST_MARKERS``) into its own committed trend-log row,
-``suite="consistency"``, carrying an ``agreement_rate`` (``eval/trend_log.py``'s
+``suite="consistency"``, carrying an ``agreement_rate`` (``eval/dashboard/trend_log.py``'s
 ``pool_consistency_metrics`` -- its own smaller pooling function, since there's no truth table here
 to produce a ``pool_correctness_metrics``-shaped precision/recall).
 
@@ -107,7 +107,7 @@
 pooled by nodeid substring (``_SCALE_TEST_MARKERS``, mapping to ``(suite, "structural" |
 "correctness")`` rather than a bare suite name) and merged into one row per dimension/level --
 ``scale_total_corpus_prb``/``scale_per_decision_prb`` (and their ``_e2e`` counterparts) -- via
-``eval/trend_log.py``'s ``pool_scale_metrics`` for the structural half and the existing
+``eval/dashboard/trend_log.py``'s ``pool_scale_metrics`` for the structural half and the existing
 ``pool_correctness_metrics`` for the correctness half.
 
 Below the per-category drill-down, ``pytest_sessionfinish`` appends an **"Improvement
@@ -139,8 +139,8 @@
 import pytest
 from dotenv import load_dotenv
 
+from eval.dashboard.trend_log import append_row, pool_consistency_metrics, pool_correctness_metrics, pool_scale_metrics
 from eval.recommendations import build_recommendations
-from eval.trend_log import append_row, pool_consistency_metrics, pool_correctness_metrics, pool_scale_metrics
 
 HERE = Path(__file__).resolve().parent
 REPORTS_DIR = HERE / "reports"
@@ -292,7 +292,7 @@ def _format_pairs_dict(pairs_by_gate: dict) -> str:
     "::test_prb_sensitive_to_semantic_perturbation[",
 )
 
-# Nodeid substring -> trend-log suite name (eval/trend_log.py). With the five former ``eval_*``
+# Nodeid substring -> trend-log suite name (eval/dashboard/trend_log.py). With the five former ``eval_*``
 # markers collapsed to one flat ``eval`` marker, every suite under `eval/` is disambiguated by its
 # own test function's nodeid rather than by marker — the same ``::test_prb_correctness[``/
 # ``::test_e2e_correctness[`` substrings ``_CORRECTNESS_TEST_MARKERS`` already uses. Deliberately
@@ -347,7 +347,7 @@ def _format_pairs_dict(pairs_by_gate: dict) -> str:
 # invariance_rate/sensitivity_rate, which _write_trend_log injects on top of pool_correctness_
 # metrics' output separately). Kept as its own dict (not folded into _TREND_LOG_SUITES) since its
 # entries are pooled by pool_consistency_metrics, not pool_correctness_metrics -- see
-# eval/trend_log.py.
+# eval/dashboard/trend_log.py.
 _CONSISTENCY_TEST_MARKERS = {
     "::test_prb_consistent_across_repeats[": "consistency",
 }
@@ -356,7 +356,7 @@ def _format_pairs_dict(pairs_by_gate: dict) -> str:
 # docs/evaluation/policy-eval-scale.md). Unlike every dict above, each Scale trend-log row
 # (e.g. "scale_total_corpus_prb") is built from *two* separate test functions -- a structural test
 # (record_property("structural_pass"/"structural_issue_count"/...), pooled by
-# eval.trend_log.pool_scale_metrics) and a correctness test (the familiar precision/recall shape,
+# eval.dashboard.trend_log.pool_scale_metrics) and a correctness test (the familiar precision/recall shape,
 # pooled by the existing pool_correctness_metrics) -- so this dict's value is a
 # (suite, "structural" | "correctness") pair rather than a bare suite name; _write_trend_log pools
 # each half separately then merges them into one row. Neither test function is parametrized (each

eval/dashboard/__init__.py

@@ -1,10 +1,10 @@
 """Static eval results dashboard — scenario drill-down + historical trend charts (issue #2542).
 
-Renders entirely from the artifacts ``eval/trend_log.py``/``eval/conftest.py`` (#2091) already
-produce — the committed, append-only ``eval/trend_log.jsonl`` and the gitignored per-run
-``eval/reports/report_<timestamp>.md`` — no new data source, no CI wiring, no server. A single
-self-contained HTML file (inline SVG, inline CSS, no external requests) a developer regenerates
-locally after an eval run.
+Renders entirely from the artifacts ``eval/dashboard/trend_log.py``/``eval/conftest.py`` (#2091)
+already produce — the committed, append-only ``eval/dashboard/trend_log.jsonl`` and the
+gitignored per-run ``eval/reports/report_<timestamp>.md`` — no new data source, no CI wiring, no
+server. A single self-contained HTML file (inline SVG, inline CSS, no external requests) a
+developer regenerates locally after an eval run.
 
 Two artifacts, two read paths: ``load_trend_log`` reads the committed trend log; ``parse_report``/
 ``parse_reports`` re-derive structured scenario data from the Markdown report(s) ``eval/conftest.py``
@@ -24,9 +24,10 @@
 from pathlib import Path
 from typing import Any
 
-from eval.trend_log import DEFAULT_PATH as TREND_LOG_DEFAULT_PATH
+from eval.dashboard.trend_log import DEFAULT_PATH as TREND_LOG_DEFAULT_PATH
 
-HERE = Path(__file__).resolve().parent
+HERE = Path(__file__).resolve().parent  # eval/dashboard/
+EVAL_DIR = HERE.parent  # eval/ -- eval/reports/ lives here, a sibling of this package, not under it
 
 
 def load_trend_log(path: Path = TREND_LOG_DEFAULT_PATH) -> list[dict[str, Any]]:
@@ -301,7 +302,7 @@ def _report_anchor(report: ParsedReport) -> str:
     return "run-" + report.run_at.strftime("%Y%m%dT%H%M%SZ")
 
 
-# Bookkeeping keys every trend-log row carries (eval/trend_log.py's `append_row`, plus
+# Bookkeeping keys every trend-log row carries (eval/dashboard/trend_log.py's `append_row`, plus
 # `scenarios_scored` that every pooling function adds) that are never themselves a plottable
 # metric -- everything else on a row *used* to be one, whatever the suite, back when every metric
 # any pooling function produced was naturally 0-1-bounded (a rate or a score). This chart hardcodes
@@ -543,7 +544,7 @@ def render_dashboard(trend_rows: list[dict[str, Any]], reports: list[ParsedRepor
         )
         trends_body = f'<div class="trends-grid">{chart_sections}</div>'
     else:
-        trends_body = "<p>No trend-log rows yet — run an eval suite to populate eval/trend_log.jsonl.</p>"
+        trends_body = "<p>No trend-log rows yet — run an eval suite to populate eval/dashboard/trend_log.jsonl.</p>"
 
     tables = "".join(render_scenario_table(report) for report in reports)
     drilldown_sections = (
@@ -565,7 +566,7 @@ def render_dashboard(trend_rows: list[dict[str, Any]], reports: list[ParsedRepor
 
 def build_dashboard(
     trend_log_path: Path = TREND_LOG_DEFAULT_PATH,
-    reports_dir: Path = HERE / "reports",
+    reports_dir: Path = EVAL_DIR / "reports",
     output_path: Path = HERE / "dashboard.html",
 ) -> Path:
     """Load the trend log + every per-run report, render the dashboard, write it to

eval/dashboard.py eval/dashboard/dashboard.pyeval/dashboard.py renamed to eval/dashboard/dashboard.py

@@ -31,3 +31,4 @@
 {"timestamp": "2026-09-28T14:23:10+00:00", "suite": "scale_per_decision_prb", "run_type": "regression", "model": "Azure/gpt-5-mini-2025-08-07", "structural_pass_rate": 1.0, "structural_issue_count": 0, "total_tokens": 14435, "mean_wall_clock_seconds": 31.791, "mean_token_coverage": 1.0, "scenarios_scored": 1, "precision": 1.0, "recall": 1.0, "denial_precision": 1.0}
 {"timestamp": "2026-09-28T14:23:10+00:00", "suite": "scale_total_corpus_e2e", "run_type": "regression", "model": "Azure/gpt-5-mini-2025-08-07", "structural_pass_rate": 1.0, "structural_issue_count": 0, "total_tokens": 9231887, "mean_wall_clock_seconds": 118.671, "mean_token_coverage": 1.0, "scenarios_scored": 1, "precision": 1.0, "recall": 0.991, "denial_precision": 1.0}
 {"timestamp": "2026-09-28T14:23:10+00:00", "suite": "scale_total_corpus_prb", "run_type": "regression", "model": "Azure/gpt-5-mini-2025-08-07", "structural_pass_rate": 1.0, "structural_issue_count": 0, "total_tokens": 9109596, "mean_wall_clock_seconds": 291.003, "mean_token_coverage": 1.0, "scenarios_scored": 1, "precision": 1.0, "recall": 0.99, "denial_precision": 1.0}
+{"timestamp": "2026-10-06T11:21:16+00:00", "suite": "correctness_e2e", "run_type": "regression", "model": "Azure/gpt-5-mini-2025-08-07", "scenarios_scored": 8, "precision": 0.969, "recall": 1.0, "denial_precision": 0.75}