Docs: record PR 46 merge in handoff

docs/HANDOFF.md

@@ -2,7 +2,7 @@
 
 Current state of the project and active work. Read this at session start. Update before compaction or at natural breakpoints.
 
-**Last updated:** 2026-06-12
+**Last updated:** 2026-06-17
 
 ## Project State
 
@@ -45,7 +45,7 @@ See `memory/verb_architecture.md` for full taxonomy with consistent terminology.
 
 Check `todos/` for the full list. Key ones:
 
-- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, updated-model 2-model L3 smoke, two-prompt 3-model L3 sample, initial parser hardening for recoverable eval-score shapes, and one post-hardening L3 validation completed; next step is a small L3 rerun after bounded raw invalid-response previews are captured, to distinguish malformed/empty/provider output from genuinely unhandled score JSON; avoid full MiniMax M3 suites until runtime/output budget controls are explicit
+- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, updated-model 2-model L3 smoke, two-prompt 3-model L3 sample, parser hardening/response-preview triage support in PR #46, and one post-hardening L3 validation completed; next step is a small L3 rerun that inspects bounded `response_preview` fields to distinguish malformed/empty/provider output from genuinely unhandled score JSON; avoid full MiniMax M3 suites until runtime/output budget controls are explicit
 - **025** — optional brainstorm lineage-reference polish if softer phrases like "builds on..." feel too process-oriented in demos
 - **018** — brainstorm divergence expansion: first-stage prompt reframing implemented behind hidden `brainstorm --prompt-variants per-model`; next run L3 benchmarks and defer domain collisions
 - **021** — evaluate TOON (`toon-format/toon`) for prompt-facing artifact export / benchmark fixtures
@@ -59,7 +59,8 @@ Triage pattern: fix P1/P2 with code, create TODOs for P3/nitpicks, reply to ever
 
 ## Recent Context
 
-- 2026-06-12 post-`dc805a7` L3 parser validation completed (`docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`). Ran one paired architecture prompt using Codex/GLM/Kimi-for-coding. Baseline `off` degraded with one GLM invalid eval score; expanded `per-model` degraded with three Kimi 429 overload/rate-limit failures and one Kimi invalid eval score. Analyzer used exactly the two fresh run dirs in `target/brainstorm-benchmark-2026-06-11-l3-parser-validation/logs/run-dirs.txt`. Conclusion: invalid evaluation summaries persist after parser hardening, but this does not prove another score-shape parsing gap. Added bounded raw response-preview capture for invalid brainstorm structured-response parse failures so the next rerun can distinguish malformed/empty/provider output from genuinely unhandled JSON shape.
+- 2026-06-17 PR #46 (`Fix brainstorm invalid eval triage`) merged through the merge queue. It hardened brainstorm eval score parsing for scaled/labeled text, nested scores, `overall_score`, and dimension fallbacks; added bounded `response_preview` capture to invalid structured-response parse failures in CLI JSON and `provider-failures.json`; added artifact regression coverage; and addressed review feedback for spaced slash scores (`8.5 / 10`), labeled scales (`Score (1-10): 8`), and char-count preview bounds. GitHub CI and CodeRabbit passed; Buildkite was queued via merge queue and the PR landed as `8e89266`.
+- 2026-06-12 post-`dc805a7` L3 parser validation completed (`docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`). Ran one paired architecture prompt using Codex/GLM/Kimi-for-coding. Baseline `off` degraded with one GLM invalid eval score; expanded `per-model` degraded with three Kimi 429 overload/rate-limit failures and one Kimi invalid eval score. Analyzer used exactly the two fresh run dirs in `target/brainstorm-benchmark-2026-06-11-l3-parser-validation/logs/run-dirs.txt`. Conclusion: invalid evaluation summaries persist after parser hardening, but this does not prove another score-shape parsing gap. PR #46 now preserves bounded raw response previews for invalid structured-response parse failures so the next rerun can distinguish malformed/empty/provider output from genuinely unhandled JSON shape.
 - 2026-06-11 GLM invalid-evaluation parser hardening completed (`todos/013`, plan `docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md`). `parse_brainstorm_evaluation_response()` now accepts recoverable score variants seen/plausible in expanded brainstorm evals: scaled score text like `"8 out of 10"`, nested score objects, `overall_score`, and a missing-overall fallback to the four required dimension scores. It rejects incomplete dimension sets even if extra numeric fields are present. Verified with `cargo fmt --all -- --check`, `cargo test -p refinery_core parse_brainstorm_evaluation -q`, `cargo test -p refinery_core brainstorm -q`, and `cargo clippy -p refinery_core --all-targets -- -D warnings`.
 - 2026-06-09 three-model L3 sample completed (`todos/013`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`) after PR #44 merged. Compared `--prompt-variants off` vs `per-model` on product and technical prompts using `pi/openai-codex/gpt-5.4:off`, `pi/zai/glm-5.1:off`, and `pi/kimi-coding/kimi-for-coding:off`, serial with `--max-concurrent 1`. Baseline runs completed clean (`18` calls each, ~7-8m). Per-model runs completed with full 12-candidate final sets but degraded evaluation status (`75` calls each, ~30-38m): product had one GLM invalid eval score; technical had one Codex SSE response-header timeout and one GLM invalid eval score. `controversy_floor_7` two-prompt averages improved mean quality `7.83 → 8.25`, min quality `7.00 → 8.00`, disagreement `0.33 → 0.75`; lexical overlap also rose `0.056 → 0.074`; meta-preamble stayed `0.0`. Promising but not enough for default changes because both expanded runs degraded.
 - 2026-06-05 updated-model L3 smoke completed (`todos/013`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`) after Pi exposed `pi/kimi-coding/kimi-for-coding` (Kimi K2.6 for coding) and `pi/minimax/MiniMax-M3`. Single-model smoke calls for both worked. A two-model product baseline (`prompt-variants off`) completed clean with `total_calls: 8`, `degraded: false`, `controversy_floor_7` mean/min quality `7.50/7.00`, lexical overlap `0.073`. A two-model prompt-reframing run completed degraded with `total_calls: 25/26`, final candidates `5`, and one MiniMax M3 round-2 proposal timeout on the legal-scrutiny variant after 900s; `controversy_floor_7` mean/min quality `8.33/8.00`, lexical overlap `0.080`, meta-preamble `0.0`. Because two-model runs have only one evaluator per candidate, disagreement/controversy is not meaningful. A four-model updated sample (Codex + GLM + Kimi-for-coding + MiniMax M3) was stopped after ~14 minutes while still in the first baseline run; partial artifacts showed round-1 progress, so treat it as a budget/runtime caution rather than a correctness failure. Keep production defaults unchanged.
@@ -155,6 +156,6 @@ Recommended order:
 
 1. If continuing Buildkite migration, review PR #39 and either trigger a real Buildkite run against `ci-linux-arm64-rust-bazel` or update the Buildkite pipeline settings to upload `.buildkite/pipeline.yml` from the repo so PR pipeline changes are exercised.
 2. Start from clean `main` and read this handoff plus the valid baseline in `docs/brainstorms/2026-05-23-brainstorm-smoke-baseline.md`.
-3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`, and `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`; then rerun a small L3 prompt-reframing validation with the Codex/GLM/Kimi-for-coding panel and inspect new `response_preview` fields for any invalid structured-response parse failures. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls.
+3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`, and `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`; then rerun a small low-call L3 validation (start with `--prompt-variants off` before expensive `per-model`) with the Codex/GLM/Kimi-for-coding panel and inspect `response_preview` fields for any invalid structured-response parse failures. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls.
 4. For future Pi-backed benchmark runs, use `--max-concurrent 1` unless Pi config locking is fixed; for OpenCode-backed models use `--max-concurrent 1` and `--idle-timeout 480` until `todos/022` is fixed.
 5. Do not implement Open Collider-style domain collisions before benchmark budget constraints are explicit; if moving to L3, start with prompt-reframing expansion from `todos/018`.

todos/013-brainstorm-strategy-benchmarks.md

@@ -4,7 +4,7 @@ priority: low
 milestone: v0.4
 depends_on: 004-verb-brainstorm
 status: in_progress
-updated: 2026-06-11
+updated: 2026-06-17
 ---
 
 # Benchmark: Brainstorm Iteration and Selection Strategies
@@ -104,11 +104,11 @@ A first-pass qualitative review over the generated blind panel review pack is co
 
 Latest L3 three-model sample is documented in `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`. It compared `--prompt-variants off` vs `per-model` on product and technical prompts with Codex, GLM, and Kimi-for-coding. Per-model improved two-prompt `controversy_floor_7` average quality floor (`7.00` → `8.00`) and disagreement (`0.33` → `0.75`), but both per-model runs degraded due to evaluation issues (GLM invalid eval scores; one Codex SSE header timeout), so it cannot support default changes.
 
-A verified parser hardening pass for recoverable GLM-style invalid evaluation scores completed on 2026-06-11. Brainstorm evaluation parsing now accepts scaled score text, nested score objects, `overall_score`, and a missing-overall fallback to the four required dimension scores while rejecting incomplete dimension sets. Verified with targeted parser tests, `cargo test -p refinery_core brainstorm -q`, and `cargo clippy -p refinery_core --all-targets -- -D warnings`.
+A verified parser hardening and invalid-response observability pass landed in PR #46 on 2026-06-17. Brainstorm evaluation parsing now accepts scaled/labeled score text, nested score objects, `overall_score`, and a missing-overall fallback to the four required dimension scores while rejecting incomplete dimension sets. Invalid structured-response parse failures now preserve a bounded `response_preview` in CLI JSON and `provider-failures.json`. PR #46 also addressed review feedback for spaced slash scores (`8.5 / 10`), labeled scales (`Score (1-10): 8`), and char-count preview bounds.
 
-A live post-hardening L3 validation on 2026-06-12 is documented in `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`. It still degraded: baseline had one GLM invalid eval score, while expanded prompt-reframing had Kimi overload/rate-limit failures plus one Kimi invalid eval score. This does not prove a remaining score-shape parser gap; the next code step is evidence capture to distinguish malformed/empty/provider output from genuinely unhandled score JSON.
+A live post-hardening L3 validation on 2026-06-12 is documented in `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`. It still degraded: baseline had one GLM invalid eval score, while expanded prompt-reframing had Kimi overload/rate-limit failures plus one Kimi invalid eval score. This does not prove a remaining score-shape parser gap; with PR #46 merged, the next degraded run should preserve raw preview evidence to distinguish malformed/empty/provider output from genuinely unhandled score JSON.
 
-Next concrete step: rerun a small L3 validation and inspect bounded `response_preview` fields for any invalid structured-response parse failures, or run a human/calibrated model-judge pass over the L2/L3 panel findings. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit.
+Next concrete step: rerun a small low-call L3 validation, starting with `--prompt-variants off`, and inspect bounded `response_preview` fields for any invalid structured-response parse failures before spending budget on another expanded `per-model` run. Alternatively, run a human/calibrated model-judge pass over the L2/L3 panel findings. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit.
 
 ## References