From 208c39cb7af0a073add9f69898de272ee591363b Mon Sep 17 00:00:00 2001 From: El-Fitz <8971906+El-Fitz@users.noreply.github.com> Date: Wed, 17 Jun 2026 17:16:06 +0000 Subject: [PATCH] docs: record PR 46 merge in handoff --- docs/HANDOFF.md | 9 +++++---- todos/013-brainstorm-strategy-benchmarks.md | 8 ++++---- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 3fbef7c..004535a 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -2,7 +2,7 @@ Current state of the project and active work. Read this at session start. Update before compaction or at natural breakpoints. -**Last updated:** 2026-06-12 +**Last updated:** 2026-06-17 ## Project State @@ -45,7 +45,7 @@ See `memory/verb_architecture.md` for full taxonomy with consistent terminology. Check `todos/` for the full list. Key ones: -- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, updated-model 2-model L3 smoke, two-prompt 3-model L3 sample, initial parser hardening for recoverable eval-score shapes, and one post-hardening L3 validation completed; next step is a small L3 rerun after bounded raw invalid-response previews are captured, to distinguish malformed/empty/provider output from genuinely unhandled score JSON; avoid full MiniMax M3 suites until runtime/output budget controls are explicit +- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, updated-model 2-model L3 smoke, two-prompt 3-model L3 sample, parser hardening/response-preview triage support in PR #46, and one post-hardening L3 validation completed; next step is a small L3 rerun that inspects bounded `response_preview` fields to distinguish malformed/empty/provider output from genuinely unhandled score JSON; avoid full MiniMax M3 suites until runtime/output budget controls are explicit - **025** — optional brainstorm lineage-reference polish if softer phrases like "builds on..." feel too process-oriented in demos - **018** — brainstorm divergence expansion: first-stage prompt reframing implemented behind hidden `brainstorm --prompt-variants per-model`; next run L3 benchmarks and defer domain collisions - **021** — evaluate TOON (`toon-format/toon`) for prompt-facing artifact export / benchmark fixtures @@ -59,7 +59,8 @@ Triage pattern: fix P1/P2 with code, create TODOs for P3/nitpicks, reply to ever ## Recent Context -- 2026-06-12 post-`dc805a7` L3 parser validation completed (`docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`). Ran one paired architecture prompt using Codex/GLM/Kimi-for-coding. Baseline `off` degraded with one GLM invalid eval score; expanded `per-model` degraded with three Kimi 429 overload/rate-limit failures and one Kimi invalid eval score. Analyzer used exactly the two fresh run dirs in `target/brainstorm-benchmark-2026-06-11-l3-parser-validation/logs/run-dirs.txt`. Conclusion: invalid evaluation summaries persist after parser hardening, but this does not prove another score-shape parsing gap. Added bounded raw response-preview capture for invalid brainstorm structured-response parse failures so the next rerun can distinguish malformed/empty/provider output from genuinely unhandled JSON shape. +- 2026-06-17 PR #46 (`Fix brainstorm invalid eval triage`) merged through the merge queue. It hardened brainstorm eval score parsing for scaled/labeled text, nested scores, `overall_score`, and dimension fallbacks; added bounded `response_preview` capture to invalid structured-response parse failures in CLI JSON and `provider-failures.json`; added artifact regression coverage; and addressed review feedback for spaced slash scores (`8.5 / 10`), labeled scales (`Score (1-10): 8`), and char-count preview bounds. GitHub CI and CodeRabbit passed; Buildkite was queued via merge queue and the PR landed as `8e89266`. +- 2026-06-12 post-`dc805a7` L3 parser validation completed (`docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`). Ran one paired architecture prompt using Codex/GLM/Kimi-for-coding. Baseline `off` degraded with one GLM invalid eval score; expanded `per-model` degraded with three Kimi 429 overload/rate-limit failures and one Kimi invalid eval score. Analyzer used exactly the two fresh run dirs in `target/brainstorm-benchmark-2026-06-11-l3-parser-validation/logs/run-dirs.txt`. Conclusion: invalid evaluation summaries persist after parser hardening, but this does not prove another score-shape parsing gap. PR #46 now preserves bounded raw response previews for invalid structured-response parse failures so the next rerun can distinguish malformed/empty/provider output from genuinely unhandled JSON shape. - 2026-06-11 GLM invalid-evaluation parser hardening completed (`todos/013`, plan `docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md`). `parse_brainstorm_evaluation_response()` now accepts recoverable score variants seen/plausible in expanded brainstorm evals: scaled score text like `"8 out of 10"`, nested score objects, `overall_score`, and a missing-overall fallback to the four required dimension scores. It rejects incomplete dimension sets even if extra numeric fields are present. Verified with `cargo fmt --all -- --check`, `cargo test -p refinery_core parse_brainstorm_evaluation -q`, `cargo test -p refinery_core brainstorm -q`, and `cargo clippy -p refinery_core --all-targets -- -D warnings`. - 2026-06-09 three-model L3 sample completed (`todos/013`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`) after PR #44 merged. Compared `--prompt-variants off` vs `per-model` on product and technical prompts using `pi/openai-codex/gpt-5.4:off`, `pi/zai/glm-5.1:off`, and `pi/kimi-coding/kimi-for-coding:off`, serial with `--max-concurrent 1`. Baseline runs completed clean (`18` calls each, ~7-8m). Per-model runs completed with full 12-candidate final sets but degraded evaluation status (`75` calls each, ~30-38m): product had one GLM invalid eval score; technical had one Codex SSE response-header timeout and one GLM invalid eval score. `controversy_floor_7` two-prompt averages improved mean quality `7.83 → 8.25`, min quality `7.00 → 8.00`, disagreement `0.33 → 0.75`; lexical overlap also rose `0.056 → 0.074`; meta-preamble stayed `0.0`. Promising but not enough for default changes because both expanded runs degraded. - 2026-06-05 updated-model L3 smoke completed (`todos/013`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`) after Pi exposed `pi/kimi-coding/kimi-for-coding` (Kimi K2.6 for coding) and `pi/minimax/MiniMax-M3`. Single-model smoke calls for both worked. A two-model product baseline (`prompt-variants off`) completed clean with `total_calls: 8`, `degraded: false`, `controversy_floor_7` mean/min quality `7.50/7.00`, lexical overlap `0.073`. A two-model prompt-reframing run completed degraded with `total_calls: 25/26`, final candidates `5`, and one MiniMax M3 round-2 proposal timeout on the legal-scrutiny variant after 900s; `controversy_floor_7` mean/min quality `8.33/8.00`, lexical overlap `0.080`, meta-preamble `0.0`. Because two-model runs have only one evaluator per candidate, disagreement/controversy is not meaningful. A four-model updated sample (Codex + GLM + Kimi-for-coding + MiniMax M3) was stopped after ~14 minutes while still in the first baseline run; partial artifacts showed round-1 progress, so treat it as a budget/runtime caution rather than a correctness failure. Keep production defaults unchanged. @@ -155,6 +156,6 @@ Recommended order: 1. If continuing Buildkite migration, review PR #39 and either trigger a real Buildkite run against `ci-linux-arm64-rust-bazel` or update the Buildkite pipeline settings to upload `.buildkite/pipeline.yml` from the repo so PR pipeline changes are exercised. 2. Start from clean `main` and read this handoff plus the valid baseline in `docs/brainstorms/2026-05-23-brainstorm-smoke-baseline.md`. -3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`, and `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`; then rerun a small L3 prompt-reframing validation with the Codex/GLM/Kimi-for-coding panel and inspect new `response_preview` fields for any invalid structured-response parse failures. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls. +3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`, and `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`; then rerun a small low-call L3 validation (start with `--prompt-variants off` before expensive `per-model`) with the Codex/GLM/Kimi-for-coding panel and inspect `response_preview` fields for any invalid structured-response parse failures. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls. 4. For future Pi-backed benchmark runs, use `--max-concurrent 1` unless Pi config locking is fixed; for OpenCode-backed models use `--max-concurrent 1` and `--idle-timeout 480` until `todos/022` is fixed. 5. Do not implement Open Collider-style domain collisions before benchmark budget constraints are explicit; if moving to L3, start with prompt-reframing expansion from `todos/018`. diff --git a/todos/013-brainstorm-strategy-benchmarks.md b/todos/013-brainstorm-strategy-benchmarks.md index 9a19cd5..ac4d4f0 100644 --- a/todos/013-brainstorm-strategy-benchmarks.md +++ b/todos/013-brainstorm-strategy-benchmarks.md @@ -4,7 +4,7 @@ priority: low milestone: v0.4 depends_on: 004-verb-brainstorm status: in_progress -updated: 2026-06-11 +updated: 2026-06-17 --- # Benchmark: Brainstorm Iteration and Selection Strategies @@ -104,11 +104,11 @@ A first-pass qualitative review over the generated blind panel review pack is co Latest L3 three-model sample is documented in `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`. It compared `--prompt-variants off` vs `per-model` on product and technical prompts with Codex, GLM, and Kimi-for-coding. Per-model improved two-prompt `controversy_floor_7` average quality floor (`7.00` → `8.00`) and disagreement (`0.33` → `0.75`), but both per-model runs degraded due to evaluation issues (GLM invalid eval scores; one Codex SSE header timeout), so it cannot support default changes. -A verified parser hardening pass for recoverable GLM-style invalid evaluation scores completed on 2026-06-11. Brainstorm evaluation parsing now accepts scaled score text, nested score objects, `overall_score`, and a missing-overall fallback to the four required dimension scores while rejecting incomplete dimension sets. Verified with targeted parser tests, `cargo test -p refinery_core brainstorm -q`, and `cargo clippy -p refinery_core --all-targets -- -D warnings`. +A verified parser hardening and invalid-response observability pass landed in PR #46 on 2026-06-17. Brainstorm evaluation parsing now accepts scaled/labeled score text, nested score objects, `overall_score`, and a missing-overall fallback to the four required dimension scores while rejecting incomplete dimension sets. Invalid structured-response parse failures now preserve a bounded `response_preview` in CLI JSON and `provider-failures.json`. PR #46 also addressed review feedback for spaced slash scores (`8.5 / 10`), labeled scales (`Score (1-10): 8`), and char-count preview bounds. -A live post-hardening L3 validation on 2026-06-12 is documented in `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`. It still degraded: baseline had one GLM invalid eval score, while expanded prompt-reframing had Kimi overload/rate-limit failures plus one Kimi invalid eval score. This does not prove a remaining score-shape parser gap; the next code step is evidence capture to distinguish malformed/empty/provider output from genuinely unhandled score JSON. +A live post-hardening L3 validation on 2026-06-12 is documented in `docs/brainstorms/2026-06-12-brainstorm-l3-parser-validation.md`. It still degraded: baseline had one GLM invalid eval score, while expanded prompt-reframing had Kimi overload/rate-limit failures plus one Kimi invalid eval score. This does not prove a remaining score-shape parser gap; with PR #46 merged, the next degraded run should preserve raw preview evidence to distinguish malformed/empty/provider output from genuinely unhandled score JSON. -Next concrete step: rerun a small L3 validation and inspect bounded `response_preview` fields for any invalid structured-response parse failures, or run a human/calibrated model-judge pass over the L2/L3 panel findings. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit. +Next concrete step: rerun a small low-call L3 validation, starting with `--prompt-variants off`, and inspect bounded `response_preview` fields for any invalid structured-response parse failures before spending budget on another expanded `per-model` run. Alternatively, run a human/calibrated model-judge pass over the L2/L3 panel findings. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit. ## References