docs: record three-model L3 brainstorm sample

docs/HANDOFF.md

@@ -2,7 +2,7 @@
 
 Current state of the project and active work. Read this at session start. Update before compaction or at natural breakpoints.
 
-**Last updated:** 2026-06-05
+**Last updated:** 2026-06-09
 
 ## Project State
 
@@ -45,7 +45,7 @@ See `memory/verb_architecture.md` for full taxonomy with consistent terminology.
 
 Check `todos/` for the full list. Key ones:
 
-- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, and updated-model 2-model L3 smoke completed; next run budgeted L3 prompt-reframing benchmarks with `score-only` as baseline, but avoid full MiniMax M3 suites until runtime/output budget controls are explicit
+- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, updated-model 2-model L3 smoke, and two-prompt 3-model L3 sample completed; next step is either 2-4 more L3 prompts with Codex/GLM/Kimi-for-coding or triage GLM invalid eval scores on expanded runs; avoid full MiniMax M3 suites until runtime/output budget controls are explicit
 - **025** — optional brainstorm lineage-reference polish if softer phrases like "builds on..." feel too process-oriented in demos
 - **018** — brainstorm divergence expansion: first-stage prompt reframing implemented behind hidden `brainstorm --prompt-variants per-model`; next run L3 benchmarks and defer domain collisions
 - **021** — evaluate TOON (`toon-format/toon`) for prompt-facing artifact export / benchmark fixtures
@@ -59,6 +59,7 @@ Triage pattern: fix P1/P2 with code, create TODOs for P3/nitpicks, reply to ever
 
 ## Recent Context
 
+- 2026-06-09 three-model L3 sample completed (`todos/013`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`) after PR #44 merged. Compared `--prompt-variants off` vs `per-model` on product and technical prompts using `pi/openai-codex/gpt-5.4:off`, `pi/zai/glm-5.1:off`, and `pi/kimi-coding/kimi-for-coding:off`, serial with `--max-concurrent 1`. Baseline runs completed clean (`18` calls each, ~7-8m). Per-model runs completed with full 12-candidate final sets but degraded evaluation status (`75` calls each, ~30-38m): product had one GLM invalid eval score; technical had one Codex SSE response-header timeout and one GLM invalid eval score. `controversy_floor_7` two-prompt averages improved mean quality `7.83 → 8.25`, min quality `7.00 → 8.00`, disagreement `0.33 → 0.75`; lexical overlap also rose `0.056 → 0.074`; meta-preamble stayed `0.0`. Promising but not enough for default changes because both expanded runs degraded.
 - 2026-06-05 updated-model L3 smoke completed (`todos/013`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`) after Pi exposed `pi/kimi-coding/kimi-for-coding` (Kimi K2.6 for coding) and `pi/minimax/MiniMax-M3`. Single-model smoke calls for both worked. A two-model product baseline (`prompt-variants off`) completed clean with `total_calls: 8`, `degraded: false`, `controversy_floor_7` mean/min quality `7.50/7.00`, lexical overlap `0.073`. A two-model prompt-reframing run completed degraded with `total_calls: 25/26`, final candidates `5`, and one MiniMax M3 round-2 proposal timeout on the legal-scrutiny variant after 900s; `controversy_floor_7` mean/min quality `8.33/8.00`, lexical overlap `0.080`, meta-preamble `0.0`. Because two-model runs have only one evaluator per candidate, disagreement/controversy is not meaningful. A four-model updated sample (Codex + GLM + Kimi-for-coding + MiniMax M3) was stopped after ~14 minutes while still in the first baseline run; partial artifacts showed round-1 progress, so treat it as a budget/runtime caution rather than a correctness failure. Keep production defaults unchanged.
 - 2026-06-04 Pi stream parsing completed (`todos/026`, plan `docs/plans/2026-06-04-002-fix-stream-parse-pi-json-events-plan.md`) after L3 prompt-reframing smoke re-exposed the 64MB Pi stdout cap. Added `process::spawn_cli_stream_lines()` with stderr draining, timeout/idle-timeout handling, and bounded error previews; `PiProvider` now feeds JSONL lines into a stateful parser shared with `extract_pi_response()`. The pre-fix 3-model L3 smoke degraded with Codex `ResponseTooLarge` (~64MB) and invalid GLM eval scores. The post-fix rerun completed `degraded: false`, `evaluation_status: peer_evaluated`, `total_calls: 75`, no provider failures, empty stderr. Smoke report: `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`. Verified with `cargo fmt --all -- --check`, `cargo test -p tundish_providers pi -q`, streaming process test, `cargo clippy -p tundish_providers --all-targets -- -D warnings`, `cargo test --workspace --no-fail-fast`, and `cargo clippy --workspace --all-targets -- -D warnings`.
 - 2026-06-04 prompt-reframing expansion implementation completed on branch `feat/brainstorm-prompt-reframing-l3` (`todos/018`, plan `docs/plans/2026-06-04-001-feat-brainstorm-prompt-reframing-expansion-plan.md`). Added hidden/internal `brainstorm --prompt-variants off|per-model`; `per-model` first asks each provider for one strategic prompt reframing, then runs every provider over the original anchor plus all generated variants. Expanded lineages use flat `ModelId`s like `provider/model+variant-1`, preserving the existing `round-N/propose-*.md` / `evaluate-*.json` benchmark artifact layout. Peer evaluation remains provider-owned, so a provider evaluates other providers' lineages but not its own. CLI text/JSON/dry-run output and `metadata.json` expose prompt-variant strategy/counts; `benchmark-brainstorm` surfaces prompt-variant metadata. Final verification observed clean: `cargo fmt --all -- --check`, `cargo check --workspace`, targeted `cargo test -p refinery_core brainstorm -q`, `cargo test -p refinery_core prompts -q`, `cargo test -p refinery_cli brainstorm -q`, full `cargo test --workspace`, targeted clippy for core/CLI, and `cargo clippy --workspace --all-targets -- -D warnings`. Manual dry-run and synthetic `benchmark-brainstorm` text/JSON smoke checks passed. Next step: run real Pi-backed L3 prompt-reframing benchmark using `score-only` baseline and `--max-concurrent 1`.
@@ -97,6 +98,6 @@ Recommended order:
 
 1. If continuing Buildkite migration, review PR #39 and either trigger a real Buildkite run against `ci-linux-arm64-rust-bazel` or update the Buildkite pipeline settings to upload `.buildkite/pipeline.yml` from the repo so PR pipeline changes are exercised.
 2. Start from clean `main` and read this handoff plus the valid baseline in `docs/brainstorms/2026-05-23-brainstorm-smoke-baseline.md`.
-3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, and `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`; then run budgeted L3 prompt-reframing benchmarks using hidden `brainstorm --prompt-variants per-model` with `score-only` as the baseline. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls.
+3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`, and `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`; then either run 2-4 more L3 prompt-reframing prompts with the Codex/GLM/Kimi-for-coding panel or triage GLM invalid evaluation scores on expanded prompts. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls.
 4. For future Pi-backed benchmark runs, use `--max-concurrent 1` unless Pi config locking is fixed; for OpenCode-backed models use `--max-concurrent 1` and `--idle-timeout 480` until `todos/022` is fixed.
 5. Do not implement Open Collider-style domain collisions before benchmark budget constraints are explicit; if moving to L3, start with prompt-reframing expansion from `todos/018`.

docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md

@@ -0,0 +1,108 @@
+---
+date: 2026-06-09
+topic: brainstorm-l3-three-model-sample
+todo: 013-brainstorm-strategy-benchmarks
+plan: 2026-05-23-001-research-brainstorm-strategy-benchmarks-plan
+related_todo: 018-brainstorm-divergence-expansion
+---
+
+# Brainstorm L3 Three-Model Sample
+
+## Summary
+
+Ran a two-prompt L3 comparison using three Pi-routed models and the merged hidden prompt-reframing expansion:
+
+```text
+pi/openai-codex/gpt-5.4:off
+pi/zai/glm-5.1:off
+pi/kimi-coding/kimi-for-coding:off
+```
+
+Compared:
+
+- baseline: `--prompt-variants off`
+- L3 expansion: `--prompt-variants per-model`
+
+Common settings:
+
+```text
+--max-rounds 2
+--panel-size 3
+--quality-floor 7.0
+--iteration-strategy score-only
+--idle-timeout 480
+--timeout 1800
+--max-concurrent 1
+```
+
+Artifact root:
+
+```text
+target/brainstorm-benchmark-2026-06-09-l3-three-model-sample/
+```
+
+Analyzer outputs:
+
+```text
+target/brainstorm-benchmark-2026-06-09-l3-three-model-sample/logs/l3-three-model-sample-analysis.json
+target/brainstorm-benchmark-2026-06-09-l3-three-model-sample/logs/l3-three-model-sample-analysis.txt
+```
+
+## Prompt Suite
+
+1. Product/strategy — privacy-first personal knowledge assistant.
+2. Technical/design — secretless multi-model brainstorm artifact format.
+
+## Run Results
+
+| Prompt | Prompt variants | Status | Eval status | Calls | Elapsed | Provider failures |
+|---|---|---|---|---:|---:|---|
+| product | `off` | `brainstormed` | `peer_evaluated` | 18 | ~7.1m | none |
+| product | `per-model` | `degraded` | `partial` | 75 | ~37.8m | GLM invalid eval score |
+| technical | `off` | `brainstormed` | `peer_evaluated` | 18 | ~7.9m | none |
+| technical | `per-model` | `degraded` | `partial` | 75 | ~30.3m | Codex SSE header timeout; GLM invalid eval score |
+
+The per-model runs produced complete final-round candidate sets despite degraded evaluation status:
+
+- product per-model: 12 candidates
+- technical per-model: 12 candidates
+
+## Production-Selector Metrics
+
+`controversy_floor_7` view:
+
+| Prompt | Prompt variants | Mean quality | Min quality | Disagreement | Lexical overlap | Meta preamble rate |
+|---|---|---:|---:|---:|---:|---:|
+| product | `off` | 8.33 | 7.50 | 0.33 | 0.056 | 0.00 |
+| product | `per-model` | 8.17 | 8.00 | 0.83 | 0.102 | 0.00 |
+| technical | `off` | 7.33 | 6.50 | 0.33 | 0.056 | 0.00 |
+| technical | `per-model` | 8.33 | 8.00 | 0.67 | 0.045 | 0.00 |
+
+Two-prompt averages:
+
+| Prompt variants | Mean quality | Min quality | Disagreement | Lexical overlap | Meta preamble rate |
+|---|---:|---:|---:|---:|---:|
+| `off` | 7.83 | 7.00 | 0.33 | 0.056 | 0.00 |
+| `per-model` | 8.25 | 8.00 | 0.75 | 0.074 | 0.00 |
+
+## Observations
+
+- The three-model L3 sample is operationally feasible but expensive: the paired two-prompt sample took ~83 minutes wall-clock with serial Pi calls.
+- Prompt reframing improved the two-prompt average quality floor (`7.00` → `8.00`) and disagreement (`0.33` → `0.75`) under `controversy_floor_7`.
+- Lexical overlap increased on the product prompt (`0.056` → `0.102`) but decreased on the technical prompt (`0.056` → `0.045`), so there is no simple diversity conclusion from this small sample.
+- Both per-model runs degraded from evaluation issues rather than proposal collapse:
+  - GLM produced invalid brainstorm evaluation scores in both per-model runs.
+  - Codex hit an SSE response header timeout during one technical evaluation.
+- The per-model product run initially looked stalled because artifact stdout is only written at process completion; inspecting child processes showed provider calls were still running, and the run completed successfully at the candidate-artifact level.
+- `meta_preamble_rate` stayed at `0.0` across all selectors and runs.
+
+## Recommendation
+
+Do not change production defaults based on this sample.
+
+Next benchmark step should be one of:
+
+1. run 2-4 more prompts with the same three-model panel to see whether the quality-floor gain survives degraded evaluation noise, or
+2. first harden/triage GLM invalid evaluation scores for expanded brainstorm evaluation prompts, since both per-model runs degraded on that failure mode.
+
+Keep `--max-concurrent 1` for Pi-backed benchmark runs.

docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md

@@ -15,6 +15,7 @@ todo: 013-brainstorm-strategy-benchmarks
 **Addendum:** 2026-05-30 — Added blind panel review pack generator (`refinery review-brainstorm-panels`) and generated the first L2 review packet.
 **Addendum:** 2026-06-01 — Completed first-pass qualitative L2 panel review; see `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`.
 **Addendum:** 2026-06-05 — Ran an updated-model L3 smoke with `pi/kimi-coding/kimi-for-coding:off` and `pi/minimax/MiniMax-M3:off`; see `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`.
+**Addendum:** 2026-06-09 — Ran a two-prompt three-model L3 comparison (`off` vs `per-model`) with Codex, GLM, and Kimi-for-coding; see `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`.
 
 ## Context
 
@@ -177,7 +178,7 @@ Result: `score-only` remained strongest on useful diversity/non-overlap; `full-v
 
 ## Next Implementation Step
 
-Continue `todos/013` with either a human/calibrated model-judge pass over `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md` or a carefully budgeted L3 prompt-reframing comparison from `todos/018`. The 2026-06-05 updated-model smoke showed Kimi-for-coding and MiniMax M3 are available through Pi, but MiniMax M3 can dominate runtime and timed out on one expanded product-prompt lineage; do not launch a full 4-model × 6-prompt L3 suite until latency/output budget controls are explicit. Do not change the production default based on the first-pass L2 review or this small L3 smoke alone.
+Continue `todos/013` with either a human/calibrated model-judge pass over `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, 2-4 more L3 prompts with the Codex/GLM/Kimi-for-coding panel, or hardening/triage for GLM invalid evaluation scores on expanded prompt-reframing runs. The 2026-06-05 updated-model smoke showed Kimi-for-coding and MiniMax M3 are available through Pi, but MiniMax M3 can dominate runtime and timed out on one expanded product-prompt lineage; do not launch a full 4-model × 6-prompt L3 suite until latency/output budget controls are explicit. The 2026-06-09 three-model sample showed promising quality-floor/disagreement gains for `per-model`, but both expanded runs degraded due to evaluator failures. Do not change the production default based on the first-pass L2 review or small L3 samples alone.
 
 ## Verification
 
@@ -193,6 +194,7 @@ Completed:
 - Blind panel review pack generator added as `refinery review-brainstorm-panels`; L2 review pack and answer key generated under the same logs directory.
 - First-pass L2 panel review documented in `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`.
 - Updated-model L3 smoke documented in `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`.
+- Three-model L3 sample documented in `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`.
 - `cargo fmt --all -- --check`
 - `cargo test -p refinery_core brainstorm`
 - `cargo test -p refinery_cli`

todos/013-brainstorm-strategy-benchmarks.md

@@ -4,7 +4,7 @@ priority: low
 milestone: v0.4
 depends_on: 004-verb-brainstorm
 status: in_progress
-updated: 2026-06-05
+updated: 2026-06-09
 ---
 
 # Benchmark: Brainstorm Iteration and Selection Strategies
@@ -102,9 +102,9 @@ The fixed six-prompt suite has now been run for all four L2 variants with Pi-bac
 
 A first-pass qualitative review over the generated blind panel review pack is complete. Result: `score-only` still looked strongest on useful diversity and non-overlap; `full-visibility` looked strongest on actionability and coverage; `own-reviews` did not dominate globally but produced the strongest debugging/process panel. Keep production default as `score-only` until stronger human/calibrated model-judge evidence says otherwise.
 
-Latest L3 smoke with updated Pi models (`pi/kimi-coding/kimi-for-coding:off`, `pi/minimax/MiniMax-M3:off`) is documented in `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`. A two-model product prompt showed prompt reframing can increase the candidate pool and apparent quality, but the run degraded on a MiniMax M3 timeout and cannot support selector/default changes.
+Latest L3 three-model sample is documented in `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`. It compared `--prompt-variants off` vs `per-model` on product and technical prompts with Codex, GLM, and Kimi-for-coding. Per-model improved two-prompt `controversy_floor_7` average quality floor (`7.00` → `8.00`) and disagreement (`0.33` → `0.75`), but both per-model runs degraded due to evaluation issues (GLM invalid eval scores; one Codex SSE header timeout), so it cannot support default changes.
 
-Next concrete step: either run a human/calibrated model-judge pass over the L2 panel review findings, or run a carefully budgeted L3 prompt-reframing comparison. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit.
+Next concrete step: either run a human/calibrated model-judge pass over the L2/L3 panel findings, run 2-4 more L3 prompts with the same three-model panel, or first harden/triage GLM invalid evaluation scores on expanded prompts. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit.
 
 ## References