diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 14df5e6..3046fd7 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -2,7 +2,7 @@ Current state of the project and active work. Read this at session start. Update before compaction or at natural breakpoints. -**Last updated:** 2026-06-05 +**Last updated:** 2026-06-09 ## Project State @@ -45,7 +45,7 @@ See `memory/verb_architecture.md` for full taxonomy with consistent terminology. Check `todos/` for the full list. Key ones: -- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, and updated-model 2-model L3 smoke completed; next run budgeted L3 prompt-reframing benchmarks with `score-only` as baseline, but avoid full MiniMax M3 suites until runtime/output budget controls are explicit +- **013** — brainstorm strategy benchmarks (in progress): design, analyzer, six-prompt v0 suite, quality-floor follow-up, meta-preamble prompt polish, benchmark-only iteration variants, L2 six-prompt variant suite, blind review pack, first-pass qualitative L2 panel review, hidden L3 prompt-reframing implementation, 3-model L3 smoke, updated-model 2-model L3 smoke, and two-prompt 3-model L3 sample completed; next step is either 2-4 more L3 prompts with Codex/GLM/Kimi-for-coding or triage GLM invalid eval scores on expanded runs; avoid full MiniMax M3 suites until runtime/output budget controls are explicit - **025** — optional brainstorm lineage-reference polish if softer phrases like "builds on..." feel too process-oriented in demos - **018** — brainstorm divergence expansion: first-stage prompt reframing implemented behind hidden `brainstorm --prompt-variants per-model`; next run L3 benchmarks and defer domain collisions - **021** — evaluate TOON (`toon-format/toon`) for prompt-facing artifact export / benchmark fixtures @@ -59,6 +59,7 @@ Triage pattern: fix P1/P2 with code, create TODOs for P3/nitpicks, reply to ever ## Recent Context +- 2026-06-09 three-model L3 sample completed (`todos/013`, `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`) after PR #44 merged. Compared `--prompt-variants off` vs `per-model` on product and technical prompts using `pi/openai-codex/gpt-5.4:off`, `pi/zai/glm-5.1:off`, and `pi/kimi-coding/kimi-for-coding:off`, serial with `--max-concurrent 1`. Baseline runs completed clean (`18` calls each, ~7-8m). Per-model runs completed with full 12-candidate final sets but degraded evaluation status (`75` calls each, ~30-38m): product had one GLM invalid eval score; technical had one Codex SSE response-header timeout and one GLM invalid eval score. `controversy_floor_7` two-prompt averages improved mean quality `7.83 → 8.25`, min quality `7.00 → 8.00`, disagreement `0.33 → 0.75`; lexical overlap also rose `0.056 → 0.074`; meta-preamble stayed `0.0`. Promising but not enough for default changes because both expanded runs degraded. - 2026-06-05 updated-model L3 smoke completed (`todos/013`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`) after Pi exposed `pi/kimi-coding/kimi-for-coding` (Kimi K2.6 for coding) and `pi/minimax/MiniMax-M3`. Single-model smoke calls for both worked. A two-model product baseline (`prompt-variants off`) completed clean with `total_calls: 8`, `degraded: false`, `controversy_floor_7` mean/min quality `7.50/7.00`, lexical overlap `0.073`. A two-model prompt-reframing run completed degraded with `total_calls: 25/26`, final candidates `5`, and one MiniMax M3 round-2 proposal timeout on the legal-scrutiny variant after 900s; `controversy_floor_7` mean/min quality `8.33/8.00`, lexical overlap `0.080`, meta-preamble `0.0`. Because two-model runs have only one evaluator per candidate, disagreement/controversy is not meaningful. A four-model updated sample (Codex + GLM + Kimi-for-coding + MiniMax M3) was stopped after ~14 minutes while still in the first baseline run; partial artifacts showed round-1 progress, so treat it as a budget/runtime caution rather than a correctness failure. Keep production defaults unchanged. - 2026-06-04 Pi stream parsing completed (`todos/026`, plan `docs/plans/2026-06-04-002-fix-stream-parse-pi-json-events-plan.md`) after L3 prompt-reframing smoke re-exposed the 64MB Pi stdout cap. Added `process::spawn_cli_stream_lines()` with stderr draining, timeout/idle-timeout handling, and bounded error previews; `PiProvider` now feeds JSONL lines into a stateful parser shared with `extract_pi_response()`. The pre-fix 3-model L3 smoke degraded with Codex `ResponseTooLarge` (~64MB) and invalid GLM eval scores. The post-fix rerun completed `degraded: false`, `evaluation_status: peer_evaluated`, `total_calls: 75`, no provider failures, empty stderr. Smoke report: `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`. Verified with `cargo fmt --all -- --check`, `cargo test -p tundish_providers pi -q`, streaming process test, `cargo clippy -p tundish_providers --all-targets -- -D warnings`, `cargo test --workspace --no-fail-fast`, and `cargo clippy --workspace --all-targets -- -D warnings`. - 2026-06-04 prompt-reframing expansion implementation completed on branch `feat/brainstorm-prompt-reframing-l3` (`todos/018`, plan `docs/plans/2026-06-04-001-feat-brainstorm-prompt-reframing-expansion-plan.md`). Added hidden/internal `brainstorm --prompt-variants off|per-model`; `per-model` first asks each provider for one strategic prompt reframing, then runs every provider over the original anchor plus all generated variants. Expanded lineages use flat `ModelId`s like `provider/model+variant-1`, preserving the existing `round-N/propose-*.md` / `evaluate-*.json` benchmark artifact layout. Peer evaluation remains provider-owned, so a provider evaluates other providers' lineages but not its own. CLI text/JSON/dry-run output and `metadata.json` expose prompt-variant strategy/counts; `benchmark-brainstorm` surfaces prompt-variant metadata. Final verification observed clean: `cargo fmt --all -- --check`, `cargo check --workspace`, targeted `cargo test -p refinery_core brainstorm -q`, `cargo test -p refinery_core prompts -q`, `cargo test -p refinery_cli brainstorm -q`, full `cargo test --workspace`, targeted clippy for core/CLI, and `cargo clippy --workspace --all-targets -- -D warnings`. Manual dry-run and synthetic `benchmark-brainstorm` text/JSON smoke checks passed. Next step: run real Pi-backed L3 prompt-reframing benchmark using `score-only` baseline and `--max-concurrent 1`. @@ -97,6 +98,6 @@ Recommended order: 1. If continuing Buildkite migration, review PR #39 and either trigger a real Buildkite run against `ci-linux-arm64-rust-bazel` or update the Buildkite pipeline settings to upload `.buildkite/pipeline.yml` from the repo so PR pipeline changes are exercised. 2. Start from clean `main` and read this handoff plus the valid baseline in `docs/brainstorms/2026-05-23-brainstorm-smoke-baseline.md`. -3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, and `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`; then run budgeted L3 prompt-reframing benchmarks using hidden `brainstorm --prompt-variants per-model` with `score-only` as the baseline. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls. +3. If continuing brainstorm strategy work, read `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, `docs/brainstorms/2026-06-04-brainstorm-l3-prompt-reframing-smoke.md`, `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`, and `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`; then either run 2-4 more L3 prompt-reframing prompts with the Codex/GLM/Kimi-for-coding panel or triage GLM invalid evaluation scores on expanded prompts. Do not launch a full MiniMax M3-heavy suite without explicit runtime/output budget controls. 4. For future Pi-backed benchmark runs, use `--max-concurrent 1` unless Pi config locking is fixed; for OpenCode-backed models use `--max-concurrent 1` and `--idle-timeout 480` until `todos/022` is fixed. 5. Do not implement Open Collider-style domain collisions before benchmark budget constraints are explicit; if moving to L3, start with prompt-reframing expansion from `todos/018`. diff --git a/docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md b/docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md new file mode 100644 index 0000000..ec4b659 --- /dev/null +++ b/docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md @@ -0,0 +1,108 @@ +--- +date: 2026-06-09 +topic: brainstorm-l3-three-model-sample +todo: 013-brainstorm-strategy-benchmarks +plan: 2026-05-23-001-research-brainstorm-strategy-benchmarks-plan +related_todo: 018-brainstorm-divergence-expansion +--- + +# Brainstorm L3 Three-Model Sample + +## Summary + +Ran a two-prompt L3 comparison using three Pi-routed models and the merged hidden prompt-reframing expansion: + +```text +pi/openai-codex/gpt-5.4:off +pi/zai/glm-5.1:off +pi/kimi-coding/kimi-for-coding:off +``` + +Compared: + +- baseline: `--prompt-variants off` +- L3 expansion: `--prompt-variants per-model` + +Common settings: + +```text +--max-rounds 2 +--panel-size 3 +--quality-floor 7.0 +--iteration-strategy score-only +--idle-timeout 480 +--timeout 1800 +--max-concurrent 1 +``` + +Artifact root: + +```text +target/brainstorm-benchmark-2026-06-09-l3-three-model-sample/ +``` + +Analyzer outputs: + +```text +target/brainstorm-benchmark-2026-06-09-l3-three-model-sample/logs/l3-three-model-sample-analysis.json +target/brainstorm-benchmark-2026-06-09-l3-three-model-sample/logs/l3-three-model-sample-analysis.txt +``` + +## Prompt Suite + +1. Product/strategy — privacy-first personal knowledge assistant. +2. Technical/design — secretless multi-model brainstorm artifact format. + +## Run Results + +| Prompt | Prompt variants | Status | Eval status | Calls | Elapsed | Provider failures | +|---|---|---|---|---:|---:|---| +| product | `off` | `brainstormed` | `peer_evaluated` | 18 | ~7.1m | none | +| product | `per-model` | `degraded` | `partial` | 75 | ~37.8m | GLM invalid eval score | +| technical | `off` | `brainstormed` | `peer_evaluated` | 18 | ~7.9m | none | +| technical | `per-model` | `degraded` | `partial` | 75 | ~30.3m | Codex SSE header timeout; GLM invalid eval score | + +The per-model runs produced complete final-round candidate sets despite degraded evaluation status: + +- product per-model: 12 candidates +- technical per-model: 12 candidates + +## Production-Selector Metrics + +`controversy_floor_7` view: + +| Prompt | Prompt variants | Mean quality | Min quality | Disagreement | Lexical overlap | Meta preamble rate | +|---|---|---:|---:|---:|---:|---:| +| product | `off` | 8.33 | 7.50 | 0.33 | 0.056 | 0.00 | +| product | `per-model` | 8.17 | 8.00 | 0.83 | 0.102 | 0.00 | +| technical | `off` | 7.33 | 6.50 | 0.33 | 0.056 | 0.00 | +| technical | `per-model` | 8.33 | 8.00 | 0.67 | 0.045 | 0.00 | + +Two-prompt averages: + +| Prompt variants | Mean quality | Min quality | Disagreement | Lexical overlap | Meta preamble rate | +|---|---:|---:|---:|---:|---:| +| `off` | 7.83 | 7.00 | 0.33 | 0.056 | 0.00 | +| `per-model` | 8.25 | 8.00 | 0.75 | 0.074 | 0.00 | + +## Observations + +- The three-model L3 sample is operationally feasible but expensive: the paired two-prompt sample took ~83 minutes wall-clock with serial Pi calls. +- Prompt reframing improved the two-prompt average quality floor (`7.00` → `8.00`) and disagreement (`0.33` → `0.75`) under `controversy_floor_7`. +- Lexical overlap increased on the product prompt (`0.056` → `0.102`) but decreased on the technical prompt (`0.056` → `0.045`), so there is no simple diversity conclusion from this small sample. +- Both per-model runs degraded from evaluation issues rather than proposal collapse: + - GLM produced invalid brainstorm evaluation scores in both per-model runs. + - Codex hit an SSE response header timeout during one technical evaluation. +- The per-model product run initially looked stalled because artifact stdout is only written at process completion; inspecting child processes showed provider calls were still running, and the run completed successfully at the candidate-artifact level. +- `meta_preamble_rate` stayed at `0.0` across all selectors and runs. + +## Recommendation + +Do not change production defaults based on this sample. + +Next benchmark step should be one of: + +1. run 2-4 more prompts with the same three-model panel to see whether the quality-floor gain survives degraded evaluation noise, or +2. first harden/triage GLM invalid evaluation scores for expanded brainstorm evaluation prompts, since both per-model runs degraded on that failure mode. + +Keep `--max-concurrent 1` for Pi-backed benchmark runs. diff --git a/docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md b/docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md index b94f896..0a354c2 100644 --- a/docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md +++ b/docs/plans/2026-05-23-001-research-brainstorm-strategy-benchmarks-plan.md @@ -15,6 +15,7 @@ todo: 013-brainstorm-strategy-benchmarks **Addendum:** 2026-05-30 — Added blind panel review pack generator (`refinery review-brainstorm-panels`) and generated the first L2 review packet. **Addendum:** 2026-06-01 — Completed first-pass qualitative L2 panel review; see `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`. **Addendum:** 2026-06-05 — Ran an updated-model L3 smoke with `pi/kimi-coding/kimi-for-coding:off` and `pi/minimax/MiniMax-M3:off`; see `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`. +**Addendum:** 2026-06-09 — Ran a two-prompt three-model L3 comparison (`off` vs `per-model`) with Codex, GLM, and Kimi-for-coding; see `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`. ## Context @@ -177,7 +178,7 @@ Result: `score-only` remained strongest on useful diversity/non-overlap; `full-v ## Next Implementation Step -Continue `todos/013` with either a human/calibrated model-judge pass over `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md` or a carefully budgeted L3 prompt-reframing comparison from `todos/018`. The 2026-06-05 updated-model smoke showed Kimi-for-coding and MiniMax M3 are available through Pi, but MiniMax M3 can dominate runtime and timed out on one expanded product-prompt lineage; do not launch a full 4-model × 6-prompt L3 suite until latency/output budget controls are explicit. Do not change the production default based on the first-pass L2 review or this small L3 smoke alone. +Continue `todos/013` with either a human/calibrated model-judge pass over `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`, 2-4 more L3 prompts with the Codex/GLM/Kimi-for-coding panel, or hardening/triage for GLM invalid evaluation scores on expanded prompt-reframing runs. The 2026-06-05 updated-model smoke showed Kimi-for-coding and MiniMax M3 are available through Pi, but MiniMax M3 can dominate runtime and timed out on one expanded product-prompt lineage; do not launch a full 4-model × 6-prompt L3 suite until latency/output budget controls are explicit. The 2026-06-09 three-model sample showed promising quality-floor/disagreement gains for `per-model`, but both expanded runs degraded due to evaluator failures. Do not change the production default based on the first-pass L2 review or small L3 samples alone. ## Verification @@ -193,6 +194,7 @@ Completed: - Blind panel review pack generator added as `refinery review-brainstorm-panels`; L2 review pack and answer key generated under the same logs directory. - First-pass L2 panel review documented in `docs/brainstorms/2026-06-01-brainstorm-l2-panel-review.md`. - Updated-model L3 smoke documented in `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`. +- Three-model L3 sample documented in `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`. - `cargo fmt --all -- --check` - `cargo test -p refinery_core brainstorm` - `cargo test -p refinery_cli` diff --git a/todos/013-brainstorm-strategy-benchmarks.md b/todos/013-brainstorm-strategy-benchmarks.md index d3341a1..1cb572b 100644 --- a/todos/013-brainstorm-strategy-benchmarks.md +++ b/todos/013-brainstorm-strategy-benchmarks.md @@ -4,7 +4,7 @@ priority: low milestone: v0.4 depends_on: 004-verb-brainstorm status: in_progress -updated: 2026-06-05 +updated: 2026-06-09 --- # Benchmark: Brainstorm Iteration and Selection Strategies @@ -102,9 +102,9 @@ The fixed six-prompt suite has now been run for all four L2 variants with Pi-bac A first-pass qualitative review over the generated blind panel review pack is complete. Result: `score-only` still looked strongest on useful diversity and non-overlap; `full-visibility` looked strongest on actionability and coverage; `own-reviews` did not dominate globally but produced the strongest debugging/process panel. Keep production default as `score-only` until stronger human/calibrated model-judge evidence says otherwise. -Latest L3 smoke with updated Pi models (`pi/kimi-coding/kimi-for-coding:off`, `pi/minimax/MiniMax-M3:off`) is documented in `docs/brainstorms/2026-06-05-brainstorm-l3-updated-model-smoke.md`. A two-model product prompt showed prompt reframing can increase the candidate pool and apparent quality, but the run degraded on a MiniMax M3 timeout and cannot support selector/default changes. +Latest L3 three-model sample is documented in `docs/brainstorms/2026-06-09-brainstorm-l3-three-model-sample.md`. It compared `--prompt-variants off` vs `per-model` on product and technical prompts with Codex, GLM, and Kimi-for-coding. Per-model improved two-prompt `controversy_floor_7` average quality floor (`7.00` → `8.00`) and disagreement (`0.33` → `0.75`), but both per-model runs degraded due to evaluation issues (GLM invalid eval scores; one Codex SSE header timeout), so it cannot support default changes. -Next concrete step: either run a human/calibrated model-judge pass over the L2 panel review findings, or run a carefully budgeted L3 prompt-reframing comparison. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit. +Next concrete step: either run a human/calibrated model-judge pass over the L2/L3 panel findings, run 2-4 more L3 prompts with the same three-model panel, or first harden/triage GLM invalid evaluation scores on expanded prompts. For L3, use `score-only` as the baseline, treat `own-reviews` as optional, and avoid launching a full 4-model × 6-prompt suite with MiniMax M3 until latency/output budget controls are explicit. ## References