extract_llm drops task-relevant lines: wrong answers in live Claude Code runs #577

Summary

In live Claude Code runs with extract_llm on, the final answer of a simple log-reading task was wrong in every run (4 of 4). The same task with the component off was wrong in 1 of 4. The extraction was "accepted" every time; nothing flagged the lost content.

Setup (synthetic task, my own runs)

• Claude Code 2.1.299 through a proxy built from fix/replay-cap-550 (fix(dash): cap extract_llm and offloader replay credit at the counterfactual client compaction (#550) #573), then LiteLLM, then Azure gpt-5.2; extractor model gpt-5-mini.
• Task: 20 generated log files (about 170 INFO lines each, 3 to 6 ERROR ... code=E-#### lines hidden among them), read with cat one file per tool call. Final message: the number of distinct error codes (expected 47).
• ON: extract_llm only, fire_on: size, min_tokens: 1000, economic_gate: false, allow_on_caching_backend: true, skip_file_reads: false. OFF: passthrough proxy.
• 4 paired sessions (ABBA). ON proxy restarted before each ON session.

Evidence

• Final count: ON 32, 32, 32, 32 against the true 47. OFF: 47 in three runs, wrong once.
• In one ON session, 20 extraction calls, all accepted, all strategy: deterministic (the model-free projection). The candidate outputs held 86 ERROR lines; the extracted versions kept 41. 18 of the 20 outputs lost at least one.
• Example (synthetic file 17): 4 error lines in, 3 out. The call elided a 4,608-character middle block, and the dropped line is in it:

2026-03-17 10:32:53 ERROR worker-6 operation failed code=E-1040 retry=exhausted

The agent's recent turns asked only for "distinct ERROR codes", and the head/tail window did not keep every ERROR line.

Hypothesis

The extractor cannot know the user's question. The accepted result is checked for derivation from the input, sanity and being smaller, but not for whether the next step of the task still has what it needs. With the gate off and fire_on: size, the deterministic projection keeps a head and tail window and elides the middle, which is exactly where sparse, task-relevant lines live. (This lab uses the settings above, not a default; the defect may be weaker in the shipped presets, which is part of what a test should show.)

Proposal

Add a task-success check to the acceptance gate, and a regression test for it:

1. Test: replay a fixed set of tool outputs with a known "needle" set (for example every line matching a pattern named in the recent turns, or the lines the following assistant turn quotes). Fail if an accepted extraction drops any needle line.
2. Gate: before accepting, check that lines matching identifiers or patterns from the last user and assistant turns (HarvestIdentifiers already builds such a keep-list) all survive; reject the deterministic window projection when they do not.
3. Report rejection: dropped_needle in extraction_calls so the dashboard shows the rate.

Related: #550 and #573 (credit cap) are unrelated to this defect; this is about correctness, not accounting.