This repository studies whether language models use the same evidence differently when only its position in a long context changes. It compares Transformer, state-space, and hybrid sequence mixers with a paired, ten-position intervention derived from the Lost in the Middle evaluation.
Paper | Evidence ledger | Artifacts | Runbooks | Code | License
The released evaluation contains 2,655 multi-document questions, each with one answer-bearing passage and nine fixed distractors. We use 800 questions for exploratory analysis and leave 1,855 questions untouched for confirmation. For each question, we move the answer-bearing passage through positions 1 to 10 while holding the question, distractors, prompt template, and decoding configuration fixed.
Primacy is mean accuracy at positions 1 and 2 minus mean accuracy at positions 5 and 6. Recency is mean accuracy at positions 9 and 10 minus mean accuracy at positions 5 and 6. Inference uses 10,000 bootstrap resamples of complete question bundles and Holm correction across the two edge tests.
Primary comparison.
Pythia-2.8B has a +5.19 percentage-point primacy edge, while Mamba-2.8B and Mamba-2 2.7B have edges of -0.13 and -1.81 points.
The paired Pythia-minus-Mamba primacy differences are +5.31 and +7.00 points, both with Holm p < 0.0001.
All three models show positive recency edges.
See the Phase 2 summary.
Matched 8B comparison.
The hybrid model has a larger primacy estimate than pure Mamba-2, but the paired hybrid-minus-pure effect is +1.88 points with a 95% confidence interval of [-0.56, +4.44] and Holm p = 0.1442.
The direction is consistent with the primary comparison, but the paired effect is statistically uncertain and does not establish that attention caused the difference.
See the Phase 3 summary and the matched model release of Waleffe et al. (2024).
Calibration and scale. The end-to-end calibration and key-value control show that the harness can detect a known position effect. Across five approximate size pairs, the family gap is near zero at the two smallest scales and appears in the three larger pairs, but capability and architecture remain confounded. See the Phase 1 summary, Phase 4 summary, and the original Pythia, Mamba, and Mamba-2 papers.
Corpus and task checks. Changing the Mamba-2.8B pretraining corpus changes overall accuracy but produces no detectable primacy or recency shape change in this executed contrast. On RULER at 2K tokens, Pythia reproduces a primacy edge, while both Mamba models saturate at perfect accuracy and therefore do not support a mixer comparison on that task. See the Phase 5 summary and Phase 6 summary.
Mechanistic evidence. Late-layer attention-sink mass tracks Pythia primacy across scale, position remains linearly decodable in both model families, and prompt variants substantially change the measured edges. These results are correlational and diagnostic, not evidence of a causal mechanism. See the Phase 7 summary.
Production systems. Nemotron-H-8B, Llama-3.1-8B, and Qwen2.5-7B all show positive primacy edges, but their many architectural and training differences make this a descriptive prevalence check rather than an architecture test. See the Phase 8 summary.
The committed summaries regenerate every canonical SVG and PDF figure deterministically.
uv sync --extra test
uv run pytest -q
uv run python paper/generate_figures.pyGPU execution, checkpoint validation, and phase-specific analysis commands are documented in the runbooks. The figure tests check deterministic regeneration, expected labels, and source-summary provenance.
- All reported ten-document QA results are exploratory, and the 1,855-question confirmatory split remains unopened.
- Negative-control and distractor-order runs have no committed outputs, and the prepared manual audit has no human labels.
- The matched 8B paired effect is statistically uncertain, and the mechanism analyses do not support a causal attention claim.
- Confidence intervals quantify variation across questions for fixed checkpoints, prompts, and decoding settings, not variation across training seeds or model checkpoints.
The paper builds on the position-intervention protocol of Liu et al. (2024) and evaluates models introduced by Biderman et al. (2023), Gu and Dao (2024), Dao and Gu (2024), and Waleffe et al. (2024). The complete scholarly bibliography is included in the paper.
The public author list and archival paper URL are not yet available. Until they are released, please cite the paper by title: Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering, New in ML workshop submission, 2026. Archival BibTeX will be added after the public author list and paper URL are available.
We thank the Indian Institute of Technology Roorkee (IIT Roorkee) for providing the computational resources that supported this research.
This project is released under the Apache License 2.0.