Skip to content

Reject corpus length mismatches when using cached references - #303

Open
sueun-dev wants to merge 1 commit into
mjpost:masterfrom
sueun-dev:fix-cached-corpus-length-20260917
Open

sueun-dev wants to merge 1 commit into
mjpost:masterfrom
sueun-dev:fix-cached-corpus-length-20260917

Conversation

@sueun-dev

Copy link
Copy Markdown

corpus_score(hyps, None) silently truncates to the shorter of the hypotheses and the cached references. With one cached reference, a matching first hypothesis and an extra incorrect hypothesis still produce a perfect BLEU/chrF score or zero TER. Passing those references explicitly already raises a length error after #294; this covers the cached path left over from that validation work (related to #220 and #281).

Check the selected reference cache's length before extracting statistics. Putting the check here also covers PairedTest, which calls the extractor directly. Tests cover shorter and longer hypothesis lists across BLEU/chrF/TER, valid cached scores with missing alternative references, explicit-reference overrides, sentence scoring, and mismatched baseline/candidate inputs in serial and multiprocessing significance tests.

Checked on Python 3.12.14, Linux arm64:

  • Final regression file: 14 failures on the original source; 64 passes with the fix.
  • python -m pytest -q --ignore=test/test_dataset.py -k 'not test_api_get_source and not test_api_get_reference': 197 passed, 6 deselected.
  • mypy sacrebleu scripts test and Ruff on all tracked Python files passed.
  • Source distribution and wheel built offline with synthetic 2.6.1.dev0 version metadata.

The dataset-download tests, CLI shell suite, optional Japanese/Korean tokenizers, and other Python/OS combinations were not run. An initial Ruff run also scanned generated build/version files in the source archive; the tracked-source check follows the repository's ignore rules.

Signed-off-by: Sueun Cho <sueun.dev@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant