Skip to content

fix: include non-default BLEU n-gram order in signatures - #305

Open
leeshengis wants to merge 1 commit into
mjpost:masterfrom
leeshengis:fix/bleu-ngram-signature
Open

leeshengis wants to merge 1 commit into
mjpost:masterfrom
leeshengis:fix/bleu-ngram-signature

Conversation

@leeshengis

Copy link
Copy Markdown

Fixes #251

Summary

Include ngram:N in BLEU reproducibility signatures when max_ngram_order differs from the default of 4, using n:N for short signatures. Default signatures remain unchanged, as discussed in the issue.

This enhancement distinguishes configurations that can produce different BLEU scores. It does not change score computation.

  • Extend existing maximum-order tests to check full and short signatures for orders 1–6 while retaining the score assertions
  • Check that the implicit default signature remains unchanged
  • Document the optional signature field

Validation

  • Regression tests before implementation: 10 non-default signature failures; 4 default-signature cases passed
  • BLEU tests: 35 passed
  • python -m pytest --ignore=test/test_dataset.py: 194 passed (including API dataset access tests)
  • mypy sacrebleu scripts test: passed
  • ruff check .: passed
  • python -m build --no-isolation: passed
  • Local CLI smoke checks: default long and short text signatures unchanged
  • Focused CLI Japanese/Korean MeCab checks: 2 passed
  • Full pytest attempted but did not finish within the local time limit while fetching test datasets; no full-suite pass is claimed
  • Full CLI suite attempted; checks progressed through BLEU/chrF comparisons into TER before the local time limit, so no full CLI-suite pass is claimed

AI assistance was used to implement and test this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add BLEU max_ngram_order to signature

1 participant