Skip to content

Allow JSON score formatting without a signature - #306

Open
ktz03 wants to merge 1 commit into
mjpost:masterfrom
ktz03:fix/json-score-empty-signature
Open

ktz03 wants to merge 1 commit into
mjpost:masterfrom
ktz03:fix/json-score-empty-signature

Conversation

@ktz03

@ktz03 ktz03 commented Oct 11, 2026

Copy link
Copy Markdown

Calling corpus_score(...).format(is_json=True) without supplying the optional signature raises ValueError: not enough values to unpack for BLEU, chrF and TER. The default empty signature is split as though it contained a key:value parameter.

Only expand signature parameters when a signature is present. JSON output without one retains the metric name, rounded score, empty signature and existing verbose information. Add regressions for omitted and explicitly empty signatures using all three public metric APIs.

Validation on Windows with Python 3.12.14:

  • The six new cases fail on the unchanged implementation, with the existing 47 cases passing; the corrected test_metrics_base.py passes all 53 cases.
  • All 192 tests pass across test_api.py, test_bleu.py, test_chrf.py, test_metrics_base.py, test_sentence_bleu.py, test_significance.py, test_ter.py and test_tokenizer_ter.py.
  • Public API checks pass for all three metrics, including normal long/short signatures, text and score-only output. Local CLI JSON output also passes normal-signature and score checks for all three metrics.
  • mypy sacrebleu scripts test: no issues in 44 source files. Full Ruff check passes. The isolated source-distribution and wheel build succeeds.

The full python -m pytest run exceeded a 900-second limit; a bounded follow-up was stopped at test_dataset.py::test_maybe_download. The full WMT/Perl CLI comparison script also exceeded a 360-second limit. Neither full suite is claimed as passing. No Linux or macOS test was run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant