Skip to content

fix: score BleuScore as a single segment so every sentence counts - #3055

Open
Ziad-Abdalla wants to merge 1 commit into
vibrantlabsai:mainfrom
Ziad-Abdalla:fix/bleu-score-single-segment
Open

Ziad-Abdalla wants to merge 1 commit into
vibrantlabsai:mainfrom
Ziad-Abdalla:fix/bleu-score-single-segment

Conversation

@Ziad-Abdalla

Copy link
Copy Markdown

Issue Link / Problem Description

BleuScore only grades the first sentence of a multi-sentence response. Ragas splits both texts on ". " and passes the reference as [[s1], [s2], ...], which is NLTK's layout. sacrebleu reads references as a list of reference streams (one entry per hypothesis), so it sees one segment with several alternative references, and everything after the first response sentence is ignored.

Example, with reference "The capital of France is Paris. It is known for the Eiffel Tower.":

response before after
"The capital of France is Paris. Bananas are a popular fruit." 1.0 0.417
"Bananas are a popular fruit. The capital of France is Paris." 0.0 0.417

It also gets more urgent soon: sacrebleu has merged a stricter reference-length check (mjpost/sacrebleu#294, not released yet). Since Ragas doesn't pin sacrebleu, once that ships the current layout will raise on any multi-sentence response.

Changes Made

  • Score the whole response against the whole reference as a single segment: corpus_bleu([response], [[reference]]). This is the same shape ChrfScore already uses, and it no longer depends on the ". " sentence splitter.
  • Applied to both implementations: ragas.metrics._bleu_score and ragas.metrics.collections._bleu_score.
  • User kwargs still pass straight through to sacrebleu.

Testing

How to Test

  • Automated tests added/updated
  • New tests/unit/test_bleu_score.py (18 tests, both implementations): later sentences affect the score, sentence order doesn't wipe out credit, unequal sentence counts don't truncate or raise, scores match whole-text BLEU, kwargs pass through, and sacrebleu is called with one segment.
  • 16 of the 18 fail on current main and pass with the fix. The existing tests/e2e/metrics_migration/test_bleu_migration.py and the chrF tests still pass. ruff format and ruff check are clean.

References

sacrebleu reads references as reference streams, not NLTK's per-sentence
lists, so splitting on ". " made BleuScore grade only the first sentence.
Pass the whole response and reference as one segment, like ChrfScore, in
both the legacy and collections implementations, and add regression tests.

Fixes vibrantlabsai#2888

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BleuScore evaluates only the first sentence (corpus_bleu reference structure bug)

1 participant