Repository navigation
fix: score BleuScore as a single segment so every sentence counts - #3055
Open
Ziad-Abdalla wants to merge 1 commit into
Open
Ziad-Abdalla wants to merge 1 commit into
Ziad-Abdalla wants to merge 1 commit into
Conversation
sacrebleu reads references as reference streams, not NLTK's per-sentence lists, so splitting on ". " made BleuScore grade only the first sentence. Pass the whole response and reference as one segment, like ChrfScore, in both the legacy and collections implementations, and add regression tests. Fixes vibrantlabsai#2888
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Link / Problem Description
BleuScoreonly grades the first sentence of a multi-sentence response. Ragas splits both texts on". "and passes the reference as[[s1], [s2], ...], which is NLTK's layout. sacrebleu readsreferencesas a list of reference streams (one entry per hypothesis), so it sees one segment with several alternative references, and everything after the first response sentence is ignored.Example, with reference
"The capital of France is Paris. It is known for the Eiffel Tower.":"The capital of France is Paris. Bananas are a popular fruit.""Bananas are a popular fruit. The capital of France is Paris."It also gets more urgent soon: sacrebleu has merged a stricter reference-length check (mjpost/sacrebleu#294, not released yet). Since Ragas doesn't pin sacrebleu, once that ships the current layout will raise on any multi-sentence response.
Changes Made
corpus_bleu([response], [[reference]]). This is the same shapeChrfScorealready uses, and it no longer depends on the". "sentence splitter.ragas.metrics._bleu_scoreandragas.metrics.collections._bleu_score.kwargsstill pass straight through to sacrebleu.Testing
How to Test
tests/unit/test_bleu_score.py(18 tests, both implementations): later sentences affect the score, sentence order doesn't wipe out credit, unequal sentence counts don't truncate or raise, scores match whole-text BLEU, kwargs pass through, and sacrebleu is called with one segment.mainand pass with the fix. The existingtests/e2e/metrics_migration/test_bleu_migration.pyand the chrF tests still pass.ruff formatandruff checkare clean.References