Skip to content

fix: parse judge replies that prefix prose before the fenced block - #7

Open
Mr-Neutr0n wants to merge 1 commit into
actava-ai:mainfrom
Mr-Neutr0n:fix/judge-parse-prose-prefixed-fences
Open

fix: parse judge replies that prefix prose before the fenced block#7
Mr-Neutr0n wants to merge 1 commit into
actava-ai:mainfrom
Mr-Neutr0n:fix/judge-parse-prose-prefixed-fences

Conversation

@Mr-Neutr0n

Copy link
Copy Markdown

Problem

_parse_judge_json required the reply to start with a backtick fence:

if cleaned.startswith("```"):
    cleaned = cleaned.split("```")[1]

Judge models frequently preface the block with a sentence (Certainly. followed by a json fence). For those replies the guard is False, the raw text hits json.loads, parsing returns None, and _judge_one scores the criterion as JUDGE_PARSE_FAILED (unmet) after burning all retries on a perfectly valid judgment.

That silently deflates HealthBench / HealthBench Professional scores for any judge model that wraps its JSON in conversational text, and it looks like a model-quality gap rather than a parser gap.

Change

  • Find the first fence anywhere in the reply, strip an optional json tag, and parse up to the closing fence.
  • When there is no fence at all, fall back to the outermost {...} object so prose-wrapped judgments still parse.
  • Non-bool criteria_met, malformed JSON, and no-object replies still return None exactly as before.

Verification

pytest tests/test_judge.py -q

Same pass/fail set as before the change plus the new test_parse_judge_json_handles_prose_around_the_block (5 cases). The 3 pre-existing failures in that file reproduce identically on clean main without optional provider deps installed; none touch the parser.


This change was prepared with AI assistance under human direction and review.

_parse_judge_json required the reply to start with a backtick fence, so
'Certainly. ```json ...' fell through to raw json.loads, returned None,
and the criterion was scored as JUDGE_PARSE_FAILED (unmet), silently
deflating HealthBench scores for judge models that preface their JSON.
Find the first fence anywhere in the reply and, when there is none, fall
back to the outermost JSON object.

Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant