Skip to content

fix: grade decorated Answer markers instead of falling through to prose - #6

Open
Mr-Neutr0n wants to merge 1 commit into
actava-ai:mainfrom
Mr-Neutr0n:fix/mcq-decorated-answer-markers
Open

fix: grade decorated Answer markers instead of falling through to prose#6
Mr-Neutr0n wants to merge 1 commit into
actava-ai:mainfrom
Mr-Neutr0n:fix/mcq-decorated-answer-markers

Conversation

@Mr-Neutr0n

Copy link
Copy Markdown

Problem

The MCQ answer-marker regex only allowed whitespace between ANSWER/IS and the letter:

(?:FINAL\s+)?ANSWER\s*(?:IS\s+)?[:=]?\s*\(?([A-J])\)?\b

Real model replies decorate the marker, and the most common form in the wild is markdown bold: **Answer:** C. That fails the marker rule, so grading falls through to the standalone last-occurrence fallback, which happily takes a letter from later prose. Concretely, with gold C:

**Answer:** C. Option D was considered and rejected.

grades as D (wrong). Same for **Final Answer:** A followed by a differential listing (B), (C), etc.

Reproduced before/after across a battery of decorated markers (**Answer:** X, Final Answer — X, fullwidth colon Answer:X, ANSWER IS :: X): old regex missed every one; new regex catches all.

Change

Allow up to six non-alphanumeric characters between the marker words and the letter:

(?:FINAL\s+)?ANSWER\s*(?:IS\s+)?[^A-Za-z0-9]{0,6}\(?([A-J])\s*\)?\b

Adversarial negatives verified unchanged: ANSWERED, digits, out-of-range letters, prose without markers all behave exactly as before. Plain forms (Answer: B, FINAL ANSWER: D, answer is (E)) unchanged. Boxed answers still win.

Verification

pytest tests/test_mcq.py -q   # 9 passed (5 existing + 1 new regression test with 6 cases)

This change was prepared with AI assistance under human direction and review.

The Answer-marker regex only allowed whitespace between ANSWER/IS and
the letter, so markdown-bold forms like '**Answer:** C' missed the rule
and the standalone last-occurrence fallback then took letters from later
prose ('Option D was ruled out'), flipping correct answers to wrong.
Allow a short run of non-alphanumeric decoration between the marker and
the letter; verified no new false positives against adversarial cases.

Signed-off-by: Mr-Neutr0n <64578610+Mr-Neutr0n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant