Skip to content

fix(miniwob): reject partial find-greatest rewards#396

Open
jstar0 wants to merge 1 commit into
ServiceNow:mainfrom
jstar0:fix/miniwob-find-greatest-success
Open

fix(miniwob): reject partial find-greatest rewards#396
jstar0 wants to merge 1 commit into
ServiceNow:mainfrom
jstar0:fix/miniwob-find-greatest-success

Conversation

@jstar0

@jstar0 jstar0 commented Jul 17, 2026

Copy link
Copy Markdown

Summary

  • add a task-level MiniWoB success threshold while preserving the existing default
  • require miniwob.find-greatest to clear its partial wrong-answer reward
  • cover the behavior with unit-level and real-browser regressions

Problem

MiniWoB++ returns a raw reward of 0.1 when find-greatest submits the wrong
card and 1.0 for the correct card. BrowserGym currently converts every
positive raw reward to 1.0, so the wrong-card partial reward is reported as a
successful benchmark episode.

Changes

AbstractMiniwobTask now exposes a success_threshold that defaults to 0.0,
which preserves existing behavior for all other MiniWoB tasks.
FindGreatestTask overrides the threshold to 0.5, separating its known
0.1 wrong-card reward from its 1.0 success reward.

The regression coverage checks both the threshold conversion directly and a
real find-greatest browser episode that deliberately submits the minimum
card.

Scope and risk

This does not change dependencies, task registration, or the reward behavior
of tasks that do not override the threshold. The only task-specific behavior
change is that find-greatest no longer promotes its documented partial
wrong-answer reward to success.

Verification

uv run black . --check
uv run pytest tests/miniwob/test_base.py -k success_threshold
MINIWOB_URL=file://<miniwob-plusplus>/miniwob/html/miniwob/ uv run pytest tests/miniwob

Results: Black left all 91 files unchanged; the focused regression passed 3
tests; the complete MiniWoB suite passed 43 tests with 4 skipped.

Fixes #392

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

miniwob.find-greatest can be marked successful after selecting a non-maximum card

1 participant