fix(miniwob): reject partial find-greatest rewards#396
Open
jstar0 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
miniwob.find-greatestto clear its partial wrong-answer rewardProblem
MiniWoB++ returns a raw reward of
0.1whenfind-greatestsubmits the wrongcard and
1.0for the correct card. BrowserGym currently converts everypositive raw reward to
1.0, so the wrong-card partial reward is reported as asuccessful benchmark episode.
Changes
AbstractMiniwobTasknow exposes asuccess_thresholdthat defaults to0.0,which preserves existing behavior for all other MiniWoB tasks.
FindGreatestTaskoverrides the threshold to0.5, separating its known0.1wrong-card reward from its1.0success reward.The regression coverage checks both the threshold conversion directly and a
real
find-greatestbrowser episode that deliberately submits the minimumcard.
Scope and risk
This does not change dependencies, task registration, or the reward behavior
of tasks that do not override the threshold. The only task-specific behavior
change is that
find-greatestno longer promotes its documented partialwrong-answer reward to success.
Verification
Results: Black left all 91 files unchanged; the focused regression passed 3
tests; the complete MiniWoB suite passed 43 tests with 4 skipped.
Fixes #392