Repository navigation
fix(examples): let the citation-check oracle run on python3 < 3.10 - #1160
Merged
bingran-you merged 2 commits intoOct 30, 2026
Merged
Conversation
The oracle's inline Python script annotates `title_from_body` with `-> str | None`. Without postponed evaluation that annotation is evaluated when the function is defined, so any `python3` older than 3.10 aborts with `TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'` before writing answer.json. Inside the task image (ubuntu:24.04, Python 3.12) this never shows, but the oracle is also run directly on the host by test_citation_check_oracle_self_answers_static_fixture, which resolves `python3` from PATH. When the venv's bin dir is not on PATH (for example `.venv/bin/python -m pytest`), that is the system interpreter, which is 3.9 on stock macOS and on several LTS Linux distributions, and the test fails. Add `from __future__ import annotations` so the hints stay unevaluated. The citation-check-network variant shares this oracle through a symlink.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
bingran-you
approved these changes
Oct 30, 2026
bingran-you
left a comment
bingran-you
left a comment
Collaborator
There was a problem hiding this comment.
Verified against current main: with Python 3.9 as python3, the citation-check oracle test fails on main with the TypeError and passes here. Fresh CI test and fixture-scenarios pass; parity and pip-audit fail only on main's existing openclaw manifest drift and dnspython advisory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The inline Python in
docs/examples/task-md/real-skillsbench/citation-check/oracle/solve.sh(from #773) annotatestitle_from_bodyas-> str | Nonewithoutfrom __future__ import annotations.On a
python3older than 3.10 the oracle exits before writinganswer.json:The task image (Python 3.12) is unaffected, but
tests/test_task_check_eval_consistency.py::test_citation_check_oracle_self_answers_static_fixturerunsbash solve.shwith the caller'sPATH.Under
uv runthat resolves to the venv'spython3, so CI passes.When pytest is started by interpreter path (
.venv/bin/python -m pytest, IDE test integrations) on a host whosepython3is 3.9 (the Xcode Command Line Tools on macOS, RHEL 9), the test fails.Change
Add
from __future__ import annotationsto the heredoc.Output is unchanged, and
citation-check-networkshares this oracle through a symlink.No other host-run example or fixture script has the same problem.
Testing
Fails on
mainwith the traceback above and passes on this branch.bench eval run --tasks-dir docs/examples/task-md/real-skillsbench/citation-check --agent oracle --sandbox dockerstill scores 1.0.No new test, because a failing case would need a pre-3.10 interpreter in CI.