Skip to content

fix: codex_cli picks its Codex model slug from a custom model_catalog_json - #185

Merged
ransomr merged 15 commits into
mainfrom
codex-custom-catalog-alignment
Oct 22, 2026
Merged

ransomr merged 15 commits into
mainfrom
codex-custom-catalog-alignment

Conversation

@ransomr

@ransomr ransomr commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

When config_overrides sets model_catalog_json, Codex reads that file instead of its release catalog. codex_cli still chose the --model slug from the release catalog (or the bundled snapshot), so the slug could be one the custom catalog lacks, or miss an entry only the custom catalog has. This was found in the review of #184 and left out of it as pre-existing (decision: Ransom, 2026-10-07).

Changes:

  • When model_config is not set, the launch reads the model_catalog_json file from the sandbox and alignment chooses from it. Without one it uses the release catalog, else the bundled snapshot, as before.
  • The key is matched as Codex and the code-mode check match it: trimmed, with the last -c pair winning. A dotted key such as model_catalog_json.x names no file.
  • The value is converted with str(), as the -c pair passed to Codex does, so a Path names its file. Only a plain path is read: after trimming surrounding whitespace, a bare path or one in a matching pair of " or ', with no backslash, quote character or control character in it, and no other whitespace at the ends of a bare path. Codex 0.160.1 (codex-rs/utils/cli/src/config_override.rs) parses the value as TOML, else uses the raw text with quotes trimmed; for every such value both readings name the same file. Any other value (TOML escapes, embedded or unmatched quotes, multi-line strings, a trailing comment, control characters) is not read: a warning names it and says only a plain absolute path, optionally quoted, is read, and alignment uses the release catalog. Matching Codex's parser for those values turned up a new edge case every review round, for paths nobody writes (decision: Ransom, 2026-10-21).
  • If the file cannot be read (a relative path, a dotted key, a value that is not a string, a missing file, or not a JSON object with a models list), alignment uses the release catalog and a warning names the path. Codex itself fails to start on a missing or invalid absolute path, so this mainly covers a relative path, which Codex resolves itself.
  • A model below gpt-5.4 whose entry binds tool_search is sent to Codex's generic prompt through the slug inspect-generic. If the custom catalog has an entry that slug matches (inspect-generic, inspect), the launch now fails with an error naming the entry, unless model_config is set.
  • The code-mode check still skips when model_catalog_json is set; it now shares its key matcher with alignment.
  • Docs: the model_config option says where the slug comes from, that model_catalog_json must be a plain absolute path in the sandbox, and what happens otherwise.

Tests in tests/test_codex_code_mode.py cover alignment against a custom catalog (a slug only it has, and one only the release catalog has), the release fallback without one, no read when model_config is set, padded keys, a Path value, and non-string TOML values. Bare, double-quoted, single-quoted and space-padded paths read the right file, with decoy catalogs at the other candidate names. Refused values (escapes, a backslash, embedded or mismatched quotes, triple-quoted strings, a trailing comment, control characters including U+001C, trailing non-ASCII whitespace) warn and read no file, with catalogs at every name they could mean. The fallback and warning when the file is unreadable are covered too, and an empty models list passes through, since Codex rejects it itself. Slow Docker tests in tests/test_codex_align.py check the first request Codex sends: a model only the custom catalog has gets that entry's instructions, and a model only the release catalog has is aliased to the custom entry, not to Codex's generic prompt. Another passes a bare path and checks that Codex and alignment both use the custom entry. Two more cover gpt-5.2: it gets the generic prompt without tool_search, and the launch fails when an inspect entry would capture that slug.

🤖 Generated with Claude Code

Agent review

  • Reviewers: Codex gpt-6.1-sol (xhigh reasoning) and Claude Opus 5.5 (high effort), each in a fresh, clean context with a short review prompt and no history of the loop. Each new head gets a first pass of Codex and Claude reviewers side by side; when nothing they found is kept, more run in parallel to confirm, Codex and Claude side by side. A separate Codex gpt-6.1-sol (xhigh reasoning) session that keeps the loop's history checks every finding against the code, decides what is raised, and writes the verdict and the regression analysis. 12 passes, all separate from the implementer (Claude Code).
  • Pass 2: one Codex reviewer session, before the panel: 1 Blocking, 0 Optional, 0 Questions; verdict changes. Implementer: B1 fixed. tests/test_codex_align.py gains slow Docker test_codex_align_custom_catalog. It uses automatic alignment, a dummy OpenAI key and a filter that answers each request. The custom catalog's only entry is gpt-9.9, a direct copy of the 0.160.1 release's gpt-6.1-sol with a marker in its instructions. Both cases assert the marker and apply_patch in Codex's first request: openai/gpt-9.9 (only the custom catalog has it) and openai/gpt-5.5 (only the release has it, aliased to the custom entry, not Codex's generic prompt). The PR body now lists these tests. Checks (tests-3.md): …
  • Pass 3: one Codex reviewer session, before the panel: 0 Blocking, 0 Optional, 0 Questions; verdict clean
  • Pass 4: first pass, 1 reviewer: 1 finding, 1 raised; aggregator: 1 Blocking, 0 Optional, 0 Questions; verdict changes. Implementer: B1 declined as asked: decoding TOML escapes would need a new runtime dependency on Python 3.10 (tomli), for a path shape nothing uses. Instead, the change refuses such paths with a warning. In _read_model_catalog_json (codex_cli.py), a double-quoted model_catalog_json value containing a backslash is not read: alignment warns ('without TOML escapes') and uses the release catalog, the same fallback as a relative path. The old code read the raw escaped name, so it could pick a file Codex does not open. …
  • Pass 5: first pass, 1 reviewer: 1 finding, 1 raised; aggregator: 1 Blocking, 0 Optional, 1 Question; verdict changes. Implementer: B1 fixed: _read_model_catalog_json (codex_cli.py) now reads the value the way Codex 0.160.1's config_override.rs does. A quoted TOML string (matching quotes, no inner copy of that quote) keeps its inner text; any other value has every quote trimmed from its ends, which is Codex's fallback. The review's bare-path case is the exception: Codex trims that apostrophe too, so the trim stays, and a test pins it. New test_alignment_reads_the_file_codex_reads covers 4 forms with a decoy catalog at the other names and asserts the custom slug and one read; …
  • Pass 6: first pass, 1 reviewer: 1 finding, 1 raised; aggregator: 1 Blocking, 0 Optional, 1 Question; verdict changes. Implementer: B1 fixed by bounding the forms read rather than adding another decoder. _read_model_catalog_json (codex_cli.py) reads a value that does not start with a quote, with quotes trimmed as Codex does. It also reads one plain pair of matching quotes, with no copy of that quote inside and no backslash in double quotes, keeping the inner text. Every other value that starts with a quote is not read: escapes, multi-line strings, and text after the closing quote get the warning and the release catalog. The docs, docstring and warning say 'bare or in one pair of quotes without escapes'. …
  • Pass 7: first pass, 1 reviewer: 0 findings; confirming batch, 3 reviewers: 1/1/0 findings, 2 raised; aggregator: 2 Blocking, 0 Optional, 1 Question; verdict changes. Implementer: B1 fixed in resolve_codex_model_slug (model_catalog.py). When a sub-5.4 model's entry binds tool_search, the resolver uses the generic slug inspect-generic. If a selected catalog entry matches that slug by prefix (e.g. inspect-generic, inspect), it now raises ValueError naming the entry, before launch; explicit model_config still wins. Tests: a unit test for exact and prefix collisions on gpt-5.2 and gpt-5.3-codex. Two Docker cases: gpt-5.2 with a custom gpt-5.2 entry gets the generic prompt without tool_search or apply_patch; …
  • Pass 8: first pass, 1 reviewer: 0 findings; confirming batch, 3 reviewers: 0/0/0 findings; aggregator: 0 Blocking, 0 Optional, 1 Question; verdict clean
  • Pass 9: first pass, 2 reviewers (1 Codex gpt-6.1-sol, 1 Claude Opus 5.5): 1/1 findings, 2 raised; aggregator: 1 Blocking, 1 Optional, 1 Question; verdict changes
  • Pass 10: first pass, 2 reviewers (1 Codex gpt-6.1-sol, 1 Claude Opus 5.5): 1/1 findings, 1 raised; aggregator: 1 Blocking, 0 Optional, 0 Questions; verdict changes. Implementer: B1 fixed: the TOML 1.1 refusal in _read_model_catalog_json (codex_cli.py) now applies only when a \xHH or \e escape is why tomllib rejects the value. The new _toml_1_1_escape swaps each such escape (not after an escaped backslash) for a TOML 1.0 escape and parses again, only to validate; the swapped value is never used as a path. If that parse still fails, the value takes Codex's raw quote-trim fallback. Four file-selection cases added to test_alignment_reads_the_file_codex_reads: "/tmp/c\d\e.json", "/tmp/\x41\d.json", unterminated "/tmp/\x41.json, and "/tmp/\e.json" trailing; …
  • Pass 11: first pass, 2 reviewers (1 Codex gpt-6.1-sol, 1 Claude Opus 5.5): 0/0 findings; confirming batch, 6 reviewers (3 Codex gpt-6.1-sol, 3 Claude Opus 5.5): 1/0/1/0/0/0 findings, 1 raised, 1 dismissed; aggregator: 1 Blocking, 0 Optional, 0 Questions; verdict changes
  • Pass 12: first pass, 1 reviewer: 1 finding, 1 dismissed; confirming batch, 4 reviewers (3 Codex gpt-6.1-sol, 1 Claude Opus 5.5): 0/0/2/0 findings, 2 dismissed; aggregator: 0 Blocking, 0 Optional, 0 Questions; verdict clean
  • Pass 13: first pass, 1 reviewer: 1 finding, 1 dismissed; confirming batch, 4 reviewers (3 Codex gpt-6.1-sol, 1 Claude Opus 5.5): 0/0/0/0 findings; aggregator: 0 Blocking, 0 Optional, 0 Questions; verdict clean
  • Outcome: converged: Codex review round 13 found nothing to act on; CI green

Regression analysis (aggregator, pass 13)

Surface What changes for it Evidence Verdict
Backslashes, valid/invalid escapes and literal backslashes Warns and falls back rather than parsing/reading the custom filename. tests/test_codex_code_mode.py:334; explicit plain-path restriction, decision: Ransom, 2026-10-21. This replaces the earlier TOML/version exception. decision
Embedded, unmatched or extra quote characters Warns and performs zero reads, including filenames containing the other quote kind. tests/test_codex_code_mode.py:338; explicitly refused by Ransom's 2026-10-21 rule. decision
Multiline/triple-quoted values Warns and performs zero reads. tests/test_codex_code_mode.py:342; explicit refused family, decision: Ransom, 2026-10-21. decision
Control characters in path values Warns and performs zero reads; round 11's decoy-file selection is gone. tests/test_codex_code_mode.py:344; independent U+001C probe selects release slug, zero reads and one warning. Decision: Ransom, 2026-10-21. decision
Comment/trailing text and bare non-ASCII edge whitespace Warned release fallback for values outside the simple path rule. tests/test_codex_code_mode.py:341,349; refuses reinterpretation instead of guessing a filename, per the explicit 2026-10-21 restriction. decision
Relative path Warns and uses release metadata even when Codex could resolve a file. tests/test_codex_code_mode.py:289; decision: Ransom, 2026-10-13. decision
OpenAI-compatible providers Name passthrough; an explicit custom setting adds at most one sandbox read. model_catalog.py:253; bounded, explicitly requested read is negligible. accepted
Other non-OpenAI providers Same passthrough and explicit bounded read. model_catalog.py:253; no provider/request behavior change. accepted
Non-OpenAI catalog warning An unreadable configured catalog can produce an inaccurate release-fallback message although the slug passes through. Round 12 probe confirms unchanged slug, one read for a missing absolute file, zero for a relative path. This unchanged low-impact diagnostic is dismissed under the later-round Optional rule. accepted

Risk areas

  • none beyond the explicit fallback costs and assessed low-impact diagnostic above. The malformed-key case is assessed under Panel as an unused caller configuration, not claimed to be false or explicitly accepted by Ransom. The requested Path-value support is implemented and verified.

Not traced

  • none. Centaur and Kubernetes were traced through shared resolution/I/O contracts rather than separately executed.

Base automatically changed from codex-code-mode-fail-fast to main October 13, 2026 15:23
When config_overrides sets model_catalog_json, Codex reads that file instead
of its release catalog, but alignment still chose the --model slug from the
release (or bundled) catalog. The slug could then be one the custom catalog
lacks, or miss an entry only the custom catalog has.

Alignment now reads the custom catalog from the sandbox and chooses from it.
When it cannot be read (a relative path, a missing file, or not a JSON
object), alignment keeps the release catalog and a warning names the path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ransomr
ransomr force-pushed the codex-custom-catalog-alignment branch from 36953dc to 3b439f0 Compare October 14, 2026 14:12
ransomr and others added 2 commits October 14, 2026 10:26
Run codex_cli with a custom model_catalog_json whose only entry carries
marked instructions, and check the first request Codex sends: a model only
the custom catalog has gets that entry, and a model only the release catalog
has is aliased to it rather than to Codex's generic fallback. The filter
answers every request, so no API key is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ransomr
ransomr marked this pull request as ready for review October 14, 2026 14:37
@ransomr
ransomr requested a review from dragonstyle October 14, 2026 14:37

@dragonstyle dragonstyle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (CASE) at 40177e08ca

Reading the custom catalog and aligning against it works for the documented case, and the tests cover it. Two input-handling gaps let alignment drift from the catalog Codex actually loads, which is the mismatch this PR fixes. Details inline.

Comment thread src/inspect_swe/_codex_cli/codex_cli.py Outdated
Comment thread src/inspect_swe/_codex_cli/codex_cli.py
@ransomr
ransomr marked this pull request as draft October 14, 2026 21:46
ransomr and others added 5 commits October 14, 2026 17:49
Codex trims -c keys and accepts dotted subkeys; the code-mode check already
matched keys that way but alignment looked the key up exactly, so a padded
key loaded the custom catalog in Codex while alignment used the release one.
Both now share one key matcher. A catalog object without a models list is
now unreadable (warning, release fallback) instead of silently aligning to
nothing; an empty models list still passes through, as Codex would see it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reader strips quotes but does not decode TOML escapes, so a quoted path
with a backslash named a different file than the one Codex opens. Decoding
would need a TOML parser on Python 3.10 (a new runtime dependency) for a
shape nothing uses, so such a path is not read: alignment warns and uses the
release catalog, as it does for other paths it cannot read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex parses the value as TOML, so a quoted string keeps any other quote
character inside it, and only a value that is not TOML has every quote
trimmed from its ends. Stripping quotes from both ends in all cases made
alignment read a different file from Codex's for a path ending in a quote.
The docs now say an escaped path is not read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pair

Three review rounds found TOML forms the quote handling misread: escapes,
then inner quotes, now multi-line strings and text after the closing quote,
which made alignment read a different file from Codex's or none. Rather
than grow a partial TOML decoder, a value that starts with a quote is read
only when it is one pair of matching quotes with no copy of that quote and
no escape inside; every other quoted value is refused with the warning.
Values that do not start with a quote keep Codex's quote trimming.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… slug

A model below gpt-5.4 whose catalog entry binds tool_search is sent to
Codex's generic prompt through the slug inspect-generic. A caller's catalog
can hold an entry that slug matches by prefix (inspect-generic, inspect),
which would bind that entry's tools again, so resolution now raises before
launch unless model_config is set.

A quoted model_catalog_json value with a control character TOML rejects
(other than tab) is also left unread: Codex falls back to trimming its
quotes, so alignment would read a different file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ransomr

ransomr commented Oct 14, 2026

Copy link
Copy Markdown
Member Author

Addressed in 56adf0c:

  • @dragonstyle's review: Both gaps are fixed (see the T1 and T2 threads), with tests for each.

Generated with Claude (claude-opus-5-5) in an implement loop whose changes Codex reviewed. Not reviewed by Ransom Richardson before posting.

ransomr and others added 6 commits October 20, 2026 17:07
Brings in #194 (Python 3.11 floor, so tomllib is available) and #201
(plain Tools in bridged_tools). Conflicts: the alignment call now passes
the sandbox and the resolved bridged_tools_specs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ransom chose to parse the value as Codex does (2026-10-15) once the
Python 3.11 floor landed (#194), so tomllib costs no dependency. Codex
parses the trimmed value as a TOML value and, when that fails, trims
quotes from the raw text; alignment now does the same, so escaped,
literal and multi-line strings name the file Codex reads. This replaces
the restricted-format reader and its refusal list.

A value that parses to something other than a string names no file.
Codex parses TOML 1.1, whose \e and \xHH escapes tomllib reads only from
Python 3.15, so a double-quoted value with a backslash that tomllib
rejects is not read rather than guessed at.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review 9 B1: the guard refused every double-quoted value with a
backslash that tomllib rejects, which skipped Codex's raw fallback for
plain invalid escapes ("/tmp/c\d.json" names /tmp/c\d.json in Codex).
Now only a \xHH or \e escape not preceded by an escaped backslash is
refused, since Codex's TOML 1.1 parser decodes those and tomllib does
so only from Python 3.15; every other parse failure takes the raw
fallback. Ransom chose this documented, warned fallback over decoding
the escapes (2026-10-20). The docs, docstring and warning now state the
exception and name the escape (review 9 O1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se failure

Review 10 B1: the guard refused any rejected value that held a \xHH or
\e escape, even when the value is invalid in TOML 1.1 too (another bad
escape, no closing quote, text after the string). Codex takes its raw
quote-trim fallback for those, so alignment read nothing and chose a
release slug. Now the escapes are swapped for a TOML 1.0 escape only to
check whether the rest parses; the value is refused only if it does,
and is never decoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Matching Codex's value parser (Rust trim, TOML 1.1 parse, raw quote-trim
fallback) kept turning up a new edge case each review round, all for
values nobody writes. Read the value only when it is an absolute path,
bare or in one pair of quotes, with no quote, backslash or control
character; Codex names the same file for every such value. Anything else
warns and uses the release catalog without reading a file.

This drops the tomllib parse, the raw fallback and the TOML 1.1 escape
check, so Python 3.15 no longer behaves differently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ransomr
ransomr marked this pull request as ready for review October 21, 2026 19:18
@ransomr
ransomr requested a review from dragonstyle October 21, 2026 19:19

@dragonstyle dragonstyle left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (CASE) at 4312f4bec1

The model_catalog_json reading now matches how Codex reads the path (plain paths, matched quotes, refused TOML 1.1 escapes and control characters), the generic-slug capture guard is right, and the callers are unaffected. One regression, inline.

Comment thread src/inspect_swe/_codex_cli/codex_cli.py
@ransomr
ransomr marked this pull request as draft October 21, 2026 19:27
The -c pair passes the value to Codex with f"{key}={value}", so a Path
launched fine before alignment read the file, but alignment called
.strip() on it and crashed the launch. Convert the value with str() the
same way, so alignment reads the file Codex reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ransomr

ransomr commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Addressed in 9f8629f:


Generated with Claude (claude-opus-5-5) in an implement loop whose changes Codex reviewed. Not reviewed by Ransom Richardson before posting.

@ransomr
ransomr marked this pull request as ready for review October 22, 2026 13:41
@ransomr
ransomr requested a review from dragonstyle October 22, 2026 13:41
@ransomr
ransomr merged commit 7453133 into main Oct 22, 2026
18 checks passed
@ransomr
ransomr deleted the codex-custom-catalog-alignment branch October 22, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants