Repository navigation
ci(integration): Add tiered L0-L3 integration gates with scope planner and codex review - #802
Conversation
…r and codex review Refactor the single integration-eval.yml into four scope-aware test levels plus a deterministic review pack, implementing the ENG-265 plan (#799). - L0 (test.yml): unchanged static/unit + a concurrency block. - L1 integration-light.yml: per-PR scoped smoke rollout + agent-judge gate. - L2 integration-scope.yml: scope-gated matrix (docker/daytona, skill modes, network + cheat lanes) selected from the diff. - L3 integration-final-review.yml: manual final gate; deterministic review pack plus a codex before/after-equivalence reviewer (fail-closed). New trusted-main CI infrastructure: - .github/scripts/integration_matrix.py: deterministic 7-set scope planner (citation/low-smoke/low-3/medium-3/high-3/nine/expanded), hard cell ceiling, aggregate-concurrency cap, DeepSeek flash/pro tiering by task difficulty. - tests/integration/rubric_checks.py + .github/scripts/build_integration_review_pack.py: rubric gates (realness, skill loading, verifier isolation, parity, capability attribution) operationalizing the benchflow-experiment-review skill; fail-closed verdict (mergeable | mergeable-with-quarantines | not mergeable). - .github/scripts/codex_review.py: codex-exec reviewer (auth.json), advisory and able only to make the verdict stricter; cheap per-rollout DeepSeek reads. - Reward-band parity against the HuggingFace SkillsBench leaderboard main run (openhands deepseek-v4-flash) as golden truth. - docs/integration-tiers.md + docs/integration-review-rubric.md. Security: planner/grader/harness always load from trusted main; only src/benchflow code-under-test comes from the PR head; plain pull_request (internal-PR-only). 65 stdlib tests pass, ruff clean.
…ecker + ADRs (port #799) Ports the verified-valuable pieces of the ENG-265 plan PR (#799) into the implementation, so #799 can be closed without losing the decisions or code. - Fail-closed verifier tamper (ADR-0002): a mechanically-detected verifier-file mutation is now included in agent_judge.realness_issues, so the realness gate hard-fails on tamper even when the LLM judge is absent/lenient (previously it was only surfaced to the judge as advisory). TDD: test_realness_gate_fails_closed_on_verifier_tamper (red -> green). - Config<->manifest drift checker: tests/integration/_consistency.py + test_integration_consistency.py, wired into run_suite.load_suite. Catches a drift this branch shipped (configs/mimo.yaml with no manifest axis); fixed by adding the experimental:[mimo] agent axis to suites/release.yaml. - ADRs: docs/adr/0001-0003 (lane execution home, verifier-tamper, network_mode conformance), with Context corrected to #802's reality and the deferred work (producer-side hash, verifier_files_mutated field, runtime egress conformance lane blocked on benchflow.sandbox._egress) recorded. Plus the conformance-vs- identity glossary aligned to task/config.py NetworkMode. ruff clean; 77 stdlib tests pass.
… e2e on a real rollout End-to-end run of the #802 pipeline (planner -> openhands+deepseek rollout -> grader) on PR #794's diff produced a real reward-1.0 rollout that the grader wrongly graded "not mergeable". Two bugs: - R-TELEMETRY false-failed: the production schema adapter read timing from run_config / timing.{started_at,ended_at} (the flat-fixture shape), but a real result.json records started_at/finished_at at the TOP LEVEL with per-phase seconds under timing.total. Adapter now reads those; R-TELEMETRY passes. - Stale-slot false-positive: classify_slots compared the rollout's task-source sha (the skillsbench task-repo commit) against the benchflow head_sha (the PR commit) — unrelated repos, so every real rollout read "stale". Staleness now compares only against an explicit task-source pin (plan.source_sha); absent a pin it is not fabricated (the workflow guarantees freshness by running at HEAD). Verified: the same real rollout now grades all-gates-green -> verdict "mergeable". Regression tests added (production top-level timing; task-source vs head_sha not conflated; stale via source_sha pin). 79 stdlib tests, ruff clean.
Ready for review — ported #799 + e2e-validated on a real rolloutPorted from #799 (now closed), TDD: fail-closed verifier-tamper gate ( End-to-end validated against PR #794 (
Regression tests added for both. 79 stdlib tests pass, ruff clean. Commits: Before merge: add |
Greptile SummaryThis PR replaces the single
Confidence Score: 4/5The deterministic gate is structurally sound and fail-closed; the L3 Codex gate has an unresolved credential routing problem flagged in a prior thread that causes it to permanently emit not mergeable (codex unavailable) on every dispatch. The core planner, credential filter, and rubric grader are well-designed with comprehensive test coverage. The new finding here — _parse_codex_verdict returning the first matching verdict instead of the worst across all candidates — only affects the advisory codex layer and cannot produce a false mergeable at the deterministic gate. The dominant risk comes from the L3 Codex gate credential bleed issue identified in a prior review thread that has not been resolved in this branch. integration-final-review.yml and codex_review.py together form the L3 required gate — the credential routing between select_integration_provider.py and the codex subprocess needs the most attention before this gate can be relied upon. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
PR[Pull Request / Push] --> L0
subgraph L0["L0 — test.yml (every PR/push, no keys)"]
L0_ruff[ruff + ty]
L0_pytest[pytest incl. planner/rubric/filter tests]
L0_audit[audit + concurrency]
end
PR --> L1
subgraph L1["L1 — integration-light.yml (always required)"]
DS[detect-scope\nno secrets, trusted main] -->|should_run=false| L1_noop[green no-op]
DS -->|should_run=true| RS[rollout-smoke\npypi-internal-preview env\nDeepSeek baseline agent]
RS --> AJ[agent_judge gate]
end
PR -->|label: integration:medium| L2
subgraph L2["L2 — integration-scope.yml (required, gated)"]
DS2[detect-scope\nno secrets] -->|gate_open=false| L2_noop[green no-op]
DS2 -->|gate_open=true| PL[plan job\ncredential filter has_cells]
PL -->|has_cells=false| L2_noop2[green no-op]
PL -->|has_cells=true| RM[run-matrix\nDeepSeek roster subset]
RM --> RP2[review-pack\ndeterministic verdict\nadvisory codex]
RP2 -->|not mergeable| L2_FAIL[gate fails]
RP2 -->|mergeable| L2_PASS[gate passes]
end
DISPATCH[workflow_dispatch\npr_number] --> L3
subgraph L3["L3 — integration-final-review.yml (manual gate)"]
PL3[plan job\ncredential filter]
RM3[run-matrix\nfull DeepSeek roster]
RP3[review-pack\ndeterministic grader\nCodex REQUIRED fail-closed]
VERDICT["worst(deterministic, codex)"]
PL3 --> RM3 --> RP3 --> VERDICT
VERDICT -->|mergeable| L3_PASS[check-run success]
VERDICT -->|not mergeable| L3_FAIL[check-run failure]
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
PR[Pull Request / Push] --> L0
subgraph L0["L0 — test.yml (every PR/push, no keys)"]
L0_ruff[ruff + ty]
L0_pytest[pytest incl. planner/rubric/filter tests]
L0_audit[audit + concurrency]
end
PR --> L1
subgraph L1["L1 — integration-light.yml (always required)"]
DS[detect-scope\nno secrets, trusted main] -->|should_run=false| L1_noop[green no-op]
DS -->|should_run=true| RS[rollout-smoke\npypi-internal-preview env\nDeepSeek baseline agent]
RS --> AJ[agent_judge gate]
end
PR -->|label: integration:medium| L2
subgraph L2["L2 — integration-scope.yml (required, gated)"]
DS2[detect-scope\nno secrets] -->|gate_open=false| L2_noop[green no-op]
DS2 -->|gate_open=true| PL[plan job\ncredential filter has_cells]
PL -->|has_cells=false| L2_noop2[green no-op]
PL -->|has_cells=true| RM[run-matrix\nDeepSeek roster subset]
RM --> RP2[review-pack\ndeterministic verdict\nadvisory codex]
RP2 -->|not mergeable| L2_FAIL[gate fails]
RP2 -->|mergeable| L2_PASS[gate passes]
end
DISPATCH[workflow_dispatch\npr_number] --> L3
subgraph L3["L3 — integration-final-review.yml (manual gate)"]
PL3[plan job\ncredential filter]
RM3[run-matrix\nfull DeepSeek roster]
RP3[review-pack\ndeterministic grader\nCodex REQUIRED fail-closed]
VERDICT["worst(deterministic, codex)"]
PL3 --> RM3 --> RP3 --> VERDICT
VERDICT -->|mergeable| L3_PASS[check-run success]
VERDICT -->|not mergeable| L3_FAIL[check-run failure]
end
Reviews (8): Last reviewed commit: "fix(integration): make the missing-base-..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| match = re.search(r"\{.*\}", raw, re.DOTALL) | ||
| if match: | ||
| try: | ||
| finding["parsed"] = json.loads(match.group(0)) | ||
| except json.JSONDecodeError: | ||
| finding["parse_error"] = "deepseek finding JSON was unparseable" | ||
| else: | ||
| finding["parse_error"] = "no JSON object in deepseek finding" |
There was a problem hiding this comment.
re.search(r"\{.*\}", raw, re.DOTALL) is greedy and spans from the first { to the last } in the entire string. When the model includes any additional curly-brace content after its JSON answer (e.g. an explanatory sentence like "I observed {"key": "val"} in the trace"), the matched span covers both objects and the text between them — json.loads then raises JSONDecodeError, the finding gets parse_error, and the structured evidence fed to the codex pass 2 is lost for that rollout. The fix is a non-greedy match or an iterative approach that tries each candidate from left to right.
| def _changed_files_from_git(base_ref: str, head_sha: str) -> list[str]: | ||
| out = subprocess.run( | ||
| ["git", "diff", "--name-only", f"{base_ref}...{head_sha}"], | ||
| cwd=REPO_ROOT, | ||
| capture_output=True, | ||
| text=True, | ||
| check=True, | ||
| ) | ||
| return [line.strip() for line in out.stdout.splitlines() if line.strip()] |
There was a problem hiding this comment.
CalledProcessError bypasses the ScopeError handler
subprocess.run(..., check=True) raises subprocess.CalledProcessError when git diff fails (e.g. because HEAD_SHA was not successfully fetched, which is silenced by || true in the workflow). That exception propagates past the except ScopeError block in main(), producing a raw Python traceback and exit code 1 instead of a ::error:: annotation. Using check=False and raising ScopeError on failure makes the error path consistent with every other planner failure mode.
| def _changed_files_from_git(base_ref: str, head_sha: str) -> list[str]: | |
| out = subprocess.run( | |
| ["git", "diff", "--name-only", f"{base_ref}...{head_sha}"], | |
| cwd=REPO_ROOT, | |
| capture_output=True, | |
| text=True, | |
| check=True, | |
| ) | |
| return [line.strip() for line in out.stdout.splitlines() if line.strip()] | |
| def _changed_files_from_git(base_ref: str, head_sha: str) -> list[str]: | |
| out = subprocess.run( | |
| ["git", "diff", "--name-only", f"{base_ref}...{head_sha}"], | |
| cwd=REPO_ROOT, | |
| capture_output=True, | |
| text=True, | |
| check=False, | |
| ) | |
| if out.returncode != 0: | |
| raise ScopeError( | |
| f"git diff {base_ref}...{head_sha} failed (rc={out.returncode}): " | |
| f"{out.stderr.strip()[:200]}" | |
| ) | |
| return [line.strip() for line in out.stdout.splitlines() if line.strip()] |
| async def one(rollout: Path) -> dict: | ||
| try: | ||
| evidence = agent_judge.load_rollout_evidence(rollout) | ||
| evidence_json = json.dumps(evidence.to_dict(), indent=2)[:8000] |
There was a problem hiding this comment.
json.dumps(evidence.to_dict(), indent=2)[:8000] cuts at a character boundary with no regard for JSON structure. This is embedded in the prompt as labelled "UNTRUSTED DATA" text, so the model reads it as raw text rather than parsing it — but a truncation mid-string (e.g. inside a long trajectory snippet) can produce visually misleading fragments. Truncating after a complete top-level key, or at a clean \n boundary, would produce more legible evidence.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
bingran-you
added
enhancement
|
Automation triage update (2026-06-18): pushed What changed:
Local verification on the PR branch:
CI has restarted. Leaving |
|
Automation follow-up (2026-06-18): the pushed bootstrap fix is now verified remotely as well. Current PR #802 checks at head
Additional local validation after installing the required
No merge taken: repo policy still requires human review before |
…ENAI_API_KEY Drop the CODEX_AUTH_JSON requirement: a personal ChatGPT OAuth blob (auth_mode: chatgpt) rotates and would stale-break the fail-closed L3 gate, and exposes a personal account in a shared org secret. The reviewer now prefers the existing repo-level OPENAI_API_KEY (durable, revocable), written by codex_review.py as an apikey auth.json so `codex exec` uses it the same way `codex login --api-key` would. CODEX_AUTH_JSON remains an optional fallback. - codex_review.py: write_codex_auth writes an apikey auth.json from OPENAI_API_KEY / CODEX_API_KEY (precedence: full blob -> api key -> on-host auth.json). New test_write_codex_auth_from_openai_api_key. - L2/L3 workflows: codex step authenticates via the repo OPENAI_API_KEY (already present); no CODEX_AUTH_JSON secret to create. - docs/ADMIN_SETUP/ADRs: codex auth = OPENAI_API_KEY, no extra secret. 101 stdlib tests pass; ruff clean.
…very agent Scope gap: the planner ran only the baseline agent (openhands) for changes that break ALL agents — the agent registry, shared ACP/protocol/install/credential infra, and provider/LLM-proxy routing. A specific agent file already mapped to its agent (codex_config.py -> codex-acp + baseline), but registry.py and providers/litellm_runtime.py silently tested openhands only. - New `all-agents` extra -> _matrix_agents fans the full credentialed roster. - scope_map: new `agent-runtime-infra` rule (registry/protocol/install/ credentials/env/errors/acp) and `all-agents` on the provider/network rule. - Specific-agent changes are unchanged (affected agent + baseline, stays cheap). - All bounded by the hard cell ceiling: registry.py -> 9 agents/40 cells, providers -> 9 agents/28 cells, no overflow. TDD regressions: test_agent_runtime_infra_fans_full_roster, test_provider_change_fans_full_roster, test_specific_agent_change_does_not_fan_full_roster. 40 planner tests pass; ruff clean; MiniYaml-safe.
… + model policy + cred-aware emission Refines the b4d10ea all-agents fan-out (which put the full 9-agent daytona roster on L2's auto-on-push label trigger and assumed creds CI lacks). - Breadth-tier: the L2 agent-runtime-infra + provider rules now emit a representative SUBSET (openhands + codex-acp + gemini + pi-acp); the FULL 9 roster runs only at L3 via expanded/nine. New `roster_subset` in scope_defaults + an `all-agents-subset` branch in _matrix_agents. - Model policy (empirically grounded): the openai-completions-family agents (openhands, openclaw, opencode, pi-acp, mimo) run on deepseek-v4-flash via the LiteLLM proxy — mimo on deepseek closes the XIAOMI gap; gemini/codex/harvey stay native (branded CLIs are protocol-locked: deepseek serves only openai-completions, so the protocol guard raises); claude-agent-acp routes through aws-bedrock's anthropic-messages surface. - Cred-aware emission (bug fix): .github/scripts/filter_credentialed_cells.py drops any cell whose agent+model credential is absent (resolve_agent_env signal) as a documented skip — un-keyed agents no longer spin up a sandbox, fail, and log a false-red slot. Wired into L2 + L3. claude skips until AWS_BEARER_TOKEN_BEDROCK is provisioned. - ADR-0004 records the breadth-tier + model-policy + cred-aware decisions. 112 stdlib tests pass; ruff clean; MiniYaml-safe.
| - name: Upload matrix plan | ||
| uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0 | ||
| with: | ||
| name: integration-scope-matrix | ||
| path: matrix.json | ||
| if-no-files-found: error | ||
| retention-days: 14 | ||
|
|
There was a problem hiding this comment.
run-matrix job failure instead of a clean skip
The plan job first runs the planner (which determines should_run=true based on cell count > 0) and then runs filter_credentialed_cells.py, which can drop cells whose provider credentials are absent. The filtered matrix is emitted directly to the run-matrix strategy without a guard against an empty result. If all planned cells are filtered out, fromJson(needs.plan.outputs.matrix) resolves to [], and GitHub Actions will fail the run-matrix job with "The matrix must define at least one combination." The review-pack job then runs on empty artifacts and produces "not mergeable" — a false-negative verdict that blocks a PR with no actual regression. Adding an emptiness check in the emit step (setting should_run back to false and skipping downstream jobs) or inside filter_credentialed_cells.py would close this gap.
…ase-url skips
- L2/L3 workflows: add a `has_cells` output from the credential-filter step.
When the filtered matrix is empty (no provider key for any planned cell),
skip run-matrix + review-pack as a green no-op instead of failing
`fromJson([])` ("matrix must define at least one combination") into a false
'not mergeable'. Closes PR #802 review r3438339786.
- filter_credentialed_cells.py: also recognize the provider-base-url-missing
message ("requires <KEY> to build the provider base URL") as a documented
missing-credential skip. Previously a missing DEEPSEEK_BASE_URL slipped past
the filter and red-failed the DeepSeek baseline agent at the first LLM call.
… gate natives) Broad roster lanes now fan the DeepSeek roster ONLY (new `deepseek_roster` in scope_defaults.yml). The non-DeepSeek agents cannot ride the DeepSeek proxy (protocol walls), so they are gated rather than fanned by default: - claude-agent-acp (Bedrock anthropic-messages) and codex-acp (OpenAI Responses) run ONLY via affected-agent -- a PR touching their own adapter -- paired with the DeepSeek baseline for before/after comparison. Never in the default fan. - gemini (native Google GenerateContent; _NATIVE_PROTOCOL_AGENTS) and harvey-lab-harness (OpenAI Responses API in its _create_adapter, same wall as codex) are dropped from the roster entirely. - roster 9 -> 7; caps.max_agents 9 -> 7; L2 breadth subset is now 3 DeepSeek reps (openhands, pi-acp, opencode). The agent-judge runs on DeepSeek-v4 only -- no other models. judge_model is deepseek-v4-flash (recorded per-cell metadata + local default); at runtime select_integration_provider.py exports BENCHFLOW_JUDGE_MODEL=openai/deepseek-v4-flash, routed via the OpenAI-compatible chat-completions surface with the DeepSeek key/base_url. GEMINI_API_KEY is no longer load-bearing on the default path. Docs (integration-tiers.md, ADR 0004, ADMIN_SETUP.md) and tests updated.
Update — DeepSeek-first roster + DeepSeek-only judge, plus credential-filter hardeningPushed two commits (
Secrets (cross-checked against Verified: stdlib planner + filter + agent-judge test suites pass, ruff clean, all workflow/config YAML valid; an end-to-end |
_matrix_agents returned from the all-agents / nine-expanded / all-agents-subset branches BEFORE the affected-agent branch, so a PR that co-changed a gated native's adapter (codex_config.py / claude*.py -> affected-agent) AND a broad-fan trigger (registry.py infra, or a pyproject.toml expanded-scope file) silently dropped that native from the matrix — and affected-agent is its ONLY path. Fix: compute the lane's base roster, then always UNION res.affected_agents onto it, so a co-changed native still gets its cell (incl. at nine/expanded). Surfaced by the pre-merge audit; regression test added. Fail-safe before (under-coverage, never a false-green), correct now.
The run-matrix provider-select step exports OPENAI_*=DeepSeek into the job env so the DeepSeek agent-judge works. But the gated-native codex-acp rollout (gpt, OpenAI Responses) KEEPS ambient OPENAI_* (it does not recompute them in resolve_agent_env), so it would auth its rollout against DeepSeek and fail. Pin the REAL OpenAI key+base for codex-acp cells ONLY (L2 + L3 run-matrix) via a matrix.cell.agent guard; DeepSeek cells keep the job env unchanged (verified no-op: openhands/pi-acp/openclaw ignore OPENAI_*, mimo/opencode recompute their own provider creds), and the later judge step keeps the DeepSeek OPENAI_*.
Pre-merge verification — 17-agent audit (per-fix × {thermo-nuclear, correctness, test, adversarial} + merge-readiness)Ran a multi-agent pre-merge audit over the four recent fixes; one agent per fix applied the thermo-nuclear code-quality-review skill. Verdicts:
The audit found two real (fail-safe, under-coverage) bugs in the gated-native path — both consequences of the new gating — now fixed (
Everything else passed, including the thermo-nuclear structural review (no file-size / spaghetti / abstraction regressions across all four fixes). Post-fix: 93 stdlib tests pass, ruff clean, both workflows valid YAML. Merge-readiness: ✅ ready — no code/config blockersVerified: all load-bearing secrets present on Manual steps before/after merge (GitHub-UI, not code):
Confirmed documented-as-deferred (not silently broken): |
| # SkillsBench leaderboard submissions (native BenchFlow result.json) from | ||
| # the HF dataset + revision recorded in scope_defaults.yml -> baseline. | ||
| # The band model covers every integration task; cheap deepseek runs band- | ||
| # compare advisory-only. Best-effort: a fetch miss leaves band parity NA | ||
| # rather than failing the gate (the deterministic floor + codex still run). | ||
| id: baseline | ||
| continue-on-error: true | ||
| run: | | ||
| set -euo pipefail | ||
| vals=$(uv run python -c "import yaml;d=yaml.safe_load(open('.github/integration/scope_defaults.yml'))['baseline'];print(d['hf_repo'],d['hf_revision'],d['submission_root'],d['config_no_skill'],d['band_model'])") | ||
| read -r HF_REPO HF_REV SUB_ROOT CONFIG BAND_MODEL <<< "$vals" | ||
| DIR="$SUB_ROOT/${CONFIG}__${BAND_MODEL}" | ||
| uvx --from huggingface_hub hf download "$HF_REPO" --repo-type dataset \ | ||
| --revision "$HF_REV" --include "$DIR/**/result.json" \ | ||
| --local-dir baseline-root |
There was a problem hiding this comment.
select_integration_provider.py (the "Select judge provider" step above) writes OPENAI_API_KEY=<deepseek key> and OPENAI_BASE_URL=<deepseek url> to $GITHUB_ENV. Those values override the job-level env: block for all subsequent steps. The codex review step does not have step-level overrides for either variable, so codex_review.py reads the DeepSeek key from os.environ["OPENAI_API_KEY"], writes it into auth.json, and then passes env=dict(env) (including OPENAI_BASE_URL=DeepSeek) to the codex exec subprocess. Codex routes to the DeepSeek endpoint with a DeepSeek key and requests a model (e.g. o4-mini) that DeepSeek does not serve, so the call fails. The codex_verdict is None path fires on every L3 dispatch, making the required gate permanently blocked.
The analogous fix is already in the run-matrix step for codex-acp cells; the same two-variable pattern needs to be applied here. The integration-scope.yml advisory codex step has the same OPENAI_BASE_URL gap (it restores OPENAI_API_KEY but not the base URL).
…ences Drop the docs/adr/ folder (0001-0004) and remove every dangling reference so no link 404s: the tiers-doc intro now states the key decisions inline (cells-not- markers, fail-closed V-TAMPER, static V-NETWORK, DeepSeek-only broad fan) instead of linking the ADRs; the ADR markdown link and the prose "(ADR-000X)" citations in integration-tiers.md / integration-review-rubric.md and two test docstrings are removed. No behavior change.
…stic test_drops_deepseek_when_base_url_missing relied on DEEPSEEK_BASE_URL being genuinely absent, but a populated CI test job supplies a fallback base for the baseline openhands cell, so it was KEPT and the assertion failed (false red in L0). Patch resolve_agent_env to raise the base-URL message directly, pinning the filter's drop+missing_key routing regardless of ambient env. Whether a bare DEEPSEEK_BASE_URL is truly absent is benchflow's concern, not the filter's; the message-shape recognition is also unit-covered by test_extract_missing_key_shapes.
Cross-checked the unaddressed greptile comments on the earlier integration PRs (#802-#810); most were already fixed or stale. Three genuine ones remained: - P1 (build_integration_review_pack): an R-OUTCOME-only deterministic reject is demoted to a healthy+quarantine slot, but grade["deterministic_reject"] was left True and serialized into agent_judge_summary.json. codex reads that and a "healthy slot with a deterministic reject" can spuriously push it to downgrade mergeable -> not mergeable. Clear the flag on demotion. + regression test. - codex_review deepseek-finding parser: replace the greedy `{.*}` DOTALL regex (which merges two JSON objects into one invalid blob) with raw_decode of the first complete object, tolerant of trailing prose. - integration_matrix: `git diff` runs with check=True but CalledProcessError bypassed the ScopeError fail-closed handler (uncaught traceback). Re-raise it as ScopeError so a failed/absent head_sha fails closed cleanly. Co-authored-by: symphony-bot <symphony@benchflow.ai>
Summary
Implements the ENG-265 plan (#799). Splits the single
integration-eval.ymlinto four scope-aware test levels plus a deterministic review pack, so light checks run per commit and the heaviest integration check runs only when a PR is final — scoped to the components a change can affect, fanned across a DeepSeek-first agent roster, and graded by a fail-closed deterministic verdict with an advisory-stricter-only Codex before/after-equivalence reviewer on top.test.ymlintegration-light.ymldetect-scopeself-skips to a green no-op)integration-scope.ymlintegration:medium(cheapdetect-scopeno-op otherwise)integration-final-review.ymlworkflow_dispatchEvery level always triggers (no
on: pathsfilter); a secret-freedetect-scopejob computes from the diff whether real work is needed and otherwise reports SUCCESS as a green no-op, so each check can be made an unconditionally required branch-protection check.Key design (grilled to closure)
origin/main; onlysrc/benchflow(code-under-test) comes from the PR head — a PR can't weaken its own gate. Plainpull_request.build_integration_review_pack.pyoverrubric_checks.py) is the required gate, emittingverdict.md=mergeable | mergeable-with-quarantines | not mergeable. An unreadable plan, a missing required slot, or any not-mergeable gate fails closed; the pack exits non-zero. Codex is an advisory-stricter-only before/after-equivalence signal (worst(deterministic, codex)): it can downgrademergeable → not mergeablebut never upgrade, and itself fails closed tonot mergeable (codex unavailable)if the binary/auth/output is missing.mainrun (openhands · deepseek-v4-flash) viacheck_skillsbench_harbor_parity— apples-to-apples with our lane. Covers 6/9 tasks; the other 3 (absent from the leaderboard's DeepSeek runs) band-compare NA and fall back to oracle determinism + the Codex equivalence verdict.rerun-baseonly forscope=expanded.integration_matrix.pymaps changed paths → one of 7 named task-sets (citation/low-smoke/low-3/medium-3/high-3/nine/expanded) + unioned axes (skill modes, docker/daytona, scope-gated network lane, cheat lane).scope=autoderives from the diff; explicit--scopeoverrides. Fail-closed hard cell ceiling (caps.max_cells,rejected_overflow+ exit 2, never a silent drop) and an aggregate-concurrency clamp (per_agent_concurrency × distinct daytona agents ≤ 24). DeepSeek flash/pro tiering by task difficulty.rubric_checks.pyoperationalizes thebenchflow-experiment-reviewskill —R-REALrealness,V-TAMPERverifier-tamper (reusingagent_judge._scan_verifier_tamper, folded fail-closed into realness),V-NETWORKstatic network-posture,P-REWARD/P-SCHEMAdocker↔daytona parity,R-TELEMETRY— honest about deterministic vs codex/residual.Agent roster, models & judge (DeepSeek-first)
deepseek_roster=openhands, pi-acp, openclaw, opencode, mimo), all ondeepseek/deepseek-v4-flash(promoted todeepseek-v4-proon hard tasks viadeepseek_tiering).mimomoved offxiaomi/mimo-v2.5onto DeepSeek (closes the XIAOMI gap).codex-acp(native OpenAIgpt-5.4-nano) andclaude-agent-acp(Bedrockanthropic-messagesclaude-haiku) cannot run on DeepSeek (OpenAI-Responses / anthropic-messages protocol walls), so they are gated: they appear in the matrix only via affected-agent (a PR touching their own adapter —codex_config.py → codex-acp,claude*.py → claude-agent-acp), paired with the DeepSeek baseline, never in the default broad fan.gemini(native GoogleGenerateContent) andharvey-lab-harness(OpenAI Responses API) are dropped entirely — neither can run on DeepSeek.caps.max_agentslowered 9 → 7.agent-runtime-infrarule: registry/protocol/install/credentials/env/errors/acp) or provider/LLM-proxy routing fan a representative DeepSeek subset (roster_subset=openhands, pi-acp, opencode+ baseline) at L2 auto-on-push; the full DeepSeek roster runs at L3/expanded/nine. The full-roster check precedes the subset/affected-agent branches so promoting an infra change to L3 fans the full roster (not the L2 subset).judge_model = deepseek-v4-flashrecorded as metadata, and at runtimeselect_integration_providerexportsBENCHFLOW_JUDGE_MODEL=openai/deepseek-v4-flash(routed via the OpenAI-compatible chat-completions surface).agent_judge.DEFAULT_JUDGE_MODELchanged fromgemini-3.1-flash-litetoopenai/deepseek-v4-flash;GEMINI_API_KEYis no longer load-bearing on the default path.Credential-aware emission & matrix guards
filter_credentialed_cells.py, runs in the plan job under thepypi-internal-previewenvironment): drops any cell whose agent+model credential (perresolve_agent_env) is absent intoskipped_uncredentialedas a documented skip, not a red slot, so un-keyed agents never spin up a false-red sandbox. Fails open (passes the matrix through unchanged) if the benchflow import fails. Recognizes both missing-credential shapes: missing API key (<KEY> required … not set) and missing base URL (requires <KEY> to build the provider base URL) — the latter catches a missingDEEPSEEK_BASE_URLthat would otherwise red-fail the DeepSeek baseline at the first LLM call.has_cells=falseand skips run-matrix + review-pack as a green no-op with a warning, instead of hard-failingfromJson([])("matrix must define at least one combination") into a falsenot mergeable(PR review r3438339786).Fail-closed gates, drift checker & ADRs (ported from #799)
agent_judge.realness_issuesnow folds the mechanically-detected tamper signal (evidence.flagged_actionsfrom_scan_verifier_tamper) into the judge-independent realness gate: any flagged write/delete/chmod of a score-defining file hard-fails the gate even when the LLM judge is absent or lenient (ADR-0002 Task A1, surfaced asV-TAMPER/R-REAL).run_suite.load_suitenow calls_consistency.config_drifts: if a siblingconfigs/dir exists and any config disagrees withsuites/release.yaml(agent membership both directions, model pins, order-insensitive task includes, task source{repo,path,ref}), the suite load raisesSuiteErrorinstead of producing a quietly inconsistent plan. No-op whenconfigs/is absent (synthetic test manifests). Caught a real defect this branch shipped (configs/mimo.yamlwith no manifest axis → fixed byaxes.agents.experimental:[mimo]).release.yamlis a declarative backlog, not a gating path), ADR-0002 (cheap fail-closed tamper regex ships now; authoritative producer-side hash deferred), ADR-0003 (V-NETWORKis a static scope-gated declaration check today; runtime egress conformance deferred, blocked onbuild_egress_overridenot yet in main), plus adocs/integration-tiers.md§12 glossary. Network vocabulary aligned to the authoritativeNetworkModeenum (no-network/allowlist/public).Hardening fixes
OPENAI_API_KEY. The fail-closed L3 Codex reviewer now authenticates with the durable, revocable repo-levelOPENAI_API_KEY(written as an apikeyauth.json, likecodex login --api-key), instead of a rotating personalCODEX_AUTH_JSONOAuth blob. Precedence:CODEX_AUTH_JSON→OPENAI_API_KEY/CODEX_API_KEY→ pre-existing on-hostauth.json; otherwise still fails closed. No separate org secret needs creating.should_run=falseto a green no-op when the trusted base branch doesn't yet containintegration_matrix.py(bootstrap PRs).not mergeable):R-TELEMETRYnow reads production timing from top-levelstarted_at/finished_at+timing.total(fixing a false telemetry failure), and the stale-slot check no longer conflates the skillsbench task-source sha with the benchflow PRhead_sha(param renamedhead_sha → expected_source_sha; with no pin no slot is marked stale, since the workflow runs at HEAD).Verified
hf download … --revision main); band-compare viacheck_skillsbench_harbor_parity.agents == 7with gemini/harvey absent; broad fan ==deepseek_roster; gated natives only via affected-agent;_matrix_agentsordering regression) and the credential-filter fail-open / missing-API-key / missing-base-URL coverage.not mergeablenow grades all-gates-green →mergeable.Test plan
test.ymllane runs ruff + ty + pytest (incl. new stdlib planner/filter/rubric/review-pack tests) greennot mergeable; planner exits 2 + setsrejected_overflowon cell-ceiling overflowSuiteErroron a manifest mismatchmainbaseline fetch + band-compare confirmedDEEPSEEK_BASE_URL) recognition unit-coveredroster_subset; affected-agent change → gated native + baseline) emits and runs the expected cells, with un-keyed cells documented-skipped (not red)has_cells=falsegreen no-op (nofromJson([])failure)workflow_dispatchfinal gate: full DeepSeek roster matrix → review pack + Codex equivalence verdict, with Codex authed via repoOPENAI_API_KEYFollow-ups / admin
OPENAI_API_KEY(written as an apikeyauth.json) — no new secret required; fail-closed until resolvable.srcchanges, one coordinated additive rollout-contract schema bump): producer-sideverifier_files_mutated: bool; runtime egressV-NETWORK-CONFORMlane (blocked onbuild_egress_overridein main);network_moderesult.json serialization.Relates to #799 (the ENG-265 plan / ADRs); supersedes #799's full 9-agent gemini/harvey roster and gemini judge.