Skip to content

fix(evidence): answer inputs-schema selector refusals as request.selector-invalid and fix evidencectl authoring diagnostics - #2610

Merged
jeremi merged 11 commits into
mainfrom
fix/evidence-followups
Nov 8, 2026
Merged

jeremi merged 11 commits into
mainfrom
fix/evidence-followups

Conversation

@jeremi

@jeremi jeremi commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Summary

Two follow-ups to the public certificate check.

1. Inputs-schema selector refusals on the relying routes

On the single and batch evaluate routes, a selector value that passes its profile but that the requirement's Source inputs schema refuses now answers 400 request.selector-invalid, as a profile refusal does, instead of 503 service.unavailable (category source-inputs).

  • It is refused during selector resolution (refuse_selector_the_source_inputs_reject), before any audit entry, credential acquisition or source request.
  • It counts against the caller's selector-failure budget, and one such item refuses the whole batch.
  • The public check route already answered this way.
  • A value that passes the inputs schema but cannot occupy a request path segment still answers 503 source.unavailable after the access-attempt audit entry; the guidance in evidence-problems.mdx and the contracts now say which case is which.

2. evidencectl authoring defects

Each known authoring defect was re-checked against main; the live ones are fixed here:

Defect Fix
target new --local dropped the project's source extract bindings, so check --target refused a fresh SQLite-extract project with source-extract-unbound The target carries extracts beside connections, and the command names both
target new failed on a bare file or directory name (--settings local-target.yaml, or a destination with no separator) An empty parent resolves to the current directory
init --openapi wrote a starter fixture its own README's check refused The fixture uses the current case fields and passes check and test
BReg tutorial: flat connection shape, wrong diff context, missing extract binding The tutorial shows the carried bindings and the nested endpoints.registry shape
A directory with no evidence.yaml failed with exit 3, a generic message and a private snapshot path Exit 1, evidence.project.not-found, naming the path given and pointing to evidencectl init
A malformed or unknown-key pack.yaml, or a Source route the pack does not declare, gave the generic inspection failure The pack decode report is located at its file, line and key; an unknown route is evidence.pack.route-missing at /sources/<id>/route, naming the declared routes
The unbound-extract refusal said to bind under sourceExtracts It says extracts, the target's key
check passed a fixture without common or privacyExpectation, and test then refused it without a location check and bundle loading refuse it as evidence.fixture.missing-block, located in the fixture; a coequal acceptance fixture needs only privacyExpectation. The fixture JSON Schema lists the blocks as required, and the evidencectl.fixtures.no-fixture action names them
A failing case with no error or publicProblem said only "reference failing case has no exact expectation" The report names the stage that failed, its internal category and its public problem

Compatibility note on the fixture rule: evidencectl package already runs every fixture through bundle-evaluate, so any packaged bundle carries both blocks. Only a hand-assembled bundle without them is now refused at load. The two in-repo conformance bundles were such bundles; they gain empty blocks and refreshed SHA256SUMS.

Left as is:

  • check --json reports status: complete while deploymentInputs.complete is false (only when .generated/secrets/ is absent, as in a fresh clone). The human output already says the compiled-target check was skipped; changing status is a contract change for script readers, so it is not done here.
  • dev start --detach: removed on purpose in 0.42.0.
  • A vendored evidence-configuration.mdx renders empty until npm run generate runs in docs/site; that is a vendoring step, not a defect here.
  • Under assuranceProfile: local, any fixture case can be removed and check and test stay green. This is by design and stated in products/evidence/reference/request-adapter/deployment-projects/FIXTURES.md.
  • An undeclared derivation output in an authored project reaches the kernel as a unit script error, so the report says the derivation script failed "for example by returning an output the definition does not declare" rather than naming the output. Naming it would need an authoring-wrapper or kernel change.

Tests

Each fix has a regression test seen failing before the fix:

  • runtime_tests::a_selector_value_the_source_inputs_schema_refuses_is_an_invalid_selector (single, batch, budget, no audit entry)
  • tests/scaffold.rs: a_new_local_target_carries_the_projects_extract_bindings, target_new_accepts_bare_settings_and_destination_names, the_generated_openapi_starter_passes_check_and_test (ignored; needs EVIDENCE_BIN)
  • tests/project_diagnostics.rs (new): missing project for check and test, unparseable pack, unknown pack key, unknown route
  • authored::tests::an_unbound_extract_names_the_authored_extracts_key_in_path_and_action
  • fixture::tests::refuses_each_rule_at_its_member_without_the_value: three rows for the missing blocks
  • schema::tests::the_fixture_schema_accepts_what_the_reader_accepts_and_refuses_what_it_refuses (--features schema): reader and schema both refuse a fixture missing either block
  • tests/fixtures.rs: a_project_with_no_fixture_file_says_so_and_where_one_goes checks that the action names both blocks
  • tests/cli.rs: a_derivation_output_the_definition_does_not_declare_is_named_as_the_cause

Checks run

Locally on macOS, at the branch head:

  • cargo fmt --check for registry-evidence, registry-evidence-authoring and registry-evidencectl
  • products/evidence/scripts/check-contracts.sh
  • cargo clippy --locked -p registry-evidence -p registry-evidencectl -p registry-evidence-authoring --all-targets -- -D warnings, and for registry-evidence with --features schema
  • cargo test --locked for registry-evidence, registry-evidencectl and registry-evidence-authoring
  • cargo test --locked -p registry-evidence --features schema --lib schema::tests
  • with EVIDENCE_BIN set: cargo test --locked -p registry-evidencectl --test public_check -- --include-ignored and --test scaffold -- --include-ignored
  • node --test products/evidence/examples/public-check/page/check.test.mjs
  • npm test and npm run check in docs/site
  • products/breg/evidence/tests/verify-composition.py against debug bregctl, evidencectl and evidence: passed (offline boundary; 37 fixture cases, 3 definitions, 2 sources; tampered import and renamed fact refused)

Notes for review

  • Commits are separated by concern; squash freely.
  • The registry-evidence schema-feature tests (schema::tests) do not run on pull requests. The fixture rule broke three of them, which pull-request CI did not show; they are fixed and were run locally.

Review follow-ups

Changed after review:

  • The fixture JSON Schema lists common and privacyExpectation (reference) and privacyExpectation (coequal) as required, through a struct-level schema transform so the block definitions stay in $defs. The Rust fields stay Option so the reader refuses a missing block at its own location.
  • The changelog entries for the three refusal changes are marked BREAKING, with what a caller or script changes.
  • The missing-project action also names the deployment directory evidencectl test accepts (one holding runtime.yaml).
  • The missing-block action reads "Add a <block> mapping to the fixture file."
  • The batch source stage says why an inputs failure there is not the caller's.
  • IMPLEMENTATION.md states where in the request order the selector refusal is returned.

Not changed:

  • The relative extract path target new --local writes is a placeholder resolved against the target directory; check --target targets/local passes on a fresh project, and no file is generated at that path.
  • The literal messages in validate_reference_error stay &'static str: the error type takes one in about 250 places, and changing it is out of scope here.

…lector-invalid on the relying routes

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…penapi fixture passes check

target new --local now carries the project extract bindings alongside its connections. Bare file and directory names in target new resolve against the current directory. The init --openapi fixture is written in the coequal grammar its flag selects.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…n shape in the BReg tutorial

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… pack or route, and an unbound extract

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tation

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…c fixes in the changelogs

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T07:59:57.448566Z 4b3828c New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…hema and the no-fixture action

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…project is found

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tor refusal in the request order

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… change

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@jeremi
jeremi added this pull request to the merge queue Nov 8, 2026
Merged via the queue into main with commit 55f9149 Nov 8, 2026
65 checks passed
@jeremi
jeremi deleted the fix/evidence-followups branch November 8, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant