Skip to content

fix: settle the CI flakes on #2609 and the native record-timer harness races on #2505 - #2611

Merged
jeremi merged 5 commits into
mainfrom
fix/ci-flakes-2505-2609
Nov 8, 2026
Merged

jeremi merged 5 commits into
mainfrom
fix/ci-flakes-2505-2609

Conversation

@jeremi

@jeremi jeremi commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Closes #2609. Refs #2505: two harness races are fixed and the diagnostics are in, but two of its CI symptoms still have no proven cause, so the issue should stay open until the next failure names one.

#2609

  1. Adapter a_reply_of_many_small_rows_is_walked_to_the_end raced the 2 s per-call script deadline in an unoptimised build. The test moves into the engine's unit tests (registry-adapter-engine/src/scripts.rs) and runs through a private invoke_until under a long caller deadline, so it proves the operation and size limits admit the densest reply the cap allows. Production calls still go through invoke_thrown, which keeps the 2 s ceiling. Trade-off: no test now asserts that a cap-sized reply finishes within 2 s on production hardware; that is a performance property, not a correctness one, and a wall-clock assertion in a debug build could not prove it anyway.
  2. BReg real_postgres_blind_index_reindex_refuses_more_rows_than_its_ceiling. The PostgreSQL server log from the failed run shows the cause: the activation's managed-ACL catalog query was cancelled by the test's 5 s statement_timeout while another test's DROP DATABASE forced a checkpoint that spent 6.7 s writing. The query reads a few hundred catalog rows. Every apply in postgres_migration.rs now goes through one timeouts() helper with a 30 s statement bound. No test in the suite proves an apply statement times out; the rehearsal timeout test uses the reviewed migration's own 200 ms declaration and is unaffected.
  3. GovStack real_breg_and_adapter_preserve_write_receipts_conditions_and_exact_recovery failed because bregctl dev could not download the pinned ThunderID image. The "Registry Adapter PostgreSQL and recovery suites" job now pulls the image named in thunderid-version.json with three attempts before the suites, so a registry hiccup fails as that step. Other jobs that start bregctl dev are unchanged.

#2505

Five symptoms, with their status:

Symptom Status Change
Fairness journey: "stale timer did not refresh within the bound" Reproduced under load The due window (5 s) was shorter than setup on a starved runner, so both replicas parked at their claims before refreshing the stale key. It is now 15 s.
A record_timer.worker.iteration_failed warning in the fairness run Mechanism found in code, not reproduced A replica that takes the anchor row after its peer committed the refresh matches no row and reports a retryable conflict. The journey now allows up to two such warnings per replica, and only with the concurrency-conflict cause.
Bulk-race tests: "did not report a new successful iteration after release" Reproduced under load The wait required every replica to be fresher than 0.5 s in one sweep. It now marks each replica once it succeeds after release, and the timeout names the held and last ages.
Replica exits with status 1 before readiness Unknown Diagnostics: the failure now carries the replica's log tail.
Uncoded WARN/ERROR line Unknown Diagnostics: log mismatches now quote the offending lines.

Product change: the record_timer.worker.iteration_failed warning now carries a closed, value-free error field. It names one of six classes: runtime unavailable, concurrency conflict, commit unresolved, handler failure, field-encryption unavailable, or refused. The commit also adds a CHANGELOG entry and a sentence in the monitor docs. Log lines are quoted only after the existing forbidden-value check passes over every replica's log.

Open product question, not changed here: in the refresh, refresh-control-cleared and cancel lanes a lost peer race logs a warning, but in the expire lane the same race returns quietly. Whether a lost race should warn at all is a separate call.

Not in this PR

  • The initial-activation path in migration.rs discards the underlying error (reconcile_runtime_acl(...).is_ok(), and From<tokio_postgres::Error> maps every database error to Connection), so an operator or a test sees a bare ApplyFailed. The cause of item 2 was only recoverable from the server log. Carrying a value-free cause (the existing PostgresFailure shape) through that path is a product error-shape change; it should be its own change.
  • govstack_consultation_served.rs a_page_at_the_declared_maximum_is_shaped_within_the_script_budget asserts under the same 2 s wall-clock deadline and may flake the same way; it has not failed yet.
  • registry-adapter-job signed_coordinator_event_native.rs postgres_source_reaches_native_coordinator_and_reviews_a_revised_row failed once on this PR's first CI run at its 30 s wall-clock wait for the native Run to finish ("the native Run did not finish as revised-reviewed"). This PR does not touch that file, it passed on re-run, and it has no earlier failure in the last 60 CI runs. It is the same class of fixed wall-clock bound.

Verification

  • cargo fmt --check: clean.
  • cargo clippy -D warnings: clean on registry-adapter-engine and registry-adapter (all targets), and on registry-breg (all targets, default features).
  • Adapter: the engine scripts unit tests (5/5) and script_budget (4/4).
  • postgres_migration against the CI PostGIS image: 61/61.
  • iteration_failures_map_to_closed_value_free_causes: 1/1.
  • Record-timer metrics: 12/12.
  • Native record-timer suite: 35/35 in a normal run while the machine was heavily loaded.
  • Repeated parallel stress rounds of the native suite: the fairness and bulk-race journeys passed after the fixes. The remaining failures in the most loaded round were other tests that hit their own fixed wall-clock bounds; this PR does not change those.
  • actionlint and release/scripts/check-gates-inventory.py: pass.
  • Known, and not from this PR: clippy --features postgres-test,tooling,schema --test postgres_migration reports a pre-existing type_complexity at postgres_migration.rs:9033. CI does not gate on that combination.

… wall clock

The many-small-rows test raced the 2 s per-call deadline in unoptimised CI builds. It moves into the engine's unit tests and runs under a long caller deadline, so it proves the operation and size limits admit the densest reply the cap allows. Production calls keep the 2 s ceiling. Refs #2609.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…uites

The GovStack BReg suite starts bregctl dev, which pulls the image on first use, so a registry hiccup failed as a test. Refs #2609.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…sts a CI checkpoint stall

The blind-index ceiling test failed in setup when the activation's managed-ACL catalog query hit the suite's 5 s statement timeout during another test's forced checkpoint. Every apply in the suite now uses one timeouts() helper with a 30 s statement bound. Refs #2609.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A record_timer.worker.iteration_failed warning carried no cause, so a CI
failure on it could not tell a lost peer race from an unavailable database.
The warning now carries a closed, value-free `error` naming one of six
classes. Refs #2505.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…fix two harness races

Log mismatches now quote the offending WARN/ERROR lines, and a replica that
exits or times out before readiness reports its log tail, both only after
the forbidden-value check passes. A code expectation can be restricted to
closed `error` values.

The fairness journey's due window was shorter than its setup on a starved
runner, so both replicas parked at their claims before refreshing the stale
key; it now leaves fifteen seconds. Each replica may lose up to two stale
refresh races to its peer, reported only as a concurrency conflict.

The progress wait required every replica to be fresher than half a second
in one sweep; it now latches each replica once it succeeds after release
and names the held and last ages on timeout. Refs #2505.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-08T08:02:36.340090Z a6237b6 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jeremi
jeremi added this pull request to the merge queue Nov 8, 2026
Merged via the queue into main with commit 07651c0 Nov 8, 2026
135 of 138 checks passed
@jeremi
jeremi deleted the fix/ci-flakes-2505-2609 branch November 8, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI flakes on 2026-11-07: Adapter script budget, BReg blind-index setup, GovStack ThunderID pull

1 participant