Skip to content

fix(frontend): fence session reset from active requests - #26497

Merged
mergify[bot] merged 4 commits into
matrixorigin:mainfrom
VioletQwQ-0:codex/issue-20022-proxy-reset-fence
Aug 5, 2026
Merged

fix(frontend): fence session reset from active requests#26497
mergify[bot] merged 4 commits into
matrixorigin:mainfrom
VioletQwQ-0:codex/issue-20022-proxy-reset-fence

Conversation

@VioletQwQ-0

@VioletQwQ-0 VioletQwQ-0 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #20022

What this PR does / why we need it:

Proxy can observe a terminal MySQL response and start ResetSession while the
original CN RoutineManager.Handler is still completing response accounting
and defers. The old lifecycle controller serialized reset/migration/close, but
did not serialize ordinary requests, so reset could close the old Session,
process, and transaction handler while Handler still owned them.

This change:

  • adds a lightweight request lease to the existing lifecycle controller;
  • makes ResetSession fail fast with the existing busy error while a request
    is active, so Proxy discards that backend and reconnects;
  • lets migration wait with its caller context, while routine close keeps its
    existing cancel-without-request-wait behavior;
  • rejects a new request before it reads Session if lifecycle work already owns
    the routine;
  • makes ExecRequest panic recovery safely handle a nil transaction handler.

Proxy cache/generation/in-flight production logic is unchanged.

Regression evidence

  • Base: 4959b7cdd853923f60b168d9ef69e242a1cadc37
    • TestRoutineManagerResetSessionRejectsRequestAfterResponseWrite failed:
      reset returned nil after the terminal response write while Handler was
      deliberately blocked before returning.
  • PR head: af17db6b3db4b19cf5b4a604395574ce21a4a602
    • the same deterministic interleaving returns busy, preserves the old
      Session/proc/TxnHandler and SessionManager entry, then succeeds once after
      Handler exits and serves the next COM_PING;
    • the race regression passed 100 consecutive iterations;
    • lifecycle-first, migration wait/cancel, close-no-wait, replacement reuse,
      rejected Proxy cache entry, and nil-TxnHandler recovery paths pass.

Validation

  • .agents/skills/mo-dev/scripts/mo-cgo-test ./pkg/frontend -count=1: PASS
  • .agents/skills/mo-dev/scripts/mo-cgo-test ./pkg/proxy -count=1: PASS
  • complete pkg/frontend and pkg/proxy with -race -count=1: PASS
  • focused request/reset interleaving with -race -count=100: PASS
  • go build -mod=readonly ./pkg/frontend ./pkg/proxy: PASS
  • go vet -mod=readonly ./pkg/frontend ./pkg/proxy: PASS
  • git diff --check: PASS
  • schema-v4 preflight: PASS review=PASS validation=PENDING
  • current exact-head required CI: PASS, including Ubuntu UT, SCA, UT coverage,
    Proxy Compose BVT, pessimistic launch BVT, and coverage.

Local deployment evidence

The local topology was launched with the real Proxy path
(mo-service -with-proxy), 2 CNs, and true new client connections.

  • direct-CN control, 100 workers / 2 s / 2 min: 5,987 queries, 0 failures;
  • Proxy cache disabled, 100 workers / 2 s / 2 min: 5,988 queries and one
    isolated 5 s context deadline; all Proxy/CN probes passed;
  • Proxy cache enabled after warm-up, 100 workers / 1 s / 60 s: 5,995 queries,
    0 failures, all 36 endpoint probes passed, and backend sessions converged
    from 119 to 17/18 rather than growing with total connections;
  • no CN/Proxy panic, pod/process restart, wait active, or
    createTxnOpUnsafe was observed in these valid runs.

A malformed first result was observed in two cold-start runs immediately after
ports opened. A later diagnostic run without s2c batching passed 5,984 queries,
but it ran after the cluster was fully warm and therefore does not prove a
buffering defect. This cold-start observation is tracked as unresolved
acceptance evidence and is not used to justify an unrelated Proxy code change.

The complete 10-minute local soak is still pending: the initial 100
connections/s setup exhausts the macOS loopback ephemeral-port range, while
the later 50 connections/s rerun was interrupted by the local disk safety
threshold. TKE validation remains pending.

QA / BVT

  • QA required: yes
  • BVT: N/A — regular SQL BVT cannot enable Proxy connection cache or create
    independent true short connections. Deterministic Frontend/Proxy concurrency
    tests provide the in-repo regression coverage.
  • Deployment validation pending:
    • local 2 CN + Proxy, warmed and readiness-gated, 100 concurrent true short
      connections for 10 minutes at a host-safe connection rate;
    • TKE 3 CN + 2 Proxy, 1000 vusers for 10 minutes;
    • TKE 300 terminals for 70 minutes, with independent Proxy/CN login probes,
      no panic/restart/continuous login failure, and bounded backend sessions.
  • After merge, keep [Bug]: config proxy cache connection mo hung #20022 open under phase/testing for Ariznawlll to run
    the deployment matrix; link matrixorigin/mo-nightly-regression#835 for the
    recurring short-connection coverage.

Current-head CR follow-up (2026-08-05)

Head e70c5a45ca7617ab668d09150184814f98218078 also routes both rolling-upgrade
MigrateConnFrom compatibility actions through the Routine lifecycle admission
before reading or mutating Session user-lock state. Skip now checks the current
admitted Session after a request/reset owner exits; Enable mutates only that
admitted Session generation.

  • Parent-head reproduction: request-first/reset-first × Skip/Enable all failed
    against 1bc7d0021fb1fbcbd00ce4af3b703711a90d9e38.
  • Focused and adjacent migration tests: PASS.
  • Focused race stress: request-first T=0.21s, B=30s, N=100 PASS;
    reset-first T=0.22s, B=30s, N=100 PASS.
  • pkg/frontend: list/build/vet/test and full-package race PASS.
  • Direct consumer pkg/cnservice: list/build/vet/test PASS.
  • git diff --check: PASS.
  • Schema-v4 preflight: PASS review=PASS validation=PENDING at exact head
    e70c5a45ca7617ab668d09150184814f98218078.
  • BVT: N/A — these legacy QueryService MigrateConnFrom action values are
    selected by old Proxy versions and cannot be invoked through SQL BVT. The
    deterministic RoutineManager.MigrateConnectionFromWithContext interleaving
    tests exercise the production RPC entry and assert the final lock/Session
    state. New exact-head CI is pending.

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the new request/lifecycle fence is bypassed by the legacy MigrateConnFrom actions.

In RoutineManager.MigrateConnectionFromWithContext, MigrateConnFromSkipUserLevelLockRelease reads the current session user-lock state before routine.migrateConnectionFromWithContext acquires mc. A reachable ordering is:

  1. A SQL request owns requestInProgress but has not acquired its user lock yet.
  2. The legacy migration RPC checks UserLevelLocksForMigration and observes no locks, then blocks in beginOperationWithContext.
  3. The SQL request completes GET_LOCK and releases the request lease.
  4. Migration resumes, sets userLevelLocksMigrated = true, and succeeds without rechecking the lock state.

That defeats the safety check and can let a connection migrate with a held lock; close then takes the migrated-lock discard path instead of releasing it. The MigrateConnFromEnableUserLevelLockRelease branch is also completely outside lifecycle admission and can mutate an old session while reset/close owns it. These compatibility actions remain reachable during rolling upgrades.

Please move both action-specific session reads/mutations behind the same lifecycle admission (with the check performed after admission), and add request-first/reset-first interleaving tests for these two actions.

@VioletQwQ-0

Copy link
Copy Markdown
Collaborator Author

@XuPeng-SH Addressed on e70c5a45ca7617ab668d09150184814f98218078.

Both legacy MigrateConnFrom actions now enter the same Routine lifecycle admission before any Session access:

  • MigrateConnFromSkipUserLevelLockRelease re-reads user-lock state from the current admitted Session and sets userLevelLocksMigrated only after that check succeeds.
  • MigrateConnFromEnableUserLevelLockRelease clears the flag only on the current admitted Session generation.

Regression coverage calls the production RoutineManager.MigrateConnectionFromWithContext entry and covers request-first/reset-first interleavings for both actions. The four cases failed on parent head 1bc7d0021fb1fbcbd00ce4af3b703711a90d9e38 and pass with this repair. The Skip cases add a lock or publish a locked replacement while migration is queued; the Enable cases assert that the old/current Session is untouched until admission.

Validation:

  • focused and adjacent migration tests: PASS
  • request-first race stress: T=0.21s, B=30s, N=100: PASS
  • reset-first race stress: T=0.22s, B=30s, N=100: PASS
  • full pkg/frontend normal + race, build, and vet: PASS
  • direct consumer pkg/cnservice test, build, and vet: PASS
  • git diff --check: PASS
  • schema-v4 preflight: PASS review=PASS validation=PENDING on exact head e70c5a45ca7617ab668d09150184814f98218078
  • BVT: N/A — old-Proxy QueryService action selection is not reachable through SQL BVT; the deterministic production-RPC-entry tests are the equivalent regression coverage

The PR remains Ready for Review. New exact-head CI is pending.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous lifecycle-fence blocker is resolved on e70c5a45ca7617ab668d09150184814f98218078.

Both legacy MigrateConnFrom actions now acquire the same request/lifecycle admission before reading or mutating the session. In particular, the skip action rechecks user-level locks on the admitted current session, and the enable action only changes that same session generation. The new request-first and reset-first tests cover both actions and would expose the stale/pre-admission access from the prior head.

I also rechecked the surrounding ownership paths: reset remains fail-fast while a request owns the session; migration waits with caller cancellation; close seals admission and cancels active work while preserving its existing non-waiting request-close behavior. I found no remaining correctness or lifecycle blocker in this diff.

Local exact-head validation passed:

  • all newly added focused tests
  • the same focused set under -race
  • full pkg/frontend
  • direct consumer pkg/cnservice
  • compile-only checks and go vet for both packages
  • git diff --check

Approve.

@mergify mergify Bot added the queued label Aug 5, 2026
@mergify

mergify Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • Entered queue2026-08-05 03:33 UTC · Rule: main · triggered by rule Automatic queue on approval for main
  • Checks passed · in-place
  • Merged2026-08-05 05:02 UTC · at 63e07c1e9cf818f388a00ee62b14386eac9f0a53 · squash

This pull request spent 1 hour 28 minutes 23 seconds in the queue, including 45 minutes 57 seconds running CI.

Required conditions to merge
  • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • github-review-approved [🛡 GitHub branch protection]
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
    • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
    • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone CI / UT Test on Ubuntu/x86
    • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
    • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone CI / SCA Test on Linux/arm64
    • check-neutral = Matrixone CI / SCA Test on Linux/arm64
    • check-skipped = Matrixone CI / SCA Test on Linux/arm64
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PROXY)
    • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PROXY)
    • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PROXY)
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone Utils CI / Coverage
    • check-neutral = Matrixone Utils CI / Coverage
    • check-skipped = Matrixone Utils CI / Coverage
  • any of [🛡 GitHub branch protection]:
    • check-success = Matrixone UT Coverage / UT Coverage on Ubuntu/x86
    • check-neutral = Matrixone UT Coverage / UT Coverage on Ubuntu/x86
    • check-skipped = Matrixone UT Coverage / UT Coverage on Ubuntu/x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/bug Something isn't working size/M Denotes a PR that changes [100,499] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants