fix(compile): preserve remote pipeline receiver ownership - #26603
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
Distributed shuffle joins can leave a dispatch in one DOP bucket targeting local receivers owned by sibling buckets on the same CN. A following global merge then treats each bucket as an independent RemoteRun tree and rejects the dispatch-bearing tree. Detect this ownership violation at the common merge boundary and group the stage by CN before creating RemoteRun units. Keep independent pipelines unchanged and retain the runtime standalone guard. Add focused topology coverage and a multi-CN FULL OUTER JOIN plus WINDOW regression. Fixes #26595
0ac1117 to
8d9b02d
Compare
LeftHandCold
left a comment
There was a problem hiding this comment.
Deep-reviewed head 8d9b02d: no blocking issues found.
I traced the complete path from newMergeScope regrouping through per-CN scope construction, standalone validation, RemoteRun root stripping/serialization, local and remote dispatch registration, receiver notification, cancellation, and cleanup.
The regrouping is gated to non-local trees with an external in-process receiver dependency. Independent remote inputs and local dependencies retain the existing topology. Each original scope is attached exactly once, receiver/end counts remain consistent, wrapper growth is bounded by CN/input count, and error paths retain sibling cancellation plus bounded cleanup.
Focused race tests for the new Dispatch/Connector counterexamples, independent-input fast path, FULL OUTER JOIN -> WINDOW topology, and the existing standalone detector passed 3 consecutive runs with -p=1. The PR is based on current main, diff check is clean, and current GitHub checks have no failures or pending jobs.
gouhongshen
left a comment
There was a problem hiding this comment.
Codex automated review
No findings. The per-CN regrouping restores remote receiver ownership while preserving local and independent-input paths, cancellation, cleanup, boundedness, and compatibility. Tests cover Dispatch, Connector, fast-path, and FULL OUTER JOIN→WINDOW topologies. Diff and formatting checks are clean; focused local execution was unavailable because this worktree lacks MatrixOne’s native CGo prerequisites.
aptend
left a comment
There was a problem hiding this comment.
Reviewed the receiver-ownership regrouping and RemoteRun lifecycle paths. The focused topology tests, package tests, race tests, build, and vet pass locally.
What type of PR is this?
Which issue(s) this PR fixes:
Fixes #26595
What this PR does / why we need it:
Root cause
A distributed shuffle FULL OUTER JOIN creates multiple DOP receiver buckets per CN. A dispatch attached to one bucket can target local receivers owned by sibling buckets. When a following global operator such as WINDOW calls the common merge path, each bucket becomes a separate RemoteRun tree, so the dispatch-bearing tree is not standalone executable and is rejected before remote start.
Fix
Validation
Lifecycle audit
Scope and risk
No API, wire, catalog, or storage-format changes. Regrouping is compile-time only and triggers only when a remote input would otherwise be non-standalone. Independent stages retain their existing topology.