[DO NOT MERGE until freeze lifts] TT-9424: Sort aggregate batches by ID to prevent PostgreSQL deadlocks - #1069
Draft
nerdydread wants to merge 1 commit into
Draft
[DO NOT MERGE until freeze lifts] TT-9424: Sort aggregate batches by ID to prevent PostgreSQL deadlocks#1069nerdydread wants to merge 1 commit into
nerdydread wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #1058, which was approved and merged prematurely as
eead02edbefore QA, so it had to be reverted by #1067 (e43a3b8f).What this contains
This is a single
git revertof #1067's squash commit, so it restores both changes that #1058 carried:sort.Slice(recs, ...)sites (pumps/sql_aggregate.go,pumps/graph_sql_aggregate.go,pumps/mcp_sql_aggregate.go,pumps/sql.go), giving every replica the same row-lock acquisition order and preventing the circular waits behindSQLSTATE 40P01.OmitIndexCreation: trueinTestMCPSQLAggregatePump_WriteData, which stops the backgroundCREATE INDEX CONCURRENTLYgoroutine from racing the subtest'sDROP TABLEcleanup.Side benefit
masteris currently red ate43a3b8f:TestMCPSQLAggregatePump_WriteData/tracks_errorsfails at roughly 1 run in 3, because the revert took theOmitIndexCreationworkaround out along with the production change. Merging this fixes that flakey test, since the fix rides along.Verification
pumps/sql_deadlock_ordering_test.go.git diff eead02ed HEAD -- pumps/is empty — byte-identical to the tree that was already reviewed and approved on TT-9424: Sort aggregate batches by ID to prevent PostgreSQL deadlocks #1058.master, so it includesac01deff(GCP IAM auth) and the two Kinesis fixes that the originalTT-9424-sql-aggregate-deadlockbranch predates.go build ./...andgo test ./pumps/ -run 'TestSQL|TestMCPSQL' -count=1pass locally.Note for reviewers
The
validate(Jira linter) check will fail while TT-9424 sits inReady for Testing— that status is not in the linter's accepted list. It is not a required check. Required checks areAggregated CI StatusandUnit Tests & Linting.Full background, test methodology, and the four-call-site verification are on TT-9424.