Skip to content

Harden rewind against race conditions - #1433

Merged
sophiatev merged 5 commits into
mainfrom
stevosyan/harden-rewind
Nov 13, 2026
Merged

sophiatev merged 5 commits into
mainfrom
stevosyan/harden-rewind

Conversation

@sophiatev

@sophiatev sophiatev commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This PR hardens rewind/work item completion against race conditions:

  1. Work item commits failed instance and history state and then fails before deleting the triggering messages. Rewind then sets the instance state back to "Pending". When reprocessing the triggering messages we assume the instance state needs to be reconciled back to "Failed" when we should in reality leave it alone
  2. Same situation as above except a new execution is what sets the status back to "Pending". We again don't want to override the new execution when reprocessing the triggering messages
  3. Rewind races with a new execution - we don't want rewind to clobber the new execution
  4. A suborchestration instance fails and sends the failure to its parent before it actually commits its failed history and instance status. Then rewound can be legitimately requested on the parent and reach the suborchestration, still before the new state has committed. We want to wait for the suborchestration's failed state to commit before proceeding with rewind

Copilot AI balanced review requested due to automatic review settings November 11, 2026 20:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Rewind recovery can strand already-rewound children, overwrite Pending state with stale Failed history, and time out on stale or missing instance rows.

3 open issues
What changed in this PR

This PR hardens Azure Storage rewind and terminal-status recovery against concurrent history changes and replacement executions.

Changes:

  • Adds conditional ETag checks to rewind and status-recovery writes.
  • Waits for failed checkpoints and retries work-item acquisition when recovery encounters conflicts.
  • Adds race-condition tests and updates tracking-store contracts and test implementations.
File Description
test/​DurableTask.AzureStorage.Tests/​ParentInstanceIdTrackingStoreTests.cs Updates repair tests to use conditional merges and ETags.
test/​DurableTask.AzureStorage.Tests/​OrchestrationStatePrefetchTests.cs Updates the test tracking-store implementation for new signatures.
test/​DurableTask.AzureStorage.Tests/​AzureStorageScenarioTests.cs Adds rewind race and cancellation tests; waits for nested rewind completion.
src/​DurableTask.AzureStorage/​Tracking/​TrackingStoreBase.cs Updates rewind and recovery method signatures.
src/​DurableTask.AzureStorage/​Tracking/​ITrackingStore.cs Defines and documents ETag requirements.
src/​DurableTask.AzureStorage/​Tracking/​InstanceStoreBackedTrackingStore.cs Adapts the recovery implementation to the revised contract.
src/​DurableTask.AzureStorage/​Tracking/​AzureTableTrackingStore.cs Adds checkpoint waiting, history validation, and conditional status writes.
src/​DurableTask.AzureStorage/​AzureStorageOrchestrationService.cs Passes recovery ETags and abandons sessions on concurrency conflicts.

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 11, 2026 21:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-table ETag coordination and recursive rewind involve non-atomic storage and queue operations, requiring human review and storage-backed concurrency validation of the accepted crash-recovery and paginated-read gaps.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 12, 2026 18:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Recursive rewind makes non-atomic updates across history and instance tables, leaving acknowledged crash/retry and paginated-read recovery gaps that require human risk assessment and Azure Storage integration validation.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 12, 2026 20:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Rewind’s non-atomic cross-table updates and recursive child waits require human review of crash-recovery risks and storage-backed concurrency and fault-injection tests that could not be run here.

No open issues 🎉

Review effort: Balanced

@sophiatev
sophiatev merged commit 3d11a0b into main Nov 13, 2026
49 checks passed
@sophiatev
sophiatev deleted the stevosyan/harden-rewind branch November 13, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants