Repository navigation
Harden rewind against race conditions - #1433
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Rewind recovery can strand already-rewound children, overwrite Pending state with stale Failed history, and time out on stale or missing instance rows.
3 open issues
What changed in this PR
This PR hardens Azure Storage rewind and terminal-status recovery against concurrent history changes and replacement executions.
Changes:
- Adds conditional ETag checks to rewind and status-recovery writes.
- Waits for failed checkpoints and retries work-item acquisition when recovery encounters conflicts.
- Adds race-condition tests and updates tracking-store contracts and test implementations.
| File | Description |
|---|---|
| test/DurableTask.AzureStorage.Tests/ParentInstanceIdTrackingStoreTests.cs | Updates repair tests to use conditional merges and ETags. |
| test/DurableTask.AzureStorage.Tests/OrchestrationStatePrefetchTests.cs | Updates the test tracking-store implementation for new signatures. |
| test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs | Adds rewind race and cancellation tests; waits for nested rewind completion. |
| src/DurableTask.AzureStorage/Tracking/TrackingStoreBase.cs | Updates rewind and recovery method signatures. |
| src/DurableTask.AzureStorage/Tracking/ITrackingStore.cs | Defines and documents ETag requirements. |
| src/DurableTask.AzureStorage/Tracking/InstanceStoreBackedTrackingStore.cs | Adapts the recovery implementation to the revised contract. |
| src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs | Adds checkpoint waiting, history validation, and conditional status writes. |
| src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs | Passes recovery ETags and abandons sessions on concurrency conflicts. |
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-table ETag coordination and recursive rewind involve non-atomic storage and queue operations, requiring human review and storage-backed concurrency validation of the accepted crash-recovery and paginated-read gaps.
No open issues 🎉
3 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Recursive rewind makes non-atomic updates across history and instance tables, leaving acknowledged crash/retry and paginated-read recovery gaps that require human risk assessment and Azure Storage integration validation.
No open issues 🎉
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Rewind’s non-atomic cross-table updates and recursive child waits require human review of crash-recovery risks and storage-backed concurrency and fault-injection tests that could not be run here.
No open issues 🎉
Review effort: Balanced


This PR hardens rewind/work item completion against race conditions: