Skip to content

fix(sharding): enqueue delete for deferred updates moving out of migrating chunk#1819

Open
Abuhaithem wants to merge 1 commit into
mongodb:masterfrom
Abuhaithem:fix-migration-missed-deletes-deferred-updates
Open

fix(sharding): enqueue delete for deferred updates moving out of migrating chunk#1819
Abuhaithem wants to merge 1 commit into
mongodb:masterfrom
Abuhaithem:fix-migration-missed-deletes-deferred-updates

Conversation

@Abuhaithem

Copy link
Copy Markdown

This pull request addresses a severe data inconsistency bug where an orphaned document can be left on a migration destination shard.

When a chunk migration is ongoing, the migration cloner tracks updates to documents inside the chunk's range. For transactions prepared on the donor in a previous term but committed in the current term, the oplog entry lacks a postImage document key. The cloner handles these by deferring their processing via _deferProcessingForXferMod and resolving them later during nextModsBatch via _processDeferredXferMods.

However, a serious race condition exists if the deferred update moves a document out of the migrating chunk, and the document is subsequently deleted before _processDeferredXferMods can run:

  1. The prepared transaction's update moves the document out of the migrating chunk (e.g., changes its shard key).
  2. Because the transaction was prepared in a previous term, postImageDocKey is empty, and the update is deferred.
  3. A subsequent operation deletes the document from its new location (outside the migrating chunk).
  4. onDeleteOp evaluates the deletion. Because the document's new shard key is out of bounds for the migrating chunk, onDeleteOp returns early and does not enqueue a deletion to xferMods.
  5. Later, _processDeferredXferMods runs. It attempts to fetch the document using Helpers::findById.
  6. Since the document was deleted, findById fails. The code previously assumed: "That delete would have been captured by the xferMods so nothing else to do here," and executed continue.

Because the deletion was skipped by onDeleteOp (as it occurred outside the chunk bounds), the destination shard is never instructed to delete the document. The migration completes with the document incorrectly cloned to the destination shard, resulting in an orphaned document.

The Fix:
This PR fixes the bug by having _processDeferredXferMods extract the shard key from the deferred update's preImageDocKey. If the preImageDocKey was within the bounds of the migrating chunk, a deletion is unconditionally enqueued for the recipient shard. This operation is fully idempotent and ensures the destination shard correctly deletes the document.

Impact & Severity

High Severity: Silently compromises data integrity for sharded clusters during migrations combined with failovers and prepared transactions. These orphaned documents could reappear in queries or violate unique constraints.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Testing

  • Verified through rigorous code-review that the synthetic delete is idempotent and perfectly resolves the race condition.
  • The unit tests pass cleanly locally.

…ating chunk

If a deferred transaction update moves a document out of a migrating chunk
and the document is subsequently deleted before `_processDeferredXferMods` runs,
the destination shard misses the deletion, leaving an orphaned document.

This commit updates `_processDeferredXferMods` to model a delete
if the document is no longer found and its pre-image was in the chunk.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant