fix: number multipart parts starting at 1 in uploadStream - #1490
Conversation
uploadStream incremented partNumber before uploading each chunk, so the number sent to the server (and pushed to eTags) was one higher than the chunk's position. A fresh upload numbered parts 2,3,4..., and on a resume the skip branch (which uses the current partNumber) and the upload branch (which used the pre-incremented one) disagreed, producing gaps or duplicate part numbers in completeMultipartUpload. Use the current partNumber for the upload and increment only afterwards, so the skip and upload branches agree and numbering is 1-based. Fixes minio#1483
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughCorrects multipart upload part-number advancement during resume operations and adds unit tests for fresh uploads and resumed uploads with matching or mismatched existing parts. ChangesMultipart upload numbering
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Warning
CodeRabbit couldn't request changes on this pull request because it doesn't have sufficient GitHub permissions.
Please grant CodeRabbit Pull requests: Read and write permission and re-run the review.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unit/upload-stream-partnumber-test.js`:
- Around line 85-101: Add a test alongside “numbers re-uploaded parts
consistently with skipped parts when resuming” that seeds an existing part with
a different etag/MD5, resumes the upload, and verifies the mismatched chunk is
re-uploaded using the same part number rather than an incremented number. Assert
the recorded sent and completed part numbers reflect that reused part number.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a559a141-fc79-42e4-9ea8-ffcc18f5857f
📒 Files selected for processing (2)
src/internal/client.tstests/unit/upload-stream-partnumber-test.js
|
@maximilize please address the review comments |
Adds the resume case where a stored part's etag is stale: the chunk must be re-uploaded at its own part number rather than shifted. Guards the same off-by-one the fix addresses (verified RED with a pre-increment mutation, GREEN with the fix).
|
Added the missing case in b588ebb — the resume path where a stored part's etag is stale, so the chunk is re-uploaded at its own part number (2) instead of being shifted to 3. It asserts Verified in a container: green with the fix, and red ( |
…load - Remove duplicate partNumber++ in uploadStream that caused fresh uploads to number parts as 1,3,5 instead of 1,2,3 (regression from minio#1490) - Fix selectObjectContent building XML body as array-of-objects instead of a flat object, which produced malformed XML rejected by S3/MinIO - Functional test: replace host-name guard on Select test with a runtime MethodNotAllowed catch so the test runs and passes on supporting servers and silently skips on servers that don't implement S3 Select
Problem
Fixes #1483 (and its duplicate #1482). In
uploadStream(src/internal/client.ts) the multipart part counter is incremented before the part is uploaded, so the number sent to the server and pushed toeTagsis one higher than the chunk's actual position:Two consequences:
2, 3, 4, …instead of1, 2, 3, …. S3 tolerates a non-1 start as long as it is consistent, so this stays latent.partNumberwhile the upload branch records it underpartNumber + 1. On a resumed upload that mixes skipped and re-uploaded chunks, the two branches disagree, producing gaps or duplicate part numbers in theeTagspassed tocompleteMultipartUpload— a corrupted completion.Fix
Use the current
partNumberfor the upload and increment only after a successful part, so the skip and upload branches agree and numbering is 1-based.Tests
tests/unit/upload-stream-partnumber-test.jsstubs the network-touching internals and drivesuploadStream:[1, 2, 3](was[2, 3, 4]);3(was4), so skipped and uploaded parts stay consistent.Both fail before the change and pass after; the full
tests/unitsuite (176 tests) stays green.Note
This overlaps the stream loop touched by #1480 and #1481. #1481 (about unknown stream size) also happens to correct the counter but has been in
CHANGES_REQUESTEDfor a while, and #1480 refactors the same loop while preserving the off-by-one. This PR is a minimal, standalone fix for the part-numbering bug with dedicated regression tests, independent of those changes.Summary by CodeRabbit