fix(s3): multipart uploads for large and streaming bodies#421
Merged
Conversation
Single PutObject signs the body as one aws-chunked stream, which S3-compatible stores reject once a chunk exceeds 16MiB, and cannot send a non-seekable body without a Content-Length. Route UploadObject through the multipart manager (16MiB parts) so unbounded streams and multi-gigabyte objects upload correctly against both AWS S3 and MinIO.
skhaz
approved these changes
Jul 4, 2026
The s3/manager multipart uploader is the stable API for streaming and large-body uploads; the s3/transfermanager replacement is not a dependency and its migration is out of scope. Annotate the two calls with a specific nolint so the deprecation warning does not fail lint.
The header middleware was attached to the uploader's ClientOptions, so object-level headers (x-amz-tagging, x-amz-server-side-encryption, ...) were applied to every multipart sub-request, including UploadPart and CompleteMultipartUpload, which reject many of them. That made uploads larger than the part size fail while small single-PutObject uploads with the same options succeeded. Inject the headers only on PutObject and CreateMultipartUpload.
…uests Scoping every header to the object-creating operations dropped headers that S3 requires on multipart subrequests: SSE-C customer-key headers must be resent on UploadPart, and x-amz-request-payer / x-amz-expected-bucket-owner apply to every subrequest. Split object-metadata headers (creation only) from these request-scoped headers so encrypted and requester-pays multipart uploads keep working.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cloudstorage.UploadObject(S3 backend) sent the body through a singles3.PutObject. Against S3-compatible stores this fails for large or streaming bodies:Content-Lengthreturns411 MissingContentLength.aws-chunkedstream is rejected by MinIO once a chunk exceeds 16MiB:chunk too big: choose chunk size <= 16MiB.The Lua
cloudstorage:upload_objectpath handsUploadObjecta plainio.Reader(string bodies wrapbytes.Reader; userdata bodies pass the raw reader), so any multi-gigabyte checkpoint or dataset upload fails.Fix
Route
UploadObjectthrough the SDK multipart manager (feature/s3/manager) with a 16MiB part size. The manager buffers the stream into parts of known length, so:Content-Length;aws-chunkedchunk larger than MinIO's 16MiB cap;All existing
PutObjectInputsemantics are preserved (content-type, cache-control, disposition, encoding, metadata,If-Match/If-None-Match, and the custom request-header middleware, now applied viaUploader.ClientOptions); theErrPreconditionFailedmapping is unchanged.Test
TestUploadObjectLargeStreamuploads a 32MiB non-seekable stream and reads it back byte-identical. It is gated onS3_ENDPOINT/S3_ACCESS_KEY/S3_SECRET_KEY(skips without a live endpoint). Verified against a live MinIO: fails on the pre-fix code (411 MissingContentLength), passes after.