Skip to content

fix(s3): multipart uploads for large and streaming bodies#421

Merged
wolfy-j merged 4 commits into
mainfrom
fix/s3-multipart-uploader
Jul 4, 2026
Merged

fix(s3): multipart uploads for large and streaming bodies#421
wolfy-j merged 4 commits into
mainfrom
fix/s3-multipart-uploader

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Problem

cloudstorage.UploadObject (S3 backend) sent the body through a single s3.PutObject. Against S3-compatible stores this fails for large or streaming bodies:

  • A non-seekable reader with no Content-Length returns 411 MissingContentLength.
  • A large body signed as one aws-chunked stream is rejected by MinIO once a chunk exceeds 16MiB: chunk too big: choose chunk size <= 16MiB.

The Lua cloudstorage:upload_object path hands UploadObject a plain io.Reader (string bodies wrap bytes.Reader; userdata bodies pass the raw reader), so any multi-gigabyte checkpoint or dataset upload fails.

Fix

Route UploadObject through the SDK multipart manager (feature/s3/manager) with a 16MiB part size. The manager buffers the stream into parts of known length, so:

  • unbounded / non-seekable streams upload without a caller-supplied Content-Length;
  • no single request carries an aws-chunked chunk larger than MinIO's 16MiB cap;
  • objects split into up to 10000 parts (160GiB for unknown-length streams; auto-grown up to S3's object max for known-length bodies).

All existing PutObjectInput semantics are preserved (content-type, cache-control, disposition, encoding, metadata, If-Match/If-None-Match, and the custom request-header middleware, now applied via Uploader.ClientOptions); the ErrPreconditionFailed mapping is unchanged.

Test

TestUploadObjectLargeStream uploads a 32MiB non-seekable stream and reads it back byte-identical. It is gated on S3_ENDPOINT/S3_ACCESS_KEY/S3_SECRET_KEY (skips without a live endpoint). Verified against a live MinIO: fails on the pre-fix code (411 MissingContentLength), passes after.

Single PutObject signs the body as one aws-chunked stream, which
S3-compatible stores reject once a chunk exceeds 16MiB, and cannot send a
non-seekable body without a Content-Length. Route UploadObject through the
multipart manager (16MiB parts) so unbounded streams and multi-gigabyte
objects upload correctly against both AWS S3 and MinIO.
@wolfy-j wolfy-j requested a review from skhaz July 4, 2026 03:04
wolfy-j added 3 commits July 4, 2026 11:08
The s3/manager multipart uploader is the stable API for streaming and
large-body uploads; the s3/transfermanager replacement is not a dependency
and its migration is out of scope. Annotate the two calls with a specific
nolint so the deprecation warning does not fail lint.
The header middleware was attached to the uploader's ClientOptions, so
object-level headers (x-amz-tagging, x-amz-server-side-encryption, ...) were
applied to every multipart sub-request, including UploadPart and
CompleteMultipartUpload, which reject many of them. That made uploads larger
than the part size fail while small single-PutObject uploads with the same
options succeeded. Inject the headers only on PutObject and
CreateMultipartUpload.
…uests

Scoping every header to the object-creating operations dropped headers that
S3 requires on multipart subrequests: SSE-C customer-key headers must be
resent on UploadPart, and x-amz-request-payer / x-amz-expected-bucket-owner
apply to every subrequest. Split object-metadata headers (creation only) from
these request-scoped headers so encrypted and requester-pays multipart uploads
keep working.
@wolfy-j wolfy-j merged commit 24fa253 into main Jul 4, 2026
4 checks passed
@wolfy-j wolfy-j deleted the fix/s3-multipart-uploader branch July 4, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants