Skip to content

fix(web): retry each upload part instead of failing the whole upload - #243

Merged
ravirajsinh45 merged 3 commits into
Techiebutler:mainfrom
Lennart-Pingpong:fix/chunk-upload-retry
Aug 15, 2026
Merged

fix(web): retry each upload part instead of failing the whole upload#243
ravirajsinh45 merged 3 commits into
Techiebutler:mainfrom
Lennart-Pingpong:fix/chunk-upload-retry

Conversation

@Lennart-Pingpong

@Lennart-Pingpong Lennart-Pingpong commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

A multi-gigabyte upload is several hundred sequential PUTs. Until now the first
non-OK response threw and aborted the entire upload without a second attempt,
discarding every byte already transferred. A single transient 500/503 from the
object store — or a brief network drop — could therefore cost an hour of work on
a slow uplink.

We hit this on our own instance: a 3.75 GB upload died at 36 % (129 of 358 parts,
1.35 GB transferred, 21 minutes in) and had to start over from zero.

Each part is now retried with exponential backoff instead.

Changes

  • uploadPartOnce() — one attempt at a single part, returns its ETag
  • uploadPart() — retries a part up to PART_MAX_ATTEMPTS (8) times with
    exponential backoff and jitter, ≈254s of tolerance per part
  • uploadAllParts() — the part loop, now shared
  • The presigned URL is re-fetched on every attempt: presign_upload_part
    expires after an hour and a large upload can outlive that, so a URL cached
    before a long backoff may already be dead
  • User-initiated cancels (AbortController) are never retried and surface
    immediately
  • A 4xx is not retried — a rejected request (bad signature, expired policy,
    wrong length) fails identically on every attempt, so repeating it eight times
    across four minutes only delays the error while the upload looks frozen. 408
    and 429 stay retryable, since both explicitly invite a later attempt
  • The part loop was duplicated verbatim between the new-asset and new-version
    upload paths — both now call uploadAllParts(), so the retry logic exists once
    (this is why the diff removes ~47 lines while adding the retry)
  • 6 new tests in apps/web/stores/__tests__/upload-store.test.ts

Verified the tests actually pin the behaviour: setting PART_MAX_ATTEMPTS = 1
turns exactly the three retry tests red and leaves the ordering and cancellation
tests green.

Testing

  • Backend tests pass — not run: no backend files touched by this PR
  • Frontend tests + build pass (pnpm test → 269 tests passing,
    pnpm exec tsc --noEmit clean, pnpm build succeeds)
  • Tested manually in browser

Manual test. Uploaded a large file against a staging instance and pulled the
Wi-Fi for >30s mid-transfer, then reconnected. Verified server-side against S3
ListParts rather than trusting the progress bar:

parts before the outage : 32   (335.5 MB)
gap in the part timeline: 09:39:15 → 09:40:43  (89s)
parts after reconnect   : 43   (450.9 MB, still climbing when we stopped it)
gaps in the part-number sequence: none

The upload survived and continued at the next part. On main the first failed
PUT would have ended it and discarded all 335 MB. The gap is 89s rather than 30s
because the retry was mid-backoff when connectivity returned and waited out the
remaining interval.

Two honest caveats: the run was cancelled deliberately rather than carried
through to /upload/complete (it was a 4.6 GB file on a slow uplink), and this
tests recovery from a network outage — not a closed tab or a suspended machine,
which no amount of retrying can address.

One observation, not part of this PR: during the backoff the UI shows
nothing — no "retrying" state, no stalled-connection hint — so it looks like the
upload died while it is in fact waiting. A user would likely cancel and start
over, which is the opposite of what the retry is for. The store knows it is
retrying; it just doesn't tell the UI. Happy to follow up separately.

Checklist

  • CHANGELOG.md entry added under ## [Unreleased]### Fixed

Screenshots

n/a — no UI changes.


Related: this is the client-side half of the problem described in #241.
Retry reduces how often an upload dies; it cannot help when the browser context
itself disappears, which is what resumable uploads would address.

Lennart-Pingpong and others added 2 commits August 14, 2026 11:05
A multi-gigabyte file is several hundred sequential PUTs. Previously the
first non-OK response threw and aborted the entire upload without a second
attempt, discarding everything already transferred - a single transient
500/503 from the object store, or a brief network drop, could cost an hour
of work on a slow uplink.

Each part is now attempted up to 8 times with exponential backoff and
jitter (~254s of tolerance per part). The presigned URL is re-fetched on
every attempt because presign_upload_part expires after an hour and a large
upload can outlive that, so a URL cached before a long backoff may already
be dead. User-initiated cancels are never retried.

The part loop was duplicated verbatim between the new-asset and new-version
upload paths; both now share uploadAllParts(), so the retry exists once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The retry treated every failure as transient, so a rejected request - bad
signature, expired policy, wrong length - was repeated eight times across
four minutes before surfacing. That delays the error message without ever
changing the outcome, and while it happens the upload looks frozen at its
current percentage.

Only network errors and 5xx are repeated now. 408 and 429 stay retryable:
both explicitly invite a later attempt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@ravirajsinh45 ravirajsinh45 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Lennart-Pingpong. This one is in good shape and we are happy with it.

We checked it out and ran it rather than reading alone: 35 test files, 271 tests, all green, and the one tsc error (globals.css side-effect import) reproduces identically on main, so it is not from this PR.

The details we specifically looked at and agree with: re-fetching the presigned URL inside each attempt is right, since presign_upload_part expires after an hour and a large upload can easily outlive that; treating 4xx as permanent while letting 408 and 429 through is the correct split; and never retrying after a cancel avoids the obvious foot-gun. Jitter on the backoff is a nice touch. Folding the duplicated part loop into one uploadAllParts is worth it on its own, and it is what #242 and #241 will build on.

We pushed one commit to your branch: the ### Fixed block sat above ### Changed, and Keep a Changelog orders it the other way. Nothing else changed.

One optional follow-up, not a blocker for this PR. The backoff is a plain setTimeout, so cancelling during a late attempt takes up to 128 seconds to register and issues one more presign call before it notices. An abort-aware sleep would make cancel feel immediate. Happy for that to be a separate change, or to leave it.

Merging once CI re-runs.

@ravirajsinh45
ravirajsinh45 merged commit 0a7a116 into Techiebutler:main Aug 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants