Skip to content

fix(gcsartifact): prevent artifact version race in concurrent Save#1172

Open
wolo-lab wants to merge 1 commit into
mainfrom
wolo/gcs-artifact-version-race
Open

fix(gcsartifact): prevent artifact version race in concurrent Save#1172
wolo-lab wants to merge 1 commit into
mainfrom
wolo/gcs-artifact-version-race

Conversation

@wolo-lab

@wolo-lab wolo-lab commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Problem

gcsService.Save assigned version numbers non-atomically: it listed the
existing versions, computed max+1, and wrote the blob at that version. Two
concurrent Save calls for the same artifact read the same version set, pick
the same next version, and the second upload silently overwrites the first
callers get duplicate version numbers and one object is lost. The code carried a
TODO acknowledging the race.

Note: adk-python's GcsArtifactService has the same unguarded
max(versions) + 1 logic, so this is Go-specific hardening rather than a port.

Summary

  • Write the new blob with a "does not exist" generation precondition
    (x-goog-if-generation-match: 0) via
    ObjectHandle.If(storage.Conditions{DoesNotExist: true}). GCS has no
    transaction spanning the list and the write, so a colliding writer now fails
    the precondition with HTTP 412 instead of clobbering the existing object.
  • On a 412, Save re-reads the versions and retries with a fresh number,
    bounded by maxSaveAttempts (returns errVersionConflict if exhausted). N
    concurrent saves therefore yield exactly versions 1..N with no lost writes.
  • Extracted writeArtifact / isPreconditionFailed helpers and added an
    ifNotExist() method to the internal gcsObject seam and its wrapper. No
    public API change

@wolo-lab
wolo-lab force-pushed the wolo/gcs-artifact-version-race branch 2 times, most recently from 1081c43 to e9db6f3 Compare July 18, 2026 20:30
Save assigned versions non-atomically: it listed the existing versions,
computed max+1, and wrote that blob. Two concurrent Save calls for the same
artifact would read the same version set, pick the same next version, and the
second write would silently overwrite the first — duplicate returned versions
and a lost object. The code carried a TODO acknowledging this.

Assign versions with optimistic concurrency instead: write the new blob with a
"does not exist" generation precondition (x-goog-if-generation-match: 0). GCS
has no transaction spanning the list and the write, so a colliding writer now
fails the precondition (HTTP 412); Save re-reads the versions and retries with
a fresh number, using a jittered backoff between attempts and bounded by
maxSaveAttempts. It returns errVersionConflict if the budget is exhausted
rather than overwriting an existing version. N concurrent saves therefore yield
exactly versions 1..N with no lost writes.

Testing Plan:
- go build -mod=readonly ./...
- go test -race -mod=readonly -count=1 -shuffle=on ./artifact/...
- golangci-lint run ./artifact/...
- go mod tidy -diff
- TestGCSArtifactServiceConcurrentSave: concurrent savers assert distinct,
  contiguous versions and that nothing is overwritten (saver count kept within
  the retry budget). Verified it fails against the pre-fix code.
- TestSaveReturnsWriteErrorWithoutRetry: a non-precondition write error is
  surfaced as-is and not retried.
- TestSaveExhaustsRetriesOnPersistentConflict: persistent 412s give up with
  errVersionConflict after maxSaveAttempts.
- TestBackoffDelayBounds: jittered backoff stays within bounds.
@wolo-lab
wolo-lab force-pushed the wolo/gcs-artifact-version-race branch from e9db6f3 to 99e5437 Compare July 18, 2026 22:29
@wolo-lab
wolo-lab marked this pull request as ready for review July 19, 2026 14:14
@wolo-lab
wolo-lab requested a review from baptmont July 19, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant