Skip to content

ci: build karnataka-gpu strict cells on hosted, test them on karnataka - #430

Open
hanthor wants to merge 1 commit into
mainfrom
fix/gpu-strict-split
Open

hanthor wants to merge 1 commit into
mainfrom
fix/gpu-strict-split

Conversation

@hanthor

@hanthor hanthor commented Nov 11, 2026

Copy link
Copy Markdown
Member

Problem

karnataka cannot upload a strict cell's ISO artifact. Pantheon run 42919271864 built the 7.53 GB ISO. upload ISO chunks then ran from 23:11 to 00:11 at about 0.54 MB/s and had sent 1.95 GB when it failed:

2026-11-11T00:11:06.2465199Z Uploaded bytes 1954545664
2026-11-11T00:11:06.3679411Z ##[error]Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

The upload's signed URL expires after about 60 minutes, so a longer step timeout cannot help. This PR reverts #425's 120-minute upload cap. karnataka's downlink is not the bottleneck: in the same run it pulled 2402042079 bytes; ... wall 449322 ms of registry layers, about 5.3 MB/s.

Change

When runner=karnataka-gpu and strict_roundtrip_test=true:

  • full-gpu-build (new, on ubuntu-24.04) runs the same browser build as full-matrix, then iso-artifact-split. It uploads the ISO chunks, the manifest and the browser evidence under the run-scoped names full-matrix already uses. Its job outputs are the browser-measured iso_sha256 and the artifact_id.
  • full-matrix (on tuna-iso-gpu) needs full-gpu-build and runs only if that job succeeded or was skipped. It skips engine, node, Playwright and the build itself. It downloads the ISO artifact and the browser evidence, and iso-artifact-verify re-checks every chunk and the whole ISO against the hosted build's measured hash. full-strict-inputs cross-checks the same bytes against the downloaded measurements. Then the guest, receipt and attestation steps run unchanged, in the same job that holds id-token.

Hosted runs and non-strict karnataka runs take exactly the path they took before. The karnataka download cap stays at 90 minutes, now with the measured downlink in its comment.

Verification

  • python3 -m pytest -q tests: 653 passed, 3 skipped, before the new test. tests/test_justfile.py: 71 passed, 2 skipped, including the new test_karnataka_strict_cell_builds_and_uploads_on_hosted_and_only_tests_on_gpu.
  • actionlint .github/workflows/ci.yml: no findings.
  • End-to-end: the niri karnataka run that follows this merge (linked below once dispatched).

karnataka cannot upload an ISO artifact: pantheon run 42919271864 pushed
1.95 of 7.53 GB in 60 min (~0.54 MB/s) before the upload's signed URL
expired ("Server failed to authenticate the request"). No step timeout
fixes that, so #425's 120-minute upload cap is reverted.

With runner=karnataka-gpu and strict_roundtrip_test, the new hosted job
full-gpu-build runs the browser build, splits and uploads the ISO chunks,
manifest and browser evidence. full-matrix on karnataka skips its own
build, downloads that run-scoped artifact (downlink ~5.3 MB/s in the same
run) and the browser evidence, and re-verifies every chunk and the whole
ISO against the hosted build's browser-measured hash before
full-strict-inputs and the guest, so the receipt binds the same bytes.
Hosted runs and non-strict karnataka runs are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant