ADR 0070: Support for zstd:chunked image compression

[Amanyd]

Draft ADR 0070 proposing dual-compression for container builds.

[konflux-ci-review]

Review

Findings

High

• [protected-path] ADR/AGENTS.md — This PR modifies AGENTS.md, which is a protected governance file. The PR has no linked issue to authorize the change. While the 1-line addition to the ADR index is a mechanical accompaniment to the new ADR, protected-path changes require human approval regardless of context.

Medium

• [edge-case] ADR/0070-dual-compression-for-container-builds.md:62 — The ADR asserts that buildah manifest add --all preserves gzip-first ordering when flattening per-arch indexes into the final multi-arch index. This is an unverified claim about buildah's internal behavior. The --all flag extracts all manifests from a referenced index, but whether the resulting order in the destination manifest list matches the source order depends on buildah's implementation, which is not documented to guarantee preservation of descriptor ordering. If buildah reorders entries during manifest add --all, the gzip-first invariant (which is critical for backward compatibility with older Docker clients) could be silently broken. The ADR does correctly note the ADR 0069 interaction regarding sort keys, but that mitigates a different concern (reproducibility sorting) rather than this one (flattening order preservation).
  Remediation: Add a note that the gzip-first ordering preservation by --all must be verified against buildah source or via integration test, or add a post-flattening validation step in build-image-index that checks the final manifest order ensures gzip entries precede zstd entries for each platform.

• [api-contract] ADR/0070-dual-compression-for-container-builds.md:73 — The dedup filter for six release tasks is described as grouping by (architecture, os) and keeping the non-zstd entry, but the ADR does not specify the mechanism for identifying which manifest is zstd vs gzip after build-image-index flattening. The io.github.containers.compression.zstd annotation is set on the zstd manifest descriptor within the per-arch index (line 60), but the ADR does not verify whether descriptor-level annotations survive buildah manifest add --all flattening into the final multi-arch index, nor whether get-image-architectures surfaces these annotations in its output. Without a specified contract, implementers of the six release task fixes have no reliable mechanism to code against.
  Remediation: Specify that the build-image-index task must preserve the io.github.containers.compression.zstd annotation on manifest descriptors during flattening, and confirm that get-image-architectures includes descriptor annotations in its output so the dedup filter can key on them.

Low

• [internal-consistency] ADR/0070-dual-compression-for-container-builds.md:86 — The Rollout section describes a Conforma validation problem: subject_build_task_matches checks each subject against IMAGE_URL@IMAGE_DIGEST, and since IMAGE_DIGEST is the per-arch index, only the index subject matches while gzip and zstd child subjects trigger violations. The ADR frames this as blocking the default flip, but the same violations would affect users who opt in to dual-compression mode before the Conforma fix lands. The ADR does not clarify whether opt-in users will experience Conforma failures or whether this is only relevant for the default change.

• [logic-error] ADR/0070-dual-compression-for-container-builds.md:80 — The push-rpm-data-to-pyxis entry states it "works since each variant has its own SBOM with the correct digest as the main component." This correctness depends on the SBOM generation changes described in lines 65-71 (Mobster running separately for each variant) being implemented first. The ADR presents this task as not needing a fix, but that is only true after the SBOM changes are in place. This ordering dependency is implicit rather than explicit.

• [consequences-completeness] ADR/0070-dual-compression-for-container-builds.md:93 — The Consequences section covers four concrete negative consequences but does not explicitly address operational complexity or long-term maintenance burden of dual-compression logic across multiple tasks (buildah, oci-copy, modelcar).

Previous run

Review

Findings

High

• [protected-path] ADR/AGENTS.md — This PR modifies AGENTS.md, which is a protected governance file. The PR has no linked issue to authorize the change. While the 1-line addition to the ADR index is a mechanical accompaniment to the new ADR, protected-path changes require human approval regardless of context.

Medium

• [edge-case] ADR/0070-dual-compression-for-container-builds.md:62 — The ADR asserts as fact that buildah manifest add --all preserves gzip-first ordering within each platform when flattening per-arch indexes into the final multi-arch index. This claim about buildah's iteration order is not verified against the buildah source code or documentation. The --all flag extracts all manifests from a referenced index, but the order in which they appear in the resulting manifest list depends on buildah's internal implementation. If buildah sorts or reorders entries during manifest add --all, the gzip-first invariant could be broken silently. The ADR does address the ADR 0069 interaction ("must either include compression type in the sort key or use a stable sort"), which resolves the cross-ADR concern from the prior review.
  Remediation: Add a note that the gzip-first ordering assumption for --all should be verified against buildah source or a test, or add a validation step in build-image-index that checks the final manifest order.

• [api-contract] ADR/0070-dual-compression-for-container-builds.md:73 — The dedup filter for six release tasks is described as grouping by (architecture, os) and keeping the non-zstd entry, but the ADR does not specify the mechanism for identifying which manifest is zstd vs gzip after flattening by build-image-index. The io.github.containers.compression.zstd annotation is set on the zstd manifest descriptor within the per-arch index (line 60), but the ADR does not verify whether descriptor-level annotations survive buildah manifest add --all flattening into the final multi-arch index, nor whether get-image-architectures surfaces these annotations in its output. Without this, implementers of the six release task fixes have no specified contract to code against.
  Remediation: Specify that the build-image-index task must preserve the io.github.containers.compression.zstd annotation on manifest descriptors during flattening, and confirm that get-image-architectures includes descriptor annotations in its output so the dedup filter can key on them.

Low

• [internal-consistency] ADR/0070-dual-compression-for-container-builds.md:86 — The Rollout section describes a Conforma validation problem: subject_build_task_matches checks each subject against IMAGE_URL@IMAGE_DIGEST, and since IMAGE_DIGEST is the per-arch index, only the index subject matches while gzip and zstd child subjects trigger violations. The ADR does not specify whether Conforma changes are a prerequisite for enabling the opt-in mode or only for flipping the default.

• [missing-authorization] ADR/0070-dual-compression-for-container-builds.md — This ADR has no linked issue in the architecture repository. The ADR references external implementation issues (build-definitions#1264, #3188, release-service-catalog#634) as demand sources, which provides implicit authorization context.

• [consequences-completeness] ADR/0070-dual-compression-for-container-builds.md:93 — The Consequences section covers four concrete negative consequences but does not explicitly address operational complexity or long-term maintenance burden of dual-compression logic across multiple tasks (buildah, oci-copy, modelcar).

• [logic-error] ADR/0070-dual-compression-for-container-builds.md:80 — The push-rpm-data-to-pyxis entry states it "works since each variant has its own SBOM with the correct digest as the main component." This correctness depends on the SBOM generation changes described in lines 65-71 (Mobster running separately for each variant) being implemented first. The ADR presents this task as not needing a fix, but that is only true after the SBOM changes are in place. This ordering dependency is implicit rather than explicit.

• [api-contract] ADR/0070-dual-compression-for-container-builds.md:93 — The Consequences section states "Docker 23.0+ clients gain tolerance for the zstd layer." Docker Engine 23.0 introduced the containerd image store as an experimental opt-in feature. Native zstd decompression support in Docker depends on the containerd image store being enabled, which was not the default until Docker 26.0. The phrase "gain tolerance" is ambiguous.

• [consequences-structure] ADR/0070-dual-compression-for-container-builds.md:93 — The Consequences section uses "Pros:" and "Cons:" labels rather than markdown headings. Recent ADRs (0067, 0054) use ### Positive and ### Negative section headings.

Previous run (2)

Review

Findings

Medium

• [edge-case] ADR/0070-dual-compression-for-container-builds.md:62 — The ADR states the build-image-index task's --all flag on buildah manifest add "is expected to extract child manifests from per-arch indexes and flatten them into the final multi-arch index, preserving the gzip-first invariant or explicit flattening logic will be added in the task if needed." The gzip-first ordering is critical for backward compatibility with older Docker clients (line 30), yet the ADR treats it as an unverified assumption about buildah manifest add --all iteration order. The hedge "or explicit flattening logic will be added if needed" defers this to implementation time without specifying a verification step. Furthermore, ADR 0069 (Reproducible Container Builds) proposes sorting the manifest list in build-image-index by platform for reproducibility. That sort does not account for compression-variant ordering. If both ADRs are implemented, the reproducibility sort could reorder gzip and zstd entries for the same platform, breaking the gzip-first invariant.
  Remediation: Specify that the build-image-index task must verify gzip-first ordering in its output index after flattening, or explicitly re-order entries, rather than relying on unspecified buildah behavior. Coordinate with ADR 0069's proposed platform sort so the sort key includes compression type (e.g., sort by (os, arch, variant, compression) with gzip before zstd). Add a concrete acceptance criterion (e.g., an integration test) that validates ordering.

• [api-contract] ADR/0070-dual-compression-for-container-builds.md:73 — The dedup filter for six release tasks "groups by (architecture, os) and keeps the non-zstd entry." The ADR does not specify how the filter identifies which manifest is zstd vs gzip. The io.github.containers.compression.zstd annotation is described as being set on the zstd manifest descriptor within the per-arch index (line 60), but get-image-architectures is called on the final multi-arch index after flattening by build-image-index. The ADR does not verify that descriptor-level annotations survive the flattening process or that get-image-architectures surfaces them in its output.
  Remediation: Specify that the build-image-index task must preserve the io.github.containers.compression.zstd annotation on manifest descriptors during flattening, and confirm that get-image-architectures includes descriptor annotations in its output so the dedup filter can key on them.

Low

• [logic-error] ADR/0070-dual-compression-for-container-builds.md:80 — The push-rpm-data-to-pyxis entry states it "works since each variant has its own SBOM with the correct digest as the main component." This depends on the SBOM generation changes described earlier (lines 65-71) being implemented first — specifically, Mobster running separately for each variant. The ADR presents this task as not needing a fix, but that is only true after the SBOM changes are in place. This ordering dependency is implicit rather than explicit.

• [scope-documentation] ADR/0070-dual-compression-for-container-builds.md — The relationship between this ADR and ADR 0069 (Reproducible Container Builds) is unclear regarding how dual-compression impacts reproducibility guarantees. Consider adding a cross-reference to ADR 0069.

• [internal-consistency] ADR/0070-dual-compression-for-container-builds.md:57 — IMAGES is used in two different semantic contexts without disambiguation. In the buildah task context (line 38), IMAGES is a task result containing individual child manifest digests. In the build-image-index context (line 62), IMAGES refers to an input parameter array of per-arch indexes.

• [api-contract] ADR/0070-dual-compression-for-container-builds.md:93 — The Consequences section states "Docker 23.0+ clients gain tolerance for the zstd layer." Docker Engine 23.0 introduced the containerd image store as an experimental opt-in feature. Zstd support depends on the containerd image store, which was not the default until Docker 26.0.

• [consequences-structure] ADR/0070-dual-compression-for-container-builds.md:93 — The Consequences section uses "Pros:" and "Cons:" labels rather than markdown headings. Most recent ADRs use ### Positive and ### Negative headings or unsegmented bullet lists.

---

Labels: New ADR proposing a feature capability (dual-compression for container builds).

Previous run (3)

Review

Findings

Medium

• [edge-case] ADR/0069-dual-compression-for-container-builds.md:62 — The ADR states the build-image-index task's --all flag on buildah manifest add "is expected to extract child manifests from per-arch indexes and flatten them into the final multi-arch index, preserving the gzip-first invariant or explicit flattening logic will be added in the task if needed." The gzip-first ordering is critical for backward compatibility with older Docker clients (line 30), yet the ADR treats it as an unverified assumption about buildah manifest add --all iteration order. The hedge "or explicit flattening logic will be added if needed" defers this to implementation time without specifying a verification step.
  Remediation: Specify that the build-image-index task must verify gzip-first ordering in its output index after flattening, or explicitly re-order entries, rather than relying on unspecified buildah behavior. Add a concrete acceptance criterion (e.g., an integration test) that validates ordering.

• [api-contract] ADR/0069-dual-compression-for-container-builds.md:73 — The dedup filter for six release tasks "groups by (architecture, os) and keeps the non-zstd entry." The ADR does not specify how the filter identifies which manifest is zstd vs gzip. The io.github.containers.compression.zstd annotation is described as being set on the zstd manifest descriptor within the per-arch index (line 60), but get-image-architectures is called on the final multi-arch index after flattening by build-image-index. The ADR does not verify that descriptor-level annotations survive the flattening process or that get-image-architectures surfaces them.
  Remediation: Specify that the build-image-index task must preserve the io.github.containers.compression.zstd annotation on manifest descriptors during flattening, and confirm that get-image-architectures includes descriptor annotations in its output.

Low

• [internal-consistency] ADR/0069-dual-compression-for-container-builds.md:57 — IMAGES is used in two different semantic contexts without disambiguation. On line 40, IMAGES refers to the buildah task result listing individual gzip and zstd child manifest digests. On line 62, IMAGES refers to the build-image-index task's input parameter, where each entry is now a per-arch index rather than a single manifest.

• [api-contract] ADR/0069-dual-compression-for-container-builds.md:93 — The Consequences section states "Docker 23.0+ clients gain tolerance for the zstd layer." Docker Engine 23.0 introduced the containerd image store as an experimental opt-in feature; zstd support depends on using the containerd image store, which is not the default storage driver. The claim overstates Docker 23.0+ compatibility.

• [consequences-structure] ADR/0069-dual-compression-for-container-builds.md:93 — The Consequences section uses "Pros:" and "Cons:" labels rather than markdown headings. Most recent ADRs in the repository use ### Positive and ### Negative headings.

• [missing-adr-reference] architecture/core/build-service.md:16 — The build-service.md related_adrs frontmatter lists ADRs 0047, 0029, and 0046, but does not reference ADR 0069. Since ADR 0069 has applies_to: build-service, consider adding it once the ADR status advances beyond Proposed.

Info

• [line-count-accuracy] ADR/AGENTS.md:60 — The AGENTS.md entry shows 105L for ADR 0069, but the file has 111 lines.

• [edge-case] ADR/0069-dual-compression-for-container-builds.md:96 — The stated threshold of "4-5 architectures" for hitting the Tekton 4KB result limit appears to be an estimate without supporting calculation.

• [cross-references] ADR/0069-dual-compression-for-container-builds.md — The ADR does not include a "Relates to" section linking to related internal ADRs (e.g., ADR 0063 Generalized Build Pipelines, ADR 0047 Caching for Container Build Layers).

Previous run (4)

Review

Findings

Medium

• [edge-case] ADR/0069-dual-compression-for-container-builds.md:62 — The ADR states the build-image-index task's --all flag "is expected to extract child manifests from per-arch indexes and flatten them into the final multi-arch index, preserving the gzip-first invariant or explicit flattening logic will be added in the task if needed." The gzip-first ordering is the sole mechanism guaranteeing backward compatibility with older Docker clients. Relying on an unspecified expectation about buildah manifest add --all iteration order is fragile — if a future buildah version changes iteration order, the gzip-first invariant silently breaks.
  Remediation: Remove the hedging language and commit to an explicit post-assembly ordering/validation step in build-image-index that ensures gzip manifests precede zstd manifests for each platform tuple.

• [internal-consistency] ADR/0069-dual-compression-for-container-builds.md:85 — The Rollout section references IMAGE_REF as the source Chains uses to determine the attestation subject ("it generates provenance with the gzip child as the attestation subject (from IMAGE_REF)"). However, IMAGE_REF is never defined in the Implementation details section — only IMAGE_DIGEST (the per-arch index) and IMAGES (listing both child digests) are specified as task results. It is unclear whether IMAGE_REF is a separate task result, a Chains-internal concept, or an error.
  Remediation: Define IMAGE_REF in the Implementation details section, or clarify which existing task result Chains uses to derive the attestation subject and why it resolves to the gzip child rather than the index digest.

• [api-contract] ADR/0069-dual-compression-for-container-builds.md:73 — The dedup filter for six release tasks groups manifests by (architecture, os) and keeps the non-zstd entry, relying on the io.github.containers.compression.zstd annotation on manifest descriptors. The ADR does not verify that get-image-architectures (used by all six tasks) actually surfaces descriptor-level annotations from the image index. If it only returns platform fields without annotations, the jq dedup filter has no field to distinguish gzip from zstd manifests.
  Remediation: Verify that get-image-architectures returns manifest descriptor annotations (or document that it must be updated to do so), and specify the exact jq expression or annotation field used for distinguishing compression variants.

Low

• [api-contract] ADR/0069-dual-compression-for-container-builds.md:93 — The Consequences section states "Docker 23.0+ clients gain tolerance for the zstd layer." Docker Engine 23.0 introduced the containerd image store as an experimental opt-in feature; standard Docker installations still depend on gzip-first ordering. The second clause correctly describes the actual behavior, so the risk of misleading implementers is low.

• [internal-consistency] ADR/0069-dual-compression-for-container-builds.md:57 — IMAGES is used in two different semantic contexts without disambiguation: as the buildah task result containing individual child manifest digests (line 37), and as the build-image-index input where each entry is a per-arch index (line 62). These are different data shapes for the same name.

• [consequences-structure] ADR/0069-dual-compression-for-container-builds.md:93 — The Consequences section uses bold "Pros:" and "Cons:" labels rather than markdown headings. ADR 0067 uses ### Positive and ### Negative headings in its Consequences section; aligning with that pattern would improve structural consistency.

• [missing-authorization] ADR/0069-dual-compression-for-container-builds.md — This PR has no linked issue. The ADR references three upstream issues (build-definitions#1264, build-definitions#3188, release-service-catalog#634) that document the problem statement, which provides implicit authorization context. Consider linking one of these issues to the PR for traceability.

Info

• [edge-case] ADR/0069-dual-compression-for-container-builds.md:96 — The claim that "builds exceeding 4-5 architectures may hit this limit" (Tekton 4KB result size limit) appears conservative. The IMAGES result size depends on more than just digest count, but 12 digests at ~80 bytes each is ~960 bytes, well under 4KB.

• [cross-references] ADR/0069-dual-compression-for-container-builds.md — The ADR does not include a "Relates to" section linking to related internal ADRs (e.g., ADR 0036 Trusted Artifacts, ADR 0027 Container Image Management Practice), which is a common pattern in this repository.

Previous run (5)

Review

Findings

Critical

• [identifier-conflict] ADR/0068-dual-compression-for-container-builds.md:1 — ADR number 0068 is already taken. The file ADR/0068-konflux-ci-dev-api-group.md exists on main (merged via PR Add ADR: Standardize on konflux-ci.dev as Official API Group #362, commit 1cafe77, "Standardize on konflux-ci.dev as Official API Group"), and ADR/AGENTS.md on main already contains the entry 0068 Pr 145L Standardize on konflux-ci.dev as Official API Group. This PR creates a second ADR 0068 file with a different name and topic, and the AGENTS.md change adds a duplicate 0068 line. Merging would produce two ADRs with the same number, breaking the uniqueness invariant and failing the lint-adr-numbers CI check.
  Remediation: Renumber this ADR to the next available number (0069 or higher). Update the filename to ADR/0069-dual-compression-for-container-builds.md, the title and heading from "68" to "69", and regenerate the AGENTS.md entry accordingly.

High

• [agents-md-entry] ADR/AGENTS.md:60 — The AGENTS.md update adds 0068 Pr 105L Support for zstd:chunked image compression but line 59 already contains 0068 Pr 145L Standardize on konflux-ci.dev as Official API Group. This creates duplicate ADR number entries in the index. See also: [identifier-conflict] finding above — this is a direct consequence of the ADR number collision.
  Remediation: After renumbering the ADR file, update the AGENTS.md entry to use the new number.

Medium

• [edge-case] ADR/0068-dual-compression-for-container-builds.md:62 — The ADR states the build-image-index task's --all flag "is expected to extract child manifests from per-arch indexes and flatten them into the final multi-arch index, preserving the gzip-first invariant or explicit flattening logic will be added in the task if needed." The gzip-first ordering is the sole mechanism guaranteeing backward compatibility with older Docker clients. Relying on an unspecified expectation about buildah manifest add --all iteration order is fragile, and the hedge ("or explicit flattening logic will be added if needed") leaves the design ambiguous.
  Remediation: Commit to a post-assembly validation or explicit sorting step in build-image-index that verifies gzip manifests appear before zstd manifests for each platform tuple, rather than leaving this as a conditional future action.

• [api-contract] ADR/0068-dual-compression-for-container-builds.md:73 — The dedup filter for six release tasks "groups by (architecture, os) and keeps the non-zstd entry," relying on the io.github.containers.compression.zstd annotation on manifest descriptors. The ADR does not verify that get-image-architectures (consumed by all six listed release tasks) actually surfaces descriptor-level annotations from the image index. If it only returns platform fields without annotations, the jq dedup filter has no field to distinguish gzip from zstd manifests.
  Remediation: Verify that get-image-architectures returns manifest descriptor annotations (or document that it must be updated to do so), and specify the exact jq expression or annotation field used for distinguishing compression variants.

Low

• [missing-authorization] ADR/0068-dual-compression-for-container-builds.md — This PR has no linked issue. The ADR references three upstream issues (build-definitions#1264, build-definitions#3188, release-service-catalog#634) that document the problem statement, which provides implicit authorization context. Consider linking one of these issues to the PR for traceability.

• [api-contract] ADR/0068-dual-compression-for-container-builds.md:93 — The Consequences section states "Docker 23.0+ clients gain tolerance for the zstd layer." Docker Engine 23.0 introduced the containerd image store as an experimental opt-in feature; standard Docker installations still depend on gzip-first ordering. The second clause ("but will continue to receive the gzip variant via first-match selection") correctly describes the actual behavior, so the risk of misleading implementers is low.

• [internal-consistency] ADR/0068-dual-compression-for-container-builds.md:57 — IMAGES is used in two different semantic contexts without disambiguation: as the buildah task result containing individual child manifest digests (line 39), and as the build-image-index input where each entry is a per-arch index (line 62). These are different data shapes for the same name, which could cause implementation confusion.

• [stale-doc] architecture/core/build-service.md:16 — The build-service frontmatter's related_adrs list does not include the new ADR number. Per CONTRIBUTING.md, service frontmatter should be updated when a new ADR references applies_to: build-service, though this can also be done by maintainers post-merge.

Info

• [edge-case] ADR/0068-dual-compression-for-container-builds.md:96 — The claim that "builds exceeding 4-5 architectures may hit this limit" (Tekton 4KB result size limit) appears conservative. The IMAGES result size depends on more than just digest count, but 12 digests at ~80 bytes each is ~960 bytes, well under 4KB.

• [cross-references] ADR/0068-dual-compression-for-container-builds.md — The ADR does not include a "Relates to" section linking to related internal ADRs (e.g., ADR 0036 Trusted Artifacts, ADR 0027 Container Image Management Practice), which is a common pattern in this repository.

Previous run (6)

Review

Findings

Medium

• [edge-case] ADR/0068-dual-compression-for-container-builds.md:66 — The ADR claims "Buildah appends manifests in the order they appear in the source index, which preserves the gzip-first invariant in the flat index." This ordering guarantee is the linchpin of backward compatibility with older Docker clients, but buildah's manifest add --all does not formally guarantee insertion order in its documentation. If a future buildah version changes iteration order, the gzip-first invariant silently breaks.
  Remediation: Add a post-assembly validation step in build-image-index that inspects the final index and verifies gzip manifests appear before zstd manifests for each platform, or explicitly sort manifests after assembly rather than relying on implicit ordering.

• [api-contract] ADR/0068-dual-compression-for-container-builds.md:73 — The dedup filter for release tasks "groups by (architecture, os) and keeps the non-zstd entry," relying on the io.github.containers.compression.zstd annotation on manifest descriptors. However, the ADR does not verify that get-image-architectures (used by all five listed release tasks) actually exposes descriptor-level annotations from the image index. If it only returns platform fields without annotations, the jq dedup filter has no way to distinguish gzip from zstd manifests.
  Remediation: Verify that get-image-architectures returns manifest descriptor annotations (or can be updated to do so), and document the exact jq filter expression or field used for distinguishing compression variants.

• [api-contract] ADR/0068-dual-compression-for-container-builds.md:93 — The Consequences section states "Docker 23.0+ clients gain tolerance for the zstd layer." This is misleading — Docker Engine 23.0 did not add native zstd support to the default storage driver. zstd support in Docker requires enabling the containerd image store (an opt-in feature introduced experimentally in 23.0, stabilized later). Standard Docker installations post-23.0 still rely on the gzip-first ordering for correct operation.
  Remediation: Clarify that Docker's zstd support requires the containerd image store feature to be enabled, and that standard Docker installations still depend on gzip-first ordering.

Low

• [internal-consistency] ADR/0068-dual-compression-for-container-builds.md:57 — IMAGES is used in two different semantic contexts without disambiguation: as the buildah task result containing child manifest digests (line 39), and as the build-image-index input where each entry is a per-arch index (line 62). This naming collision could cause implementation confusion.

• [edge-case] ADR/0068-dual-compression-for-container-builds.md:96 — The 4-5 architecture threshold for hitting the Tekton 4KB result limit appears too conservative. A 6-arch dual build produces ~12 digests at ~80 bytes each (~960 bytes), well under 4KB. If the estimate is based on measurements, those should be cited.

• [stale-doc] architecture/core/build-service.md — The build-service frontmatter's related_adrs list does not include 0068. Per CONTRIBUTING.md, service frontmatter should be updated when a new ADR references applies_to: build-service. The lint-frontmatter check may flag this.

Previous run (7)

Review

Findings

Low

• [internal-consistency] ADR/0068-dual-compression-for-container-builds.md:95 — The Consequences section states "Docker 25+" will benefit from zstd:chunked, but the Context section identifies Docker pre-23.0 as the compatibility boundary, implying 23.0+ supports zstd. The gap between 23.0 and 25.0 is unaddressed. Additionally, Docker does not use the io.github.containers.compression.zstd annotation for variant selection (that is Podman/CRI-O behavior), so Docker users will not "automatically benefit" from the zstd variant — they will continue receiving gzip via first-match. Consider attributing the automatic benefit to Podman and CRI-O specifically, and clarifying what Docker 23.0+ gains (tolerance, not preference).
• [edge-case] ADR/0068-dual-compression-for-container-builds.md:38 — Persists from prior review. The ADR acknowledges the Tekton 4KB result limit and references TEP-0164 as a future fix, but does not specify the maximum architecture count supportable in dual mode before hitting the limit, nor what happens when the limit is exceeded (silent truncation? task failure?). A concrete threshold or failure-mode note would help implementors.
• [api-contract] ADR/0068-dual-compression-for-container-builds.md:62 — The ADR claims that buildah manifest add --all flattens child manifests from per-arch indexes and that ordering within each per-arch index (gzip first) is preserved in the flat index. Ordering preservation during flattening is an implementation detail of buildah that is not guaranteed by any specification. If buildah reorders manifests, the gzip-first invariant that older Docker clients depend on would break in the final multi-arch index. This claim should be verified against buildah's actual behavior or the ordering should be explicitly enforced after flattening.
• [missing-related-adr-reference] architecture/core/build-service.md:16 — The related_adrs frontmatter does not include this ADR. Since ADR 0068 has applies_to: build-service and introduces a significant architectural change (dual compression), consider adding it once the ADR status advances beyond Proposed.

Info

• [metadata-stale] ADR/0068-dual-compression-for-container-builds.md:1 — The PR title and body still reference "ADR 0067" but the file is correctly numbered 0068 (resolving the prior review's critical number-conflict finding). Consider updating the PR title to "ADR 0068: Support for zstd:chunked image compression" for consistency.

Previous run (8)

Review

Findings

Critical

• [identifier-conflict] ADR/0067-dual-compression-for-container-builds.md:1 — ADR number 0067 is already taken. ADR/0067-nudging-relationship-singleton-crd.md ("Nudging Relationship Storage via Singleton CRD", 421 lines) was merged to main while this PR was in flight. The AGENTS.md on the PR branch now contains two entries for 0067. Merging this PR would place two ADR files with the same number in the repository, breaking the numbering convention and any tooling that uses ADR numbers as unique identifiers.
  Remediation: Renumber this ADR to 0068 (the next available number). Rename the file to ADR/0068-dual-compression-for-container-builds.md, update the title and heading from "67" to "68", and update the AGENTS.md entry accordingly.

Low

• [style] ADR/0067-dual-compression-for-container-builds.md:56 — Comma splice: "SBOM generation requires an update, the SBOM must attach to the gzip manifest digest directly" — use a colon or semicolon to separate independent clauses.
• [style] ADR/0067-dual-compression-for-container-builds.md:62 — Comma splice: "Individual compression variants do not receive separate SBOMs, they share identical content" — use a semicolon, colon, or conjunction ("as they share...").
• [api-contract] ADR/0067-dual-compression-for-container-builds.md:25 — The ADR states "Per the OCI image index spec, clients select the first matching platform entry." The OCI image-spec does not mandate first-match semantics; it says manifest selection is implementation-defined. Docker happens to use first-match behavior, but this is Docker's implementation choice, not an OCI spec guarantee. Consider attributing this to Docker specifically.
• [internal-consistency] ADR/0067-dual-compression-for-container-builds.md:93 — The "Push two individual manifests" alternative cites "increased Tekton result size (2N digests instead of N)" as a drawback, but the chosen approach also puts 2N child manifest digests in the IMAGES result. The Tekton result size is comparable in both approaches; the chosen approach's advantage is avoiding orphaned manifests and GC issues, not result size.
• [edge-case] ADR/0067-dual-compression-for-container-builds.md:38 — The ADR acknowledges the Tekton 4KB result limit but does not specify the maximum architecture count supportable in dual mode. A concrete threshold (or what happens when exceeded) would help implementors.
• [incomplete-doc] ADR/0067-dual-compression-for-container-builds.md:70 — The ADR specifies that the SBOM must attach to the gzip manifest digest, and separately notes that the zstd manifest carries the io.github.containers.compression.zstd annotation. It would be clearer to explicitly state that the SBOM step identifies the gzip manifest as the child descriptor lacking that annotation.
• [missing-related-adr-reference] architecture/core/build-service.md:16 — The related_adrs frontmatter does not include this ADR. Since it has applies_to: build-service and is architecturally significant (comparable to ADR 0047 on caching), consider adding it once the number conflict is resolved and the ADR status advances.

Previous run (9)

Review

Findings

Low

• [style] ADR/0067-dual-compression-for-container-builds.md:56 — Comma splice: "SBOM generation requires an update, the SBOM must attach to the gzip manifest digest directly" — use a colon or semicolon to separate independent clauses.
• [style] ADR/0067-dual-compression-for-container-builds.md:62 — Comma splice: "Individual compression variants do not receive separate SBOMs, they share identical content" — use a colon or semicolon to separate independent clauses.

Info

• [correctness] ADR/AGENTS.md:1 — Line count and ADR count correctly updated (64→65 ADRs, 10430→10529 total lines = +99 lines matching the new file length).

Previous run (10)

Review

Findings

Info

• [correctness] ADR/AGENTS.md:68 — Line count for ADR 0067 is listed as 96L, but the file contains 99 lines (per the diff). The total line count (10526) is also based on the 96-line figure. This is a minor metadata inaccuracy that affects context-cost estimation for AI agents.
  Remediation: Update the entry to 0067 Pr 99L Support for zstd:chunked image compression and adjust the total from 10526 to 10529.

Previous run (11)

Review

No findings.

Previous run (12)

Review

Findings

Info

• [docs-currency] architecture/core/build-service.md:16 — The related_adrs frontmatter list does not include "0067". Since ADR 0067 has applies_to: build-service, consider adding it to related_adrs once the ADR status moves beyond "Proposed."

• [style] ADR/0067-dual-compression-for-container-builds.md:1 — Minor naming inconsistency: the filename uses "dual-compression" (describing the solution) while the title uses "Support for zstd:chunked image compression" (describing the problem). Both are accurate, but aligning them would improve discoverability.

No blocking findings. The ADR is well-structured, follows existing conventions, and thoroughly covers the decision rationale, implementation details, alternatives considered, and consequences.

Previous run (13)

Review

Findings

Low

• [correctness] ADR/AGENTS.md:68 — Line count for ADR 0067 is listed as 79L but the file has 96 lines. The total line count (10509) is also incorrect — should be ~10526.
  Remediation: Update the entry to 0067 Pr 96L Support for zstd:chunked image compression and recalculate the total.

• [correctness] ADR/0067-dual-compression-for-container-builds.md:55 — Broken sentence: "The task's existing --all flag on buildah manifest add extracts child manifests from per-arch indexes will flatten them into the final multi-arch index." Missing conjunction makes the sentence ungrammatical.
  Remediation: Change to "extracts child manifests from per-arch indexes and flattens them into the final multi-arch index."

• [style] ADR/0067-dual-compression-for-container-builds.md:71 — Section heading ## Alternatives considered uses lowercase 'c'. Three of four existing ADRs with this section use ## Alternatives Considered (title case).
  Remediation: Change to ## Alternatives Considered.

Info

• [style] ADR/0067-dual-compression-for-container-builds.md:59 — Comma splice: "Individual compression variants within the index do not receive separate SBOMs, they share identical content, only differing in layer compression." Two independent clauses joined by a comma.
  Remediation: Use a semicolon or split into two sentences.

Previous run (14)

Review: #357

Head SHA: e9b9472
Timestamp: 2026-05-15T00:00:00Z
Outcome: approve

Summary

This PR adds ADR 0067 proposing dual-compression (gzip + zstd:chunked) for container image builds and updates the ADR index in AGENTS.md. The ADR is well-structured, technically coherent, and follows the repository's conventions. The AGENTS.md line/count updates are arithmetically correct (64→65 ADRs, 10430→10509 lines). No security, injection, or correctness concerns were found. Two minor style observations are noted below but do not warrant blocking.

Findings

Medium / Low / Info

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md:59 — The heading ## Alternatives considered uses lowercase "considered", while existing ADRs (e.g., 0066) use ## Alternatives Considered with a capital C. Consider capitalizing for consistency.

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md:52 — The sentence ending "...the filter only prevents duplicate metadata in Pyxis, advisory content, and related systems" is missing a trailing period.

Footer

Outcome: approve
This review applies to SHA e9b9472d59277eebfb02041555957c1160bc3cf7. Any push to the PR head clears this review and requires a new evaluation.

Previous run (15)

Review: #357

Head SHA: 55018a9
Timestamp: 2026-05-14T00:00:00Z
Outcome: approve

Summary

This PR adds ADR 0067 proposing dual-compression (gzip + zstd:chunked) for Konflux container builds, along with the corresponding AGENTS.md index update. The ADR is well-structured, technically detailed, and follows the established ADR conventions. The AGENTS.md arithmetic is correct (line count 10430→10509 matches the 79-line ADR; entry count 64→65). No correctness, security, or injection concerns were found. One minor style note below.

Findings

Info

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md:66 — Section heading uses ## Alternatives considered (lowercase "c") while the majority of ADRs in this repo use ## Alternatives Considered (title case). At least one other ADR (0064) also uses lowercase, so this is not blocking — just a consistency note.

Footer

Outcome: approve
This review applies to SHA 55018a947cdda61729e850563bb03c5dfbc76616. Any push to the PR head clears this review and requires a new evaluation.

Previous run (16)

Review: #357

Head SHA: 0ac8063
Timestamp: 2026-05-14T00:00:00Z
Outcome: approve

Summary

This PR adds ADR 0067 proposing dual-compression (gzip + zstd:chunked) for container builds and updates the ADR index in AGENTS.md. The ADR is well-structured, technically sound, and follows the repository's conventions. The AGENTS.md line/count updates are arithmetically correct. No security, correctness, or intent-alignment concerns were found.

Findings

Critical

None.

High

None.

Medium

None.

Low

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md:54 — Missing period at end of sentence ("...advisory content, and related systems").
  Remediation: Add a period at the end of the sentence.

Info

• [intent alignment] ADR/0067-dual-compression-for-container-builds.md:5 — applies_to lists only build-service, but the ADR also describes changes needed in release-service-catalog tasks. Consider adding release-service to applies_to for discoverability.

Footer

Outcome: approve
This review applies to SHA 0ac80636be2111b6853f595809d03419eaaa9f35. Any push to the PR head clears this review and requires a new evaluation.

Previous run (17)

Review: #357

Head SHA: 7ac20ef
Timestamp: 2026-05-14T00:00:00Z
Outcome: approve

Summary

This PR adds ADR 0067, a well-structured proposal for dual-compression (gzip + zstd:chunked) in container builds. The ADR follows the repository's established template and conventions, provides clear context with references to upstream specs and issues, documents implementation details with rollout gating, and considers two alternatives with sound rationale for rejecting them. No correctness, security, or injection concerns apply — this is a documentation-only change. The content is technically coherent and appropriately scoped for an architecture decision record.

Findings

Info

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md — The filename slug (dual-compression-for-container-builds) diverges slightly from the title wording (Support for zstd:chunked image compression). Both are descriptive and clear; no action required, just noting the difference.

Footer

Outcome: approve
This review applies to SHA 7ac20ef41f528793febde9e39b6aeb34fdda79b6. Any push to the PR head clears this review and requires a new evaluation.

Previous run (18)

Review: #357

Head SHA: e4778ed
Timestamp: 2026-05-14T00:00:00Z
Outcome: comment-only

Summary

This PR adds a new ADR (0067) proposing dual-compression (gzip + zstd:chunked) for container builds. The ADR is well-structured, technically detailed, and follows existing conventions. No critical or high findings. A few minor style and consistency observations are noted below.

Findings

Critical

None.

High

None.

Medium

None.

Low

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md — The filename slug (dual-compression-for-container-builds) diverges from the frontmatter title (Support for zstd:chunked image compression). Most existing ADRs keep the filename slug closely aligned with the title (e.g., 0066-ecosystem-native-distribution.md ↔ "Ecosystem-Native Distribution for Non-OCI Artifacts"). Consider aligning the filename to the title or vice versa for discoverability.

• [style/conventions] ADR/0067-dual-compression-for-container-builds.md:57 — Section heading ## Alternatives considered uses lowercase "considered". The majority of existing ADRs (4 of 6 that have this section) use title case: ## Alternatives Considered.

Info

• [correctness] ADR/0067-dual-compression-for-container-builds.md:45 — The statement "the image index contains 2N manifests instead of N" and the claim that "2N digests fit within Tekton's 4096-byte result limit" for arch counts ≤4 is plausible but worth verifying during implementation. With 4 architectures, 8 digests (each ~71 chars as sha256 hex) plus separators is ~580 bytes, well within 4096. The claim holds.

• [intent alignment] The PR description says "Draft ADR 0067" but the frontmatter status is Proposed, which is the correct ADR lifecycle status. No issue — the PR description's use of "draft" is informal and doesn't conflict with the ADR status.

• [injection defense] PR body and file content inspected — no prompt injection patterns, no non-rendering Unicode, no suspicious directives found.

Footer

Outcome: comment-only
This review applies to SHA e4778ed28b41c4ca8e00f7e6221b1d8cf7e3e891. Any push to the PR head clears this review and requires a new evaluation.

Previous run (19)

Review: #357

Head SHA: 7016222
Timestamp: 2026-05-14T00:00:00Z
Outcome: approve

Summary

This PR adds a new ADR (0067) proposing dual-compression (gzip + zstd:chunked) for container image builds. The document is well-structured, follows the established ADR template and conventions, correctly identifies backward-compatibility risks with older Docker versions, and thoroughly covers the implementation plan including release pipeline impacts. No blocking findings were identified across any review dimension.

Findings

Info

• [Style/conventions] ADR/0067-dual-compression-for-container-builds.md — The frontmatter title ("Support for zstd:chunked image compression") emphasizes the feature goal while the filename and PR title emphasize the mechanism ("dual-compression"). Both are accurate; noting the difference for awareness only.

• [Correctness] ADR/0067-dual-compression-for-container-builds.md — The ADR references four release tasks that need dedup fixes (create-pyxis-image, populate-release-notes, filter-already-released-advisory-images, extract-oot-kmods) and states the default cannot change to dual until those are resolved. This dependency is clearly documented, which is good. Tracking those fixes will be important for the rollout plan.

Footer

Outcome: approve
This review applies to SHA 70162225ca5722dbb9a83aa2e2bc54a990e6edfd. Any push to the PR head clears this review and requires a new evaluation.

Previous run (20)

Review: #357

Head SHA: c542bce
Timestamp: 2026-05-14T00:00:00Z
Outcome: approve

Summary

This PR adds ADR 0067, proposing dual-compression (gzip + zstd:chunked) for container builds. The ADR is well-structured, follows the repository's ADR template and numbering conventions (sequential after 0066), and presents a technically sound proposal. The context section clearly motivates the change with backward-compatibility concerns, the decision section provides concrete implementation details including new task parameters and release pipeline impacts, and the alternatives section explains why other approaches were rejected. No security, correctness, or style issues were found.

Findings

Info

• [style] ADR/0067-dual-compression-for-container-builds.md — The applies_to frontmatter lists only build-service, but the ADR also discusses required changes to release pipeline tasks in release-service-catalog. Consider adding release-service to applies_to for discoverability.

Footer

Outcome: approve
This review applies to SHA c542bce2c5306569dfc9fd2a1203f319d6843dc3. Any push to the PR head clears this review and requires a new evaluation.

[chmeliik]

How will all of this be represented in the SBOM? Currently, each architecture gets an SBOM where the arch-specific manifest is the main component, and the index gets an SBOM pointing to each arch-specific manifest.

Example:

# index SBOM
$ cosign download sbom quay.io/konflux-ci/task-runner:1.6.0

# x86_64 manifest SBOM
$ cosign download sbom quay.io/konflux-ci/task-runner@sha256:b4b46f35d697dd7cbcd77b051ceac41d324164edcf85d39379b7b53f492d8394

If we push two different images for each arch, which one will be the main component in the SBOM? How will the other one be represented? Will we need to attach the SBOM to both of them?

If we push a single index with two manifests for each arch, can that be the main component? Is it OK if the SBOM doesn't reference the individual manifests? Or can we represent them as sub-components of the main component (in both CycloneDX and SPDX)? Will we need to attach the SBOM to the index and both the manifests, or just the index?

[chmeliik]

@Amanyd when making changes that address some open threads, would you be able to respond in those threads?

[Amanyd]

How will all of this be represented in the SBOM? Currently, each architecture gets an SBOM where the arch-specific manifest is the main component, and the index gets an SBOM pointing to each arch-specific manifest.

Example:

# index SBOM
$ cosign download sbom quay.io/konflux-ci/task-runner:1.6.0

# x86_64 manifest SBOM
$ cosign download sbom quay.io/konflux-ci/task-runner@sha256:b4b46f35d697dd7cbcd77b051ceac41d324164edcf85d39379b7b53f492d8394

If we push two different images for each arch, which one will be the main component in the SBOM? How will the other one be represented? Will we need to attach the SBOM to both of them?

If we push a single index with two manifests for each arch, can that be the main component? Is it OK if the SBOM doesn't reference the individual manifests? Or can we represent them as sub-components of the main component (in both CycloneDX and SPDX)? Will we need to attach the SBOM to the index and both the manifests, or just the index?

As for the per-arch index now, the SBOM will just attach directly to the index digest, because both variants only differ in compression, one SBOM on the index is all we need, we don't have to generate separate SBOMs for each variant.

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 3:11 PM UTC · Completed 3:27 PM UTC
Commit: ec21706 · View workflow run →

[fullsend-ai-review]

See the review comment for full details.

[chmeliik]

LGTM from my side, good job!

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:25 PM UTC · Completed 11:42 PM UTC
Commit: ec21706 · View workflow run →

[fullsend-ai-review]

See the review comment for full details.

[ralphbean]

Oh, small thing. Needs to run hack/util/generate-agents-md.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 3:58 PM UTC · Ended 3:59 PM UTC
Commit: ec21706 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 4:02 PM UTC · Completed 4:17 PM UTC
Commit: ec21706 · View workflow run →

[fullsend-ai-review]

See the review comment for full details.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 5:11 PM UTC · Completed 5:18 PM UTC
Commit: ec21706 · View workflow run →

[fullsend-ai-retro]

Retro: PR #357 — ADR 0070: Support for zstd:chunked image compression

This was a human-authored PR (branch adr-0067-dual-compression, author Amanyd) open for ~46 days (May 14 – Jun 29) with substantial technical review from 4 human reviewers before merge.

Timeline highlights

• May 14: PR opened. konflux-ci-review[bot] auto-approved 6 times with zero findings (rubber stamp). chmeliik posted 10+ substantive inline comments covering pipeline breakage, SBOM representation, Conforma trust model, and Tekton result limits.
• May 24: fullsend-ai-review[bot] first review — 4 inline findings (line count inaccuracies, grammar, style). APPROVED.
• Jun 6: Bot's most valuable contribution — [critical] ADR number conflict (0067 already taken). CHANGES_REQUESTED. Also flagged valid but lower-severity concerns (OCI spec claim accuracy, internal consistency).
• Jun 17: Bot caught second number conflict (0068 also taken). CHANGES_REQUESTED. From this point forward, the bot repeated the same medium-severity findings (gzip-first ordering, annotation survival) across 5+ subsequent review cycles without resolution.
• Jun 26: eskultety pushed for acceptance; ralphbean provided architectural guidance on Conforma trust model.
• Jun 29: ralphbean noted hack/util/generate-agents-md needed to run. PR merged.

Review quality assessment

Bot strengths: ADR number collision detection (caught twice, genuinely valuable and blocking). Line count validation. Style/grammar nits.

Bot gaps vs. human review: Human reviewers drove ALL substantive architectural discourse:

• Pipeline breakage from empty IMAGE_DIGEST (chmeliik)
• Conforma trust model gap — arbitrary digest injection via build-image-index (chmeliik, ralphbean)
• SBOM attachment strategy and digest targeting (chmeliik, BorekZnovustvoritel)
• Tekton result size practical limits (chmeliik, eskultety)
• Cross-service provenance chain (ralphbean, arewm)
• Multi-arch isolation feasibility (eskultety)
• Parent image layer reuse impossibility (arewm)

The review agent could not reason about cross-service trust models, pipeline data flow, or Tekton runtime constraints — the domain knowledge gap is fundamental for this type of architectural document.

Token cost

9 review runs total (7 completed, 1 cancelled, 1 retro). Commit ec21706 alone was reviewed 4 separate times with essentially identical findings. Significant waste.

Existing issue coverage

The issues observed on this PR are comprehensively covered by existing open issues in fullsend-ai/fullsend:

• Redundant runs on same SHA: #963, #1418, #2599
• Repeated findings across cycles: #1013, #1285
• COMMENT verdict for low-severity re-reviews: #2029
• Comment-only (no approve) on ADR PRs: #1659
• Defer merge label on docs PRs: #1772
• Suppress dispatch during active human CHANGES_REQUESTED: #1791
• Cross-file impact analysis gaps: #1525
• Security-critical findings on large arch PRs: #898
• ADR number collision as CI lint: #992
• Suppress non-actionable repeated findings: #1881

No new proposals are warranted — the improvement areas are well-tracked. Resolving #963, #1013, and #1659 would have the highest impact on reducing the waste patterns observed on this PR.