pixi-audit-crater

Compare pixi-audit with Grype on installed, frozen Pixi environments. Grype-only findings are investigation candidates, not confirmed vulnerabilities or pixi-audit bugs. Agreement with Grype is not recall against ground truth.

The active corpus is corpora/expanded: 24 repositories, including 20 additions covering scientific Python, computer vision, robotics, C++ tooling, and package management. All 24 completed frozen installation and both scanners with no partial scanner errors in the expanded validation run; installed and audited lockfile hashes match the frozen inputs. The original Pandas, LightGBM, SciPy, and Kornia repository commits, lockfiles, and source archives are preserved byte-for-byte from corpora/scannable.

The benchmark scans only default on linux-64 and never uses the upstream execution script. Selection is based on successful installation and paired scanning, not finding counts, so this corpus measures coverage on known-scannable environments rather than representing the whole upstream list. LightGBM's minimal ca-certificates environment remains an outside-environment attribution control. Future failures stay visible; repositories are not silently dropped from routine runs.

The expansion started with corpora/expanded-candidates: the original four inputs plus 27 proposed additions from the same upstream revision's medium.json. The candidate validation run preserves all results: 17 paired successes, seven successful installations blocked by the runner's duplicate in-memory snapshot during extraction, six installation failures, and one source-size exclusion. Streaming extraction resolved all seven runner failures. The active subset's checksummed raw evidence records 24 successful paired analyses. Failed and excluded candidate inputs remain in history rather than being counted as successful scans.

Local commands

Install Pixi and Docker. GitHub release operations also require an authenticated gh CLI. The dedicated environment in this repository contains Python, PyYAML, and Tomli; it does not replace global tools.

pixi install --locked

# Restore source archives for the committed, reviewed corpus.
pixi run --locked python -m benchmark fetch-corpus corpora/expanded .work/corpus
pixi run --locked python -m benchmark verify .work/corpus

# Run both scanners. The output directory must not already exist.
pixi run --locked python -m benchmark run .work/corpus runs/local-001

# Publish this immutable run to its own checksummed GitHub release.
pixi run --locked python -m benchmark publish-run runs/local-001

# Fetch all durable runs and regenerate the static report.
pixi run --locked python -m benchmark fetch-history .work/history
pixi run --locked python -m benchmark site .work/history site

# Behavioral regression cases, kept separate from benchmark statistics.
pixi run --locked test

A run resolves the latest published pixi-audit once from https://prefix.dev/prefix-dev/prefix-labs, with conda-forge for dependencies. Every repository in that run uses the resolved build. benchmark-config.json pins the installation base image, Pixi, and Grype. Change those pins explicitly and review their effect on comparability.

Frozen sources and lockfiles do not guarantee byte-identical source-built artifacts. Vulnerability databases can change independently of scanner versions. Run evidence records installed inventory and available provenance, including unavailable database snapshot identities.

Reproduce one repository

This uses the same runner and reviewed corpus, but saves only that repository's evidence. It does not publish a partial-corpus run or add synthetic history. Change the repository ID and use a fresh output directory for another reproduction.

pixi run --locked python - <<'PY'
import json
from pathlib import Path
from benchmark.corpus import verify
from benchmark.runner import Runner

corpus_path = Path(".work/corpus").resolve()
corpus = verify(corpus_path)
output = Path(".work/reproduce-lightgbm").resolve()
output.mkdir(parents=True)
runner = Runner(output, json.loads(Path("benchmark-config.json").read_text()))
try:
    tools, provenance = runner.prepare_tools()
    (output / "tools.json").write_text(json.dumps(provenance, indent=2))
    index, repository = next(
        (i, repo) for i, repo in enumerate(corpus["repositories"])
        if repo["id"] == "lightgbm-org/LightGBM"
    )
    print(json.dumps(runner.repository(tools, corpus_path, repository, index), indent=2))
finally:
    runner.cleanup()
    (output / "tools/pixi").unlink(missing_ok=True)
PY

Reviewed corpus refresh

To select different repositories without refreshing any commits, locks, or archives, derive a new subset offline from a restored corpus:

pixi run --locked python -m benchmark fetch-corpus corpora/reviewed .work/original-corpus
pixi run --locked python -m benchmark subset .work/original-corpus corpora/proposed-subset \
  pandas-dev/pandas lightgbm-org/LightGBM scipy/scipy kornia/kornia
pixi run --locked python -m benchmark verify corpora/proposed-subset

The subset records its parent revision and explicit repository selection, preserves upstream order, and copies the exact frozen bytes. It cannot select an excluded or unknown repository. A changed corpus starts a new comparison baseline; existing runs remain immutable.

To add repositories while retaining existing repository commits, lockfiles, and source archives byte-for-byte:

pixi run --locked python -m benchmark extend .work/original-corpus corpora/proposed-expanded \
  prefix-dev/pixi pydata/sparse conda/conda-lock --list medium.json
pixi run --locked python -m benchmark verify corpora/proposed-expanded

Additions must appear in the frozen upstream list. An optional --list selects another list at that same upstream commit and must preserve the original repository entries. Only additions resolve repository HEAD; existing inputs are copied unchanged. Selection is bounded to the first 100 upstream entries and records an explicit ordered union and parent revision. Validate candidates with the normal paired runner before deriving a successful subset; failed and excluded attempts remain in immutable evidence.

Routine scans never follow repository HEAD or regenerate a lockfile. To propose different inputs, create a new directory:

pixi run --locked python -m benchmark freeze corpora/proposed --list small.json --limit 10
pixi run --locked python -m benchmark verify corpora/proposed

Review corpus.json, original upstream entries, repository commits, frozen lockfiles, and exclusions. A refresh creates a new corpus revision; it never edits an existing revision. After approval, publish the complete directory, including source archives, under corpus-<revision>:

pixi run --locked python -m benchmark publish corpora/proposed corpus-<revision>

Commit the reviewed corpus metadata, upstream list, and lockfiles. Source archives live in the checksummed release asset, not Git history. Update the workflow's reviewed corpus path when adopting a new revision. Restore verifies that the release manifest exactly matches the committed manifest.

The source snapshot uses GitHub's commit archive, which honors upstream export-ignore attributes. Committed Pixi metadata omitted by those attributes is restored from the same SHA with identical bytes; no lockfile is generated. Other export-ignored files and submodule contents are not materialized. Installation failures caused by unavailable source inputs remain visible. freeze --from-corpus <existing> reprepares the same repository SHAs into a new revision without following HEAD.

Publication and retention

The public repository is prefix-dev/pixi-audit-crater; Pages is at prefix-dev.github.io/pixi-audit-crater.

The default Diff view groups findings by repository and installed package, with pixi-audit-only findings on the left and Grype-only findings on the right. Shared findings and uncertain attribution have separate collapsed sections. Finding details explain attribution and link to raw evidence. History holds comparisons between runs; Run details holds statistics and provenance.

The repository jump list includes difference counts and identifies unavailable comparisons. Diff packages start expanded, with individual toggles and Expand all / Collapse all controls; expansion choices survive searches and reset when selecting another run. Repository and package headings stay visible while scrolling. Empty scanner columns are explained once per package, and filtered results explicitly say when there are no matching findings.

The visual theme draws from Basilisk. Cabin is self-hosted under the SIL Open Font License; headings use the system Georgia font. The pinned Cabin source revision is recorded in benchmark/web/style.css.

To publish report changes from stored evidence without running either scanner:

gh workflow run benchmark.yml --repo prefix-dev/pixi-audit-crater -f scan=false

Normal workflow runs still scan by default. Rebuilding the report verifies and reuses every existing run release; it does not alter raw reports or create a new scan record.

The Actions workflow is manual only. Installation/scanning has read-only repository permissions; publishing runs in a separate job, and Pages deployment has its own restricted permissions. Target containers do not receive host credentials, publishing tokens, Docker sockets, or writable host source mounts. Source builds may still execute code inside their disposable container. The runner does not deliberately execute project tasks.

Containers have CPU, memory, process, tmpfs, lifetime, and output limits in benchmark-config.json; exceeding them produces an explicit failed or partial result. Collection grants group-read access to installed files, then reads them with a separate UID using trusted inline collector code. Snapshot ingestion rejects sparse files, unsupported members, escaping paths, and expanded sizes beyond the configured workspace bound. Raw scanner output is retained; installed container images are not.

Installed filesystem snapshots stream from the host into confined scanner-root extraction. The scanner container does not retain a second in-memory tar copy, keeping extraction within the same memory and tmpfs limits. Absolute container links are rebased inside the scanner root; escaping links remain rejected.

Every run has a unique run-* release with evidence.tar.gz and checksums.json. Run directories and releases are append-only through the benchmark commands; existing run IDs are never overwritten. GitHub administrators can still delete or replace release assets, so this is not tamper-proof archival storage. Checksums detect corruption, not malicious replacement of both evidence and checksums.

Actions artifacts are only transport, retained for seven days. Published release evidence has no automatic expiry or deletion policy. Each release asset must fit GitHub's asset limit (currently 2 GiB); retrieval additionally bounds extracted files to 4 GiB and total extracted evidence to 20 GiB. Large corpora need a reviewed storage change before reaching these limits. A failed history retrieval or site generation prevents deployment and leaves the previous site intact. Workflow concurrency serializes publishers.

Comparison rules

• Match package ecosystem and version identity plus advisory aliases. A matching name alone is insufficient.
• Exclude confidently outside-environment findings before advisory matching and from all report counts, rankings, and historical transitions. Preserve original raw reports unchanged.
• Keep both-scanner findings, attributable Grype-only candidates, pixi-audit-only findings, and uncertain attribution separate. Uncertain findings are not counted as scanner-only differences.
• Conda file ownership relates a discovered library to a package, but does not prove that their identities or versions match. Bundled libraries remain uncertain without stronger evidence.
• Include suppressed findings in detection comparisons and retain suppression status. Unchecked packages stay visible.
• Installation failures, scanner errors, unsupported inputs, and partial results are not successful zero-finding scans.
• Compare only the same successfully scanned, comparable repositories in both runs. Show the denominator and exclusions separately from full-run totals.
• Both to Grype-only is a potential lost detection; Grype-only to both is an observed coverage gain. A newly appearing Grype-only advisory is a new gap, not automatically a regression.
• Count repository/package/advisory occurrences separately from deduplicated advisories across the corpus.

Raw evidence can contain upstream repository or scanner content. Treat it as untrusted data.

Initial evidence review

The first real run, pac-99a3007e865f489393e2d38b35636561, paired four repositories successfully. FreeCAD, ONNX, Bokeh, and Dask had installation failures; they are not counted as clean scans. LLMs-from-scratch has no root lockfile, and Rerun has no locked default/linux-64 target, so both are excluded from this corpus.

Representative checks against the raw reports and installed metadata:

• Pandas / Python 3.12.14 / CVE-2026-4224: Grype's binary and Conda discoveries identify the same installed Python package. Its NVD CPE range says < 3.13.13, but the Python CNA record marks the 3.12 branch affected only below 3.12.14. This discrepancy is not evidence of a pixi-audit miss. The benchmark retains the original Grype-only candidate rather than rewriting scanner evidence.
• SciPy / OpenSSL 3.6.0 / CVE-2026-14456: both Grype discoveries match the installed h26f9b46_0 Conda build. The OpenSSL advisory includes the 3.6 branch and identifies 3.6.4 as the fixed release. This remains an investigation candidate: package backports and application use of the affected QUIC server code were not established. Grype reports High severity while OpenSSL rates it Low.
• LightGBM: the installed default environment contains ca-certificates; all 246 findings in its raw Grype report are outside that environment and excluded from the derived report.
• Kornia: eight findings identify Rust crates bundled inside installed files. File ownership does not establish identity equivalence with the enclosing locked package, so they remain uncertain rather than attributable gaps.

These checks establish attribution and expose feed differences, not ground-truth recall or application exploitability. Both scanners can miss vulnerabilities. Missing pixi-audit database snapshot identities, source-build variability, and changing advisory databases remain explicit limitations.