diff --git a/.claude/commands/release.md b/.claude/commands/release.md index 83be9ced5..6558fb3a6 100644 --- a/.claude/commands/release.md +++ b/.claude/commands/release.md @@ -83,7 +83,7 @@ Before touching any version, run a multi-agent review of everything merged since 3. **TypeScript best practices** — no unjustified `any`/`as`; exhaustive switches; no floating promises; ESM import correctness; closed-shape outputs (no schema-invalid passthroughs); matches the eslint config's intent. 4. **Go best practices** — error wrapping (`%w`), no swallowed errors, `omitempty` consistency, struct-tag correctness, context usage, no goroutine leaks; matches the 39-linter `golangci-lint` intent. 5. **Cross-PR consistency & regression** — for the PRs merged since `BASE`: did any two touch the same area inconsistently? Are shared-code changes reflected in *all* consumers? Is Go↔TS parity preserved where both exist? Did any PR reintroduce something another removed, or silently regress a third? **Do NOT flag the CHANGELOG as missing/out-of-date — the new-version section is authored later, in Phase 5, so its absence at review time is by design, not a finding.** The one CHANGELOG-adjacent thing worth surfacing is a *consumer-visible behavior change* that Phase 5 must call out loudly (report it as a low-severity note so Phase 5 remembers it — not as a blocking gap). -6. **Docs / README accuracy (full-surface, not diff-scoped).** For every package `README.md` (root, `hdf-cli`, `hdf-diff`, and each `@mitre/hdf-*` package), verify what it documents still matches reality: every documented command/subcommand and flag actually exists in the current CLI (`hdf --help`) or public API; no *removed* command or renamed syntax is still shown; example invocations use real flags; and any embedded example output is faithful to a real run (statuses, counts, column headers, summary lines — not fabricated or stale). Because drift here predates the release window, this dimension inspects the **current** binary/API surface, not just `BASE..HEAD`. Every mismatch = a finding naming the README, the stale claim, and the correct current form. +6. **Docs / README accuracy (full-surface, not diff-scoped).** For every package `README.md` (root, `hdf-cli`, `hdf-diff`, and each `@mitre/hdf-*` package), verify what it documents still matches reality: every documented command/subcommand and flag actually exists in the current CLI (`hdf --help`) or public API; no *removed* command or renamed syntax is still shown; example invocations use real flags; and any embedded example output is faithful to a real run (statuses, counts, column headers, summary lines — not fabricated or stale). Because drift here predates the release window, this dimension inspects the **current** binary/API surface, not just `BASE..HEAD`. Every mismatch = a finding naming the README, the stale claim, and the correct current form. Also confirm the generated converter catalog page (`site/docs/guides/converters.md`, rendered by `site/generate-converters.mjs` from the committed, registry-golden-tested `site/data/converters.json`) is current: if any converter was added or removed this cycle, regenerate the manifest (`go test ./cmd/hdf/cmd -run TestConverterCatalogManifest -update-catalog`) and confirm the golden test then passes — the catalog must list exactly the registry's converters. **Orchestration** — use the `Workflow` tool (this instruction is the multi-agent opt-in). Fan out one finder per dimension (shard dimension×package when the diff is large), adversarially verify each finding with an independent skeptic prompted to *refute* (drop unless it survives — this kills best-practice nitpicks and hallucinated issues), then synthesize a deduped report grouped by dimension and severity. Pass `BASE`, the changed-file list, and the PR list in via `args`. Skeleton: @@ -108,7 +108,7 @@ const DIMENSIONS = [ { key: 'ts', prompt: `${ctx}\n\nReport TypeScript best-practice violations in the changed .ts files (unjustified any/as, non-exhaustive switch, floating promises, bad ESM imports, schema-invalid passthroughs).` }, { key: 'go', prompt: `${ctx}\n\nReport Go best-practice violations in the changed .go files (unwrapped/swallowed errors, omitempty drift, struct-tag errors, context misuse, goroutine leaks).` }, { key: 'crosspr', prompt: `${ctx}\n\nPRs merged since ${args.base}:\n${args.prs}\n\nReport cross-PR inconsistencies/regressions: same area touched inconsistently, shared-code change not reflected in all consumers, broken Go/TS parity, one PR reverting/regressing another. Do NOT report a missing or out-of-date CHANGELOG — its new-version section is written later in Phase 5, so its absence now is expected. The only CHANGELOG-adjacent finding worth raising is a consumer-visible BEHAVIOR CHANGE Phase 5 must document loudly — report that as a low-severity note, not a blocking gap.` }, - { key: 'docs', prompt: `Ignore the diff scope for this one — audit the CURRENT state. For every package README.md (root, hdf-cli, hdf-diff, each @mitre/hdf-* package), verify documented commands/subcommands/flags still exist in the real CLI (build ./hdf and run 'hdf --help') or public API, that no removed/renamed command or syntax is still shown, and that any embedded example output is faithful to a real run (status labels, counts, headers, summary lines). Report each mismatch with the README path, the stale claim, and the correct current form.` }, + { key: 'docs', prompt: `Ignore the diff scope for this one — audit the CURRENT state. For every package README.md (root, hdf-cli, hdf-diff, each @mitre/hdf-* package), verify documented commands/subcommands/flags still exist in the real CLI (build ./hdf and run 'hdf --help') or public API, that no removed/renamed command or syntax is still shown, and that any embedded example output is faithful to a real run (status labels, counts, headers, summary lines). Also verify the generated converter catalog (site/docs/guides/converters.md, from site/data/converters.json) matches the live registry: run the hdf-cli golden test 'go test ./cmd/hdf/cmd -run TestConverterCatalogManifest' and report a finding if it fails (manifest stale — regenerate with -update-catalog). Report each mismatch with the README/page path, the stale claim, and the correct current form.` }, ] phase('Review') diff --git a/.gitattributes b/.gitattributes index 9e4cc0944..fd9449f1c 100644 --- a/.gitattributes +++ b/.gitattributes @@ -12,6 +12,7 @@ hdf-cli/internal/mcp/**/testdata/**/*.jsonl eol=lf hdf-schema/test/**/*.json eol=lf hdf-generators/test/**/*.json eol=lf hdf-validators/test/**/*.json eol=lf +site/data/converters.json eol=lf # Test fixtures are stored as regular git objects (text XML/JSON compresses # well in packfiles). Large full-scan fixtures that exceed ~1 MB should not diff --git a/.gitignore b/.gitignore index 7916f059d..b66f160cb 100644 --- a/.gitignore +++ b/.gitignore @@ -56,6 +56,9 @@ hdf-cli/hdf site/schemas/*.md site/v*/ site/.vitepress/versions.json +# Converter catalog page — rendered from site/data/converters.json (which IS +# committed and golden-tested against the registry) by generate-converters.mjs. +site/docs/guides/converters.md # Per-version raw schema archive IS committed (backs the canonical $id # URL e.g. /schemas/hdf-amendments/v3.2.0/ and the per-version rendered # docs). ~70 KB per release; storage cost trivial. See site/seed-archive.mjs. diff --git a/hdf-cli/cmd/hdf/cmd/converter_catalog_test.go b/hdf-cli/cmd/hdf/cmd/converter_catalog_test.go new file mode 100644 index 000000000..a4181f0b4 --- /dev/null +++ b/hdf-cli/cmd/hdf/cmd/converter_catalog_test.go @@ -0,0 +1,103 @@ +package cmd + +import ( + "bytes" + "encoding/json" + "flag" + "os" + "path/filepath" + "sort" + "testing" +) + +// updateCatalog rewrites the committed converter manifest instead of asserting +// against it. Run after adding or removing a converter: +// +// go test ./cmd/hdf/cmd -run TestConverterCatalogManifest -update-catalog +var updateCatalog = flag.Bool("update-catalog", false, "rewrite site/data/converters.json from the live registry") + +// catalogEntry is one registered source→dest conversion, with the metadata the +// docs catalog page renders. +type catalogEntry struct { + Source string `json:"source"` + Dest string `json:"dest"` + Name string `json:"name"` + AcceptsEmpty bool `json:"acceptsEmpty"` +} + +// catalogManifest is the serialized shape of site/data/converters.json. It also +// carries the BOM inventory formats that flow through `hdf system create` (into +// an HDF System doc, not Results) so the catalog page can point SPDX/AIBOM users +// at the right command instead of leaving them to conclude those are unsupported. +type catalogManifest struct { + Converters []catalogEntry `json:"converters"` + SystemBomFormats []string `json:"systemBomFormats"` +} + +// catalogManifestPath resolves the committed manifest relative to this package +// (go test runs with the package dir as its working directory). +func catalogManifestPath() string { + return filepath.Join("..", "..", "..", "..", "site", "data", "converters.json") +} + +// buildCatalog snapshots the live converter registry and the system-BOM import +// formats into the manifest shape. +func buildCatalog(t *testing.T) catalogManifest { + t.Helper() + pairs := ListConverters() + entries := make([]catalogEntry, 0, len(pairs)) + for _, p := range pairs { + conv, err := GetConverter(p.Source, p.Dest) + if err != nil { + t.Fatalf("registered pair %s→%s has no retrievable converter: %v", p.Source, p.Dest, err) + } + e := catalogEntry{Source: p.Source, Dest: p.Dest, Name: conv.Name()} + if ae, ok := conv.(EmptyInputAccepting); ok { + e.AcceptsEmpty = ae.AcceptsEmptyInput() + } + entries = append(entries, e) + } + sort.Slice(entries, func(i, j int) bool { + if entries[i].Source != entries[j].Source { + return entries[i].Source < entries[j].Source + } + return entries[i].Dest < entries[j].Dest + }) + + bomFormats := append([]string(nil), bomFormatAliases...) + sort.Strings(bomFormats) + + return catalogManifest{Converters: entries, SystemBomFormats: bomFormats} +} + +// TestConverterCatalogManifest keeps site/data/converters.json — the source the +// docs site renders the converter catalog page from — in lockstep with the live +// registry. Adding or removing a converter fails this test until the manifest is +// regenerated (-update-catalog), so the published catalog can never silently +// drift from what the CLI actually supports. +func TestConverterCatalogManifest(t *testing.T) { + manifest := buildCatalog(t) + got, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + t.Fatalf("marshal catalog: %v", err) + } + got = append(got, '\n') + + path := catalogManifestPath() + if *updateCatalog { + if err := os.WriteFile(path, got, 0o600); err != nil { + t.Fatalf("write manifest: %v", err) + } + t.Logf("wrote %d converters + %d system BOM formats to %s", len(manifest.Converters), len(manifest.SystemBomFormats), path) + return + } + + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read manifest (%s): %v\nregenerate with: go test ./cmd/hdf/cmd -run TestConverterCatalogManifest -update-catalog", path, err) + } + if !bytes.Equal(got, want) { + t.Errorf("converter catalog manifest is stale — the registry and site/data/converters.json disagree.\n" + + "regenerate with: go test ./cmd/hdf/cmd -run TestConverterCatalogManifest -update-catalog") + } +} diff --git a/site/data/converters.json b/site/data/converters.json new file mode 100644 index 000000000..02f94bf35 --- /dev/null +++ b/site/data/converters.json @@ -0,0 +1,490 @@ +{ + "converters": [ + { + "source": "arf", + "dest": "hdf", + "name": "ARF to HDF", + "acceptsEmpty": false + }, + { + "source": "asff", + "dest": "hdf", + "name": "AWS Security Finding Format to HDF", + "acceptsEmpty": false + }, + { + "source": "aws-config", + "dest": "hdf", + "name": "AWS Config to HDF", + "acceptsEmpty": false + }, + { + "source": "burpsuite", + "dest": "hdf", + "name": "BurpSuite to HDF", + "acceptsEmpty": false + }, + { + "source": "checkov", + "dest": "hdf", + "name": "Checkov to HDF", + "acceptsEmpty": false + }, + { + "source": "ckl", + "dest": "hdf", + "name": "CKL to HDF", + "acceptsEmpty": false + }, + { + "source": "cklb", + "dest": "hdf", + "name": "CKLB to HDF", + "acceptsEmpty": false + }, + { + "source": "conveyor", + "dest": "hdf", + "name": "Conveyor to HDF", + "acceptsEmpty": false + }, + { + "source": "csaf-vex", + "dest": "hdf", + "name": "CSAF VEX to HDF Amendments", + "acceptsEmpty": false + }, + { + "source": "cyclonedx", + "dest": "hdf", + "name": "CycloneDX to HDF", + "acceptsEmpty": false + }, + { + "source": "cyclonedx-vex", + "dest": "hdf", + "name": "CycloneDX VEX to HDF Amendments", + "acceptsEmpty": false + }, + { + "source": "dbprotect", + "dest": "hdf", + "name": "DBProtect to HDF", + "acceptsEmpty": false + }, + { + "source": "defectdojo", + "dest": "hdf", + "name": "DefectDojo to HDF", + "acceptsEmpty": false + }, + { + "source": "defender-cloud", + "dest": "hdf", + "name": "Microsoft Defender for Cloud to HDF", + "acceptsEmpty": false + }, + { + "source": "defender-endpoint", + "dest": "hdf", + "name": "Microsoft Defender for Endpoint to HDF", + "acceptsEmpty": false + }, + { + "source": "dependency-track", + "dest": "hdf", + "name": "Dependency-Track to HDF", + "acceptsEmpty": false + }, + { + "source": "deptrack", + "dest": "hdf", + "name": "Dependency-Track to HDF", + "acceptsEmpty": false + }, + { + "source": "fortify", + "dest": "hdf", + "name": "Fortify to HDF", + "acceptsEmpty": false + }, + { + "source": "gitlab", + "dest": "hdf", + "name": "GitLab Security Report to HDF", + "acceptsEmpty": false + }, + { + "source": "gitlab-dast", + "dest": "hdf", + "name": "GitLab Security Report to HDF", + "acceptsEmpty": false + }, + { + "source": "gitlab-sast", + "dest": "hdf", + "name": "GitLab Security Report to HDF", + "acceptsEmpty": false + }, + { + "source": "gosec", + "dest": "hdf", + "name": "gosec to HDF", + "acceptsEmpty": false + }, + { + "source": "grype", + "dest": "hdf", + "name": "Grype to HDF", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "asff", + "name": "HDF Results to ASFF Findings", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "ckl", + "name": "HDF to CKL", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "cklb", + "name": "HDF to CKLB", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "csv", + "name": "HDF to CSV", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "ecs", + "name": "HDF Results to ECS", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "hdf", + "name": "HDF vauto to HDF v3", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "ocsf", + "name": "HDF Results to OCSF Findings", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "oscal-sar", + "name": "HDF Results to OSCAL SAR", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "splunk", + "name": "HDF Results to Splunk (CIM/HEC)", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "xccdf", + "name": "HDF to XCCDF", + "acceptsEmpty": false + }, + { + "source": "hdf", + "dest": "xml", + "name": "HDF to XML", + "acceptsEmpty": false + }, + { + "source": "hdf-amendments", + "dest": "csaf-vex", + "name": "HDF Amendments to CSAF VEX", + "acceptsEmpty": false + }, + { + "source": "hdf-amendments", + "dest": "cyclonedx-vex", + "name": "HDF Amendments to CycloneDX VEX", + "acceptsEmpty": false + }, + { + "source": "hdf-amendments", + "dest": "openvex", + "name": "HDF Amendments to OpenVEX", + "acceptsEmpty": false + }, + { + "source": "hdf-amendments", + "dest": "oscal-poam", + "name": "HDF Amendments to OSCAL POA\u0026M", + "acceptsEmpty": false + }, + { + "source": "hipcheck", + "dest": "hdf", + "name": "Hipcheck to HDF", + "acceptsEmpty": false + }, + { + "source": "inspec", + "dest": "hdf", + "name": "InSpec exec-json to HDF", + "acceptsEmpty": false + }, + { + "source": "invicti", + "dest": "hdf", + "name": "Netsparker/Invicti to HDF", + "acceptsEmpty": false + }, + { + "source": "ionchannel", + "dest": "hdf", + "name": "Ion Channel to HDF", + "acceptsEmpty": false + }, + { + "source": "jfrog-xray", + "dest": "hdf", + "name": "JFrog Xray to HDF", + "acceptsEmpty": false + }, + { + "source": "junit", + "dest": "hdf", + "name": "JUnit to HDF", + "acceptsEmpty": false + }, + { + "source": "legacyhdf", + "dest": "hdf", + "name": "InSpec exec-json to HDF", + "acceptsEmpty": false + }, + { + "source": "msdo", + "dest": "hdf", + "name": "Microsoft Defender for DevOps to HDF", + "acceptsEmpty": false + }, + { + "source": "msft-defender-cloud", + "dest": "hdf", + "name": "Microsoft Defender for Cloud to HDF", + "acceptsEmpty": false + }, + { + "source": "msft-defender-devops", + "dest": "hdf", + "name": "Microsoft Defender for DevOps to HDF", + "acceptsEmpty": false + }, + { + "source": "msft-defender-endpoint", + "dest": "hdf", + "name": "Microsoft Defender for Endpoint to HDF", + "acceptsEmpty": false + }, + { + "source": "msft-secure-score", + "dest": "hdf", + "name": "Microsoft Secure Score to HDF", + "acceptsEmpty": false + }, + { + "source": "nessus", + "dest": "hdf", + "name": "Nessus to HDF", + "acceptsEmpty": false + }, + { + "source": "netsparker", + "dest": "hdf", + "name": "Netsparker/Invicti to HDF", + "acceptsEmpty": false + }, + { + "source": "neuvector", + "dest": "hdf", + "name": "NeuVector to HDF", + "acceptsEmpty": false + }, + { + "source": "nikto", + "dest": "hdf", + "name": "Nikto to HDF", + "acceptsEmpty": false + }, + { + "source": "openvex", + "dest": "hdf", + "name": "OpenVEX to HDF Amendments", + "acceptsEmpty": false + }, + { + "source": "oscal", + "dest": "hdf", + "name": "OSCAL (auto-detect) to HDF", + "acceptsEmpty": false + }, + { + "source": "oscal-assessment-plan", + "dest": "hdf", + "name": "OSCAL Assessment Plan to HDF Plan", + "acceptsEmpty": false + }, + { + "source": "oscal-assessment-results", + "dest": "hdf", + "name": "OSCAL Assessment Results to HDF", + "acceptsEmpty": false + }, + { + "source": "oscal-catalog", + "dest": "hdf", + "name": "OSCAL Catalog to HDF Baseline", + "acceptsEmpty": false + }, + { + "source": "oscal-component-definition", + "dest": "hdf", + "name": "OSCAL Component Definition to HDF Baseline", + "acceptsEmpty": false + }, + { + "source": "oscal-poam", + "dest": "hdf", + "name": "OSCAL POA\u0026M to HDF Amendments", + "acceptsEmpty": false + }, + { + "source": "oscal-profile", + "dest": "hdf", + "name": "OSCAL Profile to HDF Baseline", + "acceptsEmpty": false + }, + { + "source": "oscal-sar", + "dest": "hdf", + "name": "OSCAL Assessment Results to HDF", + "acceptsEmpty": false + }, + { + "source": "oscal-ssp", + "dest": "hdf", + "name": "OSCAL System Security Plan to HDF System", + "acceptsEmpty": false + }, + { + "source": "prisma", + "dest": "hdf", + "name": "Prisma Cloud to HDF", + "acceptsEmpty": false + }, + { + "source": "sarif", + "dest": "hdf", + "name": "SARIF to HDF", + "acceptsEmpty": false + }, + { + "source": "scoutsuite", + "dest": "hdf", + "name": "ScoutSuite to HDF", + "acceptsEmpty": false + }, + { + "source": "snyk", + "dest": "hdf", + "name": "Snyk to HDF", + "acceptsEmpty": false + }, + { + "source": "sonarqube", + "dest": "hdf", + "name": "SonarQube to HDF", + "acceptsEmpty": false + }, + { + "source": "spdx-vex", + "dest": "hdf", + "name": "SPDX VEX to HDF Amendments", + "acceptsEmpty": false + }, + { + "source": "splunk", + "dest": "hdf", + "name": "Splunk to HDF", + "acceptsEmpty": false + }, + { + "source": "trivy", + "dest": "hdf", + "name": "Trivy to HDF", + "acceptsEmpty": false + }, + { + "source": "trufflehog", + "dest": "hdf", + "name": "TruffleHog to HDF", + "acceptsEmpty": true + }, + { + "source": "twistlock", + "dest": "hdf", + "name": "Twistlock to HDF", + "acceptsEmpty": false + }, + { + "source": "veracode", + "dest": "hdf", + "name": "Veracode to HDF", + "acceptsEmpty": false + }, + { + "source": "xccdf", + "dest": "hdf", + "name": "XCCDF to HDF (auto-detect)", + "acceptsEmpty": false + }, + { + "source": "xccdf-benchmark", + "dest": "hdf", + "name": "XCCDF Benchmark to HDF Baseline", + "acceptsEmpty": false + }, + { + "source": "xccdf-results", + "dest": "hdf", + "name": "XCCDF Results to HDF", + "acceptsEmpty": false + }, + { + "source": "xray", + "dest": "hdf", + "name": "JFrog Xray to HDF", + "acceptsEmpty": false + }, + { + "source": "zap", + "dest": "hdf", + "name": "OWASP ZAP to HDF", + "acceptsEmpty": false + } + ], + "systemBomFormats": [ + "cyclonedx", + "cyclonedx-mlbom", + "spdx", + "spdx-ai" + ] +} diff --git a/site/generate-converters.mjs b/site/generate-converters.mjs new file mode 100644 index 000000000..4117bf866 --- /dev/null +++ b/site/generate-converters.mjs @@ -0,0 +1,144 @@ +// Generate the converter catalog page (docs/guides/converters.md) from the +// registry manifest at data/converters.json. +// +// The manifest is produced and drift-guarded by a Go golden test in hdf-cli +// (TestConverterCatalogManifest): adding or removing a converter fails that +// test until the manifest is regenerated, so this page can never claim a +// converter the CLI does not ship, or omit one it does. This script only +// renders — it never invents data. Wired into `pnpm generate`; the output is +// git-ignored and rebuilt at site-build time, like the schema pages. +// +// Run: node generate-converters.mjs + +import fs from 'fs'; +import path from 'path'; +import { fileURLToPath } from 'url'; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const MANIFEST = path.resolve(__dirname, 'data/converters.json'); +const OUTPUT = path.resolve(__dirname, 'docs/guides/converters.md'); + +const AMENDMENTS_SOURCE = 'hdf-amendments'; + +// Render a markdown table of `format → converter name` rows. The Empty-input +// column is only meaningful for imports (it never applies to HDF export), so it +// is opt-in via `withEmpty`. +function table(headerFormat, rows, withEmpty) { + const head = withEmpty + ? `| ${headerFormat} | Converter | Empty input OK |` + : `| ${headerFormat} | Converter |`; + const rule = withEmpty ? '| --- | --- | --- |' : '| --- | --- |'; + const lines = [head, rule]; + for (const r of rows) { + lines.push( + withEmpty + ? `| \`${r.format}\` | ${r.name} | ${r.acceptsEmpty ? '✓' : '✗'} |` + : `| \`${r.format}\` | ${r.name} |`, + ); + } + return lines.join('\n'); +} + +// Friendly label for each `hdf system create --from` token. The token list is +// registry-derived (golden-tested); an unknown token still renders, falling back +// to the token itself so a newly-added format is never silently dropped. +const BOM_FORMAT_LABELS = { + cyclonedx: 'plain CycloneDX SBOM', + spdx: 'plain SPDX 2.x SBOM', + 'cyclonedx-mlbom': 'CycloneDX ML-BOM (AIBOM — a machine-learning-model component)', + 'spdx-ai': 'SPDX 3.0 AI/Dataset document (AIBOM)', +}; + +function main() { + const manifest = JSON.parse(fs.readFileSync(MANIFEST, 'utf-8')); + const entries = manifest.converters; + const bomFormats = manifest.systemBomFormats ?? []; + + // Import: anything → hdf that is not itself an HDF-family source. + const imports = entries + .filter((e) => e.dest === 'hdf' && e.source !== 'hdf' && e.source !== AMENDMENTS_SOURCE) + .map((e) => ({ format: e.source, name: e.name, acceptsEmpty: e.acceptsEmpty })); + + // Export: hdf → anything. + const exports = entries + .filter((e) => e.source === 'hdf') + .map((e) => ({ format: e.dest, name: e.name, acceptsEmpty: e.acceptsEmpty })); + + // Amendments export: hdf-amendments → anything. + const amendments = entries + .filter((e) => e.source === AMENDMENTS_SOURCE) + .map((e) => ({ format: e.dest, name: e.name, acceptsEmpty: e.acceptsEmpty })); + + const bomRows = bomFormats.map((f) => `| \`${f}\` | ${BOM_FORMAT_LABELS[f] ?? f} |`); + const bomTable = ['| Format token | Input |', '| --- | --- |', ...bomRows].join('\n'); + + const md = `# Converter Catalog + +The \`hdf\` CLI converts security assessment data between formats. This page lists +every converter that ships in this build. It is generated from the live +converter registry, so it always matches what \`hdf convert --help\` reports. + +Convert to HDF with an auto-detected input format: + +\`\`\`bash +hdf convert scan.json -o results.json +\`\`\` + +or name the format explicitly with \`--from\` / \`--to\` using the format token in +the tables below (e.g. \`--from nessus\`, \`--to splunk\`): + +\`\`\`bash +hdf convert --from nessus --to hdf scan.nessus -o results.json +\`\`\` + +The **Empty input OK** column marks converters that treat empty input as a valid +"no findings" signal (exit-code-first scanners) rather than an error; these must +be paired with an explicit \`--from\` because empty input carries nothing to +auto-detect. + +## Import to HDF (${imports.length}) + +Source formats that convert **into** HDF. Pass the format token to \`--from\`. + +${table('Source format', imports, true)} + +## Export from HDF (${exports.length}) + +Formats that HDF Results convert **out** to. Pass the format token to \`--to\`. + +${table('Target format', exports, false)} + +## Amendments export (${amendments.length}) + +Formats produced from an HDF amendments document (waivers, attestations, POA&Ms). + +${table('Target format', amendments, false)} + +## Ingesting SBOMs (inventory) — \`hdf system create\` + +Software Bill of Materials **inventory** documents are not in the tables above: +they carry no assessment results, so they are not converters. They build an HDF +**System** document (its \`components[]\`) through \`hdf system create\` (or +\`hdf system add-component\`), not the converter registry. If you are looking for +**SPDX** or an **AIBOM**, this is the path. + +\`hdf system create --from \` accepts (omit \`--from\` to auto-detect): + +${bomTable} + +**Vulnerability-bearing CycloneDX is a converter, not this path.** A CycloneDX +document that carries \`vulnerabilities\` (or VEX) converts to HDF **Results** via +\`cyclonedx\` / \`cyclonedx-vex\` in the tables above; \`cyclonedx\` (to HDF Results) +rejects a no-vulnerability inventory SBOM and points you here instead. SPDX has no +vulnerability-to-Results path — it only ever flows to the System model. +`; + + fs.mkdirSync(path.dirname(OUTPUT), { recursive: true }); + fs.writeFileSync(OUTPUT, md); + console.log( + `Generated ${path.relative(process.cwd(), OUTPUT)} — ` + + `${imports.length} import, ${exports.length} export, ${amendments.length} amendments.`, + ); +} + +main(); diff --git a/site/generate-schema-docs.mjs b/site/generate-schema-docs.mjs index b3a0f036c..5e0dd42ed 100644 --- a/site/generate-schema-docs.mjs +++ b/site/generate-schema-docs.mjs @@ -21,6 +21,7 @@ import fs from 'fs'; import path from 'path'; import { fileURLToPath } from 'url'; +import { resolveTypeName, renderEnumValues, anchorId } from './schema-render.mjs'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const SCHEMAS_DIR = path.resolve(__dirname, '../hdf-schema/dist/schemas'); @@ -231,6 +232,18 @@ function renderSchemaPage(schema, name, urlPrefix) { const isHistorical = urlPrefix !== ''; const meta = SCHEMA_META[name] || { title: name, description: '' }; const version = idVersion(schema.$id); + + // Type names that have a heading on this page (local defs + embedded-primitive + // inner defs). resolveTypeName links a $ref only when its target is in here, + // so a cross-page/unknown ref degrades to plain code instead of a dead anchor. + const knownTypes = new Set(); + for (const [key, defn] of Object.entries(schema.$defs || {})) { + if (key.startsWith('https://')) { + for (const inner of Object.keys(defn.$defs || {})) knownTypes.add(inner); + } else { + knownTypes.add(key); + } + } const downloadUrl = isHistorical ? `/schemas/${name}/${version}/index.json` : `/schemas/${name}.schema.json`; @@ -259,7 +272,7 @@ function renderSchemaPage(schema, name, urlPrefix) { lines.push('|-------|------|----------|-------------|'); const required = new Set(schema.required || []); for (const [field, prop] of Object.entries(schema.properties)) { - const type = resolveTypeName(prop); + const type = resolveTypeName(prop, knownTypes); const req = required.has(field) ? '**yes**' : 'no'; const desc = (prop.description || '').replace(/\n/g, ' ').replace(/\|/g, '\\|'); lines.push(`| \`${field}\` | ${type} | ${req} | ${desc} |`); @@ -292,19 +305,20 @@ function renderSchemaPage(schema, name, urlPrefix) { lines.push('## Types'); lines.push(''); for (const [typeName, defn] of Object.entries(localDefs)) { - lines.push(`### ${typeName.replace(/_/g, '\\_')}`); + lines.push(`### ${typeName.replace(/_/g, '\\_')} {#${anchorId(typeName)}}`); lines.push(''); if (defn.description) { lines.push(defn.description); lines.push(''); } + renderEnumValues(defn, lines); const props = collectProperties(defn); if (Object.keys(props).length > 0) { const req = collectRequired(defn); lines.push('| Field | Type | Required | Description |'); lines.push('|-------|------|----------|-------------|'); for (const [field, prop] of Object.entries(props)) { - const type = resolveTypeName(prop); + const type = resolveTypeName(prop, knownTypes); const isReq = req.has(field) ? '**yes**' : 'no'; const desc = (prop.description || '').replace(/\n/g, ' ').replace(/\|/g, '\\|'); lines.push(`| \`${field}\` | ${type} | ${isReq} | ${desc} |`); @@ -336,24 +350,31 @@ function renderSchemaPage(schema, name, urlPrefix) { const shortId = embeddedId .replace('https://mitre.github.io/hdf-libs/schemas/primitives/', '') .replace(/\/v\d+\.\d+\.\d+$/, ''); - lines.push(`### ${shortId}`); + // Explicit, namespaced anchor for the group heading. A single-type + // primitive whose file slug equals the type's kebab name (e.g. + // affected-package / Affected_Package) would otherwise auto-slug this + // heading to the same id as the child type's `{#anchorId}` below and + // fail the VitePress build on a duplicate id. Cross-links target the + // type anchors, not this one. + lines.push(`### ${shortId} {#primitive-${shortId}}`); lines.push(''); const innerDefs = embedded.$defs || {}; if (Object.keys(innerDefs).length > 0) { for (const [typeName, defn] of Object.entries(innerDefs)) { - lines.push(`#### ${typeName.replace(/_/g, '\\_')}`); + lines.push(`#### ${typeName.replace(/_/g, '\\_')} {#${anchorId(typeName)}}`); lines.push(''); if (defn.description) { lines.push(defn.description); lines.push(''); } + renderEnumValues(defn, lines); const props = collectProperties(defn); if (Object.keys(props).length > 0) { const req = collectRequired(defn); lines.push('| Field | Type | Required | Description |'); lines.push('|-------|------|----------|-------------|'); for (const [field, prop] of Object.entries(props)) { - const type = resolveTypeName(prop); + const type = resolveTypeName(prop, knownTypes); const isReq = req.has(field) ? '**yes**' : 'no'; const desc = (prop.description || '').replace(/\n/g, ' ').replace(/\|/g, '\\|'); lines.push(`| \`${field}\` | ${type} | ${isReq} | ${desc} |`); @@ -392,31 +413,6 @@ function semverCompareDesc(a, b) { return patb - pata; } -function resolveTypeName(prop) { - if (!prop) return 'any'; - if (prop.$ref) { - const ref = prop.$ref; - if (ref.startsWith('#/$defs/')) return `\`${ref.replace('#/$defs/', '')}\``; - const parts = ref.split('/'); - const typePart = parts[parts.length - 1]; - return `\`${typePart}\``; - } - if (prop.const) return `\`"${prop.const}"\``; - if (prop.enum) return prop.enum.map(v => `\`"${v}"\``).join(' \\| '); - if (prop.type === 'array') { - const itemType = prop.items ? resolveTypeName(prop.items) : 'any'; - return `${itemType}[]`; - } - if (prop.type === 'object' && prop.additionalProperties) { - const valType = resolveTypeName(prop.additionalProperties); - return `Map`; - } - if (prop.oneOf) return prop.oneOf.map(resolveTypeName).join(' \\| '); - if (prop.anyOf) return prop.anyOf.map(resolveTypeName).join(' \\| '); - if (prop.type) return `\`${prop.type}\`${prop.format ? ` (${prop.format})` : ''}`; - return 'any'; -} - function collectProperties(defn) { const props = { ...(defn.properties || {}) }; if (defn.allOf) { diff --git a/site/package.json b/site/package.json index e0cb6ac77..684aa1f1d 100644 --- a/site/package.json +++ b/site/package.json @@ -3,10 +3,11 @@ "version": "2.0.0", "private": true, "scripts": { - "generate": "node generate-schema-docs.mjs", + "generate": "node generate-schema-docs.mjs && node generate-converters.mjs", "dev": "pnpm generate && vitepress dev", "build": "pnpm generate && vitepress build", - "preview": "vitepress preview" + "preview": "vitepress preview", + "test:ts": "node --test" }, "dependencies": { "@mitre/hdf-schema": "workspace:^" diff --git a/site/schema-render.mjs b/site/schema-render.mjs new file mode 100644 index 000000000..9a217ab24 --- /dev/null +++ b/site/schema-render.mjs @@ -0,0 +1,68 @@ +// Pure rendering helpers for the schema-docs generator (generate-schema-docs.mjs), +// extracted so they can be unit-tested without executing the generator's +// top-level file I/O. + +// anchorId produces a deterministic, VitePress-independent heading id for a +// named schema type. The generator emits `### Name {#}` on the +// definition and links `$ref` cells to `#`, so both sides always +// agree regardless of VitePress's internal slugify (which we deliberately do not +// depend on). +export function anchorId(name) { + return String(name) + .toLowerCase() + .replace(/[\s_]+/g, '-') + .replace(/[^a-z0-9-]/g, '') + .replace(/-+/g, '-') + .replace(/^-|-$/g, ''); +} + +// renderEnumValues appends an "allowed values" block for a named enum def. Named +// enum defs carry no object properties, so without this they render as just a +// heading + description — hiding that they are enums and which values are legal. +export function renderEnumValues(defn, lines) { + if (!defn || !Array.isArray(defn.enum)) return; + const base = defn.type ? `\`${defn.type}\`` : '`string`'; + lines.push(`**Enum** (${base}) — allowed values:`); + lines.push(''); + for (const value of defn.enum) { + lines.push(`- \`"${value}"\``); + } + lines.push(''); +} + +// refTypeName extracts the target type name from a $ref, local (`#/$defs/X`) or +// embedded-primitive (`https://…#/$defs/Y`). +function refTypeName(ref) { + if (ref.startsWith('#/$defs/')) return ref.replace('#/$defs/', ''); + const parts = ref.split('/'); + return parts[parts.length - 1]; +} + +// resolveTypeName renders a property's type for a doc table cell. When the type +// is a $ref to a type rendered on the same page (its name is in knownTypes), it +// becomes a clickable intra-page anchor link; otherwise it degrades to plain +// code — never a broken link. Array/oneOf/anyOf recurse, so wrapped refs are +// linked too. knownTypes is a Set of the type names that have a heading on the +// current page. +export function resolveTypeName(prop, knownTypes = new Set()) { + if (!prop) return 'any'; + if (prop.$ref) { + const name = refTypeName(prop.$ref); + if (knownTypes.has(name)) return `[\`${name}\`](#${anchorId(name)})`; + return `\`${name}\``; + } + if (prop.const) return `\`"${prop.const}"\``; + if (prop.enum) return prop.enum.map((v) => `\`"${v}"\``).join(' \\| '); + if (prop.type === 'array') { + const itemType = prop.items ? resolveTypeName(prop.items, knownTypes) : 'any'; + return `${itemType}[]`; + } + if (prop.type === 'object' && prop.additionalProperties) { + const valType = resolveTypeName(prop.additionalProperties, knownTypes); + return `Map`; + } + if (prop.oneOf) return prop.oneOf.map((p) => resolveTypeName(p, knownTypes)).join(' \\| '); + if (prop.anyOf) return prop.anyOf.map((p) => resolveTypeName(p, knownTypes)).join(' \\| '); + if (prop.type) return `\`${prop.type}\`${prop.format ? ` (${prop.format})` : ''}`; + return 'any'; +} diff --git a/site/test/schema-render.test.mjs b/site/test/schema-render.test.mjs new file mode 100644 index 000000000..deac91d5b --- /dev/null +++ b/site/test/schema-render.test.mjs @@ -0,0 +1,77 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { resolveTypeName, anchorId, renderEnumValues } from '../schema-render.mjs'; + +test('anchorId slugifies type names deterministically (underscore -> hyphen, lowercased)', () => { + assert.equal(anchorId('Content_Type'), 'content-type'); + assert.equal(anchorId('External_Evidence_Reference'), 'external-evidence-reference'); + assert.equal(anchorId('Checksum'), 'checksum'); +}); + +test('resolveTypeName linkifies a local $ref to a known on-page type', () => { + const known = new Set(['Content_Type']); + assert.equal( + resolveTypeName({ $ref: '#/$defs/Content_Type' }, known), + '[`Content_Type`](#content-type)', + ); +}); + +test('resolveTypeName linkifies an embedded-primitive $ref to a known type', () => { + const known = new Set(['Checksum']); + assert.equal( + resolveTypeName( + { $ref: 'https://mitre.github.io/hdf-libs/schemas/primitives/common/v3.4.0#/$defs/Checksum' }, + known, + ), + '[`Checksum`](#checksum)', + ); +}); + +test('resolveTypeName falls back to plain code when the ref target is not on the page', () => { + assert.equal(resolveTypeName({ $ref: '#/$defs/Nowhere' }, new Set()), '`Nowhere`'); +}); + +test('resolveTypeName links array item refs via recursion', () => { + const known = new Set(['Content_Reference']); + assert.equal( + resolveTypeName({ type: 'array', items: { $ref: '#/$defs/Content_Reference' } }, known), + '[`Content_Reference`](#content-reference)[]', + ); +}); + +test('resolveTypeName links oneOf member refs via recursion', () => { + const known = new Set(['Signature']); + assert.equal( + resolveTypeName({ oneOf: [{ $ref: '#/$defs/Signature' }, { type: 'string' }] }, known), + '[`Signature`](#signature) \\| `string`', + ); +}); + +test('resolveTypeName renders inline enums as values (unchanged behavior)', () => { + assert.equal(resolveTypeName({ enum: ['a', 'b'] }, new Set()), '`"a"` \\| `"b"`'); +}); + +test('resolveTypeName renders a scalar type with format (unchanged behavior)', () => { + assert.equal( + resolveTypeName({ type: 'string', format: 'uri-reference' }, new Set()), + '`string` (uri-reference)', + ); +}); + +test('renderEnumValues emits the allowed-values block in exact order', () => { + const lines = []; + renderEnumValues({ type: 'string', enum: ['x', 'y'] }, lines); + assert.deepEqual(lines, [ + '**Enum** (`string`) — allowed values:', + '', + '- `"x"`', + '- `"y"`', + '', + ]); +}); + +test('renderEnumValues emits nothing for a non-enum def', () => { + const lines = []; + renderEnumValues({ type: 'object' }, lines); + assert.equal(lines.length, 0); +});