[Diags]: Add provenance scanner for diagnostics links

[tomvothecoder]

Goal

Provide an operational path for SimBoard to discover and link diagnostics published by zppy's SimBoard-compatible publishing mode. This follows the broader diagnostics-link MVP in #174.

Scope

• Discover newest paired zppy provenance from diagnostics archives selected by the checked-in DIAGNOSTICS_ARCHIVES_BY_MACHINE static site registry under backend/app/scripts/ingestion/.
  • Registry maps accepted SimBoard machine names and aliases to a complete archive filesystem root and matching public archive base URL.
  • Seed or refresh the registry during development from Mache mache/machines/*.cfg files with usable [web_portal] values. SimBoard does not fetch or parse Mache at scanner runtime.
  • Scan bounded production/ and development/ trees, with optional case-group directories, without following symlinks outside the configured archive root.
• Select the newest valid timestamped provenance.<timestamp>.cfg / provenance.<timestamp>.settings pair per archive case directory. Do not fall back to older provenance when the newest pair is incomplete.
• Parse case identity and authoritative diagnostics_url only from the selected .settings file. Validate the settings format, archive layout, and URL scheme, authority, and archive-path boundary.
• Treat published output presence as readiness: require a non-provenance diagnostic artifact or non-empty diagnostic subdirectory. Do not use zppy status files; they are not published archive artifacts and do not reliably identify one provenance run.
• Persist scanner state centrally in SimBoard. A scanner-specific internal API must atomically upsert the case-scoped diagnostic link and successful provenance state.
  • State is keyed by canonical machine and archive-relative case-directory path.
  • It records selected settings filename, timestamp, fingerprint, URL, and successful submission time.
  • State is linked to its scanner-managed ExternalLink with cascade deletion. Removing that link invalidates state; a later scan can recreate a still-published link.
• Keep repeated runs safe: skip unchanged successful state, retry output-not-ready or transient failures, and use database constraints/transactions for concurrent scanner safety.
• Add focused tests and operational documentation.

zppy Publishing Contract

See zppy PR #841.

• zppy adds an opt-in [simboard] configuration section. It is a configuration hook, not an HPC job.

• When simboard.enabled = True and [default] www is unset, zppy derives its archive location from the current machine's Mache web_portal.base_path.

• Published diagnostics are organized as:

  <web_portal_base_path>/diagnostics_archive/<simulation_type>/<case_group>/<case>/
  

  simulation_type distinguishes development from production; <case_group>/ is optional.

• An explicitly configured [default] www remains authoritative.

• For non-dry-run executions, zppy writes timestamped paired provenance files beside published diagnostics. The cfg establishes the timestamped pair; settings carry case_name, machine, hpc_username, optional case_group, and diagnostics_url.

• Multiple pairs may exist for one case after repeated zppy runs. SimBoard selects the newest pair by filename timestamp. A newer pair updates central provenance state; the stable URL upsert avoids duplicate links.

• Development and production archive paths may create separate diagnostic links for one SimBoard case. The scanner does not remove obsolete links; operators remove them manually.

  Example:

  global_time_series/
  provenance.20260805_234735_275469.cfg
  provenance.20260805_234735_275469.settings
  provenance.20260805_234749_729830.cfg
  provenance.20260805_234749_729830.settings
  

• SimBoard treats zppy's provenance and archive layout as the upstream publishing contract. It does not reconstruct zppy behavior or crawl public HTTP directory listings.

API and Storage Boundary

• Preserve the existing POST /api/v1/diagnostics/link contract.
• Add scanner-specific internal state-read and link endpoints for service accounts.
• Add database-backed DiagnosticProvenanceState with a migration and transactional link/state persistence.
• Keep diagnostics linkage semantics in catalog models/APIs; keep the site-side scanner under backend/app/scripts/ingestion/.

Out of Scope

• Changes to zppy publishing behavior or provenance contract.
• Mache runtime dependency or runtime configuration retrieval.
• Historical backfill beyond normal scanner operation.
• Changes to the existing diagnostics-link endpoint contract or frontend.
• Diagnostics content ingestion or interpretation.