A small, efficient system that aggregates subsystem-owned now into coherent, durable, AI-readable history.
snapper-ai turns operational state into operational memory. It is a snapper of histories: subsystems publish their present state, and snapper-ai records that state as a sequence of coherent system-wide snaps.
The -ai suffix names the primary consumer: an AI charged with reasoning and inference across system histories. The deterministic, AI-free capture path of snapper-ai provides operational memory that AIs can retrieve and reason over directly. Pages, plots, reports, and incident reviews use the same history.
Operational systems usually know what is happening now and record some exact events. They rarely preserve a direct answer to a simpler question: what did the whole system look like at an earlier time?
Reconstructing that answer later can require expensive database queries, correlation across unrelated records, or facts that were never recorded with usable lifetimes. Recovering the PanDA jobs running at a past instant, for example, requires counting jobs whose start time precedes it and whose end time follows it. Many other kinds of state have no recoverable history at all.
snapper-ai records history while the present state is already known. It turns historical state from an inference problem into a retrieval problem.
The package is the whole product — store, capture, queries, series
assembly, views, and templates — and is experiment-agnostic throughout:
it imports no host models and knows no experiment vocabulary. A host
application supplies the specifics by registration (snapper_ai.registry):
one ScopeProvider per scope (curve extraction from snap state, curve
labels and families, component card builders with a host card template,
episodic activity lanes, reference resolution) and host-wide hooks
(user preferences, configuration values, scheduler status, health page
URL). Every registration is optional, with generic fallbacks — an
unregistered component renders as a quantity table. A deployment
outside the original host needs a base.html template, a
path('snapper/', include('snapper_ai.urls')) mount, one provider
registration per scope, and a feed into snapper_ai.capture. The swf
platform's providers live in swf-monitor
(monitor_app/snapper_providers.py), registered from the host
AppConfig.ready(). The host integration guide, with the swf
integration as its worked example, is
docs/INTEGRATION.md.
The immediate catalyst was the ePIC E0-E1 global-state model in the streaming workflow testbed. Its vertical cut through concurrent state lanes is the global state at an instant. Following those cuts through time is the operational memory that snapper-ai records.
The first concrete lanes are the datataking states of the independently namespaced testbeds sharing that platform. The application also serves the PanDA workload-management ecosystem and the epicprod production system, where production progress and distributed processing activity are important to the collaboration, valuable in diagnosis, and rich input for AI reasoning and reporting. The design therefore broadened into an application-agnostic service, with ePIC/SWF and PanDA as its first integration.
subsystem owns and maintains its now
|
| publish complete component state
v
current component registry
|
| aligned capture decision
v
immutable system snap history
|
v
people, applications, and AI agents
Each subsystem owns a bounded JSON component describing its current state. Its registration supplies the shape, meaning, ownership, units, freshness, and provenance needed to interpret that state. The mutable component registry holds the latest accepted value from every owner.
Registration begins with a concrete retrieval need: state that consumers want back, in a representation and temporal granularity that will be useful when returned.
A submitted component state must be curated so that every change within it is a change worth recording. The owner chooses the projection and its resolution; snapper-ai then treats every canonical change as meaningful. Publishing a raw, fast-changing gauge is therefore an explicit choice to drive snaps at that rate.
After an assessment, the owner either publishes a new curated state or reports that it has nothing new. A no-change report refreshes assessment and liveness metadata while leaving component content and revision unchanged. A snap at every opportunity is also valid when that is the history consumers want.
At fixed, clock-aligned opportunities, snapper-ai checks a small revision vector. A canonical component change makes a snap due. A periodic baseline records the system even through long quiet periods. Quiet opportunities end before JSON assembly, keeping the system inexpensive when little is happening.
A system snap is one immutable, versioned JSON structure containing a stable cut across the registered components in a scope. Event streams retain exact transitions; snaps retain coherent sampled state.
Coherence means a consistent read of the component registry. Components may have been assessed at different times, so every component carries its own assessment time and, when applicable, source time.
snapper-ai provides authoritative system state across time, giving people and AIs direct knowledge of how complex systems evolve.
Five properties define the product:
- Owner-published state. Every component comes from the subsystem that maintains it.
- Coherent capture. Every snap is a stable system-wide cut across those independently maintained states.
- Efficient history. Cheap aligned checks, change-driven snaps, and periodic baselines fit both active and quiet systems.
- Evolutionary robustness. Component schemas and capture practice evolve under explicit versions and policy epochs; old history remains interpretable.
- AI-native retrieval. AIs receive authoritative temporal context with timestamps, hashes, provenance, and coverage status.
Capture stays deliberately simple: generic JSON, explicit metadata, content hashes, bounded component state, and straightforward database ordering.
The core retrieval surface answers questions such as:
- What is the latest recorded system state?
- What was the state at this time?
- What changed between these moments?
- How did this component evolve over the last day?
- What was pending when an incident began?
- Is a component's current state different from the latest recorded state?
The corresponding operations are latest, state at, changes between, component history, and context around. Results include actual snap and source times, schema and policy versions, provenance, and observer-coverage status. Exact transition sequences remain available from event streams joined around a snap.
The first registration set is intentionally small:
| Scope | Initial components |
|---|---|
| both | system health and assessment freshness |
| testbed | datataking state by namespace, workflow activity, agent state, data activity |
| epicprod | PanDA activity, production and campaign state, operational state |
The first high-value epicprod projection is current PanDA activity: jobs and tasks by state, running jobs and cores, active task types, sites, and recent outcomes. Recording that maintained now supplies concurrency history that is otherwise expensive or impossible to recover later.
The report page for each scope, as deployed on the SWF monitor and served publicly: epicprod · testbed.
The epicprod report page: a 24-hour window of PanDA activity. Stacked curve panels show in-flight jobs and tasks by state, with per-type and type-by-state curve selections and the health lane above the panels.
The testbed report page: a seven-day window of per-namespace activity lanes carrying workflow datataking and processing tiles, with the health lane and, below the tracks, the time cut rendered as per-run component cards.
Any host page can embed a scope's curve families as a compact, read-only rendering of the same recorded history: the report page's stacked panels without lanes, controls, or preferences, with matching curve colors and coverage-gap spans, and a click anywhere on the plot opening the full report page at the matching window. The host supplies one context call and one template include (docs/INTEGRATION.md). The SWF monitor's PanDA activity overview embeds the epicprod jobs and tasks families over the page's own time-window selection:
The epicprod jobs and tasks curve families embedded on the PanDA activity overview, following the page's selected window.
snapper-ai is packaged as a reusable Django application backed by PostgreSQL. It contains no SWF, PanDA, testbed, or epicprod domain model. A hosting Django project supplies runtime settings, database connection, routing, authentication, logging, migrations, and process deployment.
The initial SWF application installs this generic package into the existing SWF monitor runtime and uses its swfdb PostgreSQL database. That deployment is an integration choice defined in SWF_EPICPROD_INTEGRATION.md, not part of the generic Snapper product contract.
The execution plan is the authoritative roadmap and progress record. The implementation record preserves durable decisions and their code and integration consequences.
The development workflow defines the required branch boundary between snapper-ai and its SWF core-repository integrations, shared checkout safety, and the coordinated-baseline deployment rule.
The technical design defines registration, publication, capture, snap representation, evolution, query semantics, AI evidence, efficiency, and the implementation boundary.
The SWF and epicprod integration identifies the deployment contracts that connect the generic service to its first users.
The original detailed exploration remains available as a superseded early design.
snapper-ai is licensed under the Apache License 2.0.