token-tracking.md

Token collection and usage tracking

Hive tracks token usage for local budgeting, dashboard cost estimates, and hub-level fleet rollups. The implementation lives in pkg/tokens, pkg/dashboard/cost.go, and pkg/hub/usage.go.

Collection sources

The collector starts from data.metrics_dir and rescans every 30 seconds. It merges:

• Hive JSONL session files in data.metrics_dir (*.jsonl) using the flat SessionEntry schema.
• Claude Code session JSONL under data.claude_sessions_dir when configured, plus every per-agent /data/home/agents/<name>/.claude/projects beside it (each per-UID agent's ~/.claude is a real directory since #17031); sessions in a per-agent tree are attributed to that agent when the project directory name carries no work-dir hint. It reads assistant message.usage blocks from recent files (30-day mtime window).
• Copilot CLI events.jsonl under data.copilot_sessions_dir when configured. It reads session.shutdown.modelMetrics and avoids double-counting sessions already captured live by the proxy.
• Bob CLI chat recordings under data.bob_sessions_dir (default /data/home/.bob) and every per-agent /data/home/agents/<name>/.bob beside it, when configured. It reads tmp/*/chats/*.json, maps Bob project hashes back to trusted agent folders (falling back to the owning agent of a per-agent tree), and records explicit tokens fields (falling back to content-size estimates only when a recording has no token data).
• In the optional Claude Managed Agents hosted path, pkg/tokens/managed_agents_ledger.go writes managed-agents-*.jsonl after deduplicating vendor span.model_request_end IDs. Its managed-agents-ledger.json sidecar keeps cumulative USD list cost, active seconds, and web-search counts and enforces the private-spoke USD 25/day cap.

A flat session entry can include role, agent, model, input_tokens, output_tokens, cache_read, and cache_creation. When agent is absent, Hive infers the agent from session path or configured detection keywords.

Sourcing decision: hybrid ccusage parsing with Hive attribution

Proposed — awaiting maintainer sign-off on #6234/#6242. The recommended sourcing path is option (c), a hybrid: use a pinned ccusage subprocess for the backend usage formats it already covers, while Hive keeps its attribution, budget, and price-label semantics.

The earlier coordination hazard is now moot: PR #6142, which removed dead scanner variants, has already merged to v5. That may require a small seam re-add when this work lands, but it no longer needs to block the sourcing decision.

Proposed contract:

• Hive invokes ccusage --json --offline through a narrow source interface, with the ccusage version pinned and the JSON output shape asserted by tests.
• ccusage parses the nine CLI backends it covers today, closing the rotation gap for backends whose token data currently goes dark.
• Hive keeps the attribution layer: path-to-agent mapping, session de-dupe, per-agent/per-model aggregation, budget windows, dashboard labels, and hub rollups remain Hive-owned.
• Hive keeps its price table for labeling and policy decisions. Provider-native numbers stay labeled native; table-derived numbers stay estimated; missing prices remain explicit instead of pretending to be invoices.
• Hive keeps bob_scanner.go while ccusage lacks Bob coverage, so adopting ccusage does not regress an already-metered backend.
• The companion RFC design doc is expected at src/docs/design/token-metering-ccusage.md; this page records the sourcing decision so implementation PRs have one target.

Sequencing, aligned with #6234:

1. Source seam. Add an internal interface that can return normalized usage entries from either current scanners or the ccusage subprocess.
2. ccusage implementation. Add the pinned subprocess runner using --json --offline, fixtures for representative covered backends, and tests that fail on output-shape drift.
3. Parallel-run diff. Run existing scanners and ccusage side by side for overlapping sources, logging structured deltas without changing budget input.
4. Per-backend cutover. Move one backend at a time to ccusage when the diff is understood; keep Bob on Hive's scanner until upstream coverage exists.

Stored summary

The collector keeps the latest aggregate in memory and writes /data/token-summary.json by default. The summary includes:

• total input/output/cache-read/cache-create tokens,
• per-agent and per-model totals,
• detailed per-agent/per-model token buckets,
• session count and recent session metadata.

data.metrics_dir is therefore both an input directory for Hive/inference JSONL and the durable location for per-agent inference usage files.

Dashboard and API

• /api/status includes token/cost fields used by the dashboard.
• /api/cost returns estimated cost from token counts × the static price table plus native spend for gateways that report it (OpenRouter /key, LiteLLM /key/info). Estimated rows are labelled estimated; rows whose model has no exact table entry are marked with unpriced_models and shown as tier estimates. Native gateway rows are labelled native. The estimated payload also includes window_start / window_end when session timestamps are available, so dashboard tables can state the covered usage window. A recording block reports whether the running agents' models are actually being recorded: agent_models (alias-resolved models configured on running agents), unrecorded_models (those with no usage in the current token summary), recent_models (models whose usage grew in the last 24 h), last_priced_growth_at (unix ms when a model with an exact list price last grew in the persisted cost history), last_agent_model_growth_at, stale_after_ms and stale (true when the history spans more than 24 h and none of the agents' models grew in that time; the Cost card then shows a warning and marks the run-rate "recording incomplete").
• /api/repo-activity is phase 1 of per-repo cost attribution: it reports audited output counts per repo and per (repo, agent) from repo= audit entries, plus an explicit unattributed bucket for output events with no repo. It reports activity only, not dollars; cost must not be smeared across repos until timestamped token joins exist.
• /api/repo-cost is phase 3: a cached interval join of the same audited repo= events against the per-message usage timeline, with mandatory unattributed and backend_unsupported buckets so Σ buckets == total.
• /api/launch-cost is launch attribution (#12128, RFC #12134): spend placed against the issue / PR / head SHA each kick was launched for. See Launch attribution below.
• Cost estimates are not invoices. Subscription plans, self-hosted inference, negotiated rates, and provider billing semantics can differ from list prices.

Price table refresh

The per-model list prices live in src/pkg/tokens/pricing.go (modelPrices, keyed by the normalised model id, plus priceTableDate, surfaced as price_table_date in /api/cost and as the "list prices as of" date in the Cost panel). They are kept current automatically:

• Source. OpenRouter's public model catalogue, GET https://openrouter.ai/api/v1/models (no key; per-token USD pricing.prompt / completion, plus input_cache_read / input_cache_write where published). No vendor HTML is scraped. Catalogue ids are mapped onto table keys with the same tokens.NormalizeModelID the dashboard uses, so anthropic/claude-opus-4.8 and claude-opus-4-8 are one row; :free / :batch variants, ~vendor/model-latest router aliases, zero-priced and variable-priced (-1) rows are ignored, and reasoning-effort spellings (gemini-3-7-flash-high) price at their base model.
• Cadence. .github/workflows/price-table-refresh.yml runs go run ./internal/tools/pricetable every Monday (and on workflow_dispatch). When any rate moved it rewrites only those literals, bumps priceTableDate, adds a changelog.d/changed-price-table-<date>.md fragment and opens or force-updates one PR on bot/price-table-refresh whose body is the report: every change as old → new per-million input / output / cache read / cache write, and every row OpenRouter does not list (those keep their hand-set value). The PR carries needs-human and is never auto-merged — compare the figures against the vendor's own price page before merging. After 30 quiet days the tool re-dates the table without changing a rate so the date keeps meaning "last reconciled"; v2-ci.yml fails any PR once the table is more than 60 days old (go run ./internal/tools/pricetable --check-age 60), so a refresh that stops running is noticed within weeks.
• Cache-write convention. Anthropic/OpenAI publish a cache-write premium at or above the input rate, which the table takes as-is. Google publishes a per-hour storage rate instead; it has different semantics, so rows that followed the header convention (cache write = input) keep following it and the report notes the skipped figure.
• List price is a ceiling. The estimate multiplies persisted token history by these rates. Subscription seats (Claude Max, Copilot premium requests), self-hosted inference and negotiated rates all bill less than list, so the Cost panel's figure is an upper bound on per-token spend, not a bill.

Launch attribution

The kick path records a LaunchRecord (pkg/tokens/launch.go) each time the governor or the review swarm delivers a kick: agent, trigger (governor-eval, review-<kind>), cadence-target repo, and the work refs the kick named. Governor kicks carry the issue refs placed in the message (owner/repo#N, or the verbatim external key for Linear/Jira work); review-swarm kicks carry one PR ref with its head SHA. Records are appended to /data/launch-attribution.jsonl (bounded ring, compacted in place) and attached to the token collector (Collector.SetLaunchLog / RecordLaunch / Launches).

pkg/dashboard/collect/launch_cost.go joins those records against the same UsageEvent timeline /api/repo-cost uses, with the same pricing table and the same honesty rules:

• A launch is an opening event: usage is owned by the most recent recorded launch for the same agent, until the next launch or LaunchOwnershipMax (6h), whichever comes first. Usage before an agent's first retained launch, past the horizon, or outside the repo-cost window (14 days) is unattributed; backends without a per-message timeline land in backend_unsupported. Σ by_launch + unattributed + backend_unsupported == total.
• by_ref aggregates every launch that named a ref. launches is the repeat-spend counter (how many kicks named it, including kicks that produced no usage), head_shas lists the distinct PR heads launched against, launch_tokens / launch_usd are the undivided totals of every launch naming the ref (upper bound), and share_tokens / share_usd split each launch evenly across the refs it named (exact for single-PR review kicks).
• USD is null when nothing was attributed — never $0.00.
• ?ref=owner/repo#N narrows by_ref and by_launch to one work item; totals stay whole-window.

The dashboard Cost panel renders the top rows as "cost by work item". Manual send-kick and startup prompts do not record launches, so their spend stays unattributed. This is visibility only: nothing here gates or budgets a kick, and the governor window budget remains the single enforced budget.

Metering diagnostics and zero-token health

The collector keeps a non-secret Diagnostics snapshot alongside the aggregate (pkg/tokens, Collector.Diagnostics()), so consumers can distinguish "zero tokens because nothing used a model" from "zero tokens because metering itself is unhealthy":

• last_scan_error — the flat-JSONL scan of data.metrics_dir failed. This aborts the whole scan cycle: the previous aggregate is kept and no other field is refreshed until the next cycle.
• last_claude_scan_error, last_copilot_scan_error, last_bob_scan_error — a per-source scan failed. Only that source is skipped for the cycle; the other sources still merge. Each field is rebuilt every scan, so a source error clears on the next successful pass over that source.
• live_capture_enabled — the Copilot proxy's live usage capture is active (set at boot via SetCopilotLiveCapture).

The deep-health tokens check sent in spoke heartbeats (HealthSummary in pkg/dashboard/server.go) uses this to replace the bare "zero consumed" warning with a reason, evaluated in precedence order:

Status	Detail	Meaning
skip	zero consumed — all agents paused	Expected quiet: every enabled agent is paused.
skip	zero consumed — no agents due in the current governor mode	Expected quiet: only on-demand or off-schedule agents remain.
warn	zero consumed — token parser/sink error: <err>	Flat-JSONL scan failing (then Claude/Copilot/Bob parser errors, each named).
warn	zero consumed — sessions still open or live-capture has not accounted usage yet	Live capture is on and sessions exist; usage may land shortly.
warn	zero consumed — sessions made no model calls	Sessions were scanned but recorded no token usage.
warn	zero consumed — token sink or live metering disabled/misconfigured	A metered agent is running with live capture off — check metering wiring.
warn	zero consumed — no model calls recorded	Fallback when no better explanation applies.

On the hub's fleet page, the tokens column tooltip reads that check's detail out of the heartbeat health checks and renders No tokens used: <reason> for hives at or below the no-token threshold (tokenHealthReason / tokenUsageTitle in pkg/hub/saas.go).

Caveat: the reasoned detail is heartbeat-path only. The local /api/health/deep tokens check still emits the plain zero tokens consumed — agents may not be working warning without consulting Diagnostics.

See health-checks.md for the operator-facing summary of the agent and token deep-health checks.

Hub rollups

Each spoke heartbeat sends one scalar tokens_24h value. Despite the historical name, current code sends the cumulative total from the spoke's token summary. The hub stores it as totalTokens24h, samples a 7-day fleet history every 15 minutes, and serves /api/saas/usage with rollups by org/repo, owner, and cluster plus zero-consumption hives.

The hub cannot compute dollars or per-model/per-agent attribution because the heartbeat does not include model or agent token splits. Use the individual spoke's /api/cost for those details.