[P1] Make monitoring delivery and telemetry coverage an explicit production contract

[Git-on-my-level]

Problem

Omi’s current Grafana alert set has useful service rules, but the monitoring and delivery path is not itself trustworthy enough to make outages loud and obvious.

Read-only production review on 2026-07-12 found:

1. All 54 Grafana alert rules explicitly select Omi - Services Alerting (Telegram). Notification policy routes exist for instatus_component, but explicit rule receivers appear to bypass those label-based Instatus routes. A public status component may therefore not update during a confirmed incident.
2. Prometheus scrape health is materially degraded: kubelet, CoreDNS, GPU, and service metric targets include large down/unknown sets. No service-rule inventory item protects expected target coverage for critical exporters.
3. backend-sync lacks reliable availability/completion coverage: no traffic-zero/readiness/queue-age/completion SLI. Its semantic fallback metric path is paused because backend-sync metrics are not scraped by GKE Prometheus.
4. The prior Stackdriver 5xx NoData -> Alerting false-positive was addressed by fix(monitoring): resolve 5XX alert false positives from Stackdriver no-data #9567, but this clarified a broader rule: datasource/query absence, expected low-volume event absence, and customer-service failure require different semantics.
5. Several availability rules use metric or vector(0), which can intentionally or accidentally conflate absent telemetry with a zero-valued product signal.

This issue complements existing #9138, which owns persistent managed-GKE control-plane/rule noise. It focuses on the product-monitoring delivery/coverage contract and should link to, not duplicate, #9138.

Scope

Make monitoring a monitored dependency: explicit alert-routing behavior, expected telemetry coverage, documented no-data semantics, and customer-facing delivery verification. This issue does not require inventing every product SLI; transcription and sync semantic metrics are tracked separately.

Proposed deliverables

1. Alert routing and delivery contract

• Decide whether component alerts route through policy labels, rule-level receivers, or both; encode one authoritative model.
• Ensure confirmed critical incidents reach both an accountable paging/escalation path and the intended Instatus component.
• Retain Telegram broadcast, but do not let it be the only ownership/escalation path for P1 conditions.
• Add controlled, non-customer-impacting alert tests for every component route: API, live transcription, speech processing, plugins, and AI/chat.
• Document grouping, deduplication, acknowledgement/escalation expectations, and test evidence.

2. Monitoring-pipeline health and expected-target inventory

Create a declared inventory of critical telemetry sources and expected coverage for at least:

• backend-listen metrics;
• pusher metrics;
• Parakeet metrics;
• kube-state-metrics;
• Stackdriver/Cloud Monitoring datasource or exporter;
• GPU metrics where they are required for speech processing.

Add actionable coverage alerts for absent/insufficient expected targets. Separately classify known managed-GKE control-plane scrape limitations under #9138; do not page a generic TargetDown for expected unreachable endpoints.

3. Explicit no-data semantics

For each critical rule, document and test one of:

• NoData means service/telemetry outage and must page;
• NoData means event absence is expected and must be OK;
• a separate telemetry-health alert represents missing data while the service rule remains outcome-based.

Do not use or vector(0) to imply an actual zero product signal unless absence-as-zero is intentional, tested, and annotated. Separate exporter absence, readiness loss, and traffic/session anomaly rules.

4. Backend-sync operational coverage

Add a deliberate supported metric/log-based source for backend-sync acceptance/completion and operational backlog: queue age/depth, retry/exhaustion, oldest nonterminal work, readiness/revision health, and traffic where meaningful. It may be Cloud Tasks/Cloud Monitoring/log-based rather than GKE Prometheus, but must be a verified maintained path.

5. Runbook and verification

Create a concise public-safe monitoring runbook with:

• how to distinguish product outage, datasource outage, and expected idleness;
• alert route test procedure;
• critical target inventory and owner;
• dashboard/runbook links carried in alerts;
• response when the monitoring pipeline itself is degraded.

Acceptance criteria

• A controlled alert verifies delivery to the intended Instatus component and the accountable escalation/broadcast path for every component class.
• Rule-level receiver overrides and notification-policy behavior are documented and tested; routing cannot silently bypass component status updates.
• Critical telemetry sources have expected-target/coverage checks, with known managed-GKE exclusions explicitly classified rather than ignored.
• Missing datasource/exporter data is distinguishable from an application error or customer outage.
• Every critical availability rule has documented no-data behavior and avoids unintentional absence-as-zero coercion.
• Backend-sync has an actionable, maintained operational completion/backlog signal even without GKE Prometheus scraping.
• Existing prod monitoring has persistent false-positive control-plane alerts and severity noise #9138 acceptance work is reused; no duplicate control-plane-noise solution is introduced.
• Alert tests and runbook are safe to execute without customer impact and do not expose tokens, private webhook endpoints, or user data.

Related

• prod monitoring has persistent false-positive control-plane alerts and severity noise #9138 — managed-GKE control-plane/scrape/rule false positives and severity noise.
• feat(monitoring): comprehensive alerting overhaul — severity, cleanup, traffic-zero, instatus #9554 / fix(monitoring): align alert queries to dashboard panels #9562 / feat(monitoring): add backend-listen WS connection drop alert #9565 — recent monitoring rule overhaul and dashboard alignment.
• fix(monitoring): resolve 5XX alert false positives from Stackdriver no-data #9567 — Stackdriver 5xx no-data false-positive correction.
• Fix HOSTED_PARAKEET_API_URL: https → http for Cloud Run services #9548 — silent functional transcription outage motivating customer-visible alert delivery.

---

Safety / agent notes

• Do not include Grafana bearer tokens, webhook URLs/secrets, private dashboard links, credentials, customer identifiers, audio, or transcript content in code, fixtures, logs, or issue comments.
• Prefer dry-run/test alert labels and non-customer-facing test components for routing verification.
• Grafana/contact-point/notification-policy changes, pager configuration, public status changes, production deploys, and monitoring infrastructure mutations require separate maintainer approval.

[Git-on-my-level]

2026-07-14 read-only alert-hygiene update

Fresh Grafana API inventory returned exactly three active series:

• Backend-sync - 5XX error rate is high — critical, but state reason is NoData (active since 2026-07-13T17:26:50Z).
• Backend-integration - 5XX error rate is high — critical, but state reason is NoData (active since 2026-07-12T12:52:00Z).
• Pusher max concurrent requests are high — warning; no evaluation-error/NoData reason returned.

The two 5XX series are classified as telemetry/evaluation defects, not verified customer-impact incidents. The conservative production observer explicitly excludes Grafana NoData/query/error states from customer-impact paging, while retaining Cloud Run 5xx surge and external API schema/latency checks as independent outcome signals. No alert routing, Grafana configuration, workload, traffic, or public-status change was made.

Follow-up ownership: this issue owns version-controlled, reviewed no-data semantics and telemetry-health coverage; #9138 remains the owner for managed-GKE control-plane/scrape noise. The alert rule artifacts in this repository already declare noDataState: OK for the corresponding backend-sync 5XX rule; production drift/remaining NoData -> Alerting behavior must be reconciled through a reviewed provisioning change, not silently suppressed.

Read-only evidence only; no tokens, URLs, payloads, or customer data retained.