Add build-monitor-service-hook-deliveries alert to source control#6610
Add build-monitor-service-hook-deliveries alert to source control#6610missymessa wants to merge 4 commits into
Conversation
The existing Grafana alert (UID: build-monitor-service-hook-deliveries) was querying for 'ServiceHookNotificationStatus' custom events that no longer exist in the codebase. This caused false-positive alerts via noDataState. Updated the alert to query actual HTTP request telemetry from the DotNetEng-Status App Insights instance for the POST AzurePipelines/BuildComplete endpoint, which is the real webhook handler for AzDO service hook deliveries. Changes: - Query: customEvents/ServiceHookNotificationStatus -> requests/BuildComplete - noDataState: Alerting -> OK - Time bins: \ -> explicit 15m - Data source: dotnet-eng -> DotNetEng-Status-Prod (parameterized) Fixes: https://dev.azure.com/dnceng/internal/_workitems/edit/10629 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
What's up with the formatting and text of the description? It seems like it has errors. |
garath
left a comment
garath
left a comment
There was a problem hiding this comment.
Hm, I'm confused about this PR.
Adds the �uild-monitor-service-hook-deliveries Grafana alert rule to source control (Production + Staging).
This alert is already codified over in historical.dashboard.json. It's shown in the Grafana dashboard here and its currently showing squiggles of success, which indicates the alert is working.
The existing Grafana alert was querying for
ServiceHookNotificationStatuscustom events that no longer exist in the codebase — the telemetry was removed at some point but the alert was never updated.
That event is generated over in MetricsObserver. I checked the App Insights and see that it is still recording events.
Am I missing something? What is this PR fixing?
|
Closing this PR. On investigation the premise doesn't hold:
This PR also duplicates the existing alert (same uid/panelId/NotificationId) in a second repo/deploy path, and repoints to If we want to reduce noData fragility, the lower-risk follow-up is to tune the existing rule in dotnet-helix-service rather than add a parallel one here. |
2 participants
Summary
Adds the �uild-monitor-service-hook-deliveries Grafana alert rule to source control (Production + Staging).
What changed
The existing Grafana alert was querying for \ServiceHookNotificationStatus\ custom events that no longer exist in the codebase — the telemetry was removed at some point but the alert was never updated. It only fired via
oDataState: Alerting\ (false positive when no data flows), not from actual failure detection.
Updated the alert to:
equests\ table) from DotNetEng-Status App Insights for the \POST AzurePipelines/BuildComplete\ endpoint
oDataState\ from \Alerting\ to \OK\
Production
The alert has already been updated in production Grafana via the API. This PR tracks the change in source control so it's deployed consistently.
Fixes: https://dev.azure.com/dnceng/internal/_workitems/edit/10629