Add hedge-trigger, final-outcome, attempted-payload and excluded-leader metrics by koloss2001 · Pull Request #87 · grafana/warpstream-go

koloss2001 · 2026-09-23T20:21:35Z

Summary

Since v0.1.1 wgo has gained several bug fixes and behaviour changes, including:

a fix for a shared hedge flush crediting records to the wrong caller (fix(wgo): don't credit a shared hedge flush or a failed split ProduceSync chunk. #92)
a fallback to a live agent when a partition's leader is missing from metadata (Fall back to a live agent when a partition's leader is unknown #88, Fall back when a topic loses every leader, count the drop, and refresh again #96)
backoff for on-demand metadata refreshes while a leader stays excluded (Back off on-demand Metadata refreshes while a leader stays excluded #98)
rejecting only the unroutable partitions of a ProduceSync call (Reject only the unroutable partitions in a ProduceSync call #100), and out-of-range partitions (Reject an out-of-range partition instead of guessing a fallback agent #107)

The existing metrics cannot show how these paths behave. This PR adds metrics to quantify the changes, monitor the new behaviour and troubleshoot it, fine-tune wgo, and measure how well it mitigates WarpStream issues.

It does not change routing, hedging, timing or retry behaviour.

Metric	Labels	What it tells you
`warpstream_produce_hedge_triggers_total`	`trigger`: `latency`, `primary_failure`, `demoted_probe`	What starts extra hedging.
`warpstream_produce_hedge_trigger_wins_total`	`trigger`	How often each trigger's hedge wins. Wins / triggers is the win rate.
`warpstream_produce_requests_failed_total`	`reason`: `candidates_exhausted`, `terminal_error`, `write_timeout`, `internal_error`	Why a Hedger call failed. Counted once per failed call, not per `Produce` call or wire request. Successes and cancellations are not counted; successes are in `warpstream_produce_requests_attempts`.
`warpstream_agentpool_agents_changed_total`	`direction`: `added`, `removed`	Agents added or removed by a metadata refresh.
`warpstream_agentpool_excluded_leaders` (gauge)	none	Partitions whose leader is currently excluded. Goes back to 0 when it clears.
`warpstream_produce_attempt_records_total`, `warpstream_produce_attempt_bytes_total`	`attempt`: `primary`, `hedge`	Records and compressed bytes sent, including failed and canceled attempts.

Things to know

warpstream_hedge_wins_total now counts only wins that were actually returned. Before, a fallback that finished but lost to the primary could still count. The existing hedge win-ratio panel will need to be updated.
excluded_leaders replaces warpstream_agentpool_leader_dropped_total. The old counter is not in v0.1.1 or any deployed version. Its rate depended on how often we refresh, so it fell as the backoff slowed refreshes, which looked like recovery.
Failure-reason order: terminal error first (from the primary or a retry), then work deadline, then candidates exhausted. Cancellation is not counted. A deadline that has been reached counts as expired even if its timer has not fired yet.

cursor

Stale Bugbot comment from a previous run.

stephclay · 2026-09-24T01:45:23Z

Missing doc update

This PR adds 3 new metrics (warpstream_produce_hedge_triggers_total, warpstream_produce_final_outcome_total, warpstream_agentpool_agents_changed_total) but does not update docs/internal/metrics.md. pkg/AGENTS.md requires this doc to stay current with metric changes.

— via Claude Code

koloss2001 · 2026-09-24T22:12:50Z

on the doc update:
This was resolved in #68 : metrics.md documents parity/category design, not individual custom metrics.
I clarified AGENTS.md accordingly.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 16669d5. Configure here.}

stephclay · 2026-09-30T15:06:38Z

Pre-existing gap: routing-mismatch guard skips final-outcome metrics

pkg/wgo/hedger.go:144-148 (unchanged by this PR, already on main) returns early on a routing mismatch, before any of the attempt/final-outcome metrics closures run. If this guard ever triggers, the failure is invisible in warpstream_produce_final_outcome_total, breaking the one-increment-per-failed-produce invariant this PR adds. Low likelihood since it requires an internal routing bug, but worth a follow-up since it's adjacent to the metrics work here.

— via Claude Code

Conflicts in client.go and client_test.go: take main's per-partition rejection handling from #100 and keep one no_agent_assigned final outcome per rejected group until the final-outcome reasons are reworked. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…churn

…ion on deadline ties and cancel

koloss2001 · 2026-10-08T06:59:12Z

Pre-existing gap: routing-mismatch guard skips final-outcome metrics
pkg/wgo/hedger.go:144-148 (unchanged by this PR, already on main) returns early on a routing mismatch, before any of the attempt/final-outcome metrics closures run. If this guard ever triggers, the failure is invisible in warpstream_produce_final_outcome_total, breaking the one-increment-per-failed-produce invariant this PR adds. Low likelihood since it requires an internal routing bug, but worth a follow-up since it's adjacent to the metrics work here.

Fixed: the guard now counts one internal_error and returns before any dispatch. It is kept out of the attempts histogram because no attempt ran. Test asserts the outcome, no dispatch, and no attempt observation.

Copilot

🟢 Approval recommended

The implementation and coverage are coherent; only minor test-naming convention fixes remain.

2 open findings

Rename test to follow agent-pool naming convention · New
Rename test to follow Hedger naming convention · New

What changed in this PR

Adds observability for hedging, final outcomes, attempted payloads, agent churn, and excluded leaders.

Changes:

Adds and wires new Prometheus metrics.
Corrects hedge-win accounting and classifies failed cascades.
Adds comprehensive metric and integration tests.

File	Description
`AGENTS.md`	Clarifies metrics documentation policy.
`pkg/wgo/agentpool.go`	Computes agent membership churn.
`pkg/wgo/agentpool_test.go`	Tests membership differences.
`pkg/wgo/client.go`	Publishes excluded-leader gauge values.
`pkg/wgo/client_test.go`	Tests churn, exclusions, and payload metrics.
`pkg/wgo/encoded_topic_partition.go`	Aggregates encoded payload statistics.
`pkg/wgo/hedger.go`	Instruments triggers, wins, payloads, and outcomes.
`pkg/wgo/hedger_test.go`	Covers new hedging metric semantics.
`pkg/wgo/metrics.go`	Defines and registers new metrics.
`pkg/wgo/metrics_test.go`	Verifies metric families and labels.
`pkg/wgo/produce_result.go`	Exposes terminal accumulator errors.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot

🔵 Needs a closer look

Final-outcome classification can incorrectly report candidate exhaustion when the primary returned a terminal error.

0 open findings

2 resolved since last review

Rename test to follow Hedger naming convention
Rename test to follow agent-pool naming convention

Previously missed (1)

In code that hasn't changed since last review

Preserve primary terminal error in merged hedged stop reason

pkg/wgo/hedger.go:266

hedged.stop only describes the fallback accumulator. If the primary returns a non-retriable/unknown error and the fallback merely exhausts its candidates, this records candidates_exhausted even though the documented precedence says terminal_error wins. The no-proactive and racing branches have the same gap; carry the primary failure classification into the merged stop before observing the final outcome.

🧠 Review effort: Balanced

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

aldernero · 2026-10-08T19:50:04Z

+
+// diffAgentMembership counts NodeIDs that appeared or disappeared between two
+// sorted, unique agent lists.
+func diffAgentMembership(old, new []int32) (added, removed int) {


diffAgentMembership assumes sorted, unique input. refresh sorts newAgents but doesn't dedupe it. If a Metadata response ever lists a NodeID twice, [5,5] followed by [5] counts one removed with no real membership change, which inflates agents_changed_total. diffRemovedAgents is unaffected because it goes through agentSet. Probably rare in practice, but a slices.Compact after the sort would make the documented precondition true. (Or derive the counts from the same set-based diff refresh already does.)

Fixed. refresh now deduplicates NodeIDs after sorting. It was worse than described: the pool itself held the duplicate and membership_changed was also counted. Added a test that duplicates a broker in the Metadata response.

I'm reverting this. A correct Metadata response contains one entry per NodeID, and a duplicate is treated as malformed, so the sorted list already satisfies diffAgentMembership. Compacting also changes fallback routing for a malformed response (hash % len(agents) and the secondary walk), which this PR otherwise doesn't touch. I've removed the slices.Compact call and its test.

…dupe NodeIDs on refresh

Copilot

🟡 Changes recommended

Cancellation can incorrectly hide an earlier terminal primary failure from the final-outcome metric.

2 open findings

Routing changes when duplicate broker IDs are compacted · New
Cancellation masks the primary non-retriable error · New

🧠 Review effort: Balanced

…d drop NodeID compaction

Copilot

🟢 Approval recommended

The metric accounting is consistent with documented semantics and has comprehensive coverage.

0 open findings

2 resolved since last review

Cancellation masks the primary non-retriable error
Routing changes when duplicate broker IDs are compacted

🧠 Review effort: Balanced

stephclay · 2026-10-09T14:31:50Z

FYI: The issues raised by my earlier review are all resolved, so that only leaves the newer findings from Vernon to address.

aldernero

LGTM!

Add hedge-trigger, produce-failure, and agent-pool churn metrics

bb312b1

koloss2001 requested review from a team as code owners September 23, 2026 20:21

cursor Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread pkg/wgo/hedger.go Outdated

stephclay reviewed Sep 24, 2026

View reviewed changes

Comment thread pkg/wgo/hedger.go Outdated

Comment thread pkg/wgo/hedger.go Outdated

exclude caller cancellation from final outcomes

16669d5

cursor Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread pkg/wgo/hedger.go

track write timeouts in final outcomes

ed18375

koloss2001 requested a review from stephclay September 28, 2026 15:39

stephclay reviewed Sep 30, 2026

View reviewed changes

Comment thread pkg/wgo/hedger.go Outdated

Comment thread pkg/wgo/client.go Outdated

koloss2001 mentioned this pull request Oct 2, 2026

Back off on-demand Metadata refreshes while a leader stays excluded (release-0.1) #97

Merged

koloss2001 and others added 7 commits October 7, 2026 19:06

Classify hedge triggers and final outcomes from the actual decision

89da2f7

Assert final outcomes on hedger race paths and cover live agent-pool …

00dd6b5

…churn

Add per-trigger hedge win counter

b880d9f

Replace leader_dropped_total with an excluded_leaders gauge

6b3fd94

Count attempted primary and hedge payload (records and compressed bytes)

d109506

Count hedge wins in the owning race and fix final-outcome classificat…

f4a614c

…ion on deadline ties and cancel

koloss2001 changed the title ~~Add hedge-trigger, produce-failure, and agent-pool churn metrics~~ Add hedge-trigger, final-outcome, attempted-payload and excluded-leader metrics Oct 8, 2026

koloss2001 requested a balanced review from Copilot October 8, 2026 07:27

Copilot started reviewing on behalf of koloss2001 October 8, 2026 07:28 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread pkg/wgo/agentpool_test.go Outdated

Comment thread pkg/wgo/hedger_test.go Outdated

renamed tests

246d707

koloss2001 requested review from stephclay and a balanced review from Copilot October 8, 2026 07:36

Copilot started reviewing on behalf of koloss2001 October 8, 2026 07:37 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

aldernero reviewed Oct 8, 2026

View reviewed changes

Settle final-outcome reason once the primary's result is known and de…

b88a842

…dupe NodeIDs on refresh

koloss2001 requested a balanced review from Copilot October 8, 2026 23:53

Copilot started reviewing on behalf of koloss2001 October 8, 2026 23:54 View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread pkg/wgo/agentpool.go Outdated

Comment thread pkg/wgo/hedger.go

Rename the failure-reason counter to produce_requests_failed_total an…

21f50fd

…d drop NodeID compaction

koloss2001 requested review from aldernero and a balanced review from Copilot October 9, 2026 03:32

Copilot started reviewing on behalf of koloss2001 October 9, 2026 03:33 View session

Copilot AI reviewed Oct 9, 2026

View reviewed changes

koloss2001 mentioned this pull request Oct 9, 2026

Add partition-route and routing-miss counters #111

Open

aldernero approved these changes Oct 9, 2026

View reviewed changes

koloss2001 merged commit 2149a7b into main Oct 9, 2026
17 checks passed

koloss2001 deleted the koloss2001/hedge-trigger-instrumentation branch October 9, 2026 15:59

Conversation

koloss2001 commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Things to know

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephclay commented Sep 24, 2026

Uh oh!

koloss2001 commented Sep 24, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephclay commented Sep 30, 2026

Uh oh!

koloss2001 commented Oct 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

aldernero Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

koloss2001 Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

koloss2001 Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

stephclay commented Oct 9, 2026

Uh oh!

aldernero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

koloss2001 commented Sep 23, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading