feat(retry): add retry budget to prevent retry storms by joshrotenberg · Pull Request #172 · joshrotenberg/tower-resilience

joshrotenberg · 2025-12-31T06:30:46Z

Add retry budget functionality to limit total retries across all requests, preventing cascading failures when a downstream service is struggling.

Features

RetryBudget trait for pluggable budget implementations
TokenBucketBudget: simple token-based budget with deposit/withdraw
AimdBudget: adaptive budget with multiplicative decrease on exhaustion
RetryBudgetBuilder for ergonomic budget creation
BudgetExhausted event for observability
.on_budget_exhausted() callback in builder

Usage

use tower_resilience_retry::{RetryLayer, RetryBudgetBuilder};
use std::time::Duration;

// Create a budget: 100 tokens max, starts full
let budget = RetryBudgetBuilder::new()
    .token_bucket()
    .max_tokens(100)
    .initial_tokens(100)
    .build();

// Use with retry layer
let layer = RetryLayer::<MyError>::builder()
    .max_attempts(5)
    .exponential_backoff(Duration::from_millis(100))
    .budget(budget)
    .on_budget_exhausted(|attempt| {
        println!("Retry {} skipped - budget exhausted", attempt);
    })
    .build();

How It Works

The budget is shared across all service clones (via Arc)
Each retry attempt withdraws a token from the budget
Successful requests deposit a token back into the budget
When budget is exhausted, retries are skipped and the error is returned immediately

This prevents "retry storms" where a failing service gets hammered with retries from many concurrent requests.

Budget Types

Token Bucket

Simple token-based budget. Good for most use cases.

let budget = RetryBudgetBuilder::new()
    .token_bucket()
    .max_tokens(100)      // Burst capacity
    .initial_tokens(100)  // Start full
    .build();

AIMD (Additive Increase Multiplicative Decrease)

Adaptive budget that shrinks when exhausted.

let budget = RetryBudgetBuilder::new()
    .aimd()
    .min_budget(10)       // Floor
    .max_budget(1000)     // Ceiling
    .decrease_factor(0.5) // Halve on exhaustion
    .build();

References

Addresses tower-rs/tower#705 (stalled since 2022)
Closes feat: add retry budget to prevent retry storms #171

Add retry budget functionality to limit total retries across all requests, preventing cascading failures when a downstream service is struggling. Features: - RetryBudget trait for pluggable budget implementations - TokenBucketBudget: simple token-based budget with deposit/withdraw - AimdBudget: adaptive budget with multiplicative decrease on exhaustion - RetryBudgetBuilder for ergonomic budget creation - BudgetExhausted event for observability - on_budget_exhausted() callback in builder Usage: let budget = RetryBudgetBuilder::new() .token_bucket() .max_tokens(100) .initial_tokens(100) .build(); let layer = RetryLayer::<MyError>::builder() .max_attempts(5) .budget(budget) .build(); The budget is shared across all service clones, providing global rate limiting for retries. Successful requests deposit tokens back into the budget, while each retry attempt withdraws a token. Closes #171

github-actions · 2025-12-31T06:42:32Z

Benchmark Results

Comparing PR branch against main

⚠️ Performance Regressions Detected

One or more benchmarks show statistically significant regressions.

Full Benchmark Results

baseline_no_middleware  time:   [19.734 ns 19.783 ns 19.839 ns]
                        change: [−0.5547% −0.1742% +0.1789%] (p = 0.37 > 0.05)
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

circuitbreaker_closed   time:   [585.72 ns 587.04 ns 588.28 ns]
                        change: [−6.3818% −6.0597% −5.6661%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

bulkhead_permits_available
                        time:   [695.09 ns 696.09 ns 697.34 ns]
                        change: [+4.0459% +4.5236% +5.0273%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

retry_no_retries_needed time:   [287.34 ns 287.72 ns 288.12 ns]
                        change: [−7.6047% −7.2046% −6.8042%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

ratelimiter_permits_available
                        time:   [390.41 ns 391.16 ns 391.91 ns]
                        change: [+7.8064% +8.2688% +8.7178%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) low mild
  1 (1.00%) high severe

cache_hit               time:   [678.40 ns 684.92 ns 693.16 ns]
                        change: [−10.087% −9.4001% −8.6243%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe

timelimiter_completes_quickly
                        time:   [420.44 ns 421.24 ns 422.07 ns]
                        change: [+6.7097% +6.9599% +7.2154%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

composition_circuit_breaker_and_bulkhead
                        time:   [1.3168 µs 1.3203 µs 1.3242 µs]
                        change: [+0.9129% +1.3135% +1.6852%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

reconnect_no_failures   time:   [288.67 ns 289.91 ns 291.04 ns]
                        change: [+0.5276% +1.1741% +1.7687%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

reconnect_with_failure  time:   [278.09 ns 278.71 ns 279.40 ns]
                        change: [−1.6703% −1.3452% −1.0081%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

healthcheck_get_healthy time:   [3.4294 µs 3.4472 µs 3.4635 µs]
                        change: [−2.9019% −2.3725% −1.8967%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low severe
  6 (6.00%) high mild

worst_case_circuit_breaker_open
                        time:   [553.84 ns 559.56 ns 565.65 ns]
                        change: [−11.612% −10.807% −9.9579%] (p = 0.00 < 0.05)
                        Performance has improved.

worst_case_bulkhead_full
                        time:   [726.42 ns 728.50 ns 730.67 ns]
                        change: [−8.7122% −8.2356% −7.8171%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

worst_case_cache_miss   time:   [478.38 ns 479.53 ns 480.64 ns]
                        change: [+2.1635% +2.5399% +2.9212%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

worst_case_rate_limiter_exhausted
                        time:   [334.05 ns 335.83 ns 337.78 ns]
                        change: [−3.3617% −2.4988% −1.6745%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

worst_case_retry_exhausted
                        time:   [290.09 ns 290.49 ns 290.94 ns]
                        change: [−0.0734% +0.4610% +0.9768%] (p = 0.09 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

worst_case_timelimiter_completes_quickly
                        time:   [411.66 ns 412.41 ns 413.28 ns]
                        change: [+0.2862% +0.6699% +1.0600%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

composition_three_layers
                        time:   [1.4732 µs 1.4753 µs 1.4776 µs]
                        change: [−0.6010% −0.3212% +0.0183%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

fallback_no_failure     time:   [193.55 ns 194.01 ns 194.50 ns]
                        change: [−6.9384% −6.1856% −5.4345%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

fallback_with_failure   time:   [289.67 ns 292.32 ns 295.47 ns]
                        change: [+13.001% +14.285% +15.470%] (p = 0.00 < 0.05)
                        Performance has regressed.

fallback_from_error     time:   [299.41 ns 299.75 ns 300.11 ns]
                        change: [+7.0269% +7.2506% +7.4883%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high severe

hedge_no_delay_needed   time:   [24.036 µs 24.443 µs 24.829 µs]
                        change: [−3.8156% −2.0859% −0.3488%] (p = 0.02 < 0.05)
                        Change within noise threshold.

hedge_parallel_mode     time:   [24.153 µs 24.241 µs 24.329 µs]
                        change: [−1.8411% −0.9791% −0.1525%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

hedge_with_slow_primary time:   [11.139 ms 11.165 ms 11.189 ms]
                        change: [−0.2466% +0.0611% +0.3766%] (p = 0.70 > 0.05)
                        No change in performance detected.

Criterion.rs uses statistical analysis with confidence intervals and p-values to detect significant changes.

joshrotenberg merged commit 1dd9ae8 into main Dec 31, 2025
10 checks passed

joshrotenberg deleted the feat/retry-budget branch December 31, 2025 15:32

github-actions Bot mentioned this pull request Dec 31, 2025

chore: release #163

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(retry): add retry budget to prevent retry storms#172

feat(retry): add retry budget to prevent retry storms#172
joshrotenberg merged 1 commit into
mainfrom
feat/retry-budget

joshrotenberg commented Dec 31, 2025

Uh oh!

github-actions Bot commented Dec 31, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

joshrotenberg commented Dec 31, 2025

Features

Usage

How It Works

Budget Types

Token Bucket

AIMD (Additive Increase Multiplicative Decrease)

References

Uh oh!

github-actions Bot commented Dec 31, 2025

Benchmark Results

⚠️ Performance Regressions Detected

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant