Skip to content

Commit 0c65328

Browse files
kaisbaccourclaude
andcommitted
Back off the network whenever any sync target fails
A target that keeps failing while another on the same network succeeds was paced at the normal interval (or at once while the other caught up), so it kept re-reading its capped window. Any failure now counts as a failed cycle, which bounds partial failures too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 6775689 commit 0c65328

1 file changed

Lines changed: 15 additions & 7 deletions

File tree

  • crates/common/src/raindex_client/local_db/pipeline/runner/scheduler

‎crates/common/src/raindex_client/local_db/pipeline/runner/scheduler/mod.rs‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,11 @@ pub(crate) enum CycleResult {
2828
/// block because of the per-cycle cap: run the next cycle at once and do
2929
/// not mark the chain ready yet.
3030
Behind,
31-
/// Nothing on the network made progress (`run_once` errored, or every
32-
/// target failed): back off.
31+
/// `run_once` errored or any target failed: back off. A failing target
32+
/// keeps its watermark, so retrying at the normal pace (or at once while
33+
/// another target catches up) would re-read its window again and again;
34+
/// backing off the whole network bounds that even when other targets on
35+
/// it are healthy.
3336
Failed,
3437
}
3538

@@ -39,7 +42,7 @@ impl CycleResult {
3942
Err(_) => Self::Failed,
4043
Ok(RunOutcome::NotLeader) => Self::CaughtUp,
4144
Ok(RunOutcome::Report(report)) => {
42-
if !report.failures.is_empty() && report.successes.is_empty() {
45+
if !report.failures.is_empty() {
4346
Self::Failed
4447
} else if report.successes.iter().any(|s| !s.outcome.caught_up) {
4548
Self::Behind
@@ -53,8 +56,8 @@ impl CycleResult {
5356

5457
/// Paces one network's sync loop. A failed cycle keeps its old watermark,
5558
/// so retrying at the normal interval re-reads the same blocks over and
56-
/// over; this backs off while nothing on the network makes progress and
57-
/// resets as soon as anything does.
59+
/// over; this backs off while any target keeps failing and resets once a
60+
/// cycle completes without failures.
5861
#[derive(Debug, Default)]
5962
pub(crate) struct CycleBackoff {
6063
consecutive_failures: u32,
@@ -168,10 +171,15 @@ mod tests {
168171
CycleResult::of(&report(vec![], vec![failure()])),
169172
CycleResult::Failed
170173
);
171-
// One healthy raindex keeps the network at the normal pace.
174+
// A failing target backs the network off even when another succeeds,
175+
// so a partial failure cannot re-read its window every cycle.
172176
assert_eq!(
173177
CycleResult::of(&report(vec![success(true)], vec![failure()])),
174-
CycleResult::CaughtUp
178+
CycleResult::Failed
179+
);
180+
assert_eq!(
181+
CycleResult::of(&report(vec![success(false)], vec![failure()])),
182+
CycleResult::Failed
175183
);
176184
}
177185

0 commit comments

Comments
 (0)