Skip to content

feat: let the sync operator set an outage traffic allowance - #71

Open
timwu20 wants to merge 5 commits into
mainfrom
feat/outage-traffic-allowance
Open

timwu20 wants to merge 5 commits into
mainfrom
feat/outage-traffic-allowance

Conversation

@timwu20

@timwu20 timwu20 commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Implements the FR-3 outage handling agreed on 2026-10-05 (design: ChainSafe/canton-extending-mainnet#142), in place of #61 and #66. Closes ChainSafe/canton-extending-mainnet#129.

During a global-synchronizer outage, the operator sets outage-traffic-allowance (a positive number of bytes) in the sync operator app's config and restarts the app, following the runbook. Once the global synchronizer is back, the operator removes it and restarts again. Nothing detects the outage, and there is no Daml change.

  • On start, the app sets the limit of every member with a purchase on record to exactly its purchased total plus the allowance, or to its purchased total when no allowance is set.
    • It walks the members only until every limit is in place, then does nothing per member until the next start.
    • Each grant reads the member's purchased total again just before it's sent, so a purchase granted in the meantime isn't taken back.
    • Mediators are left out, since MediatorUnlimitedTrafficTrigger keeps their limits unlimited.
    • So are members the sequencer has no traffic state for yet.
  • While it's set, purchases pay down the credit, because the reconcile trigger only ever raises a limit to the purchased total. A purchase ingested while the limits are being set back is picked up by the reconcile trigger and the next pass.
  • Warnings:
    • every outage-traffic-allowance-warning-interval (5 minutes) while the allowance is set, giving the allowance, the members it covers and the total credit outstanding;
    • when traffic is bought for a member, or the DSO merges a member's traffic contracts, while the allowance is set, since either means the global synchronizer is reachable again;
    • once, when the allowance is set, naming the members with a purchase on record but no traffic state on the sequencer, which only get the allowance after another restart;
    • for each member whose limit doesn't reach its target, at most once per warning interval, with the same text at info in between, whether the allowance is set or removed. On a synchronizer whose sequencers several operators run, this is what an operator sees until enough of them send the same limit: until they have all set, or removed, the same allowance and seen the same purchases.
  • Config: both settings must be positive. An allowance of 0 would count as set, and an interval of 0 would run the warning in a tight loop.
  • Validator top-up:
    • When a member's remainder is below zero, the top-up buys that shortfall together with the normal top-up in one purchase, and the funds check prices both. Before, it bought one top-up per minTopupInterval and left the member blocked for several intervals.
    • If the wallet doesn't cover both, it buys the top-up alone, which still pays the shortfall down one interval at a time, and warns.
    • When even the top-up alone doesn't fit, the insufficient-funds warning names both amounts.
  • Store: listTotalPurchasedMemberTraffic sums the purchased totals per member on the operator's synchronizer in one query.

Two consequences for the runbook:

  • Every start without the allowance sets limits back to the purchased totals, so it also takes back traffic granted by hand above the purchases.
  • A restart with the allowance still set, after purchases have landed, gives each member the full allowance again on top of its new total.

Tests.

  • SyncOperatorTrafficIntegrationTest runs the whole flow:
    • The operator's participant is disconnected from the global synchronizer, and the app restarts with a 50 KB allowance. Each limit rises by exactly the allowance, and a second restart moves nothing.
    • Bob transacts past what was bought for him and is refused at the allowance. Alice transacts on what was bought for her without touching the allowance.
    • After reconnecting, a purchase for bob logs the warning and leaves his limit where it was.
    • With the allowance removed, the limits go back to the purchased totals and alice keeps transacting. Bob is refused until one top-up buys his shortfall (27.5 KB) together with his 12 KB top-up.
    • To change the config mid-test, a second instance of the app with the allowance set shares the first one's store and is started by hand. It's added in the test rather than in sync-operator-topology.conf, which other tests load.
  • Unit tests for the targets, the members left out, the warning rate limit, the shortfall purchase with its fallback and cost, and the store query.

Not run locally: SyncOperatorLsuIntegrationTest. Testing with several operators is a follow-up once the multi-node topology lands (ChainSafe/canton-extending-mainnet#106).

During a global-synchronizer outage, the operator sets
outage-traffic-allowance in the sync operator app's config and restarts
the app. On start, the app sets each member's limit to its purchased
total plus the allowance, or back to the purchased total once it is
removed. It warns while the allowance is set, when a purchase lands
meanwhile, and for any limit that does not reach its target. The
validator's top-up buys a member's shortfall together with its normal
top-up in one purchase.

Closes ChainSafe/canton-extending-mainnet#129

Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
- Read a member's purchased total again before its grant, after the
  traffic state, so a purchase granted since the sweep retrieved its
  tasks is in the target instead of taken back until the next poll.
- When the sweep settles with the allowance set, warn once about
  members with a purchase on record but no traffic state on the
  sequencer, which it leaves out.
- Word the purchase warning for the DSO's merges as well, and log it
  once the grant succeeds, so a retried grant warns once.
- Warn about a limit that could not be set at most once per warning
  interval for each member, with the same text at info in between.
- Require a positive allowance and warning interval: 0 counted as set,
  and a 0 interval ran the periodic warning in a tight loop.

Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
… shortfall

The top-up bought the configured amount plus the whole shortfall or
nothing, so a wallet that covered the top-up alone left a member below
zero for good, where before it paid the shortfall down one interval at
a time. A member below zero now gets the top-up alone when the wallet
does not cover both, with a warning that it stays below zero for now.
The insufficient-funds warning fires only when even the top-up alone
does not fit.

Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
…helper [ci]

Alice now pings during the outage before the check that she ran on her
own traffic, so it covers a member transacting rather than only her
background traffic. The helper that drops the splitwell apps takes the
validators to disconnect from splitwell, and this test uses it instead
of repeating it with bob left connected.

Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
- When even the top-up alone does not fit, the insufficient-funds
  warning also names the top-up and the shortfall together for a member
  below zero, so the wallet is funded in one round.
- Both purchases are priced from one read of the rules and the open
  round, and minWalletBalanceForTopup is back to its signature on main.
- A task whose target a grant has already met returns TaskNoop.
- The warnings for a limit that could not be set say that operators may
  also not all have seen the same purchases yet.

Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
@timwu20
timwu20 marked this pull request as ready for review October 7, 2026 01:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P2-E5.9] Operator-set outage traffic allowance (FR-3)

1 participant