Repository navigation
Conversation
During a global-synchronizer outage, the operator sets outage-traffic-allowance in the sync operator app's config and restarts the app. On start, the app sets each member's limit to its purchased total plus the allowance, or back to the purchased total once it is removed. It warns while the allowance is set, when a purchase lands meanwhile, and for any limit that does not reach its target. The validator's top-up buys a member's shortfall together with its normal top-up in one purchase. Closes ChainSafe/canton-extending-mainnet#129 Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
- Read a member's purchased total again before its grant, after the traffic state, so a purchase granted since the sweep retrieved its tasks is in the target instead of taken back until the next poll. - When the sweep settles with the allowance set, warn once about members with a purchase on record but no traffic state on the sequencer, which it leaves out. - Word the purchase warning for the DSO's merges as well, and log it once the grant succeeds, so a retried grant warns once. - Warn about a limit that could not be set at most once per warning interval for each member, with the same text at info in between. - Require a positive allowance and warning interval: 0 counted as set, and a 0 interval ran the periodic warning in a tight loop. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
… shortfall The top-up bought the configured amount plus the whole shortfall or nothing, so a wallet that covered the top-up alone left a member below zero for good, where before it paid the shortfall down one interval at a time. A member below zero now gets the top-up alone when the wallet does not cover both, with a warning that it stays below zero for now. The insufficient-funds warning fires only when even the top-up alone does not fit. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
…helper [ci] Alice now pings during the outage before the check that she ran on her own traffic, so it covers a member transacting rather than only her background traffic. The helper that drops the splitwell apps takes the validators to disconnect from splitwell, and this test uses it instead of repeating it with bob left connected. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
- When even the top-up alone does not fit, the insufficient-funds warning also names the top-up and the shortfall together for a member below zero, so the wallet is funded in one round. - Both purchases are priced from one read of the rules and the open round, and minWalletBalanceForTopup is back to its signature on main. - A task whose target a grant has already met returns TaskNoop. - The warnings for a limit that could not be set say that operators may also not all have seen the same purchases yet. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>
timwu20
marked this pull request as ready for review
October 7, 2026 01:26
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the FR-3 outage handling agreed on 2026-10-05 (design: ChainSafe/canton-extending-mainnet#142), in place of #61 and #66. Closes ChainSafe/canton-extending-mainnet#129.
During a global-synchronizer outage, the operator sets
outage-traffic-allowance(a positive number of bytes) in the sync operator app's config and restarts the app, following the runbook. Once the global synchronizer is back, the operator removes it and restarts again. Nothing detects the outage, and there is no Daml change.MediatorUnlimitedTrafficTriggerkeeps their limits unlimited.outage-traffic-allowance-warning-interval(5 minutes) while the allowance is set, giving the allowance, the members it covers and the total credit outstanding;minTopupIntervaland left the member blocked for several intervals.listTotalPurchasedMemberTrafficsums the purchased totals per member on the operator's synchronizer in one query.Two consequences for the runbook:
Tests.
SyncOperatorTrafficIntegrationTestruns the whole flow:sync-operator-topology.conf, which other tests load.Not run locally:
SyncOperatorLsuIntegrationTest. Testing with several operators is a follow-up once the multi-node topology lands (ChainSafe/canton-extending-mainnet#106).