From 43405f6a483cfd9d43f8cd9af98981e0196a2cb1 Mon Sep 17 00:00:00 2001 From: grunch Date: Tue, 11 Aug 2026 23:22:12 -0300 Subject: [PATCH 1/2] docs: add payment circuit breaker spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec for a damage-containment mechanism: if a bug lets an attacker pull sats out of the node, mostrod should notice that outflow is outrunning inflow and halt every outgoing payment on its own, so an operator has time to investigate instead of waking up to a drained node. Documentation only — no code changes. Rollout is split into four atomic PRs (§8), starting with an observe-only phase that ships dark. Key design decisions recorded: - The gate lives in LndConnector::send_payment, the single chokepoint every Lightning outflow passes through. Halting the scheduler jobs is not sufficient: the largest outflow (the trade payout) is driven by a user-supplied message on the release hot path, not by a job. - Two detection layers. A per-order invariant (outflow never exceeds inflow) checked synchronously before dispatch, and rolling-window velocity / net-outflow rules evaluated by a watcher job. Balance reconciliation against LND is deliberately out of scope; §4.3 records what that costs. - The latch is persistent and manual-reset only. A breaker that clears on restart is exactly the condition an attacker would induce. - Fail closed, and write-ahead: a reserved ledger row counts against every budget as if it had succeeded, so a double-pay bug is caught by the gate on the second attempt rather than in the post-mortem. - Tripping freezes outgoing payments and nothing else. No hold invoice is cancelled, no corrective transfer is attempted. Known gaps are stated rather than assumed: every rule derives from mostrod's own accounting, so a bug that moves funds without writing a ledger row is invisible to all of them; and Cashu mode has no equivalent chokepoint, so it is not covered by these phases. --- docs/PAYMENT_CIRCUIT_BREAKER.md | 433 ++++++++++++++++++++++++++++++++ docs/README.md | 1 + 2 files changed, 434 insertions(+) create mode 100644 docs/PAYMENT_CIRCUIT_BREAKER.md diff --git a/docs/PAYMENT_CIRCUIT_BREAKER.md b/docs/PAYMENT_CIRCUIT_BREAKER.md new file mode 100644 index 00000000..4349c8d8 --- /dev/null +++ b/docs/PAYMENT_CIRCUIT_BREAKER.md @@ -0,0 +1,433 @@ +# Payment Circuit Breaker — Implementation Spec + +> Damage containment for a hypothetical fund-draining bug. This document is the +> single source of truth as the feature is rolled out across several PRs. Each +> phase below maps to one **small, atomic PR** that can be reviewed, tested, and +> released independently. +> +> Not to be confused with the per-provider circuit breaker in +> [PRICE_PROVIDERS.md](./PRICE_PROVIDERS.md) §6.5, which trips on a failing +> price API and has nothing to do with fund movement. + +## 1. Goal + +If an attacker finds a bug that lets them pull sats out of the node, Mostro must +**notice that more sats are leaving than entering and halt every outgoing +payment on its own**, without operator intervention, so the operator has time to +investigate instead of waking up to a drained node. + +This is **containment, not prevention**. A draining bug remains a draining bug; +the circuit breaker only bounds how much leaves before a human can react. It is +worth building precisely because the worst realistic scenario is not "we lost +sats" but "we lost sats all night while nobody was watching". + +## 2. Guiding principles + +1. **Gate at the lowest chokepoint, not at the callers.** Every Lightning + outflow in the daemon passes through `LndConnector::send_payment` + (`src/lightning/mod.rs:227`). That is where the gate lives. Gating callers + individually guarantees that whoever adds the next payment path forgets one. +2. **Fail closed.** If the ledger cannot be read, the DB is unavailable, or the + breaker state cannot be determined, the gate **rejects the payment**. A gate + that fails open is not a gate. +3. **The latch is persistent and manual-reset only.** Once tripped, the state + survives a daemon restart and only an admin can clear it. A breaker that + resets on restart is exactly the condition an attacker will induce. +4. **Write-ahead accounting.** The ledger row is written *before* the payment is + dispatched, and an unconfirmed row counts against the budget. A double-pay + bug is then caught by the gate on the second attempt rather than discovered + in the post-mortem. +5. **Freeze, never unwind.** Tripping halts outgoing payments. It does **not** + cancel hold invoices, does not settle anything, and does not attempt any + corrective transfer. Mass irreversible action under a tripped breaker is the + last thing anyone wants automated. +6. **Observe before enforcing.** Phase 0 ships in observe-only mode so operators + can calibrate thresholds against real traffic before the gate can reject a + legitimate payment. +7. **Tests accompany every phase.** Rust unit tests co-located with the module; + `cargo test`, `cargo fmt`, `cargo clippy --all-targets --all-features` must + stay green. + +## 3. Outflow surface + +Every point where sats leave the node today: + +| Outflow | Call site | Trigger | +|---|---|---| +| Trade payout to buyer | `src/app/release.rs:589` | `release` handler (hot path, user-driven) | +| Dev fee | `src/app/dev_fee.rs:993` | `job_process_dev_fee_payment` (`src/scheduler.rs:1087`) | +| Bond payout to counterparty | `src/app/bond/slash.rs`, `src/app/bond/flow.rs` | `job_process_bond_payouts` (`src/scheduler.rs:1121`) | +| Failed-payment retries | `job_retry_failed_payments` (`src/scheduler.rs:213`) | scheduler | +| Dispute resolution payout | `src/app/admin_settle.rs` → release path | admin/solver | + +All of them funnel into `LndConnector::send_payment`. + +**Halting the scheduler jobs is not sufficient.** The largest outflow — the +trade payout — is driven by a user-supplied Nostr message on the hot path, not +by a job. Jobs are additionally short-circuited (Phase 2) to avoid burning retry +budgets and flooding logs, but that is hygiene, not the control. + +`cancel_hold_invoice` is deliberately **not** gated: it releases an HTLC back to +whoever locked it and moves no funds out of the node. + +Inflows are recorded at `settle_hold_invoice` call sites — the escrow settle +path (`src/util.rs:1330`) and the bond slash paths (`src/app/bond/slash.rs:798`, +`src/app/bond/slash.rs:1115`). + +## 4. Detection layers + +Two layers, evaluated at different points. + +### 4.1 Per-order invariant (synchronous, inside the gate) + +> For any order, the sats paid out must never exceed the sats taken in. + +``` +Σ outflow(order_id) + pending_amount ≤ Σ inflow(order_id) + tolerance_sats +``` + +This is the fast, precise layer. Nearly every conceivable draining bug violates +it: paying an order twice, paying an order whose escrow was never settled, +paying more than the escrow held, replaying a payout against a different +invoice. It is checked **before** dispatch, so the bad payment never leaves. + +The direction is naturally safe: routing fees and the node fee mean a healthy +order always has `out < in`. Default `tolerance_sats = 0`. + +**Bond ledger key.** Range-order bonds carry both `order_id` and +`child_order_id` (see `migrations/20260423120000_anti_abuse_bond.sql`). The +ledger **must** key on the parent/root `order_id` for both the slash inflow and +the payout outflow, otherwise a legitimate child payout looks like an outflow +against an order with zero inflow and trips the breaker. + +### 4.2 Velocity and net outflow (asynchronous, watcher job) + +Rolling-window aggregates over the ledger, evaluated every +`check_interval_seconds`: + +| Rule | Config key | +|---|---| +| Single payment exceeds cap | `max_payment_sats` | +| Outflow in the last hour exceeds cap | `max_outflow_sats_per_hour` | +| Outflow in the last 24h exceeds cap | `max_outflow_sats_per_day` | +| Payment count in the last hour exceeds cap | `max_payments_per_hour` | +| Net outflow (`Σ out − Σ in`) over 24h exceeds cap | `max_net_outflow_sats_24h` | + +`max_payment_sats` is additionally enforced **synchronously in the gate** — a +single oversized payment must be stopped before it leaves, not detected a minute +later. + +The net-outflow rule is the direct expression of the requirement ("more sats +leaving than entering"). Because inflow and outflow are roughly balanced +per-order in healthy operation, a sustained positive net outflow is a leak. + +### 4.3 Deliberately out of scope: balance reconciliation against LND + +An earlier draft included a third layer that compared LND's +`ChannelBalance`/`WalletBalance` against what the internal ledger says should be +there. **It is excluded from this spec by decision.** + +The cost of that exclusion should be recorded honestly: both remaining layers +are derived from Mostro's own accounting, so **if the draining bug lives in the +accounting itself, the breaker is blind to it**. A bug that moves sats without +ever writing a `payment_ledger` row is not detected by any rule here. The gate +placement in §2.1 is what mitigates this — a payment that does not go through +`send_payment` is the only way to bypass the ledger, and no such path exists +today. Adding one must be treated as a security-relevant change. + +Note that the per-payment `lookup_payment_status` call used by the reaper +(§5.4) is *not* this layer — it queries the status of one known payment hash, +not the node's balance. + +## 5. Design + +### 5.1 New module: `src/circuit_breaker.rs` + +```rust +/// What kind of outflow a payment represents. Recorded on the ledger row so +/// the operator can tell at a glance which subsystem leaked. +pub enum PaymentKind { TradePayout, DevFee, BondPayout } + +/// Context the gate needs that `send_payment`'s bolt11 + amount cannot supply. +pub struct PaymentIntent { + pub kind: PaymentKind, + /// Parent/root order id — see §4.1 on range-order bonds. + pub order_id: Uuid, + /// Bond id for `BondPayout`, `None` otherwise. + pub ref_id: Option, + pub amount_sats: i64, +} + +pub enum BreakerState { Closed, Tripped { reason: TripReason, tripped_at: i64 } } + +pub enum TripReason { + OrderInvariant { order_id: Uuid, out: i64, inflow: i64 }, + PaymentTooLarge { amount: i64, cap: i64 }, + HourlyOutflow { total: i64, cap: i64 }, + DailyOutflow { total: i64, cap: i64 }, + HourlyCount { count: i64, cap: i64 }, + NetOutflow { net: i64, cap: i64 }, + /// Set by an admin via RPC. Lets an operator halt payments manually. + Manual { note: String }, +} +``` + +`PaymentGuard` holds the pool and caches the latch in an `AtomicU8` so the hot +path does not hit the DB just to learn the breaker is closed. The DB row remains +the source of truth; the atomic is a cache refreshed on write and on boot. + +### 5.2 `send_payment` signature change + +```rust +pub async fn send_payment( + &mut self, + payment_request: &str, + amount: i64, + intent: &PaymentIntent, // new + listener: Sender, +) -> Result<(), MostroError> +``` + +Threading an explicit intent through every call site is intentional: it makes it +impossible to add a new outflow without stating what it is and which order funds +it. Call sites to update: `src/app/release.rs:589`, `src/app/dev_fee.rs:993`, +and the bond payout path in `src/app/bond/`. + +Rejections surface as +`MostroInternalErr(ServiceError::LnPaymentError("circuit breaker tripped: …"))`, +reusing the existing variant so no `mostro-core` change is required. + +### 5.3 Gate sequence (inside `send_payment`, before `decode_invoice`) + +1. Read the latch. If `Tripped`, log and return an error. No LND call. +2. If `amount > max_payment_sats`, trip and reject. +3. Query `Σ in` / `Σ out` (including `reserved`) for `intent.order_id`. If the + §4.1 invariant would be violated, trip and reject. +4. Insert the ledger row as `reserved`. +5. Dispatch to LND. +6. Terminal result confirms the row (`confirmed` with the real fee, or `failed`). + +Any error in steps 1–4 — DB unavailable, query failure, insert failure — +rejects the payment (principle 2). + +In observe-only mode (`enforce = false`), steps 1–4 still evaluate and log at +`warn!`/`error!`, and step 4 still writes the row, but nothing is rejected and +the latch is not set. + +### 5.4 Reserved-row reaper + +`send_payment` streams payment updates to a caller-owned listener, so the +confirmation in step 6 happens in the caller. Relying on every caller to +confirm correctly would reintroduce exactly the "someone will forget" problem +this design avoids. + +Instead, **a `reserved` row counts against every budget as if it succeeded.** +Confirmation is an accuracy improvement, not a safety requirement. A reaper +inside the watcher job resolves rows still `reserved` after +`reserved_row_timeout_seconds` by calling +`LndConnector::lookup_payment_status` (`src/lightning/mod.rs:325`) on the stored +hash, and marks them `confirmed` or `failed`. Until it does, the conservative +assumption stands. + +### 5.5 Trip actions + +1. Persist `Tripped` to `circuit_breaker_state` and update the atomic cache. +2. `error!` log with the full `TripReason`. +3. Notify admins over the existing message queue. +4. Scheduler payment jobs skip their tick (Phase 2). +5. `release` and `admin-settle` handlers reject early with a clear user-facing + message, rather than letting the request travel to LND to fail there. +6. Publish `payments_paused = true` on the NIP-33 info event (`src/nip33.rs`, + alongside the existing `bond_*` policy tags) so clients can warn users. + +Explicitly **not** done on trip: cancelling hold invoices, settling anything, +pausing inbound flows other than order creation (see below), auto-resetting. + +New order creation is also refused while tripped. Accepting users into a system +that provably cannot pay them out only widens the blast radius. + +### 5.6 Reset + +Admin-only, via RPC (`proto/admin.proto`, `src/rpc/service.rs`), subject to the +existing admin auth and rate limiting: + +- `CircuitBreakerStatus` — current state, reason, `tripped_at`, and the + window aggregates that drove it. +- `CircuitBreakerReset` — clears the latch. Requires an operator note, which is + persisted for the audit trail. **No timeout-based auto-reset exists.** + +## 6. Schema + +`migrations/20260811120000_payment_circuit_breaker.sql`: + +```sql +-- Append-only record of every sat entering or leaving the node. +CREATE TABLE IF NOT EXISTS payment_ledger ( + id char(36) primary key not null, + -- 'in' (hold invoice settled) | 'out' (payment sent) + direction varchar(3) not null, + -- 'trade-payout' | 'dev-fee' | 'bond-payout' | 'escrow-settle' | 'bond-slash' + kind varchar(16) not null, + -- Parent/root order id. Range-order bond rows key on the parent, never the + -- child — see spec §4.1. + order_id char(36) not null, + -- Bond id for bond rows, NULL otherwise. + ref_id char(36), + amount_sats integer not null, + -- Actual routing fee, known only after a payment confirms. NULL otherwise. + fee_sats integer, + payment_hash char(64), + -- 'reserved' | 'confirmed' | 'failed'. A 'reserved' row counts against every + -- budget as if it had succeeded (spec §5.4). + state varchar(10) not null, + -- Unix timestamps in seconds, matching the rest of the schema. + created_at integer not null, + updated_at integer not null +); + +CREATE INDEX IF NOT EXISTS idx_payment_ledger_order ON payment_ledger(order_id); +CREATE INDEX IF NOT EXISTS idx_payment_ledger_created ON payment_ledger(created_at); +CREATE INDEX IF NOT EXISTS idx_payment_ledger_state ON payment_ledger(state); + +-- Single-row latch. Persistent so a restart cannot clear a trip. +CREATE TABLE IF NOT EXISTS circuit_breaker_state ( + id integer primary key check (id = 1), + -- 'closed' | 'tripped' + state varchar(8) not null, + -- Serialized TripReason. NULL while closed. + reason text, + tripped_at integer, + -- Operator note supplied at reset time, retained as an audit trail. + reset_note text, + reset_at integer, + updated_at integer not null +); + +INSERT OR IGNORE INTO circuit_breaker_state (id, state, updated_at) + VALUES (1, 'closed', 0); +``` + +Example rows (synthetic) for a completed trade of 50 000 sats: + +``` +id direction kind order_id amount_sats fee_sats state created_at +7f1c…-…-0001 in escrow-settle 3a9e…-0007 50100 NULL confirmed 1786500000 +7f1c…-…-0002 out trade-payout 3a9e…-0007 50000 12 confirmed 1786500004 +7f1c…-…-0003 out dev-fee 3a9e…-0007 50 1 confirmed 1786500061 +``` + +## 7. Configuration + +New optional `[circuit_breaker]` block. Absent block = feature off, byte-for-byte +today's behavior. `src/config/types.rs`, template in `settings.tpl.toml`, +documented in `docs/STARTUP_AND_CONFIG.md`. + +```toml +[circuit_breaker] +# Master switch. Off = no ledger writes, no gate, no watcher job. +enabled = true +# false = observe-only: evaluate and log, never reject and never latch. +# Run this way first to calibrate the thresholds below against real traffic. +enforce = false +# Watcher job cadence (seconds). +check_interval_seconds = 60 +# Slack allowed on the per-order invariant (sats). 0 is correct in normal +# operation, since routing fees make a healthy order's outflow strictly +# smaller than its inflow. +tolerance_sats = 0 +# Also enforced synchronously in the gate, not only by the watcher. +max_payment_sats = 1000000 +max_outflow_sats_per_hour = 5000000 +max_outflow_sats_per_day = 20000000 +max_payments_per_hour = 200 +# Ceiling on (outflow - inflow) over a rolling 24h window. This is the direct +# "more sats leaving than entering" rule. +max_net_outflow_sats_24h = 1000000 +# How long a ledger row may sit 'reserved' before the reaper resolves it +# against LND. It counts against every budget until then. +reserved_row_timeout_seconds = 300 +``` + +Validating deserializers reject non-positive caps and negative tolerances at +startup, following the `slash_node_share_pct` precedent in +`src/config/types.rs` — a typo that disables a safety limit must stop the +daemon, not silently widen it. + +## 8. Phases + +Each phase is one PR. + +### Phase 0 — Ledger, write-ahead, observe-only + +- Migration, `[circuit_breaker]` config block, `src/circuit_breaker.rs`. +- `PaymentIntent` threaded through `send_payment` and all call sites. +- Ledger rows written on both sides: `reserved`/`confirmed` at outflow points, + `in` rows at the `settle_hold_invoice` call sites of §3. +- All rules evaluate and log. **Nothing is rejected. The latch is never set.** + +Ships dark and safe. Its purpose is to produce real numbers for §7. + +### Phase 1 — Per-order invariant, persistent latch, live gate + +- `circuit_breaker_state` read/write plus the `AtomicU8` cache, loaded at boot. +- §4.1 invariant and `max_payment_sats` enforced synchronously in the gate. +- `enforce = true` becomes meaningful: violations trip and reject. +- Fail-closed behavior on DB/query errors. + +### Phase 2 — Velocity rules, watcher job, propagation + +- `job_circuit_breaker_watch` registered in `start_scheduler` + (`src/scheduler.rs:26`), inside the `!Settings::is_cashu_enabled()` block + alongside the other Lightning-only jobs. +- Rolling-window rules of §4.2 plus the reserved-row reaper. +- `job_process_dev_fee_payment`, `job_process_bond_payouts`, and + `job_retry_failed_payments` skip their tick while tripped. +- `release` / `admin-settle` handlers and order creation reject early with a + user-facing message. + +### Phase 3 — Operator surface + +- `CircuitBreakerStatus` / `CircuitBreakerReset` in `proto/admin.proto` and + `src/rpc/service.rs`. +- Admin notification on trip. +- `payments_paused` tag on the NIP-33 info event. +- Operator runbook section in `docs/LIGHTNING_OPS.md`. + +## 9. Failure modes and calibration + +- **False positives.** Routing fees, unusual bond timing, and legitimate + large trades all move the aggregates. This is why Phase 0 is observe-only and + why `max_payment_sats` should be set relative to `max_order_amount`, not + guessed. A tripped breaker is an outage; the thresholds must be earned from + data. +- **Detection latency versus damage.** A watcher on a 60-second cadence gives an + attacker a 60-second window. That is exactly why the per-order invariant and + the single-payment cap are enforced *synchronously in the gate*, not by the + job. The job catches the slow, distributed leak; the gate catches the fast one. +- **The blind spot.** Restated from §4.3 because it matters: every rule here is + computed from Mostro's own ledger. A bug that moves funds without a ledger row + is invisible to all of them. +- **Cashu mode is not covered.** The gate is Lightning-only. Cashu outflows + (`src/cashu/`) pass through no equivalent chokepoint and are out of scope for + Phases 0–3. Extending containment there is a follow-up and should be tracked + separately rather than assumed. + +## 10. Test plan + +Per phase, co-located Rust unit tests: + +- Ledger: inflow/outflow rows written on both sides; `reserved` counted as + spent; reaper resolves stale rows via a mocked `lookup_payment_status`. +- Invariant: payout equal to inflow passes; payout exceeding inflow by one sat + trips; a second payout against an already-paid order trips; range-order bond + child payout keyed on the parent order does **not** trip. +- Latch: trip persists across a simulated restart; no code path clears it except + the admin reset; reset requires a note. +- Fail-closed: a failing pool makes the gate reject rather than allow. +- Observe-only: with `enforce = false`, a violating payment is logged and still + dispatched, and the latch stays closed. +- Velocity: each rule of §4.2 trips at its threshold and not below it. +- Propagation: while tripped, the payment jobs no-op, `release` rejects, and the + info event carries `payments_paused`. +- Disabled: with no `[circuit_breaker]` block, no ledger rows are written and + every existing test stays green. diff --git a/docs/README.md b/docs/README.md index d72b088c..5c36bd6d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,6 +9,7 @@ Quick links to architecture and feature guides. - Orders & Actions: ORDERS_AND_ACTIONS.md - Admin RPC & Disputes: ADMIN_RPC_AND_DISPUTES.md - Anti-Abuse Bond: ANTI_ABUSE_BOND.md (opt-in maker/taker Lightning bond; off by default) +- Payment Circuit Breaker: PAYMENT_CIRCUIT_BREAKER.md (halts outgoing payments when outflow outruns inflow; off by default) - RPC Interface Reference: RPC.md - NIP-01 Kind 0 Metadata: NIP01_KIND0_METADATA.md From 4770837c4e60853a6aab0c84555e8e56c231c78c Mon Sep 17 00:00:00 2001 From: grunch Date: Wed, 12 Aug 2026 22:32:59 -0300 Subject: [PATCH 2/2] docs: address circuit breaker spec review - Replace every path:line reference with path + symbol (AGENTS.md policy). Several were already stale: lookup_payment_status had moved 325 -> 333 and job_process_dev_fee_payment 1087 -> 820. - Label both bare fenced blocks as text (MD040). - Make the check-and-reserve step one serialized transaction, so the trade payout and the dev-fee job cannot both reserve against the same order_id. - Derive the payment hash before reserving, so no reservation can be left unreconcilable by the reaper. - Only definitive terminal LND statuses clear a reservation; Ok(None) parks in a new 'unknown' state that keeps counting, instead of being freed for retry. - Make inflows idempotent on the settled invoice hash via a partial unique index, so an 'already settled' retry cannot inflate the allowance. - Define routing fees inside the invariant, reserved at routing_fee_cap_sats. - Drop the duplicated PaymentIntent.amount_sats field. - Add an append-only circuit_breaker_event table for the audit trail. - State that caps are strict, scope the order-creation block to Lightning mode, validate the duration settings, and extend the test plan to match. - Link the guide from the docs index. --- docs/PAYMENT_CIRCUIT_BREAKER.md | 223 ++++++++++++++++++++++++++------ docs/README.md | 2 +- 2 files changed, 185 insertions(+), 40 deletions(-) diff --git a/docs/PAYMENT_CIRCUIT_BREAKER.md b/docs/PAYMENT_CIRCUIT_BREAKER.md index 4349c8d8..19928616 100644 --- a/docs/PAYMENT_CIRCUIT_BREAKER.md +++ b/docs/PAYMENT_CIRCUIT_BREAKER.md @@ -25,8 +25,9 @@ sats" but "we lost sats all night while nobody was watching". 1. **Gate at the lowest chokepoint, not at the callers.** Every Lightning outflow in the daemon passes through `LndConnector::send_payment` - (`src/lightning/mod.rs:227`). That is where the gate lives. Gating callers - individually guarantees that whoever adds the next payment path forgets one. + (`src/lightning/mod.rs`, `fn send_payment`). That is where the gate lives. + Gating callers individually guarantees that whoever adds the next payment + path forgets one. 2. **Fail closed.** If the ledger cannot be read, the DB is unavailable, or the breaker state cannot be determined, the gate **rejects the payment**. A gate that fails open is not a gate. @@ -54,10 +55,10 @@ Every point where sats leave the node today: | Outflow | Call site | Trigger | |---|---|---| -| Trade payout to buyer | `src/app/release.rs:589` | `release` handler (hot path, user-driven) | -| Dev fee | `src/app/dev_fee.rs:993` | `job_process_dev_fee_payment` (`src/scheduler.rs:1087`) | -| Bond payout to counterparty | `src/app/bond/slash.rs`, `src/app/bond/flow.rs` | `job_process_bond_payouts` (`src/scheduler.rs:1121`) | -| Failed-payment retries | `job_retry_failed_payments` (`src/scheduler.rs:213`) | scheduler | +| Trade payout to buyer | `src/app/release.rs`, `fn do_payment` | `release` handler (hot path, user-driven) | +| Dev fee | `src/app/dev_fee.rs`, `fn send_dev_fee_payment` | `fn job_process_dev_fee_payment` (`src/scheduler.rs`) | +| Bond payout to counterparty | `src/app/bond/slash.rs`, `src/app/bond/flow.rs` | `fn job_process_bond_payouts` (`src/scheduler.rs`) | +| Failed-payment retries | `fn job_retry_failed_payments` (`src/scheduler.rs`) | scheduler | | Dispute resolution payout | `src/app/admin_settle.rs` → release path | admin/solver | All of them funnel into `LndConnector::send_payment`. @@ -71,8 +72,9 @@ budgets and flooding logs, but that is hygiene, not the control. whoever locked it and moves no funds out of the node. Inflows are recorded at `settle_hold_invoice` call sites — the escrow settle -path (`src/util.rs:1330`) and the bond slash paths (`src/app/bond/slash.rs:798`, -`src/app/bond/slash.rs:1115`). +path (`src/util.rs`, `fn settle_seller_hold_invoice`) and the bond slash paths +(`src/app/bond/slash.rs`, `fn slash_one` and +`fn resolve_range_maker_bond_at_close`). ## 4. Detection layers @@ -82,7 +84,7 @@ Two layers, evaluated at different points. > For any order, the sats paid out must never exceed the sats taken in. -``` +```text Σ outflow(order_id) + pending_amount ≤ Σ inflow(order_id) + tolerance_sats ``` @@ -94,6 +96,24 @@ invoice. It is checked **before** dispatch, so the bad payment never leaves. The direction is naturally safe: routing fees and the node fee mean a healthy order always has `out < in`. Default `tolerance_sats = 0`. +**Routing fees are inside the invariant, reserved at their cap.** `fee_sats` is +only known once a payment confirms, but the gate runs before dispatch, so a +reservation of `amount_sats` alone could approve a payment whose amount plus its +routing fee later exceeds the inflow. Both sides of the comparison are therefore +defined in fee-inclusive terms: + +- `pending_amount` for the payment under evaluation is + `amount + routing_fee_cap_sats(amount)` — the same ceiling `send_payment` + already hands LND as `fee_limit_sat`, so the reservation can never be + under-stated by the actual route. +- `Σ outflow` sums `amount_sats + COALESCE(fee_sats, routing_fee_cap_sats(amount_sats))`, + so a row that is still `reserved` keeps charging the cap and a `confirmed` row + drops back to the fee actually paid. + +Confirmation therefore only ever *releases* budget, never consumes more than was +reserved. The equal-to-inflow boundary case in §10 is evaluated against this +fee-inclusive figure. + **Bond ledger key.** Range-order bonds carry both `order_id` and `child_order_id` (see `migrations/20260423120000_anti_abuse_bond.sql`). The ledger **must** key on the parent/root `order_id` for both the slash inflow and @@ -113,6 +133,11 @@ Rolling-window aggregates over the ledger, evaluated every | Payment count in the last hour exceeds cap | `max_payments_per_hour` | | Net outflow (`Σ out − Σ in`) over 24h exceeds cap | `max_net_outflow_sats_24h` | +**Every cap is strict (`>`), never `>=`.** A value exactly equal to its +configured cap is allowed; only a value above it trips. A cap is the largest +tolerated value, so `max_payment_sats = 1000000` permits a 1 000 000 sat payment +and trips at 1 000 001. §10 tests both sides of that boundary. + `max_payment_sats` is additionally enforced **synchronously in the gate** — a single oversized payment must be stopped before it leaves, not detected a minute later. @@ -149,13 +174,17 @@ not the node's balance. pub enum PaymentKind { TradePayout, DevFee, BondPayout } /// Context the gate needs that `send_payment`'s bolt11 + amount cannot supply. +/// Deliberately carries **no amount**: `send_payment` already takes one, and a +/// second copy here could disagree with it — the ledger would then check one +/// figure while LND sent another. The gate uses the `amount` parameter as the +/// single source of truth and rejects any non-positive value before writing +/// anything, so a negative amount cannot corrupt the outflow aggregates. pub struct PaymentIntent { pub kind: PaymentKind, /// Parent/root order id — see §4.1 on range-order bonds. pub order_id: Uuid, /// Bond id for `BondPayout`, `None` otherwise. pub ref_id: Option, - pub amount_sats: i64, } pub enum BreakerState { Closed, Tripped { reason: TripReason, tripped_at: i64 } } @@ -190,26 +219,52 @@ pub async fn send_payment( Threading an explicit intent through every call site is intentional: it makes it impossible to add a new outflow without stating what it is and which order funds -it. Call sites to update: `src/app/release.rs:589`, `src/app/dev_fee.rs:993`, -and the bond payout path in `src/app/bond/`. +it. Call sites to update: `src/app/release.rs` (`fn do_payment`), +`src/app/dev_fee.rs` (`fn send_dev_fee_payment`), and the bond payout path in +`src/app/bond/`. Rejections surface as `MostroInternalErr(ServiceError::LnPaymentError("circuit breaker tripped: …"))`, reusing the existing variant so no `mostro-core` change is required. -### 5.3 Gate sequence (inside `send_payment`, before `decode_invoice`) +### 5.3 Gate sequence (inside `send_payment`) 1. Read the latch. If `Tripped`, log and return an error. No LND call. -2. If `amount > max_payment_sats`, trip and reject. -3. Query `Σ in` / `Σ out` (including `reserved`) for `intent.order_id`. If the - §4.1 invariant would be violated, trip and reject. -4. Insert the ledger row as `reserved`. -5. Dispatch to LND. +2. Reject a non-positive `amount` outright. If `amount > max_payment_sats`, trip + and reject (strict comparison, §4.2). +3. Decode the invoice to derive its `payment_hash`. A decode failure rejects + here, before anything has been written. +4. In **one serialized transaction**: re-read `Σ in` / `Σ out` (including + `reserved`) for `intent.order_id`, evaluate the §4.1 invariant against + `amount + routing_fee_cap_sats(amount)`, and — only if it holds — insert the + `reserved` ledger row carrying the `payment_hash` from step 3. A violation + rolls the transaction back, trips, and rejects. +5. Dispatch to LND, only after that transaction has committed. 6. Terminal result confirms the row (`confirmed` with the real fee, or `failed`). Any error in steps 1–4 — DB unavailable, query failure, insert failure — rejects the payment (principle 2). +**Why step 4 is one transaction.** Reading the totals and inserting the +reservation as two separate statements leaves a window in which two payouts for +the same order each observe the same totals, each conclude there is room, and +both dispatch. This is not hypothetical: the trade payout runs on the `release` +hot path while the dev-fee job pays out against **the same `order_id`**, so the +two can genuinely overlap. The check and the reservation are therefore one +atomic unit — a single `BEGIN IMMEDIATE` transaction serialized per `order_id` +— and dispatch happens only once it has committed. Splitting or reordering +steps 4 and 5 reintroduces exactly the double-payment the invariant exists to +contain. + +**Why the hash is derived before reserving.** The reaper of §5.4 reconciles a +stale `reserved` row by looking its `payment_hash` up at LND. A row written +before the hash is known cannot be reconciled by anything, and because reserved +rows count against every budget, such an orphan would charge the order forever +and eventually trip the breaker on its own. Deriving the hash first costs +nothing — decoding is local and touches no network — and guarantees every +reservation is reconcilable. The reservation still precedes any LND dispatch, +so the write-ahead property of principle 4 is unchanged. + In observe-only mode (`enforce = false`), steps 1–4 still evaluate and log at `warn!`/`error!`, and step 4 still writes the row, but nothing is rejected and the latch is not set. @@ -224,10 +279,32 @@ this design avoids. Instead, **a `reserved` row counts against every budget as if it succeeded.** Confirmation is an accuracy improvement, not a safety requirement. A reaper inside the watcher job resolves rows still `reserved` after -`reserved_row_timeout_seconds` by calling -`LndConnector::lookup_payment_status` (`src/lightning/mod.rs:325`) on the stored -hash, and marks them `confirmed` or `failed`. Until it does, the conservative -assumption stands. +`reserved_row_timeout_seconds` by calling `LndConnector::lookup_payment_status` +(`src/lightning/mod.rs`, `fn lookup_payment_status`) on the stored hash. Until +it does, the conservative assumption stands. + +**Only a definitive terminal answer may clear a reservation.** The lookup has +three outcomes and they are not interchangeable: + +| LND result | Row becomes | Why | +|---|---|---| +| `Ok(Some(Succeeded))` | `confirmed`, with the real `fee_sats` | Definitive | +| `Ok(Some(Failed))` | `failed`, budget released | Definitive | +| `Ok(Some(InFlight))` | stays `reserved` | Not terminal yet; retry next tick | +| `Ok(None)` | `unknown`, kept charged, admin notified | LND has no record — see below | +| `Err(_)` | stays `reserved` | Transport failure says nothing about the payment | + +`Ok(None)` is the dangerous one. It means LND has no record of the hash, which +covers both "never attempted" and "attempted, succeeded, and the record was +since pruned" — the two are indistinguishable from the daemon's side. Marking +such a row `failed` would release its budget and permit a retry, which is +precisely how a circuit breaker meant to *prevent* double payments would cause +one. It is therefore parked in `unknown`: it keeps counting against every budget +exactly as `reserved` did, and it is surfaced to the operator for manual +resolution rather than resolved by guessing. `unknown` is a terminal state for +the reaper — it never re-attempts the lookup — but it is not a terminal state +for the operator, who can settle it once LND's history or the counterparty +confirms what happened. ### 5.5 Trip actions @@ -246,15 +323,31 @@ pausing inbound flows other than order creation (see below), auto-resetting. New order creation is also refused while tripped. Accepting users into a system that provably cannot pay them out only widens the blast radius. +That refusal is **Lightning-mode only, and cannot reach a Cashu order.** Escrow +mode is node-wide and exclusive (`Settings::escrow_mode`): a node with `[cashu]` +enabled runs in Cashu mode, where `send_payment` is never called, the watcher +job is never registered (§8, Phase 2), and the breaker consequently can never +trip. The order-creation block is reached only on a Lightning-mode node, where +every order is Lightning-backed by definition. `accept_event` being a shared +prologue for both `run` and `run_cashu` is therefore not a hazard here — but the +check must still be written against the breaker state, which is inert in Cashu +mode, rather than assumed to run only on Lightning nodes. + ### 5.6 Reset Admin-only, via RPC (`proto/admin.proto`, `src/rpc/service.rs`), subject to the existing admin auth and rate limiting: - `CircuitBreakerStatus` — current state, reason, `tripped_at`, and the - window aggregates that drove it. -- `CircuitBreakerReset` — clears the latch. Requires an operator note, which is - persisted for the audit trail. **No timeout-based auto-reset exists.** + window aggregates that drove it, read from the single-row latch. +- `CircuitBreakerReset` — clears the latch. Requires an operator note. + **No timeout-based auto-reset exists.** + +The latch row holds **current state only**; every trip and reset overwrites the +previous one. The audit trail is a separate append-only table, +`circuit_breaker_event` (§6): one row per trip and per reset, never updated. An +operator investigating a drain needs the *first* trip, and a breaker that tripped +twice would otherwise have overwritten exactly the record that mattered. ## 6. Schema @@ -277,8 +370,8 @@ CREATE TABLE IF NOT EXISTS payment_ledger ( -- Actual routing fee, known only after a payment confirms. NULL otherwise. fee_sats integer, payment_hash char(64), - -- 'reserved' | 'confirmed' | 'failed'. A 'reserved' row counts against every - -- budget as if it had succeeded (spec §5.4). + -- 'reserved' | 'confirmed' | 'failed' | 'unknown'. 'reserved' and 'unknown' + -- both count against every budget as if they had succeeded (spec §5.4). state varchar(10) not null, -- Unix timestamps in seconds, matching the rest of the schema. created_at integer not null, @@ -289,15 +382,26 @@ CREATE INDEX IF NOT EXISTS idx_payment_ledger_order ON payment_ledger(order_id); CREATE INDEX IF NOT EXISTS idx_payment_ledger_created ON payment_ledger(created_at); CREATE INDEX IF NOT EXISTS idx_payment_ledger_state ON payment_ledger(state); --- Single-row latch. Persistent so a restart cannot clear a trip. +-- Inflows are idempotent on the settled invoice's hash. A retry that finds the +-- invoice already settled (the "already settled" branch of the bond slash +-- paths) must not book the same incoming HTLC twice: a duplicated inflow row +-- inflates the per-order allowance and would license a second payout. Writers +-- use INSERT OR IGNORE against this index. Outflows are excluded because a +-- failed payment may legitimately be retried under the same hash. +CREATE UNIQUE INDEX IF NOT EXISTS idx_payment_ledger_inflow_hash + ON payment_ledger(payment_hash) WHERE direction = 'in' AND payment_hash IS NOT NULL; + +-- Single-row latch holding CURRENT STATE ONLY. Persistent so a restart cannot +-- clear a trip. Each trip and reset overwrites these fields; the history lives +-- in circuit_breaker_event below (spec §5.6). CREATE TABLE IF NOT EXISTS circuit_breaker_state ( id integer primary key check (id = 1), -- 'closed' | 'tripped' state varchar(8) not null, - -- Serialized TripReason. NULL while closed. + -- Serialized TripReason of the current trip. NULL while closed. reason text, tripped_at integer, - -- Operator note supplied at reset time, retained as an audit trail. + -- Note from the most recent reset. NULL until one happens. reset_note text, reset_at integer, updated_at integer not null @@ -305,11 +409,29 @@ CREATE TABLE IF NOT EXISTS circuit_breaker_state ( INSERT OR IGNORE INTO circuit_breaker_state (id, state, updated_at) VALUES (1, 'closed', 0); + +-- Append-only audit trail: one row per trip and per reset, never updated or +-- deleted. This is what an operator reads when reconstructing an incident — +-- the first trip is the interesting one, and the latch alone would have +-- overwritten it on the second. +CREATE TABLE IF NOT EXISTS circuit_breaker_event ( + id char(36) primary key not null, + -- 'trip' | 'reset' + event varchar(5) not null, + -- Serialized TripReason for 'trip'; NULL for 'reset'. + reason text, + -- Operator note for 'reset'; NULL for 'trip'. + note text, + created_at integer not null +); + +CREATE INDEX IF NOT EXISTS idx_circuit_breaker_event_created + ON circuit_breaker_event(created_at); ``` Example rows (synthetic) for a completed trade of 50 000 sats: -``` +```text id direction kind order_id amount_sats fee_sats state created_at 7f1c…-…-0001 in escrow-settle 3a9e…-0007 50100 NULL confirmed 1786500000 7f1c…-…-0002 out trade-payout 3a9e…-0007 50000 12 confirmed 1786500004 @@ -348,10 +470,16 @@ max_net_outflow_sats_24h = 1000000 reserved_row_timeout_seconds = 300 ``` -Validating deserializers reject non-positive caps and negative tolerances at -startup, following the `slash_node_share_pct` precedent in -`src/config/types.rs` — a typo that disables a safety limit must stop the -daemon, not silently widen it. +Validating deserializers reject the following at startup, following the +`slash_node_share_pct` precedent in `src/config/types.rs` — a typo that disables +a safety limit must stop the daemon, not silently widen it: + +- non-positive caps (`max_payment_sats`, the outflow and count caps, + `max_net_outflow_sats_24h`); +- negative `tolerance_sats`; +- non-positive `check_interval_seconds` — zero would busy-loop the watcher job; +- non-positive `reserved_row_timeout_seconds` — zero would make every + reservation instantly stale, so the reaper would race live payments to LND. ## 8. Phases @@ -376,8 +504,8 @@ Ships dark and safe. Its purpose is to produce real numbers for §7. ### Phase 2 — Velocity rules, watcher job, propagation -- `job_circuit_breaker_watch` registered in `start_scheduler` - (`src/scheduler.rs:26`), inside the `!Settings::is_cashu_enabled()` block +- `job_circuit_breaker_watch` registered in `src/scheduler.rs` + (`fn start_scheduler`), inside the `!Settings::is_cashu_enabled()` block alongside the other Lightning-only jobs. - Rolling-window rules of §4.2 plus the reserved-row reaper. - `job_process_dev_fee_payment`, `job_process_bond_payouts`, and @@ -418,15 +546,32 @@ Per phase, co-located Rust unit tests: - Ledger: inflow/outflow rows written on both sides; `reserved` counted as spent; reaper resolves stale rows via a mocked `lookup_payment_status`. +- Reaper status handling (§5.4 table): `Succeeded` confirms with the real fee; + `Failed` releases the budget; `InFlight` and `Err(_)` leave the row + `reserved`; `Ok(None)` moves it to `unknown` and it **keeps counting against + every budget** — the regression test asserts that a pruned-but-succeeded + payment is never released for retry. +- Inflow idempotency: replaying a settled invoice (the "already settled" retry + branch) writes exactly one `in` row, and the per-order allowance is unchanged + by the replay. - Invariant: payout equal to inflow passes; payout exceeding inflow by one sat trips; a second payout against an already-paid order trips; range-order bond - child payout keyed on the parent order does **not** trip. + child payout keyed on the parent order does **not** trip. Boundary cases are + evaluated fee-inclusive per §4.1. +- Concurrency: two payouts for the same `order_id` submitted simultaneously — + exactly one reserves and dispatches, the other is rejected. Asserts the §5.3 + check-and-reserve transaction is genuinely atomic. - Latch: trip persists across a simulated restart; no code path clears it except the admin reset; reset requires a note. +- Audit trail: two consecutive trips leave two `circuit_breaker_event` rows, and + the first one is still readable after the second. - Fail-closed: a failing pool makes the gate reject rather than allow. - Observe-only: with `enforce = false`, a violating payment is logged and still dispatched, and the latch stays closed. -- Velocity: each rule of §4.2 trips at its threshold and not below it. +- Config validation: zero or negative `check_interval_seconds` and + `reserved_row_timeout_seconds` are rejected at startup. +- Velocity: each rule of §4.2 passes at exactly its cap and trips one unit + above it (strict `>`, per §4.2). - Propagation: while tripped, the payment jobs no-op, `release` rejects, and the info event carries `payments_paused`. - Disabled: with no `[circuit_breaker]` block, no ledger rows are written and diff --git a/docs/README.md b/docs/README.md index 5c36bd6d..5e21b354 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,7 +9,7 @@ Quick links to architecture and feature guides. - Orders & Actions: ORDERS_AND_ACTIONS.md - Admin RPC & Disputes: ADMIN_RPC_AND_DISPUTES.md - Anti-Abuse Bond: ANTI_ABUSE_BOND.md (opt-in maker/taker Lightning bond; off by default) -- Payment Circuit Breaker: PAYMENT_CIRCUIT_BREAKER.md (halts outgoing payments when outflow outruns inflow; off by default) +- [Payment Circuit Breaker](./PAYMENT_CIRCUIT_BREAKER.md) (halts outgoing payments when outflow outruns inflow; off by default) - RPC Interface Reference: RPC.md - NIP-01 Kind 0 Metadata: NIP01_KIND0_METADATA.md