feat: liveness-based consumer removal, snapshot resync, and safe mode

docs/consumer-lifecycle.md

@@ -2,6 +2,8 @@
 
 This document describes the full lifecycle of a consumer chain in VAAS, from registration to deletion.
 
+For how an unresponsive or lagging launched consumer is handled -- the liveness grace period, removal sweep, snapshot resync, and consumer safe mode -- see [consumer-liveness.md](consumer-liveness.md).
+
 ## Phases
 
 ```
@@ -104,13 +106,17 @@ is stored and used for all subsequent VSC packet delivery.
 3. The relayer relays the packets to the consumer.
 4. The consumer applies the validator set changes on `EndBlock`.
 
+VSC packets are diffs by default. If a consumer falls behind on acknowledgements, the provider instead sends an absolute snapshot of the full validator set so the consumer resyncs in a single packet, and the consumer is removed only after a sustained liveness failure rather than on a single packet timeout. See [consumer-liveness.md](consumer-liveness.md).
+
 ---
 
 ## Phase 4: STOPPED
 
-**Trigger:** `MsgRemoveConsumer` submitted by the consumer owner.
+**Triggers:** either
+- `MsgRemoveConsumer` submitted by the governance authority (removing a consumer requires the gov authority), or
+- the automatic liveness sweep, when a launched consumer has produced no successful VSC acknowledgement for longer than the liveness grace period (see [consumer-liveness.md](consumer-liveness.md)).
 
-**Requirements:** consumer must be in `LAUNCHED` phase. Only the owner can submit.
+**Requirements:** consumer must be in `LAUNCHED` phase.
 
 **What happens on-chain:**
 1. Phase is set to `STOPPED`.
@@ -129,7 +135,8 @@ is stored and used for all subsequent VSC packet delivery.
 1. Up to 200 due consumers are dequeued per block.
 2. For each consumer, `DeleteConsumerChain` runs:
    - Deletes: IBC client ID mapping, consumer genesis, key assignments, equivocation
-     evidence minimum height, pending VSC packets, validator set, removal time.
+     evidence minimum height, pending VSC packets, validator set, removal time, and the
+     liveness state (last-ack time and the sent/acked VSC-id counters).
    - **Preserves** (for block explorer use): chain ID, phase, owner address, metadata,
      initialization parameters.
 3. Phase is set to `DELETED`.
@@ -143,5 +150,5 @@ is stored and used for all subsequent VSC packet delivery.
 | `REGISTERED` | `MsgCreateConsumer` | Any account | Consumer created, owner assigned |
 | `INITIALIZED` | `spawn_time` set | On-chain (automatic) | Queued for launch at spawn_time |
 | `LAUNCHED` | `spawn_time` elapsed | On-chain (BeginBlock) | Genesis built; operator starts consumer; relayer creates IBC path |
-| `STOPPED` | `MsgRemoveConsumer` | Owner | Queued for deletion after unbonding period |
+| `STOPPED` | `MsgRemoveConsumer` (gov) or liveness sweep | Governance / on-chain | Queued for deletion after unbonding period |
 | `DELETED` | Unbonding period elapsed | On-chain (BeginBlock) | State cleaned up |

docs/consumer-liveness.md

@@ -0,0 +1,192 @@
+# Consumer Liveness, Removal, and Resync
+
+This document describes how the provider handles an unresponsive or lagging consumer
+chain: when (and why) a consumer is removed, how a consumer that briefly falls behind
+heals itself, and how a consumer protects itself while its validator set may be stale.
+
+It complements [consumer-lifecycle.md](consumer-lifecycle.md), which covers the overall
+`REGISTERED -> INITIALIZED -> LAUNCHED -> STOPPED -> DELETED` progression. This document
+zooms in on what keeps a `LAUNCHED` consumer healthy and what happens when it is not.
+
+## Design rationale
+
+The provider sends a Validator Set Change (VSC) packet to each launched consumer every
+epoch. Two facts about that delivery shape the design:
+
+1. **The packet timeout is effectively 24h.** VSC packets are sent with a timeout of
+   `min(VaasTimeoutPeriod, MaxTimeoutDelta)`, and IBC v2 hard-caps `MaxTimeoutDelta` at 24h.
+   A packet timeout is therefore a weak signal -- a 24h outage of a single relayer or RPC
+   produces one -- so removing a consumer must not hinge on a single delivery failure.
+2. **VSC packets are diffs.** Each packet carries only the validators that changed since the
+   previous one, applied on top of the consumer's current set. A lost diff is never re-sent
+   on its own, so a departed validator could otherwise linger in the consumer's active set --
+   able to unbond on the provider and validate the consumer with no slashable stake.
+
+The model has three parts that answer these facts: a provider-local liveness clock that
+decides removal on the provider's own block time rather than on per-packet delivery; a
+self-healing snapshot resync so a lost diff cannot strand a stale validator; and a
+consumer-side safe mode for the window in which a consumer's set may be out of date.
+
+## 1. Liveness clock and removal sweep
+
+The provider tracks, per consumer, the block time of the last **successful** VSC
+acknowledgement (`lastAck`). It is seeded when the consumer enters `LAUNCHED` and refreshed
+every time the consumer acknowledges a VSC packet. A successful ack is the strongest
+possible progress signal: it proves the consumer received *and applied* the update.
+
+Each `BeginBlock`, the provider sweeps every launched consumer and stops any whose `lastAck`
+is older than a **grace period**:
+
+```
+grace = providerUnbondingPeriod * LivenessGraceFraction
+```
+
+derived from the provider's own unbonding period (about 14 days at the 21-day default and the
+`0.66` default fraction). `LivenessGraceFraction` is a provider module param: its default
+mirrors the IBC client trusting-period fraction, so the grace ends around the point where
+recovery stops being possible anyway while still leaving margin below the unbonding
+(slashable) window, but operators can lower it -- on short-lived test or dev chains, for
+instance -- to make the sweep fire in seconds rather than days. The sweep runs on the
+provider's own block clock, so it does not depend
+on a live IBC client or an available relayer -- it removes a consumer whether the silence is
+caused by a dead consumer, a dead relayer, or an expired client.
+
+When the grace period is exceeded, the consumer is moved to `STOPPED` (then `DELETED` after
+the unbonding period) exactly as a governance removal would do. See
+[consumer-lifecycle.md](consumer-lifecycle.md) for the `STOPPED -> DELETED` mechanics.
+
+**Guarantee:** a launched-but-unresponsive consumer is removed within the grace period,
+which is strictly less than the provider's unbonding period. So every validator that was in
+the consumer's set as of its last successful sync is still slashable on the provider at the
+time of removal.
+
+## 2. Snapshot resync (self-healing)
+
+Because VSC packets are diffs, a consumer that misses packets during an outage could diverge
+from the provider's true validator set. To prevent that, the provider tracks two per-consumer
+counters: the highest VSC id it has **sent** and the highest the consumer has **acked**. A
+consumer is **behind** when:
+
+```
+highestAcked < highestSent
+```
+
+While a consumer is behind, the provider does not send a diff. It sends an **absolute
+snapshot**: the complete current validator set, flagged with `is_snapshot = true` on the
+packet. The consumer, on receiving a snapshot, **replaces** its set with it -- setting each
+listed validator to its absolute power and removing (power 0) any validator it currently has
+that the snapshot omits. Applying a snapshot is idempotent: a behind consumer converges to
+exactly the provider's current set in one packet, regardless of which intermediate packets it
+missed. Once an ack catches `highestAcked` up to `highestSent`, the provider resumes sending
+ordinary diffs.
+
+The wire format gains only a single boolean (`is_snapshot`); the validator-update list is
+reused (a diff when false, the full set when true). The global VSC id counter and the
+consumer's out-of-order dedup are unchanged.
+
+**Guarantee:** a transient outage does not corrupt the consumer's set. As soon as
+connectivity returns, the next snapshot heals it -- including removing any validator that
+left the provider during the outage.
+
+## 3. IBC callbacks are log-only
+
+The sweep is the single authority for removing an unresponsive consumer, so the IBC
+delivery-failure callbacks only log and return:
+
+- **Packet timeout** (`OnTimeoutPacketV2`): logs and returns; the next epoch retries.
+- **Error acknowledgement**: logs and returns; the consumer keeps its grace budget.
+- **Local send failure** (including an expired client): logs, leaves the pending packets
+  queued, and returns; the next epoch retries.
+
+This keeps an expired client from stranding a consumer: an expired client produces no acks,
+so the sweep removes such a consumer once its grace elapses.
+
+## 4. Consumer safe mode
+
+While a consumer is behind, its local validator set may not match the provider's. The
+consumer protects itself during that window. It records the block time of the last VSC packet
+it received, and considers itself **stale** when it has not received one for longer than a
+consumer-local threshold (`SafeModeThreshold`, default 3 hours). The threshold is required to
+be strictly below the provider's liveness grace, enforced at `MsgCreateConsumer` /
+`MsgUpdateConsumer` time (see section 5), so a lagging consumer always enters safe mode before
+the provider's sweep would remove it.
+
+While stale (or while flagged in debt for unpaid fees), the consumer's transaction admission
+gate restricts incoming transactions to `/ibc.core.*` and `/cosmos.gov.*` messages only,
+rejecting value-bearing application transactions. This bounds the damage of a possibly-stale
+set: the consumer refuses to do value-bearing work under a validator set it cannot trust,
+while keeping the recovery path open -- `/ibc.core.*` still admits the incoming VSC packets
+that let the consumer resync, and `/cosmos.gov.*` keeps governance available. A freshly
+launched consumer is never treated as stale before its first VSC.
+
+## 5. Parameters and bounds
+
+| Parameter | Where | Bound | Default |
+|---|---|---|---|
+| `VaasTimeoutPeriod` | provider module param (VSC packet timeout) and per-consumer init param (consumer evidence packet timeout) | `(0, MaxTimeoutDelta (24h)]` | 1h |
+| consumer `UnbondingPeriod` | per-consumer init param | `(0, providerUnbondingPeriod]` | provider default minus 1 day |
+| `LivenessGraceFraction` | provider module param | `(0, 1)` | `0.66` |
+| consumer `SafeModeThreshold` | per-consumer init param | `(0, provider liveness grace)` | 3h |
+
+`VaasTimeoutPeriod` is validated to `(0, 24h]` at both the provider module param boundary
+and the per-consumer initialization-parameter boundary, so the configured value is honest
+rather than silently capped. The default is one epoch-scale hour: an undelivered packet is
+superseded by the next epoch's packet and a late one is dropped by the consumer's dedup, so
+a long timeout buys nothing. No lower floor is imposed either: a persistently unreachable
+consumer is handled by the liveness sweep, so a short timeout is not dangerous.
+
+The consumer `UnbondingPeriod` is bounded against the provider's at `MsgCreateConsumer` /
+`MsgUpdateConsumer` time (it must not exceed the provider's), so a consumer cannot be
+configured to outlive the provider's slashable window. No lower floor is enforced: an
+unbonding short enough to make the relayer-derived trusting period impractical is an
+operator concern (and is useful for short-lived test or dev chains), not a protocol minimum.
+
+The consumer `SafeModeThreshold` is bounded at the same `MsgCreateConsumer` /
+`MsgUpdateConsumer` boundary: it must be strictly less than the provider's liveness grace
+(`unbonding * LivenessGraceFraction`). Both values are cross-chain, but the provider holds
+each at configuration time -- the threshold is an init parameter it ships to the consumer,
+and the grace derives from its own unbonding and fraction -- so the relationship is enforced
+provider-side rather than left to operator discipline. This guarantees the consumer's own
+safe mode engages before the provider's sweep removes it.
+
+## 6. Operator guidance: the liveness query
+
+Operators can observe a consumer's liveness without waiting for removal:
+
+```
+providerd query provider consumer-liveness <consumer-id>
+```
+
+It returns, all derived (no stored extra state):
+
+- `last_ack_time` -- block time of the consumer's last successful VSC ack.
+- `grace_period` -- the derived grace for this provider.
+- `removal_eta` -- `last_ack_time + grace_period`: when the sweep will remove the consumer
+  if no further ack arrives.
+- `degraded` -- true once `last_ack_time` is older than half the grace period; an early
+  warning that a consumer is trending toward removal.
+
+A `degraded` consumer that is genuinely reachable should recover on its own once VSC packets
+flow again (the next snapshot resyncs it and refreshes `lastAck`). If it does not recover
+before `removal_eta`, it will be stopped.
+
+## 7. Guarantees and limitations
+
+**Guarantees**
+
+- A consumer is removed only after sustained ack-silence approaching, but staying within,
+  the provider's slashable window -- never on a single transient packet failure.
+- A consumer that recovers within the grace period self-heals to the provider's exact
+  validator set via snapshot resync, with no operator or governance action.
+- While a consumer's set may be stale, it refuses value-bearing transactions but keeps the
+  IBC recovery path open.
+
+**Out of scope (candidates for follow-up work)**
+
+- **Owner/governance force-resume and an explicit `DEGRADED` phase.** Automatic recovery
+  covers the common case; a manual "buy more time" path and a distinct degraded marker are
+  deferred. The `lastAck` field is the groundwork for them.
+- **Reviving a consumer whose IBC client has expired** (beyond the trusting period). This
+  needs substitute-client / client-recovery governance and is a separate effort.
+- **Relayer redundancy (multi-RPC / failover).** The single-RPC risk that motivated the
+  original concern is a relayer-tooling matter, not a chain-side one.

proto/vaas/consumer/v1/genesis.proto

@@ -29,6 +29,11 @@ message GenesisState {
   // Flag indicating whether the consumer VAAS module starts in pre-VAAS state
   bool preVAAS = 5;
   vaas.v1.ProviderInfo provider = 6 [ (gogoproto.nullable) = false ];
+  // LastVSCRecvTime is the block time of the last VSC packet the consumer
+  // received; it drives IsVSCStale (safe mode). Round-tripped on restart so a
+  // state-export upgrade preserves the staleness clock. Absent for a new chain
+  // that has not received a VSC yet.
+  google.protobuf.Timestamp last_vsc_recv_time = 7 [ (gogoproto.stdtime) = true ];
 }
 
 // HeightValsetUpdateID represents a mapping internal to the consumer VAAS module

proto/vaas/provider/v1/genesis.proto

@@ -86,6 +86,17 @@ message ConsumerState {
   // absent otherwise. Used to re-derive the keeper's removal-time queue.
   google.protobuf.Timestamp removal_time = 12
       [ (gogoproto.stdtime) = true ];
+
+  // Liveness clock, present once a consumer has launched: LastAckTime is the
+  // block time of the consumer's most recent successful VSC acknowledgement
+  // (the provider's per-consumer grace clock), and HighestSentVscId /
+  // HighestAckedVscId are the resync counters that decide whether the consumer
+  // is behind. Round-tripped so a state-export restart preserves the grace
+  // window and resync state instead of resetting them.
+  google.protobuf.Timestamp last_ack_time = 13
+      [ (gogoproto.stdtime) = true ];
+  uint64 highest_sent_vsc_id = 14;
+  uint64 highest_acked_vsc_id = 15;
 }
 
 // ValsetUpdateIdToHeight defines the genesis information for the mapping

proto/vaas/provider/v1/provider.proto

@@ -26,21 +26,27 @@ message Params {
   // TrustingPeriodFraction is used to compute the consumer and provider IBC
   // client's TrustingPeriod from the chain defined UnbondingPeriod
   string trusting_period_fraction = 1;
+
+  // LivenessGraceFraction sets the consumer liveness grace period as a fraction
+  // of the provider unbonding period (grace = unbonding * fraction). A consumer
+  // with no successful VSC acknowledgement for longer than the grace is removed.
+  string liveness_grace_fraction = 2;
+
   // Sent IBC packets will timeout after this duration
-  google.protobuf.Duration vaas_timeout_period = 2
+  google.protobuf.Duration vaas_timeout_period = 3
       [ (gogoproto.nullable) = false, (gogoproto.stdduration) = true ];
 
   // The number of blocks that comprise an epoch.
-  int64 blocks_per_epoch = 3;
+  int64 blocks_per_epoch = 4;
 
   // The maximal number of validators that will be passed
   // to the consensus engine on the provider.
-  int64 max_provider_consensus_validators = 4;
+  int64 max_provider_consensus_validators = 5;
 
   // The per-block fee amount charged for consumer chain operation. The denom
   // is not a parameter: it is fixed at module wiring (see Keeper.feeDenom) and
   // cannot be changed without a binary upgrade.
-  string fees_per_block_amount = 5 [
+  string fees_per_block_amount = 6 [
     (cosmos_proto.scalar) = "cosmos.Int",
     (gogoproto.customtype) = "cosmossdk.io/math.Int",
     (gogoproto.nullable) = false,
@@ -51,7 +57,7 @@ message Params {
   // fees_per_block.Amount (the floor is fees_per_block.Amount *
   // min_deposit_blocks). Zero disables the check. The floor applies to every
   // funder including the gov authority.
-  uint64 min_deposit_blocks = 6;
+  uint64 min_deposit_blocks = 7;
 }
 
 // AddressList contains a list of consensus addresses
@@ -174,6 +180,10 @@ message ConsumerInitializationParameters {
   // This param is a part of the cosmos sdk staking module. In the case of
   // a VAAS enabled consumer chain, the VAAS module acts as the staking module.
   int64 historical_entries = 7;
+  // How long the consumer may go without receiving a VSC packet before its
+  // tx admission gate enters safe mode (only ibc.core and gov messages pass).
+  google.protobuf.Duration safe_mode_threshold = 8
+      [ (gogoproto.nullable) = false, (gogoproto.stdduration) = true ];
 }
 
 // ConsumerIds contains consumer ids of chains

proto/vaas/provider/v1/query.proto

@@ -6,6 +6,7 @@ option go_package = "github.com/allinbits/vaas/x/vaas/provider/types";
 import "google/api/annotations.proto";
 import "gogoproto/gogo.proto";
 import "google/protobuf/timestamp.proto";
+import "google/protobuf/duration.proto";
 import "vaas/provider/v1/genesis.proto";
 import "vaas/provider/v1/provider.proto";
 import "vaas/v1/shared_consumer.proto";
@@ -132,6 +133,13 @@ service Query {
   rpc ConsumerFeePoolClaims(QueryConsumerFeePoolClaimsRequest) returns (QueryConsumerFeePoolClaimsResponse) {
     option (google.api.http).get = "/vaas/provider/v1/consumer_fee_pool_claims/{consumer_id}";
   }
+
+  // QueryConsumerLiveness returns the liveness state of a launched consumer.
+  rpc QueryConsumerLiveness(QueryConsumerLivenessRequest)
+      returns (QueryConsumerLivenessResponse) {
+    option (google.api.http).get =
+        "/vaas/provider/consumer_liveness/{consumer_id}";
+  }
 }
 
 message QueryConsumerGenesisRequest {
@@ -358,3 +366,11 @@ message QueryConsumerFeePoolClaimsResponse {
   repeated DepositorClaim claims = 1 [(gogoproto.nullable) = false];
   cosmos.base.query.v1beta1.PageResponse pagination = 2;
 }
+
+message QueryConsumerLivenessRequest { uint64 consumer_id = 1; }
+message QueryConsumerLivenessResponse {
+  google.protobuf.Timestamp last_ack_time = 1 [(gogoproto.stdtime) = true, (gogoproto.nullable) = false];
+  google.protobuf.Duration grace_period = 2 [(gogoproto.stdduration) = true, (gogoproto.nullable) = false];
+  google.protobuf.Timestamp removal_eta = 3 [(gogoproto.stdtime) = true, (gogoproto.nullable) = false];
+  bool degraded = 4;
+}

proto/vaas/v1/shared_consumer.proto

@@ -39,6 +39,11 @@ message ConsumerParams {
     // which should be smaller than that of the provider in general.
     google.protobuf.Duration unbonding_period = 4
         [ (gogoproto.nullable) = false, (gogoproto.stdduration) = true ];
+
+    // How long the consumer may go without receiving a VSC packet before its
+    // tx admission gate enters safe mode (only ibc.core and gov messages pass).
+    google.protobuf.Duration safe_mode_threshold = 5
+        [ (gogoproto.nullable) = false, (gogoproto.stdduration) = true ];
 }
 
 // ConsumerGenesisState defines shared genesis information between provider and