If there's some issue preventing us from getting bootstrapped because there is no snapshot, no existing accountsdb, and a failure to acquire a snapshot and feed it into replay, we just see logs like this and then it hangs forever:
info: Starting Service `net`, pid: 2563680
info: Starting Service `gossip`, pid: 2563681
info: Starting Service `shred_receiver`, pid: 2563682
info: Starting Service `replay`, pid: 2563683
info: Starting Service `snapshot`, pid: 2563684
info: Starting Service `accounts_db`, pid: 2563685
info: Starting Service `telemetry`, pid: 2563686
info: Starting Service `exec`, pid: 2563687
time=2026-07-21T23:38:32.209Z level=info scope=main service=net message="binding 0.0.0.0:8001"
time=2026-07-21T23:38:32.209Z level=info scope=main service=net message="binding 0.0.0.0:8002"
time=2026-07-21T23:38:32.209Z level=info scope=main port=8001 identity=3W1V1Cxf3PfvpRyk3becDgKnVZYhc7EJfNqcYVvX297W shred_version=1516 entrypoints="{ 35.203.170.30:8001, 109.94.99.177:8001 }" service=gossip message="Gossip started"
time=2026-07-21T23:38:32.210Z level=info scope=main service=shred_receiver message="Waiting for shreds on port 8002"
time=2026-07-21T23:38:32.210Z level=info scope=main service=snapshot message="snapshot path ./validator"
time=2026-07-21T23:38:34.854Z level=info scope=main service=accounts_db message="accounts_db started into file: ./validator/accounts.db"
time=2026-07-21T23:38:34.855Z level=info scope=Rooted.init service=accounts_db message="loading from existing rooted db"
time=2026-07-21T23:38:34.857Z level=info scope=Rooted.writeJournal service=accounts_db message="writing journal: .{ .magic = .valid, .state = .empty, .writing_slot = 0, .committed_slot = 0, .committed_offset = 0, .blockhash_max_age = 300 }"
time=2026-07-21T23:38:34.862Z level=info scope=main service=accounts_db message="no existing rooted db. reading from snapshot"
This is insufficiently informative to the operator. We need some logs to indicate that something is wrong. If there are any services that are actively blocked while awaiting for bootstrapping inputs, this should be indicated to the operator
- gossip should log a message if it's not getting any data from the cluster
- snapshot service should log that it's awaiting snapshot peers from gossip
- snapshot service should log if none of the snapshot peers are usable
- accountsdb should log that it is awaiting a snapshot
- replay should log that it's awaiting the runtime metadata from accountsdb
none of these should be logged very frequently, maybe once every 10 seconds at most. And they should not be messages that would trigger an alert (e.g. by logging as error level), not at first anyway. They can become alert-triggering messages if they exceed some timeout, for example:
- gossip gets no messages after one minute
- snapshot gets no snapshot peers after one minute
- accountsdb gets no snapshot after 5 minutes
- replay gets no snapshot data after 10 minutes
(these are just rough arbitrary guesses at what might be reasonable. it should be based on actual experience running sig)
for any service that logs when it's awaiting data, it should also log an info level message once it receives that data
If there's some issue preventing us from getting bootstrapped because there is no snapshot, no existing accountsdb, and a failure to acquire a snapshot and feed it into replay, we just see logs like this and then it hangs forever:
This is insufficiently informative to the operator. We need some logs to indicate that something is wrong. If there are any services that are actively blocked while awaiting for bootstrapping inputs, this should be indicated to the operator
none of these should be logged very frequently, maybe once every 10 seconds at most. And they should not be messages that would trigger an alert (e.g. by logging as
errorlevel), not at first anyway. They can become alert-triggering messages if they exceed some timeout, for example:(these are just rough arbitrary guesses at what might be reasonable. it should be based on actual experience running sig)
for any service that logs when it's awaiting data, it should also log an info level message once it receives that data