Skip to content

Call SetFinality when processing a milestone#2226

Open
sduchesneau wants to merge 3 commits into
0xPolygon:developfrom
streamingfast:fix/finality
Open

Call SetFinality when processing a milestone#2226
sduchesneau wants to merge 3 commits into
0xPolygon:developfrom
streamingfast:fix/finality

Conversation

@sduchesneau

@sduchesneau sduchesneau commented May 14, 2026

Copy link
Copy Markdown
Contributor

Description

bc.CurrentFinalBlock() is currently always nil on bor nodes, because the only writers of blockchain.SetFinalized in upstream geth are the Engine API (eth/catalyst), which bor does not use. As a result:

  • The headFinalizedBlockGauge / headSafeBlockGauge metrics never advance.
  • Any consumer that uses the standard geth BlockChain API to read finality — live tracers (core.BlockchainLogger), external indexers, downstream tooling — sees no finality on bor, even though Heimdall milestones are by definition the chain's strongest finality signal.

Bor already knows what's finalized: eth/api.go:getFinalizedBlockNumber derives it from Downloader().GetWhitelistedMilestone() to answer the eth_getBlockByNumber("finalized") RPC. This PR closes the gap by also writing that information into the canonical place (bc.SetFinalized) the moment a milestone has been successfully processed in ethHandler.handleMilestone, right after the existing ProcessMilestone call. The header is fetched defensively via GetHeaderByHash; an unknown hash is a no-op.

This PR only writes to CurrentFinalBlock() from milestones, because milestones are the strongest/live signal. The RPC finalized-tag path in getFinalizedBlockNumber (milestone → checkpoint fallback) is unchanged.

Changes

  • Bugfix (non-breaking change that solves an issue)
  • Hotfix
  • New feature
  • Breaking change
  • Changes only for a subset of nodes

Breaking changes

None at the protocol or RPC level. Two observable effects:

  1. headFinalizedBlockGauge and headSafeBlockGauge will now advance on bor nodes (they were stuck at 0). Dashboards may need to acknowledge this.
  2. The SetHead invalidated finalized block log path in core/blockchain.go becomes reachable. By construction it only triggers on a rewind below a confirmed Heimdall milestone, which is already a pathological situation — making it visible is the desired behavior.

he JSON-RPC eth_getBlockByNumber("finalized") response is unchanged: it still flows through getFinalizedBlockNumber and the downloader whitelist.

Testing

  • Unit tests
  • Manual on mainnet:
    • eth_getBlockByNumber("finalized") continues to return the latest milestoned block (unchanged).
    • headFinalizedBlockGauge advances after the first milestone is processed (new behavior).
    • Live tracer / BlockChain.CurrentFinalBlock() returns the milestone header (new behavior, previously nil).

Copilot AI review requested due to automatic review settings May 14, 2026 20:07

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@pratikspatil024

Copy link
Copy Markdown
Member

@claude review once

@claude

claude Bot commented May 15, 2026

Copy link
Copy Markdown

Claude finished @pratikspatil024's task in 1m 33s —— View job


Review: Call SetFinalized when processing a milestone

  • Gather context and read changed files
  • Review the diff against origin/develop
  • Analyze correctness, security, and edge cases
  • Post review findings

Overall: This is a well-motivated, clean fix. The PR description accurately identifies the gap — Bor's milestone finality was never written to BlockChain.SetFinalized, leaving CurrentFinalBlock() permanently nil. The placement is correct: right after ProcessMilestone succeeds, with a defensive nil check on the header lookup.

Correctness

Looks good. The change is placed after ProcessMilestone(num, hash) at eth/handler_bor.go:129, which only runs when verifier.verify has succeeded (errors return early at lines 97–117). The GetHeaderByHash nil check at line 132 is the right defensive pattern — if the block has been pruned or is somehow missing, it's a no-op rather than a panic.

Thread safety is fine: SetFinalized uses atomic.Pointer[types.Header] for the in-memory store (core/blockchain.go:390, :1130), and rawdb.WriteFinalizedBlockHash is a single DB write.

Persistence across restarts

SetFinalized calls rawdb.WriteFinalizedBlockHash (core/blockchain.go:1133), and loadLastState reads it back on startup (core/blockchain.go:992-998). So the finalized marker will survive node restarts — good.

Suggestions (non-blocking)

1. Consider also calling SetSafe

In the Engine API path (eth/catalyst/api.go:403-417), both SetFinalized and SetSafe are set. On startup, loadLastState also sets both from the finalized hash (core/blockchain.go:994-997). However, this PR only calls SetFinalized, leaving currentSafeBlock as nil until the next restart (when loadLastState will copy finalized → safe).

For Bor, milestones represent the strongest finality signal, so the finalized block is by definition also "safe." Adding eth.blockchain.SetSafe(header) alongside SetFinalized would keep headSafeBlockGauge advancing in real-time too, and match the behavior consumers expect (safe ≤ finalized).

if header := eth.blockchain.GetHeaderByHash(hash); header != nil {
    eth.blockchain.SetFinalized(header)
    eth.blockchain.SetSafe(header)
}

2. Re-processing on the 1-second ticker

handleMilestone is called every 1 second via the ticker at eth/backend.go:893-901 to re-process the same milestone. With this PR, that means SetFinalized + WriteFinalizedBlockHash (a DB write) fires every second for the same block. This is functionally harmless (idempotent write of the same value), but worth noting. If you want to avoid the redundant DB writes, a simple guard like checking bc.CurrentFinalBlock() before writing would do — but this is a micro-optimization, not a correctness concern.

3. No monotonicity check

There's no check that the new finalized block number is ≥ the current finalized block number. In practice this should always hold because milestones are monotonically increasing, but a defensive check would make the invariant explicit:

if header := eth.blockchain.GetHeaderByHash(hash); header != nil {
    if current := eth.blockchain.CurrentFinalBlock(); current == nil || header.Number.Uint64() > current.Number.Uint64() {
        eth.blockchain.SetFinalized(header)
    }
}

Again, non-blocking — the milestone protocol already guarantees monotonicity.

No issues found

  • No security concerns (no new external input, no panics in consensus path)
  • No concurrency issues (atomic store, single DB write)
  • Defensive nil check is present
  • Correctly placed after verification succeeds

LGTM with the optional SetSafe addition as a nice-to-have.


@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small, defensive 5-line change that mirrors the existing ProcessMilestone call.

Extended reasoning...

Overview

Single-file, 5-line addition in eth/handler_bor.go. After the existing h.downloader.ProcessMilestone(num, hash) call in handleMilestone, the PR fetches the corresponding header via GetHeaderByHash and, if non-nil, calls eth.blockchain.SetFinalized(header). This brings bor's finality signal into the canonical geth BlockChain API (CurrentFinalBlock()), which was previously always nil on bor since the only upstream writer is the (unused) Engine API.

Security risks

None identified. SetFinalized is a simple setter that stores the header pointer, writes the finalized hash to rawdb, and updates a metrics gauge — no consensus or state-transition impact. GetHeaderByHash is a safe read and returns nil for unknown hashes; the unknown-hash case is handled as a defensive no-op. No new external inputs, auth, or crypto surface area.

Level of scrutiny

Low. The change is purely additive metadata propagation: it makes existing geth machinery (metrics, the SetHead safety guard, tracer-visible finality) reflect a value bor already computes elsewhere (getFinalizedBlockNumber via the downloader whitelist). The two observable effects called out in the description (gauge advancing, SetHead invalidated finalized block log becoming reachable on rewinds below a milestone) are intended and benign.

Other factors

SonarQube quality gate passed. Maintainer (pratikspatil024) explicitly requested review on this PR. The change has been manually tested on mainnet per the PR description. No bugs flagged by the bug hunting system.

@pratikspatil024
pratikspatil024 requested a review from a team May 15, 2026 07:56
write time

handleMilestone now resolves the milestone block via GetBlockByNumber
and
verifies the canonical hash matches milestone.Hash before calling
SetFinalized/SetSafe to avoid promoting a stale header if a reorg
lands before this write.

Calls SetSafe alongside SetFinalized so both advance in real time.
Comment thread eth/handler_bor.go Outdated
@sonarqubecloud

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

This PR is stale because it has been open 21 days with no activity. Remove stale label or comment or this will be closed in 14 days.

@github-actions github-actions Bot added the Stale label Jul 2, 2026
@maoueh

maoueh commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

This is still valid, PR approved was just missing a merge window.

@github-actions github-actions Bot removed the Stale label Jul 14, 2026
@pratikspatil024

Copy link
Copy Markdown
Member

@claude review once

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the change is a small, well-reviewed addition that wires milestone processing into BlockChain.SetFinalized. The three findings are all nits (a stale comment, a non-blocking monotonicity suggestion already raised and accepted as non-blocking in the original review, and a subtle safe==finalized resurfacing on restart via the pre-existing loadLastState path) — none affect correctness or introduce security/consensus risk, and the core placement/nil-checks are correct.

Extended reasoning...

Overview

The diff adds a single call, eth.blockchain.SetFinalized(block.Header()), in handleMilestone (eth/handler_bor.go), right after ProcessMilestone succeeds. It re-resolves the milestone's end block by number and checks the hash matches before publishing, guarding against a reorg between verification and this write. This wires Heimdall milestone finality into the standard geth BlockChain finality bookkeeping (atomic pointer + headFinalizedBlockGauge + rawdb.WriteFinalizedBlockHash), which was previously always nil/zero on Bor since only the (unused) Engine API called SetFinalized upstream.

Security risks

No new external input surface is introduced — the milestone data was already verified by verifier.verify before this call executes, and the block is re-resolved defensively by canonical number->hash mapping rather than trusting a stale reference. The only consumers of CurrentFinalBlock() are metrics, live tracers, and downstream tooling — Bor's actual reorg-safety mechanism (the whitelist finality.IsValidChain) is untouched. The RPC eth_getBlockByNumber("finalized") path is also unchanged (still goes through the downloader whitelist). I confirmed the three inline findings are all real relative to the current code (checked SetFinalized/SetSafe/loadLastState in core/blockchain.go), but all are Nit-severity: none allow an external actor to corrupt state, bypass validation, or crash the node.

Level of scrutiny

This is a narrowly-scoped, single-function change (7 added lines) touching a non-consensus-critical bookkeeping path (finality metrics/API surface, not block validation or whitelist logic). It underwent a full review cycle with two maintainers (kamuikatsurgi, maoueh) explicitly debating and resolving the SetSafe question, and a maintainer reconfirmed on 2026-07-13 that it remains valid and was previously approved pending a merge window. Given the limited blast radius and the depth of prior human review, a light-touch confirmation is appropriate here rather than blocking again on nits that don't change behavior in any harmful way.

Other factors

All three inline findings are explicitly non-blocking suggestions (documentation staleness, a defensive monotonicity check already proposed and consciously deferred in the original review, and a restart-time edge case in pre-existing upstream loadLastState logic that this PR merely activates for the first time on Bor). None require a design decision beyond what's already been discussed in the thread.

Comment thread eth/handler_bor.go
Comment on lines +131 to +136
// Publish the milestone block as finalized and safe. Re-resolve via the
// canonical number→hash mapping so a reorg between verification and this
// write can't promote a stale or non-canonical header.
if block := eth.blockchain.GetBlockByNumber(num); block != nil && block.Hash() == hash {
eth.blockchain.SetFinalized(block.Header())
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR's first non-nil SetFinalized call on Bor writes the finalized hash to the database via rawdb.WriteFinalizedBlockHash. On the very next node restart, BlockChain.loadLastState reads that persisted hash and sets currentSafeBlock equal to currentFinalBlock, reintroducing exactly the safe == finalized state that was explicitly removed from this PR after review discussion. This makes eth_getBlockByNumber("safe") return nil at runtime but a non-nil block after any restart following a processed milestone — an inconsistency worth documenting or explicitly clearing.

Extended reasoning...

The bug: SetFinalized (core/blockchain.go:1128-1139) unconditionally persists the finalized block hash via rawdb.WriteFinalizedBlockHash whenever called with a non-nil header. Separately, BlockChain.loadLastState (core/blockchain.go:989-999) reads that same persisted hash on every node startup and, if present, stores the corresponding block into both bc.currentFinalBlock and bc.currentSafeBlock — the inline comment there literally says 'the safe block is not stored on disk and it is set to the last known finalized block on startup.'

Why this PR activates a dormant path: Before this PR, SetFinalized was never called anywhere in Bor's production code path — the only other caller is the Engine API (eth/catalyst/api.go), which Bor does not use. So the rawdb finalized-hash key was never written on Bor, and the loadLastState safe-restore branch quoted above was permanently dead code for this chain. This PR adds the first Bor-side call to SetFinalized (in handleMilestone, eth/handler_bor.go:131-136), which means the first successfully processed milestone now writes that key for the first time in Bor's history.

Why this defeats the PR's own explicit decision: The PR originally also called SetSafe(header) alongside SetFinalized. During review, kamuikatsurgi asked to remove it: 'Remove .SetSafe since we don't have soft finality in Bor.' When maoueh pushed back that safe == finalized might be a reasonable outcome, kamuikatsurgi reaffirmed the removal: 'we don't want teams who are not aware of certain mechanics of the chain take wrong decisions based on safe == finalised.' The author complied, and the merged diff only calls SetFinalized. But this only prevents safe == finalized at runtime between restarts — it does nothing to prevent loadLastState from reintroducing it on the next restart, because that path is keyed off the same persisted rawdb value this PR now writes for the first time.

Concrete walkthrough:

  1. Node starts with this PR deployed. currentSafeBlock is nil, matching the reviewed intent (eth_getBlockByNumber("safe") returns 'safe block not found').
  2. A milestone is processed successfully; handleMilestone calls eth.blockchain.SetFinalized(block.Header()). This stores the header in currentFinalBlock and writes its hash to disk via rawdb.WriteFinalizedBlockHash.
  3. Node operator restarts the node (routine maintenance, upgrade, crash recovery — nothing pathological).
  4. loadLastState runs, reads the now-populated ReadFinalizedBlockHash, resolves the block, and stores it into both currentFinalBlock and currentSafeBlock.
  5. eth_getBlockByNumber("safe") — resolved directly from b.eth.blockchain.CurrentSafeBlock() in eth/api_backend.go:118-124/186-193 and internal/ethapi/bor_api.go:462-464, with no Bor-specific override the way the 'finalized' tag has — now returns the milestone block instead of nil. headSafeBlockGauge advances too.

Why nothing in the current code prevents this: loadLastState is shared upstream geth logic; this PR only touches eth/handler_bor.go and never touches loadLastState or clears currentSafeBlock after startup. There is no Bor-specific override for the RPC 'safe' tag analogous to the one that exists for 'finalized' in getFinalizedBlockNumber, so whatever CurrentSafeBlock() holds is returned as-is.

Impact: This is a semantic/observability inconsistency, not a crash or fund-loss bug — the resulting safe value is conservative (safe == finalized ≤ head) and no state corruption occurs. But it silently reintroduces the exact behavior (safe == finalized) that two maintainers explicitly debated and rejected in this PR's own review thread, and it does so nondeterministically depending on whether the node has been restarted since the feature shipped — runtime behavior (safe nil) diverges from post-restart behavior (safe == finalized) with no code change in between.

Suggested fix: Either explicitly clear currentSafeBlock after loadLastState runs on Bor (or add a Bor-specific override for the 'safe' RPC tag, mirroring how 'finalized' is already special-cased in getFinalizedBlockNumber), or accept safe == finalized consistently and document it — the current merged state produces neither of those outcomes consistently.

Comment thread eth/handler_bor.go
Comment on lines +131 to +136
// Publish the milestone block as finalized and safe. Re-resolve via the
// canonical number→hash mapping so a reorg between verification and this
// write can't promote a stale or non-canonical header.
if block := eth.blockchain.GetBlockByNumber(num); block != nil && block.Hash() == hash {
eth.blockchain.SetFinalized(block.Header())
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 SetFinalized(block.Header()) is called unconditionally whenever a milestone's end block resolves to a canonical block, with no check that the new finalized number is >= the currently stored CurrentFinalBlock(). Because MultiHeimdallClient (and the WS failover client) can fail over to a lagging heimdall endpoint, a stale-but-still-canonical milestone could regress currentFinalBlock, the persisted finalized hash, and headFinalizedBlockGauge. This was already flagged as a non-blocking suggestion in an earlier review pass (suggestion #3); a simple header.Number.Uint64() > current.Number.Uint64() guard would close the gap.

Extended reasoning...

The gap: handleMilestone calls eth.blockchain.SetFinalized(block.Header()) (eth/handler_bor.go:131-136) as soon as the milestone's end block resolves to a canonical block via GetBlockByNumber/hash-match. Neither of the two gates upstream of this call enforce monotonicity:

  • verifier.verify (eth/bor_checkpoint_verifier.go:69-96) only checks that the local chain head is not behind the milestone's end block (errChainOutOfSync) and that the end-block hash matches canonical history. An older milestone whose end block is still part of current canonical history satisfies both checks just fine.
  • whitelist finality.Process (eth/downloader/whitelist/finality.go:69-80) unconditionally overwrites Number/Hash, so it doesn't provide a monotonicity backstop either.

Trigger path: Bor supports MultiHeimdallClient (consensus/bor/heimdall/failover_client.go), which cascades FetchMilestone across multiple independently-polled heimdall endpoints, including a fallback pass over 'unhealthy' clients. If the active/selected endpoint is (or becomes) a lagging replica, FetchMilestone can return a milestone whose EndBlock is lower than one already finalized locally. handleMilestone re-resolves that lower EndBlock via GetBlockByNumber, finds it's still canonical (it is — it's just older), and calls SetFinalized with a header whose number is less than the current CurrentFinalBlock(). The same can happen via the WS subscription client (consensus/bor/heimdallws/client.go), which also fails over between multiple heimdall URLs and can redeliver a stale milestone on reconnect.

Step-by-step proof:

  1. Node is configured with two heimdall endpoints, A (up to date) and B (lagging by several milestones), via MultiHeimdallClient.
  2. A milestone with EndBlock=1000 is fetched from A, passes verifier.verify, and SetFinalized sets CurrentFinalBlock() to block 1000.
  3. Endpoint A has a transient error (or health-registry rotation) and the failover client cascades to B.
  4. B's latest known milestone is EndBlock=900 (block 900 is still canonical — no reorg has happened).
  5. verifier.verify(ctx, eth, h, 800, 900, hash900) passes: chain head (>1000) is not behind 900, and block 900's hash matches canonical history.
  6. handleMilestone calls eth.blockchain.SetFinalized(block900.Header()), and core/blockchain.go's SetFinalized unconditionally stores it and persists it via rawdb.WriteFinalizedBlockHash, moving headFinalizedBlockGauge from 1000 back to 900.
  7. Any consumer reading CurrentFinalBlock() directly (live tracers, external indexers) between steps 6 and the next healthy poll observes finality moving backward, violating the implicit invariant that finalized blocks are monotonically non-decreasing.

Why this doesn't block merge: SetFinalized here only feeds the standard geth BlockChain finality bookkeeping (atomic pointer + metric + rawdb hash) — it is not consulted by Bor's actual reorg-safety mechanism, which is the separate whitelist (finality.IsValidChain/IsValidPeer) and remains untouched by this PR. The RPC eth_getBlockByNumber("finalized") path (getFinalizedBlockNumber) also doesn't read CurrentFinalBlock() — it goes through the downloader's whitelisted milestone, so it's unaffected. The regression is also self-correcting: the very next successful poll of a caught-up endpoint (milestones are polled roughly every ~1s) moves CurrentFinalBlock() forward again. And upstream geth's own SetFinalized (used from the Engine API) has no monotonicity guard either, so this isn't a new invariant that Bor is uniquely breaking.

Suggested fix: guard the call with a simple monotonicity check, as already suggested non-blockingly in the original review thread:

if block := eth.blockchain.GetBlockByNumber(num); block != nil && block.Hash() == hash {
    if current := eth.blockchain.CurrentFinalBlock(); current == nil || block.NumberU64() > current.Number.Uint64() {
        eth.blockchain.SetFinalized(block.Header())
    }
}

This is a one-line robustness improvement, not a correctness-blocking fix, since nothing durably breaks if it's omitted.

Comment thread eth/handler_bor.go
Comment on lines +131 to +136
// Publish the milestone block as finalized and safe. Re-resolve via the
// canonical number→hash mapping so a reorg between verification and this
// write can't promote a stale or non-canonical header.
if block := eth.blockchain.GetBlockByNumber(num); block != nil && block.Hash() == hash {
eth.blockchain.SetFinalized(block.Header())
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The block comment above the SetFinalized call still says the milestone block is published as "finalized and safe", but the SetSafe call was removed per reviewer request (Bor has no soft finality), leaving only SetFinalized. Please drop "and safe" from the comment so it matches the code.

Extended reasoning...

What the bug is: The comment at eth/handler_bor.go:131 reads:

// Publish the milestone block as finalized and safe. Re-resolve via the
// canonical number→hash mapping so a reorg between verification and this
// write can'\''t promote a stale or non-canonical header.

but the code beneath it only calls eth.blockchain.SetFinalized(block.Header()) — there is no SetSafe call anywhere in this block.

How this happened: Looking at the PR timeline, the original version of this diff called both SetFinalized and SetSafe, matching the wording "finalized and safe." Reviewer @kamuikatsurgi asked to remove SetSafe ("Remove .SetSafe since we don'''t have soft finality in Bor"), and after some discussion with @maoueh, the removal was confirmed as the right call: "while safe == finalised, we don'''t want teams who are not aware of certain mechanics of the chain take wrong decisions based on safe == finalised." That removal landed in commit bc0fd91 ("remove SetSafe as was suggested by @kamuikatsurgi"), which is visible in the recent git log. The code was correctly updated, but the comment text was never revisited and still references the old, now-removed behavior.

Why this matters: A future reader of this function — without access to the PR discussion — will read "Publish the milestone block as finalized and safe" and reasonably conclude that safe-block tracking happens here. That'''s exactly the kind of incorrect assumption @kamuikatsurgi'''s review comment was trying to prevent (teams making decisions based on safe == finalized semantics that Bor deliberately does not provide). A stale comment that implies functionality which doesn'''t exist can lead to wrong assumptions downstream, e.g. someone assuming headSafeBlockGauge advances here when it doesn'''t.

Step-by-step proof:

  1. Read the comment at line 131-133: it claims the block is published "as finalized and safe."
  2. Read the code at lines 134-136: only eth.blockchain.SetFinalized(block.Header()) executes; grep the function body for SetSafe — zero matches.
  3. Check git history: git log --oneline -5 shows bc0fd91 remove SetSafe as was suggested by @kamuikatsurgi as the most recent commit, confirming SetSafe was removed intentionally and recently, but the comment wasn'''t updated in the same commit.
  4. Conclusion: the comment is factually stale relative to the code it documents.

Fix: Simply update the comment to remove "and safe", e.g.: // Publish the milestone block as finalized. Re-resolve via the canonical number→hash mapping so a reorg between verification and this write can'\''t promote a stale or non-canonical header.

This is purely a documentation/comment mismatch — no runtime, correctness, or security impact, and does not block merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants