CHANGELOG.md

Changelog

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Added

• ForkableHub.WithoutPartials and stream.WithoutPartialBlocks: stream from a hub fed with partial blocks without getting them, the last partial of each block arriving as the complete block, as for a relayer client that does not ask for partials. Lets an app that does not handle partial blocks share its hub with one that does.
• hub.SubscriptionCatchUpTimeout (default 30s): a hub subscription is now closed with hub.ErrSubscriptionBehind when its consumer has not emptied its waiting blocks for that long, which catches a consumer that is stuck or slower than the chain. A consumer draining a burst within the timeout is not affected. ErrSubscriptionBehind matches ErrSubscriptionChannelFull with errors.Is, so callers that reconnect on one reconnect on both.
• hub.SubscriptionMaxBufferedBlocks (default 10000): how many blocks a hub subscription can have waiting before it is closed with ErrSubscriptionChannelFull. Both are meant to be set once at process startup.
• MultiplexedSourceWithRetryIntervals: new option setting, per source, the minimum time between two connection attempts, rounded up to the next 5s increment since sources are checked every 5s. Useful for a rescuer or fallback source that is down most of the time and would otherwise be redialed every 5s.
• forkable.WithFinalizedBlockNumMetric: new option reporting, on the live path, the LIB number of the block that just became head. Takes a forkable.Uint64Metric interface so consumers can provide their own metric implementation.
• DefaultMergedBlocksBundleSize: new package variable (default 100) controlling the number of blocks per merged-blocks file assumed by readers when no explicit size is given. Like GetProtocolFirstStreamableBlock, it is meant to be set once at process startup.
• stream.WithMergedBlocksBundleSize: new stream option to set the merged-blocks bundle size for a single stream (overrides the process-wide default; used by substreams tier2 which serves multiple chains at once).
• FileSource now fails fast with a clear error when a merged-blocks file contains a block beyond the configured bundle size (store files bigger than the configured size).
• stream.ErrUnavailable / stream.NewErrUnavailable: error type for a request this process could not serve yet, for servers to map to their transport's retryable status (codes.Unavailable).
• ErrCursorAboveHead, CursorHeadWaitTimeout: a cursor block above the live source's head is waited for (default 5s) before being reported.
• CheckCursorResolvable: reports whether a cursor names a block anything can still produce, against a LiveBlockKnower (the hub) and an optional ForkedBlockKnower (a FileSourceFactory). Both are new optional interfaces, implemented by hub.ForkableHub and FileSourceFactory respectively, so callers resolving cursors outside JoiningSource can make the same call.
• FileSourceFactory.HasForkedBlock: says whether the forked-blocks store holds the block at a given number whose ID ends with a given suffix.
• SanitizeBundleSize: guards the merged-blocks math against a bundle size of 0 (misconfigured DefaultMergedBlocksBundleSize or FileSourceWithBundleSize(0)), which would otherwise divide-by-zero panic in the hub or loop forever in FileSource; falls back to 100.
• hub.WithOneBlockDownloadConcurrency: sets how many one-block files the hub downloads at once on startup (default 32).
• blockstream.WithBurstFunc: function to compute the burst when the source sends its block request (instead of when the source is created)

Changed

• blockstream.Source now sends gRPC keepalive pings after 30s without data (was 5m from dgrpc) and drops the connection after 10s without an ack, so a relayer that disappears without closing the connection (e.g. host reboot) is detected in about 40s instead of 5 minutes.
• The hub no longer reads the SOURCE_CHAN_SIZE environment variable; set hub.SubscriptionMaxBufferedBlocks instead. The default goes from 100 to 10000 blocks: on fast chains, a burst of live blocks after a short pause from the live source filled 100 slots before the consumer could send the first block, closing healthy subscriptions. A consumer that cannot keep up is now caught by SubscriptionCatchUpTimeout instead.
• OneBlockFile.CanonicalName no longer includes the LIB number (num-id-prevID instead of num-id-prevID-lib). Readers whose view of the LIB differs, as on Polygon PoS / Amoy, write the same block with different LIB numbers, and these copies used to be seen as different blocks.
• DBinBlockReader.Read no longer copies the block payload out of the message it decodes: Payload.Value (and the legacy PayloadBuffer) now point into the buffer the message was read into. Reading merged blocks is 30-47% faster and allocates half as much.

Fixed

• A live stream starting at a block number with no block (a missed or skipped slot) now starts at the next block instead of waiting for the merged bundle containing it.

• hub.ForkableHub no longer breaks when its one-block bootstrap races with the merger deleting the files of a bundle it just merged. Concretely: bootstrap retries a few times when the listing comes back with a gap or a listed file is gone by the time it is opened (both expected during that race), a failed bootstrap no longer leaves the forkable holding a bogus LIB it then can't reconcile with live blocks (the hub now only swaps in the forkable it built once bootstrap actually succeeds, falling back to a fresh one otherwise), and cannot link block after reconnection is only fatal once the hub has actually been ready — before that, an unlinkable block is handled like any other, already subject to WithMaxConsecutiveUnlinkableBlocks. Previously a consumer starting or restarting during that race could see the hub die or never become ready.

• The hub no longer walks the one-block store for a block it already has below LIB. With redundant live sources, a source re-sending a final block that the hub still keeps below LIB (the lowest one, whose parent was already pruned) triggered a store lookup and a block not linkable after one-block lookup warning, even though the forkable drops blocks below LIB anyway. The hub now compares against LIB instead of the lowest kept block. Forkable.LIBNum is added for this.

• hub.WithMaxConsecutiveUnlinkableBlocks now counts blocks rather than messages: an intermediate flash block — PartialIndex != 0 without LastPartial — no longer advances the counter. Every partial of a block fails the same link check, so on a chain delivering four per block the hub gave up after a quarter of the blocks the limit names. A plain block and a block's final partial still count, and any linkable block still resets the count.

• Hub subscriptions with with_partials=false no longer stall on flash/partial-block chains. Previously every block with PartialIndex != 0 (including the closing LastPartial) was dropped, so a no-partial subscriber only advanced on separate PartialIndex==0 full blocks, which can lag the sealed head by tens of seconds. The subscription now drops only intermediate partials and delivers each LastPartial as a full block (partial markers cleared on a thin copy that shares the payload; the shared original block is never mutated).

• FileSource no longer stalls when a merged-blocks file cannot be read (a zero-byte or otherwise corrupted file). The goroutine reading a file calls Shutdown() on failure without ever closing that file's blocks channel, and the consuming loop was a plain range over it, so depending on which of the two won the race the request could hang forever instead of returning the error. The loop now selects on Terminating() as well.

• FileSource with FileSourceErrorOnMissingMergedBlocksFile no longer truncates its output: on a missing file it now drains every already-read block through the ordered stream before surfacing the error, instead of calling Shutdown() immediately (which aborted in-flight reader goroutines and discarded blocks).

• JoiningSource no longer leaves a stream silent when its cursor names a block nothing can produce. A cursor block inside the live buffer's range whose ID the buffer does not know (a corrupted or forged cursor, or one from a chain the process never saw) made the hub decline the source, and the file source it fell back to then waited for the merged-blocks file holding that block number — a whole bundle, some twenty minutes on Ethereum — before failing anyway. Such a cursor now fails immediately with ErrResolveCursor, which the stream package already surfaces as an invalid argument. Cursors on a fork the live buffer no longer holds are unaffected: the forked-blocks store is consulted before giving up, and a cursor below the live range still goes to the file source.

• forkable.Forkable now always clamps a block's lib_num down to its own block number when it is greater (invalid; equal is still valid), logs an error, instead of letting the bad value reach BlockInCurrentChain/MoveLIB, where it moved LIB past head and silently stalled the forkable forever (healthz stays green, nothing is ever emitted again) while it kept linking incoming blocks. Mirrors utils.ClampLibNum in firehose-core (streamingfast/firehose-core#284), which guards the same invariant at the reader.

Changed

• hub.ForkableHub logs linking live block using one blocks at Info instead of Debug, so a lookup in the one-block store after an unlinkable live block is visible in production logs.

• hub.ForkableHub logs processing block at Debug, except for one line every 10 seconds kept at Info to show progress; on flash-block chains the line fired several times per block.

• A cursor above head is now retryable, in case we are lagging behind and another instance is already serving that block.

• ForkableHub bootstrap now rounds its lowest kept block down to the configured merged-blocks bundle size instead of a hardcoded 100.

• ForkableHub now downloads one-block files 32 at a time, still processing them in block order, both when bootstrapping and when filling the gap before a live block it cannot link. It used to download them one at a time, which was too slow on fast chains with remote storage.

• BlockTimestampGate: new gate that lets blocks through once a block's timestamp meets or exceeds a given time.Time, supporting both inclusive and exclusive gate types.

• hub.WithLogger: new ForkableHub option to set the logger used by the hub (and, by default, propagated to its inner forkable). Previously the hub always logged under the package-level bstream logger, making lines such as processing block indistinguishable across components (relayer, firehose, tier1, ...). Callers should pass their component logger.

2026-01-02

• Added field with_partials to sf.bstream.v1.BlockRequest protobuf message: blockstream no longer sends partial blocks by default.

2025-12-12

Added

• Support for partial blocks (e.g., Flash Blocks), with special "StepPartial". These blocks are only sent when they are above the HEAD, and other blocks are never linked to them. They are always eventually replaced by a full block with StepNew.

2024-08-19

Added

• New middleware handlers in joining source to allow for user to hook into the stream and perform custom actions.

2023-12-08

Major Refactoring

• BREAKING Removed the bstream.Block object completely: the pbbstream.Block takes its place.
• BREAKING Moved the pbbstream package from github.com/streamingfast/pbgo to here, under sf.bstream.v1.Block

2023-11-08

Removed

• BREAKING Removed bstream.Block.PreviousRef method as it was invalid since it was assuming that the previous number is always minus one the current block num which is not true on a lot of supported chains (Solana and NEAR for example).

Fixed

• BREAKING we now enforce block continuity in filesource while reading merged-blocks: before, a corrupted merged-block-file would have been read as-is and serve wrong blocks.

Added

• Added FileSourceWithSecondaryBlocksStores Option to allow a fallback location
• .SetNearBlocksCount(count) and .Clone() on Tracker object.
• Tracker object to streamline queries about different targets (like network head, database lib, relayer blockstream head, whatever other BlockRef tags), ask the question about them being near one another (to select between live mode or catch-up mode). Also streamlines the requests of a start block, with a bunch of different backend implementations that can answer to the questions regarding where to start.
• JoiningSourceWithTracker to avoid joining to live when live and file sources are very far apart.
• HeadBlockRefGetter and LIBBlockRefGetter that targets a HeadInfo service, and satisfies the Tracker BlockRefGetter func signature.

Changed

• StreamGetter now requires a boolean param to know if it must decode the block
• BREAKING blockstream/v2 server now takes an array of blocksStores, to give the filesource as secondaryBlocksStores option
• Renamed HeadBlockRefGetter to StreamHeadBlockRefGetter and NetworkHeadBlockRefGetter. Choose what you need.
• Renamed LIBBlockRefGetter to StreamLIBBlockRefGetter and NetworkLIBBlockRefGetter. Choose what you need.
• Renamed Tracker.ResolveRelativeBlock to Tracker.GetRelativeBlock, to avoid confusion with the AddResolver function, which has nothing to do with GetRelativeBlock (which uses Get() and the Getters only).
• Greatly improve logging behavior of the various source implementations, this should greatly improved debuggability of the library.
• BREAKING All Source must now implement a SetLogger(logger *zap.Logger) method.
• BREAKING Removed all Name, SetName, and *Name options on all source and across bstream. Replaced by a proper *zap.Logger instance instead. Re-configure using the logger, you can use SetLogger(zlog.With("name", "my-source-name")) to emulate the old behavior.

[v0.0.1] - 2020-06-22

Added

• StartBlockResolver: interface for quickly finding out from which block to start (to cover all possible forks before your required start block)
• ParallelStartResolver: implementation of StartBlockResolver to interrogate multiple StartBlockResolvers at once, useful when all dfuse components are not "up" yet.
• SetHeadInfo on BlockStream to allow using GetHeadInfo() before stream actually starts
• WithName option on NewForkable for better logging
• License changed to Apache 2.0

Changed

• BREAKING CHANGE: JoiningSourceRateLimit now takes a time.Duration as a second argument, instead of an int amount of milliseconds.

(before dfuse-for-EOSIO)

Changed

• forkable.NewWithLIB(...) replaced by forkable.New(forkable.WithExclusiveLIB(...))
• Forkable.SetFilters() replaced by the forkable.WithFilters() Option, to bassed to New().
• Forkable.EnsureBlockFlows() replaced by the forkable.EnsureBlockFlows() Option.
• Forkable.EnsureAllBlocksTriggerLongestChain() replaced by the forkable.EnsureAllBlocksTriggerLongestChain() Option.

Added

• forkable.WithInclusiveLIB() as an Option to forkable.New()