Skip to content

Allow resize-merged-blocks with equal bundle sizes #290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sduchesneau wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Allow resize-merged-blocks with equal bundle sizes #290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ If you were at `firehose-core` version `1.0.0` and are bumping to `1.1.0`, you s

### Changed

- `tools resize-merged-blocks` now accepts equal `--source-bundle-size` and `--target-bundle-size` to rewrite merged-blocks files with different settings (ex: compression). It asks for confirmation first; pass `--force` to skip the prompt.

- `firehose` and `substreams-tier1` launched in the same process now share one forkable hub instead of each holding its own copy of the live blocks; on Solana that saves about 0.5 GiB per process. Firehose still streams complete blocks only.

- The firehose hub keeps as many final blocks as the substreams-tier1 hub: two merged-blocks files worth, at least 200, instead of at least 500. Clients resuming between 200 and 500 blocks below the last irreversible block now read those blocks from merged-blocks files.
Expand Down
9 changes: 7 additions & 2 deletions cmd/tools/mergeblock/tools_resize_merged_blocks.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ func NewToolsResizeMergedBlocksCmd[B firecore.Block](chain *firecore.Chain[B], r
cmd.Flags().Uint64("source-bundle-size", 100, "Number of blocks per merged-blocks file in the source store")
cmd.Flags().Uint64("target-bundle-size", 0, "Number of blocks per merged-blocks file to write to the destination store (required)")
cmd.Flags().Uint64("first-streamable-block", 0, "First streamable block of the chain, used to allow a <start> below the first target boundary")
cmd.Flags().Bool("force", false, "Skip the confirmation prompt when --source-bundle-size and --target-bundle-size are equal")

return cmd
}
Expand Down Expand Up @@ -77,8 +78,12 @@ func createResizeMergedBlocksE(rootLog *zap.Logger) firecore.CommandExecutor {
if err := firecore.ValidateMergedBlocksBundleSize(targetSize); err != nil {
return fmt.Errorf("invalid target-bundle-size: %w", err)
}
if sourceSize == targetSize {
return fmt.Errorf("source-bundle-size and target-bundle-size are both %d, nothing to do", sourceSize)
if sourceSize == targetSize && !sflags.MustGetBool(cmd, "force") {
message := fmt.Sprintf("source-bundle-size and target-bundle-size are both %d, files will be rewritten as-is (ex: to change compression). Continue?", sourceSize)
if confirmed, _ := cli.PromptConfirm(message); !confirmed {
rootLog.Info("skipped by user")
return nil
}
}
if targetSize%sourceSize != 0 && sourceSize%targetSize != 0 {
return fmt.Errorf("bundle sizes must divide evenly (source %d, target %d)", sourceSize, targetSize)
Expand Down
Loading