Skip to content

Commit c70fa66

Browse files
committed
Integrate atomic SQL bootstrap and deferred browser indexes (#2897)
Browser bootstrap now imports SQL through the published sqlite-web bulk API and builds secondary indexes after the first fresh-database import. This completes the browser integration layer of [RAI-2651](https://linear.app/makeitrain/issue/RAI-2651/integrate-bulk-sql-bootstrap-and-deferred-indexes-in-raindex-browser), following merged #2887 and #2893 and [sqlite-web#35](rainlanguage/sqlite-web#35), published as `0.0.4` after [sqlite-web#36](rainlanguage/sqlite-web#36). **Live effect:** faster initial browser DB import · **Risk:** medium (stored data, browser bootstrap) · **Ships:** next raindex/webapp release ## Decisions - Import UTF-8-safe chunks of at most 256 KiB in one worker transaction, cancelling on failure or dropped Rust futures. Cleanup waits for the current SDK request before rollback and retains the executor lock until rollback settles. The browser avoids building an object for each SQL statement; native CLI and incremental writes retain their statement APIs. - Drop and restore explicit secondary indexes inside the first fresh-database import transaction. Primary/unique constraints remain enforced; later imports retain indexes when their preflight sees an existing watermark. Two tabs starting different networks simultaneously can both see an empty DB and cause an extra index rebuild; data remains correct. Attempt one final `ANALYZE` after initial target provisioning; log statistics failures without discarding already committed import reports. - Wait for cross-tab import contention every 250 ms for up to five minutes. Known-safe reads use an explicit `query_json_retryable` API; generic `query_json` and writes do not replay a timeout with an unknown commit result. The idempotent cache-size setter also opts into retries. Temporary worker errors from integrity checks never reset the database. Cap failed dump imports at three attempts per target, then fall back to RPC sync. - Switch the webapp Vercel runtime to supported Node 22 and update its architecture note, as approved to unblock preview testing. - Pin the released `@rainlanguage/sqlite-web` to exactly `0.0.4`. Align the Wasm test lockfile with the existing `wasm-bindgen 0.2.122` runner so tests execute rather than silently reporting zero tests; update the affected mocks and stale test fixtures. ## Risks - Existing stale-target refresh still clears that target before opening the atomic import; an interrupted refresh can require RPC replay from its deployment block. Atomic stale-target replacement is a separate recovery follow-up. - Incremental applies retain their existing ANALYZE frequency. This PR’s final best-effort analysis covers provisioning, including imports with no catch-up window; it does not claim a steady-state analysis-cost reduction. - Persistent SDK initialization failure still follows the existing cache reset policy; distinguishing worker failure from corruption is a separate recovery follow-up documented in RAI-2651. - Compressed/decompressed SQL is still held in memory. This change bounds the import messages, not the full download or decoded dump allocation. - Worker loss during commit can leave the commit result unknown. Retries inspect persisted target watermarks; an import failure remains eligible for at most three provisioning attempts per target in the current runner session. Readiness follows committed index reconstruction. Planner analysis is best effort; its failure does not hide successful provisioning. A permanently pending SDK request can also keep cancellation cleanup pending. ## Proof - Real browser import from an empty OPFS DB using the frozen filtered SQL: all table counts matched the reference, with matching result hashes for 2,160 orders (547 active, 1,613 inactive) and 1,714 vaults. All 41 explicit indexes and planner statistics were present. Orders/Vaults lists and details rendered; warm reload reused the DB without downloading the dump again. The fixture held the chain head fixed and rejected live indexing/quotes. - Fresh final Nix checks on submitted `8b4438e35`: 1,925 native workspace tests (599 local DB tests), 140 Node/Wasm tests, 130 actual browser/Wasm tests, and 1,168 JavaScript tests passed. Workspace formatting and Clippy with all targets/features, SDK/components/webapp builds, and UI lint/type/style checks passed. Two fresh staged Codex reviewers and three simplification passes were clean; no OpenCode reviewers were used. - CodeRabbit cancellation/ANALYZE findings and Marvin cross-tab no-wipe/bounded-import-retry findings are fixed. The follow-up covers dump preflight SELECT timeout retries and uses an idempotent cache-size script with an empty SELECT for the SDK’s JSON response path. Real published sqlite-web 0.0.4 browser verification returned `[]`, confirmed cache size `-25000`, and completed a subsequent atomic import/query successfully. - Marvin formally APPROVED submitted `8b4438e35` with no blockers. Its two non-blocking notes on preexisting stale refresh and incremental ANALYZE frequency are documented below. CodeRabbit subsequently found that generic JSON timeout retries could replay a committed write; this follow-up makes retries explicit and tests that plain, CTE, and multi-statement `INSERT ... RETURNING` are invoked once after a timeout. CodeRabbit’s fresh review of `8b4438e35` completed with no actionable comments; its review gate is green. All nine review threads are resolved. - Live Node 22 preview deployed successfully. From a fresh preview origin, Base and Robinhood dumps downloaded and both networks reached ACTIVE/healthy readiness; 3,838 order events, 934 running vault balances, all 41 explicit indexes, and 55 planner-statistics rows were present. Warm reload reused the DB without another SQL dump download. Orders and Vaults lists rendered with those two networks selected (576 active orders); an order detail showed its input/output vaults and a live quote, and the related vault detail showed its balance, order relationship, and deposit history. A Robinhood USDG vault detail also rendered its related orders and recent take-order balance changes while live sync remained ACTIVE. Preview: https://rain-orderbook-v6-k10byupab-rain-x-h20.vercel.app/orders - Preview limitation: default “All items” also selects Ethereum, while the remote registry has no Ethereum raindex. The UI receives `raindex with network key: ethereum not found` and displays an empty list. Deselecting Ethereum in Networks renders the local DB lists. Network enumeration/query routing and the registry URL are unchanged by this PR; this is documented for separate registry/UI follow-up. - **Not verified:** a new production-scale bootstrap latency/memory benchmark. The earlier local fixture held the chain head fixed; the deployed preview used live RPC sync. ## Rollout 1. Release raindex and deploy the webapp with the exact SDK pin; verify fresh bootstrap reaches ACTIVE and lists/details load. Existing SQL dumps and manifest format remain supported. 2. The producer update is already deployed: filtered, grouped dumps and their manifest were verified under RAI-2652. No additional producer change is required for this integration. 3. Roll back to the previous raindex/webapp release if bootstrap regresses; the database schema is unchanged. ## Checks - [x] Kept to the browser import integration and its verification fixtures, plus the approved Node 22 preview runtime fix. - [x] The app build emits `nodejs22.x`; deployed preview and configured-network lists/details were verified. CodeRabbit is clean, Marvin formally approved, and code/build/preview CI passes on `8b4438e35`. Existing Sol static warnings remain the previously accepted failure; the human review gate is pending. See the default-network registry limitation above. - [x] Tested the new import, failure/retry, index and analysis behavior. - [x] Linked the tracking issue and upstream/predecessor PRs. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * WebAssembly local databases can import SQL dumps in chunks, with rollback and retry support if an import fails. * Database analysis is deferred during dump imports and runs once afterward; analysis failures don’t fail the sync. * **Bug Fixes** * Failed dump imports can be retried during provisioning, with RPC synchronization continuing if retries are exhausted. * SQL dump import failures now provide a clear error message. * Temporary database worker unavailability no longer triggers an unnecessary database reset. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 266b57b commit c70fa66

49 files changed

Lines changed: 1461 additions & 177 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Cargo.lock‎

Lines changed: 45 additions & 21 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/cli/src/commands/local_db/cli.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ mod tests {
216216
metadata_rpcs: Vec::new(),
217217
cfg: sync_config,
218218
dump_str: None,
219+
defer_analyze: false,
219220
block_number_threshold: 10000,
220221
manifest_end_block: 1,
221222
};

‎crates/cli/src/commands/local_db/pipeline/bootstrap.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,7 @@ mod tests {
167167

168168
let cfg = BootstrapConfig {
169169
raindex_id: sample_ob_id(),
170+
dump_sql: None,
170171
dump_stmt: None,
171172
latest_block: 0,
172173
block_number_threshold: TEST_BLOCK_NUMBER_THRESHOLD,
@@ -198,6 +199,7 @@ mod tests {
198199

199200
let cfg = BootstrapConfig {
200201
raindex_id: sample_ob_id(),
202+
dump_sql: None,
201203
dump_stmt: None,
202204
latest_block: 0,
203205
block_number_threshold: TEST_BLOCK_NUMBER_THRESHOLD,
@@ -231,6 +233,7 @@ mod tests {
231233

232234
let cfg = BootstrapConfig {
233235
raindex_id: sample_ob_id(),
236+
dump_sql: None,
234237
dump_stmt: Some(SqlStatementBatch::from(vec![dump_stmt.clone()])),
235238
latest_block: 0,
236239
block_number_threshold: TEST_BLOCK_NUMBER_THRESHOLD,
@@ -266,6 +269,7 @@ mod tests {
266269

267270
let cfg = BootstrapConfig {
268271
raindex_id: sample_ob_id(),
272+
dump_sql: None,
269273
dump_stmt: Some(SqlStatementBatch::from(vec![dump_stmt.clone()])),
270274
latest_block: 0,
271275
block_number_threshold: TEST_BLOCK_NUMBER_THRESHOLD,
@@ -301,6 +305,7 @@ mod tests {
301305

302306
let cfg = BootstrapConfig {
303307
raindex_id: sample_ob_id(),
308+
dump_sql: None,
304309
dump_stmt: None,
305310
latest_block: 0,
306311
block_number_threshold: 1,

‎crates/cli/src/commands/local_db/pipeline/runner/environment.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ mod tests {
8484
window_overrides: WindowOverrides::default(),
8585
},
8686
dump_str: None,
87+
defer_analyze: false,
8788
block_number_threshold: 10000,
8889
manifest_end_block: 1,
8990
},

‎crates/cli/src/commands/local_db/pipeline/runner/export.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,7 @@ mod tests {
149149
metadata_rpcs: Vec::new(),
150150
cfg: sync_config,
151151
dump_str: None,
152+
defer_analyze: false,
152153
block_number_threshold: 10000,
153154
manifest_end_block: 1,
154155
},
@@ -236,6 +237,7 @@ mod tests {
236237
metadata_rpcs: Vec::new(),
237238
cfg: sync_config,
238239
dump_str: None,
240+
defer_analyze: false,
239241
block_number_threshold: 10000,
240242
manifest_end_block: 1,
241243
},

‎crates/cli/src/commands/local_db/pipeline/runner/manifest.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,7 @@ mod tests {
190190
window_overrides: WindowOverrides::default(),
191191
},
192192
dump_str: None,
193+
defer_analyze: false,
193194
block_number_threshold: 10000,
194195
manifest_end_block: 1,
195196
},

‎crates/cli/src/commands/local_db/pipeline/runner/mod.rs‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,7 @@ where
293293
.await
294294
.map_err(|error| mk_failure(TargetStage::DumpDownload, error))?;
295295
SyncInputs {
296-
dump_str: Some(dump_sql),
296+
dump_str: Some(std::sync::Arc::new(dump_sql)),
297297
..inputs
298298
}
299299
}
@@ -548,8 +548,13 @@ mod tests {
548548
)));
549549
db.execute_batch(&batch.ensure_transaction()).await?;
550550

551-
self.telemetry
552-
.record_bootstrap_dump(config.dump_stmt.as_ref().map(dump_sql));
551+
self.telemetry.record_bootstrap_dump(
552+
config
553+
.dump_sql
554+
.as_ref()
555+
.map(|sql| sql.as_str().to_owned())
556+
.or_else(|| config.dump_stmt.as_ref().map(dump_sql)),
557+
);
553558
Ok(())
554559
}
555560

@@ -694,7 +699,11 @@ mod tests {
694699
panic!("stub bootstrap panic");
695700
}
696701

697-
let dump_stmt = config.dump_stmt.as_ref().map(dump_sql);
702+
let dump_stmt = config
703+
.dump_sql
704+
.as_ref()
705+
.map(|sql| sql.as_str().to_owned())
706+
.or_else(|| config.dump_stmt.as_ref().map(dump_sql));
698707
self.telemetry.record_bootstrap_dump(dump_stmt);
699708
self.ensure_tables(db).await?;
700709

‎crates/common/src/local_db/mod.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,9 @@ pub enum LocalDbError {
140140
#[error(transparent)]
141141
LocalDbQueryError(#[from] LocalDbQueryError),
142142

143+
#[error("SQL dump import failed: {0}")]
144+
DumpImportFailed(#[source] LocalDbQueryError),
145+
143146
#[error(transparent)]
144147
IoError(#[from] std::io::Error),
145148

@@ -292,6 +295,7 @@ impl LocalDbError {
292295
LocalDbError::ManifestFetch(err) => format!("Failed to fetch manifest: {}", err),
293296
LocalDbError::TaskJoin(err) => format!("Task join error: {}", err),
294297
LocalDbError::LocalDbQueryError(err) => format!("Database query error: {}", err),
298+
LocalDbError::DumpImportFailed(err) => format!("SQL dump import failed: {}", err),
295299
LocalDbError::IoError(err) => format!("I/O error: {}", err),
296300
LocalDbError::FromHexError(err) => format!("Hex decoding error: {}", err),
297301
LocalDbError::SqlBuildError(err) => format!("SQL build error: {}", err),

‎crates/common/src/local_db/pipeline/adapters/apply.rs‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ pub struct ApplyPipelineTargetInfo {
2222
pub raindex_id: RaindexIdentifier,
2323
pub start_block: u64,
2424
pub target_block: u64,
25+
pub defer_analyze: bool,
2526
pub hash: B256,
2627
}
2728

@@ -104,7 +105,9 @@ pub trait ApplyPipeline {
104105

105106
// Ensure SQLite planner stats are up to date so reads don't suffer from
106107
// poor query plans.
107-
batch.add(SqlStatement::new("ANALYZE"));
108+
if !target_info.defer_analyze {
109+
batch.add(SqlStatement::new("ANALYZE"));
110+
}
108111

109112
Ok(batch)
110113
}
@@ -166,6 +169,7 @@ mod tests {
166169
raindex_id: raindex_id.clone(),
167170
start_block,
168171
target_block,
172+
defer_analyze: false,
169173
hash: SAMPLE_HASH_B256,
170174
}
171175
}
@@ -930,6 +934,24 @@ mod tests {
930934
);
931935
}
932936

937+
#[test]
938+
fn deferred_analysis_omits_per_target_analyze() {
939+
let pipeline = DefaultApplyPipeline::new();
940+
let raindex_id = sample_raindex_id();
941+
let mut target = build_target_info(&raindex_id, 1, 100);
942+
target.defer_analyze = true;
943+
944+
let batch = pipeline
945+
.build_batch(&target, &[], &[], &[], &[])
946+
.expect("batch ok");
947+
948+
assert!(batch.statements().iter().all(|s| s.sql() != "ANALYZE"));
949+
assert!(batch
950+
.statements()
951+
.iter()
952+
.any(|s| s.sql().starts_with("INSERT INTO target_watermarks")));
953+
}
954+
933955
#[test]
934956
fn statements_order_raw_then_tokens_then_decoded_then_watermark() {
935957
let pipeline = DefaultApplyPipeline::new();

‎crates/common/src/local_db/pipeline/adapters/bootstrap.rs‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,12 @@ use crate::local_db::RaindexIdentifier;
1515
use async_trait::async_trait;
1616
use raindex_app_settings::local_db_manifest::DB_SCHEMA_VERSION;
1717
use std::collections::HashSet;
18+
use std::sync::Arc;
1819

1920
#[derive(Debug, Clone)]
2021
pub struct BootstrapConfig {
2122
pub raindex_id: RaindexIdentifier,
23+
pub dump_sql: Option<Arc<String>>,
2224
pub dump_stmt: Option<SqlStatementBatch>,
2325
pub latest_block: u64,
2426
pub block_number_threshold: u32,
@@ -53,7 +55,7 @@ pub trait BootstrapPipeline {
5355
DB: LocalDbQueryExecutor + ?Sized,
5456
{
5557
let rows = db
56-
.query_json::<Vec<DbMetadataRow>>(&fetch_db_metadata_stmt())
58+
.query_json_retryable::<Vec<DbMetadataRow>>(&fetch_db_metadata_stmt())
5759
.await?;
5860
if let Some(row) = rows.first() {
5961
let expected = db_schema_version.unwrap_or(DB_SCHEMA_VERSION);
@@ -76,7 +78,7 @@ pub trait BootstrapPipeline {
7678
where
7779
DB: LocalDbQueryExecutor + ?Sized,
7880
{
79-
let existing: Vec<TableResponse> = db.query_json(&fetch_tables_stmt()).await?;
81+
let existing: Vec<TableResponse> = db.query_json_retryable(&fetch_tables_stmt()).await?;
8082
let existing_set: HashSet<String> = existing
8183
.into_iter()
8284
.map(|t| t.name.to_ascii_lowercase())
@@ -88,7 +90,7 @@ pub trait BootstrapPipeline {
8890

8991
let last_synced_block = if existing_set.contains("target_watermarks") {
9092
let rows: Vec<TargetWatermarkRow> = db
91-
.query_json(&fetch_target_watermark_stmt(raindex_id))
93+
.query_json_retryable(&fetch_target_watermark_stmt(raindex_id))
9294
.await?;
9395
rows.first().map(|r| r.last_block)
9496
} else {
@@ -124,7 +126,7 @@ pub trait BootstrapPipeline {
124126
where
125127
DB: LocalDbQueryExecutor + ?Sized,
126128
{
127-
let rows: Vec<IntegrityCheckRow> = db.query_json(&integrity_check_stmt()).await?;
129+
let rows: Vec<IntegrityCheckRow> = db.query_json_retryable(&integrity_check_stmt()).await?;
128130
let is_healthy = rows
129131
.first()
130132
.map(|row| row.quick_check.eq_ignore_ascii_case("ok"))
@@ -748,6 +750,7 @@ mod tests {
748750
let db = MockDb::default();
749751
let cfg = BootstrapConfig {
750752
raindex_id: RaindexIdentifier::new(1, Address::ZERO),
753+
dump_sql: None,
751754
dump_stmt: None,
752755
latest_block: 0,
753756
block_number_threshold: 10_000,

0 commit comments

Comments
 (0)