Repository navigation
Commit 57e82e0
committed
Coordinate browser bootstrap across networks before rebuilding indexes (#2898)
Browser startup imports the missing networks together on a fresh database, then rebuilds shared secondary indexes once. This layer above #2897 addresses the cross-network startup cost in [RAI-2651](https://linear.app/makeitrain/issue/RAI-2651/integrate-bulk-sql-bootstrap-and-deferred-indexes-in-raindex-browser).
**Live effect:** coordinated initial browser imports · **Risk:** medium (stored data, startup scheduling) · **Ships:** next raindex/webapp release after #2897
## Decisions
- On a fresh DB with at least two owned networks, download eligible seeds in parallel and import successful downloads atomically in one bounded sqlite-web session. Rebuild the 41 non-unique secondary indexes once; PRIMARY KEY/UNIQUE constraints remain enforced. Populated DBs start independent runners and retain their indexes. Smaller cohorts hand their guards to individual runners and preserve fresh-DB deferral without cohort downloads/timeouts.
- Hold shared bootstrap and per-network ownership through provisioning; transfer each network guard directly into its runner. A shared import Web Lock serializes watermark/index preflight through commit or cancellation, including fallback imports. Recheck DB state after waiting for ownership; unsupported ownership retains indexes.
- Failed downloads retry per target; import/manifest failure returns to normal provisioning. Catch-up analyzes applied data without an extra cohort ANALYZE. Producer artifacts, schema, native import and incremental writes retain their behavior.
## Risks
- Small cold networks wait for the largest participating seed. Decoded cohort SQL stays in memory, and reads wait behind the atomic import. Coordinated manifest and seed requests are bounded to two minutes each; per-network runner requests keep their existing behavior, and fallback may lose shared deferral.
- A seed already at finalized head receives planner statistics after the next applied window. Worker loss during commit can leave its result unknown; subsequent provisioning checks persisted watermarks.
## Proof
- Frozen Base + Robinhood comparison on `3a85402`: parent **60.2 s**, cohort **33.9 s** to both ACTIVE; one 30.9 s import, 897 chunks, one rebuild and no extra ANALYZE. All table counts/index definitions/watermarks match; integrity is `ok`. Warm reload: **0.73 s**, zero seed imports. Fixtures: 42.4 MB gzip / 234 MB SQL, about 470 MB stored origin usage.
- These are local observations, not a guaranteed speedup. Earlier runs ranged from 31.7–37.0 s, and one took 147.8 s during heavy observed Spotlight activity; parent timing also varied. No production latency or peak-memory claim.
- Nix checks and staged independent local reviews cover atomic rollback, uniqueness, cancellation, partial downloads, leadership handoff and cross-tab import preflight. Latest `e240135` [deployed preview](https://rain-orderbook-v6-gaj9om8b9-rain-x-h20.vercel.app/orders) passed a cold import, Orders/Vaults lists, Base order detail with a trade and Robinhood vault detail with balance history; both networks reached ACTIVE and are healthy/ready. The Base seed download took 23.2 s in this functional run, so it is not an import-speed comparison. Warm reload downloads no SQL seed and keeps list data/readiness. Selection was restricted to Base/Robinhood because the existing Ethereum subgraph request remained pending.
- Runtime head `1f71d21` also skips coordination on any populated DB, including after waiting for the bootstrap lock, and names synthetic stop/error tests accurately. Fresh Nix checks on `e240135` pass: 1,926 native, 153 Node/Wasm, 151 actual browser/Wasm and 1,168 JS tests; builds, lint, types and Rust static checks pass. Renewed staged local review is clean (two independent Codex reviewers; no OpenCode).
- The final single-network bypass was exercised with a frozen Base fixture on `e240135`: one manifest fetch, one seed download, one import, one 41-index rebuild, matching watermark, integrity `ok`, and Base healthy/ready/ACTIVE. No live indexing; this is functional evidence, not a paired speed comparison.
- Final `e240135` addresses Marvin’s last minor follow-up: fewer than two owned networks bypass cohort downloads while retaining direct guard handoff and the existing fresh-DB deferred import. All 12 review threads are resolved. Both final-commit bot gates pass: CodeRabbit has no actionable comments and Marvin approved with no open findings. Application CI and the deployed preview pass. Human reviews requested from Jakub, Juan, Alex and Rouz. The known Solidity static failure is accepted separately; human review remains pending.
## Rollout
1. Merge after #2897, release raindex and deploy the webapp. Existing manifests/dumps and sqlite-web 0.0.4 remain supported; this layer needs no producer change.
2. Verify both networks reach ACTIVE, a single rebuild spans the cold cohort, and warm startup downloads no seed.
3. Roll back the raindex/webapp release if startup regresses; schema is unchanged.1 parent c70fa66 commit 57e82e0
9 files changed
Lines changed: 1232 additions & 41 deletions
File tree
- crates/common
- src
- local_db/query
- raindex_client/local_db
- pipeline
- runner
- scheduler
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
147 | 169 | | |
148 | 170 | | |
149 | 171 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
26 | 33 | | |
27 | 34 | | |
28 | 35 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
50 | 56 | | |
51 | 57 | | |
52 | 58 | | |
| |||
89 | 95 | | |
90 | 96 | | |
91 | 97 | | |
| 98 | + | |
| 99 | + | |
92 | 100 | | |
93 | 101 | | |
94 | 102 | | |
| |||
121 | 129 | | |
122 | 130 | | |
123 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
124 | 140 | | |
125 | 141 | | |
126 | 142 | | |
| |||
159 | 175 | | |
160 | 176 | | |
161 | 177 | | |
| 178 | + | |
162 | 179 | | |
163 | 180 | | |
164 | 181 | | |
| |||
241 | 258 | | |
242 | 259 | | |
243 | 260 | | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
244 | 266 | | |
245 | 267 | | |
246 | 268 | | |
| |||
394 | 416 | | |
395 | 417 | | |
396 | 418 | | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
397 | 475 | | |
398 | 476 | | |
399 | 477 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
272 | 273 | | |
273 | 274 | | |
274 | 275 | | |
275 | | - | |
| 276 | + | |
276 | 277 | | |
277 | 278 | | |
278 | 279 | | |
| |||
0 commit comments