Repository navigation
Commit c70fa66
committed
Integrate atomic SQL bootstrap and deferred browser indexes (#2897)
Browser bootstrap now imports SQL through the published sqlite-web bulk API and builds secondary indexes after the first fresh-database import. This completes the browser integration layer of [RAI-2651](https://linear.app/makeitrain/issue/RAI-2651/integrate-bulk-sql-bootstrap-and-deferred-indexes-in-raindex-browser), following merged #2887 and #2893 and [sqlite-web#35](rainlanguage/sqlite-web#35), published as `0.0.4` after [sqlite-web#36](rainlanguage/sqlite-web#36).
**Live effect:** faster initial browser DB import · **Risk:** medium (stored data, browser bootstrap) · **Ships:** next raindex/webapp release
## Decisions
- Import UTF-8-safe chunks of at most 256 KiB in one worker transaction, cancelling on failure or dropped Rust futures. Cleanup waits for the current SDK request before rollback and retains the executor lock until rollback settles. The browser avoids building an object for each SQL statement; native CLI and incremental writes retain their statement APIs.
- Drop and restore explicit secondary indexes inside the first fresh-database import transaction. Primary/unique constraints remain enforced; later imports retain indexes when their preflight sees an existing watermark. Two tabs starting different networks simultaneously can both see an empty DB and cause an extra index rebuild; data remains correct. Attempt one final `ANALYZE` after initial target provisioning; log statistics failures without discarding already committed import reports.
- Wait for cross-tab import contention every 250 ms for up to five minutes. Known-safe reads use an explicit `query_json_retryable` API; generic `query_json` and writes do not replay a timeout with an unknown commit result. The idempotent cache-size setter also opts into retries. Temporary worker errors from integrity checks never reset the database. Cap failed dump imports at three attempts per target, then fall back to RPC sync.
- Switch the webapp Vercel runtime to supported Node 22 and update its architecture note, as approved to unblock preview testing.
- Pin the released `@rainlanguage/sqlite-web` to exactly `0.0.4`. Align the Wasm test lockfile with the existing `wasm-bindgen 0.2.122` runner so tests execute rather than silently reporting zero tests; update the affected mocks and stale test fixtures.
## Risks
- Existing stale-target refresh still clears that target before opening the atomic import; an interrupted refresh can require RPC replay from its deployment block. Atomic stale-target replacement is a separate recovery follow-up.
- Incremental applies retain their existing ANALYZE frequency. This PR’s final best-effort analysis covers provisioning, including imports with no catch-up window; it does not claim a steady-state analysis-cost reduction.
- Persistent SDK initialization failure still follows the existing cache reset policy; distinguishing worker failure from corruption is a separate recovery follow-up documented in RAI-2651.
- Compressed/decompressed SQL is still held in memory. This change bounds the import messages, not the full download or decoded dump allocation.
- Worker loss during commit can leave the commit result unknown. Retries inspect persisted target watermarks; an import failure remains eligible for at most three provisioning attempts per target in the current runner session. Readiness follows committed index reconstruction. Planner analysis is best effort; its failure does not hide successful provisioning. A permanently pending SDK request can also keep cancellation cleanup pending.
## Proof
- Real browser import from an empty OPFS DB using the frozen filtered SQL: all table counts matched the reference, with matching result hashes for 2,160 orders (547 active, 1,613 inactive) and 1,714 vaults. All 41 explicit indexes and planner statistics were present. Orders/Vaults lists and details rendered; warm reload reused the DB without downloading the dump again. The fixture held the chain head fixed and rejected live indexing/quotes.
- Fresh final Nix checks on submitted `8b4438e35`: 1,925 native workspace tests (599 local DB tests), 140 Node/Wasm tests, 130 actual browser/Wasm tests, and 1,168 JavaScript tests passed. Workspace formatting and Clippy with all targets/features, SDK/components/webapp builds, and UI lint/type/style checks passed. Two fresh staged Codex reviewers and three simplification passes were clean; no OpenCode reviewers were used.
- CodeRabbit cancellation/ANALYZE findings and Marvin cross-tab no-wipe/bounded-import-retry findings are fixed. The follow-up covers dump preflight SELECT timeout retries and uses an idempotent cache-size script with an empty SELECT for the SDK’s JSON response path. Real published sqlite-web 0.0.4 browser verification returned `[]`, confirmed cache size `-25000`, and completed a subsequent atomic import/query successfully.
- Marvin formally APPROVED submitted `8b4438e35` with no blockers. Its two non-blocking notes on preexisting stale refresh and incremental ANALYZE frequency are documented below. CodeRabbit subsequently found that generic JSON timeout retries could replay a committed write; this follow-up makes retries explicit and tests that plain, CTE, and multi-statement `INSERT ... RETURNING` are invoked once after a timeout. CodeRabbit’s fresh review of `8b4438e35` completed with no actionable comments; its review gate is green. All nine review threads are resolved.
- Live Node 22 preview deployed successfully. From a fresh preview origin, Base and Robinhood dumps downloaded and both networks reached ACTIVE/healthy readiness; 3,838 order events, 934 running vault balances, all 41 explicit indexes, and 55 planner-statistics rows were present. Warm reload reused the DB without another SQL dump download. Orders and Vaults lists rendered with those two networks selected (576 active orders); an order detail showed its input/output vaults and a live quote, and the related vault detail showed its balance, order relationship, and deposit history. A Robinhood USDG vault detail also rendered its related orders and recent take-order balance changes while live sync remained ACTIVE. Preview: https://rain-orderbook-v6-k10byupab-rain-x-h20.vercel.app/orders
- Preview limitation: default “All items” also selects Ethereum, while the remote registry has no Ethereum raindex. The UI receives `raindex with network key: ethereum not found` and displays an empty list. Deselecting Ethereum in Networks renders the local DB lists. Network enumeration/query routing and the registry URL are unchanged by this PR; this is documented for separate registry/UI follow-up.
- **Not verified:** a new production-scale bootstrap latency/memory benchmark. The earlier local fixture held the chain head fixed; the deployed preview used live RPC sync.
## Rollout
1. Release raindex and deploy the webapp with the exact SDK pin; verify fresh bootstrap reaches ACTIVE and lists/details load. Existing SQL dumps and manifest format remain supported.
2. The producer update is already deployed: filtered, grouped dumps and their manifest were verified under RAI-2652. No additional producer change is required for this integration.
3. Roll back to the previous raindex/webapp release if bootstrap regresses; the database schema is unchanged.
## Checks
- [x] Kept to the browser import integration and its verification fixtures, plus the approved Node 22 preview runtime fix.
- [x] The app build emits `nodejs22.x`; deployed preview and configured-network lists/details were verified. CodeRabbit is clean, Marvin formally approved, and code/build/preview CI passes on `8b4438e35`. Existing Sol static warnings remain the previously accepted failure; the human review gate is pending. See the default-network registry limitation above.
- [x] Tested the new import, failure/retry, index and analysis behavior.
- [x] Linked the tracking issue and upstream/predecessor PRs.
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
* **New Features**
* WebAssembly local databases can import SQL dumps in chunks, with rollback and retry support if an import fails.
* Database analysis is deferred during dump imports and runs once afterward; analysis failures don’t fail the sync.
* **Bug Fixes**
* Failed dump imports can be retried during provisioning, with RPC synchronization continuing if retries are exhausted.
* SQL dump import failures now provide a clear error message.
* Temporary database worker unavailability no longer triggers an unnecessary database reset.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->1 parent 266b57b commit c70fa66
49 files changed
Lines changed: 1461 additions & 177 deletions
File tree
- crates
- cli/src/commands/local_db
- pipeline
- runner
- common/src
- local_db
- pipeline
- adapters
- runner
- query
- raindex_client
- local_db
- pipeline
- runner
- scheduler
- query
- packages/webapp
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
216 | 216 | | |
217 | 217 | | |
218 | 218 | | |
| 219 | + | |
219 | 220 | | |
220 | 221 | | |
221 | 222 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
| 170 | + | |
170 | 171 | | |
171 | 172 | | |
172 | 173 | | |
| |||
198 | 199 | | |
199 | 200 | | |
200 | 201 | | |
| 202 | + | |
201 | 203 | | |
202 | 204 | | |
203 | 205 | | |
| |||
231 | 233 | | |
232 | 234 | | |
233 | 235 | | |
| 236 | + | |
234 | 237 | | |
235 | 238 | | |
236 | 239 | | |
| |||
266 | 269 | | |
267 | 270 | | |
268 | 271 | | |
| 272 | + | |
269 | 273 | | |
270 | 274 | | |
271 | 275 | | |
| |||
301 | 305 | | |
302 | 306 | | |
303 | 307 | | |
| 308 | + | |
304 | 309 | | |
305 | 310 | | |
306 | 311 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| 87 | + | |
87 | 88 | | |
88 | 89 | | |
89 | 90 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
| 152 | + | |
152 | 153 | | |
153 | 154 | | |
154 | 155 | | |
| |||
236 | 237 | | |
237 | 238 | | |
238 | 239 | | |
| 240 | + | |
239 | 241 | | |
240 | 242 | | |
241 | 243 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
| 193 | + | |
193 | 194 | | |
194 | 195 | | |
195 | 196 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
293 | 293 | | |
294 | 294 | | |
295 | 295 | | |
296 | | - | |
| 296 | + | |
297 | 297 | | |
298 | 298 | | |
299 | 299 | | |
| |||
548 | 548 | | |
549 | 549 | | |
550 | 550 | | |
551 | | - | |
552 | | - | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
553 | 558 | | |
554 | 559 | | |
555 | 560 | | |
| |||
694 | 699 | | |
695 | 700 | | |
696 | 701 | | |
697 | | - | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
698 | 707 | | |
699 | 708 | | |
700 | 709 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
143 | 146 | | |
144 | 147 | | |
145 | 148 | | |
| |||
292 | 295 | | |
293 | 296 | | |
294 | 297 | | |
| 298 | + | |
295 | 299 | | |
296 | 300 | | |
297 | 301 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
104 | 105 | | |
105 | 106 | | |
106 | 107 | | |
107 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
108 | 111 | | |
109 | 112 | | |
110 | 113 | | |
| |||
166 | 169 | | |
167 | 170 | | |
168 | 171 | | |
| 172 | + | |
169 | 173 | | |
170 | 174 | | |
171 | 175 | | |
| |||
930 | 934 | | |
931 | 935 | | |
932 | 936 | | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
933 | 955 | | |
934 | 956 | | |
935 | 957 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
56 | | - | |
| 58 | + | |
57 | 59 | | |
58 | 60 | | |
59 | 61 | | |
| |||
76 | 78 | | |
77 | 79 | | |
78 | 80 | | |
79 | | - | |
| 81 | + | |
80 | 82 | | |
81 | 83 | | |
82 | 84 | | |
| |||
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
91 | | - | |
| 93 | + | |
92 | 94 | | |
93 | 95 | | |
94 | 96 | | |
| |||
124 | 126 | | |
125 | 127 | | |
126 | 128 | | |
127 | | - | |
| 129 | + | |
128 | 130 | | |
129 | 131 | | |
130 | 132 | | |
| |||
748 | 750 | | |
749 | 751 | | |
750 | 752 | | |
| 753 | + | |
751 | 754 | | |
752 | 755 | | |
753 | 756 | | |
| |||
0 commit comments