perf(search): reduce indexing and ranking overhead

[JustYannicc]

What changed

Consolidates the search, file indexing, launcher file-candidate assembly, and command discovery performance work into one upstream PR.

The combined change removes root ranking diagnostics, memoizes frecency defaults, reuses and precompiles root command scoring data, indexes browser search hot paths, avoids full script reads for Raycast headers, keeps healthy file-index intervals incremental, speeds up file delete tombstoning and path-like queries, maps launcher file result commands by path, and adds focused performance/correctness harnesses for those paths.

Why

These changes reduce repeated ranking, indexing, lookup, and discovery work on launcher hot paths. Keystrokes should avoid unnecessary full browser scans, repeated command rescoring, repeated root command field normalization/tokenization, O(n^2) launcher file-result command lookups, large path scans, interval rebuilds when the watcher is healthy, and full reads of large script command files during discovery.

Compatibility impact

Runtime behavior is intended to stay compatible. Existing scoring and ordering paths remain the source of truth after indexed candidate selection, root command scoring keeps alias exact sorting, always-on-top handling, title tie-breaks, and second-pass match metadata, file search keeps full fallbacks for query shapes that cannot be represented safely by the index, script execution keeps shebang/bash behavior, and launcher file result command matching keeps strict path matching, result order, command data, missing-command fallback behavior, and first-command-wins duplicate path semantics. No user-facing strings or i18n keys changed. The new scripts are test/performance harnesses plus a perf:file-search package script.

How tested

• Codex LSP status: TypeScript language server installed.
• Codex LSP diagnostics: no errors for src/renderer/src/hooks/useLauncherCommandModel.ts or src/renderer/src/utils/launcher-file-candidates.ts after folding perf(files): map launcher file result commands by path #552.
• git diff --check origin/main...HEAD: passed.
• node --test scripts/test-launcher-file-result-command-map.mjs: passed.
• node scripts/test-root-search-ranking.mjs: passed.
• node scripts/test-root-search-perf.mjs: passed.
• node --test 'scripts/test-*.mjs': passed, 45 passing tests and 1 expected Electron-launch skip.
• pnpm exec tsc -p tsconfig.renderer.json --noEmit --pretty false: attempted after dependencies were generated, but the local pnpm wrapper re-entered install policy and failed on ignored build scripts.
• ./node_modules/.bin/tsc -p tsconfig.renderer.json --noEmit --pretty false: still fails on existing repo-wide renderer type debt from origin/main, including existing errors in App.tsx, CameraExtension.tsx, LiquidGlassSurface.tsx, browser profile typing in useLauncherCommandModel.ts, useLauncherKeyboardControls.ts, and several Raycast runtime/settings/quicklink files.

Performance evidence

• Launcher file command-map clean PR evidence from perf(files): map launcher file result commands by path #552: exported map-builder lookup on 3,000 results/commands and 500 iterations reported .find median 18.121 ms vs path-map median 0.024 ms, with equal checksum 11445000.
• Launcher file candidate assembly integrated harness: node --test scripts/test-launcher-file-result-command-map.mjs on this branch reported path-map median 28.471 ms vs find-backed median 48.703 ms on 3,000 results, 1.71x speedup, with equal checksum 99347250.
• Full suite launcher file candidate assembly rerun: path-map median 27.020 ms vs find-backed median 47.626 ms on 3,000 results, 1.76x speedup, with equal checksum 99347250.
• Root command scoring direct harness: 6,000 commands / 8 queries, legacy filter + two-pass command scoring median 2340.07 ms, compatibility wrapper median 736.23 ms, indexed command scoring median 110.06 ms, command score index compile median 77.58 ms, legacy-vs-indexed speedup 21.26x.
• Root command scoring full-suite harness: legacy median 2332.86 ms, compatibility wrapper median 740.70 ms, indexed median 108.73 ms, command score index compile median 77.74 ms, legacy-vs-indexed speedup 21.46x.
• Browser indexed harness from the full suite: 24,001 entries / 300 tabs, full-scan parity preserved; indexed query averages 1.27-12.26 ms vs full-scan averages 120.20-138.11 ms.
• File delete benchmark: origin/main 10,760.35 ms vs this branch 14.41 ms on 68,162 entries / 4,041 delete paths.
• Script discovery benchmark: origin/main discovery 85.2 ms and 40.0 MiB read vs this branch 10.0 ms and 1.3 MiB read; cached execution shebang lookup 5.2 ms / 1.0 MiB vs 2.8 ms / 0 B.
• File-index interval harness: origin/main healthy interval rebuilds=3 and watcher-error recovery rebuilds=0; this branch healthy interval rebuilds=0 and watcher-error recovery rebuilds=1.
• Path-like file query harness: origin/main 11,429.38 ms total / 31.748 ms per query vs this branch 17.21 ms total / 0.048 ms per query on 61,201 entries / 360 queries.
• File-search perf harness: initial index 60.584 ms; normal query p95 0.591 ms; path-like query p95 1.168 ms; watcher update batch 0.805 ms; delete batch 2.540 ms.

Stack validation

#552's clean commit 5ce121e was folded into #535 by cherry-pick as cb0f2ea2e5c52678a819870d3e7f73e00ff36e7b on codex/consolidate-search-indexing. The branch was pushed to JustYannicc:codex/consolidate-search-indexing, and #535 remains targeted at SuperCmdLabs/SuperCmd:main. Generated root pnpm-lock.yaml and pnpm-workspace.yaml files were removed before push and are not tracked.

Standalone renderer typecheck still fails on existing repo-wide renderer type debt from origin/main; this PR keeps that validation cleanup separate. Touched-file LSP diagnostics for the #552 fold-in are clean.

Replaces

• Replaces perf(search): remove root ranking diagnostics #481 perf(search): remove root ranking diagnostics
• Replaces perf(files): optimize file search delete batches #489 perf(files): optimize file search delete batches
• Replaces perf(frecency): memoize sorting defaults #493 perf(frecency): memoize sorting defaults
• Replaces perf(files): avoid interval file-index rebuilds #495 perf(files): avoid interval file-index rebuilds
• Replaces perf(script-command): avoid full script reads for headers #505 perf(script-command): avoid full script reads for headers
• Replaces perf(root-search): optimize command scoring #506 perf(root-search): optimize command scoring
• Replaces perf(file-search): speed up path-like queries #508 perf(file-search): speed up path-like queries
• Replaces test(file-search): add performance harness #513 test(file-search): add performance harness
• Replaces perf(browser-search): index hot query path #520 perf(browser-search): index hot query path
• Replaces perf(files): map launcher file result commands by path #552 perf(files): map launcher file result commands by path (folded in as cb0f2ea from clean commit 5ce121e; current integrated harness reports path-map median 28.471 ms vs find-backed median 48.703 ms, 1.71x speedup, with exact checksum parity)
• Replaces perf(root-search): precompile command scoring fields #557 perf(root-search): precompile command scoring fields (folded in as d1589f4; current root scoring harness reports indexed median 110.06 ms vs legacy 2340.07 ms, 21.26x speedup, with exact result-signature assertions passing)

[shobhit99]

Reviewed the full diff (mostly test/perf harnesses; ~750 lines of real source changes across 8 files). Focused on ranking/relevance parity, index correctness, and the internal>browser precedence rule. Traced the actual match predicates and downstream assembly rather than trusting the harnesses alone.

Strengths

• Root command scoring split is faithful. createRootCommandScoreIndex builds rankingFields (alias 1.06 / keyword 0.7) and scoringFields (alias 1.08 / keyword 0.68), exactly reproducing the old two-pass weights. Since matched depends only on kind/baseScore (weight never changes whether a field matches), ranked.matched === scored.matched always, so the scored.matched ? … : ranked… fallback is inert and matchKind/matchScore come out identical. Sort keys (exact-alias → alwaysOnTop → score → title) and ascending candidate order are preserved.
• scoreSingleField refactor is equivalent. The removed hasBoundaryFuzzyMatch condition (A) tokens.some(startsWith term) was already covered by the earlier token-prefix branch, so by the time control reaches word-boundary-fuzzy it reduces to condition (B) — exactly the new field.boundaryInitials.startsWith(term.compact) check. Precompiled term/field values match the old inline normalizations.
• Browser index is a correct superset. len-2 query → word-start prefix bucket (and the scoring path only yields a positive score for len-2 via prefix/word-boundary, so mid-word-only entries the index drops also score 0 in a full scan); len≥3 → trigram intersection (superset of substring); nickname candidates are unioned in for bookmarks. Candidate IDs stay ascending by entryId, so stable-sort tie-breaks are preserved. Parity test (test-browser-search-performance.mjs) backs this across representative queries incl. gi/gh.
• Path-like file candidate index is sound. getPathLikeBoundaryTerms deliberately drops the index-0 term (which may be mid-token and thus not a real path-token prefix) and keeps only separator-preceded terms — each guaranteed to be a prefix of some indexed pathToken. Using the smallest bucket yields a superset; the scoring loop re-checks normalizedPath.includes(expandedNeedle), and it falls back to a full scan (null) when no safe term exists.
• Tombstone rewrite is equivalent. Nested-path collapse + hasDeletedPathAncestor component-walk matches the old startsWith(path + sep) semantics; single-path fast path handles a trailing-sep edge better than the old p + sep.
• Interval/watcher change is well-guarded (incremental flush only when watcher healthy + a full rebuild happened within 6h; watcher-error closes/nulls the watcher and forces a throttle-bypassing rebuild).
• Diagnostic cruft removed cleanly (SC-RANK build stamp + SC-RANK2 useEffect); frecency now uses one Date.now() per sort (more consistent, same order); no i18n keys touched.

Concerns / potential bugs

• Latent (masked) divergence for punctuation-only queries — command-helpers.tsx rankCommandsWithIndex empty-query branch + useLauncherCommandModel.ts commandCandidates. For a query that is non-empty after trim() but normalizes to empty (pure separators, e.g. ..., @@, %% — SEARCH_TOKEN_SPLIT_REGEX strips them all), shouldIndexRootCommands is still true, and the if (!normalizedQuery) branch now returns every command (matchScore 0). The old commandCandidates had an explicit if (!scored.matched) return null guard that dropped them all; the refactor removed that guard (relying on ranked.matched, which the empty-query branch bypasses). Net: the candidate set goes from none → all commands. In practice this is invisible because both result paths in root-search-sections.ts (lines 171, 186) gate on isRootResultPromotionCandidate (finalScore ≥ 500 for commands), and a matchScore-0 command tops out around ~415 (tierBoost ≤155 + alwaysOnTop 80 + frecency ≤180). So no user-facing regression today — but it leans on the promotion floor to stay hidden and wastes a little work per keystroke on such queries. Suggest restoring an explicit matched/score guard (e.g. skip the empty-normalized branch when hasSearchQuery, or filter matchScore <= 0) for robustness.
• Script header read is now bounded — script-command-runner.ts readScriptCommandHeader (256 KB / 120 lines). The old code read the whole file before slicing 120 lines. A script whose @raycast.* metadata sits beyond 256 KB within the first 120 lines (e.g. one enormous leading comment) would no longer be discovered. Extremely unlikely for real Raycast headers (they sit at the very top) — flagging as a narrowing, not a blocker.
• Shebang staleness — same file. interpreter/interpreterArgs are captured at discovery and cached (12 s TTL) rather than re-read from disk at execution. Editing a script's shebang won't take effect until the cache expires (old code re-read it fresh each run). Minor. The fs.accessSync(R_OK) correctly preserves the old "fail if the file vanished" behavior.

Suggestions (optional)

• The len-2 browser url/title prefix path assumes the token index (built from searchFields) and normalizedUrl/normalizedQuery agree on leading characters; they derive from the same source text so this holds, but a comment noting the invariant would help future edits avoid a silent drop.

🟡 Approve with minor comments — the perf work preserves ranking/ordering and the internal>browser precedence rule; the one real divergence is masked by the promotion floor, and the script-runner edges are low-risk. Worth restoring the explicit matched guard before merge.

🤖 Automated review by Claude Code (Opus 4.8).

[JustYannicc]

Follow-up for the review feedback is pushed in f51c3f3.

Addressed:

• Restored the explicit guard for non-empty command queries that normalize to empty (..., @@, punctuation-only). rankCommandsWithIndex now returns no command matches for those inputs instead of flowing all commands through with score 0.
• Added root-search regression coverage for punctuation-only queries in the root command parity/perf harness.
• Reviewed the bounded script header read. I kept the 256 KiB / 120-line cap as an intentional perf guard; Raycast metadata is expected at the top of the script, and existing coverage still verifies bounded reads for large scripts.
• Tightened the shebang-cache behavior: execution now re-reads only the first line before spawn, so shebang changes/removals take effect inside the 12s discovery cache window without returning to the old full-script read.
• Added script-runner tests for bounded first-line reads, updated shebangs within the cache window, and removed-shebang fallback behavior.

Verification:

• node --test scripts/test-file-search-index.mjs scripts/test-file-search-delete-batch.mjs scripts/test-launcher-file-result-command-map.mjs scripts/test-script-command-runner.mjs scripts/test-root-search-ranking.mjs ✅ 17 passed
• git diff --check ✅