Feat/indexer core consolidation - #1
Merged
Merged
Conversation
Introduce @statewalker/indexer-core, a workspace-internal package that consolidates engine-agnostic code previously forked across backends. See openspec/changes/indexer-core-consolidation/ for full design/specs. Phase 1 - Scaffold: * New @statewalker/indexer-core (private:true, not published) * Add as workspace:* dep in the 5 backends Phase 2 - Extract pure helpers: * compositeKey, matchesPrefix, sanitizePrefix, validateDimensionality * toAsyncIterable (replaces 10 unsafe `as AsyncIterable<>` casts in deleteDocuments implementations) * Persistence byte helpers (toBytes, singleChunk, readEntryBytes) * resolveDocId DEFERRED to Phase 6 (absorbed into SqlRetrieverBase) Phase 3 - Unify hybrid merge: * One mergeByRRF/mergeByWeights/mergeHybrid in indexer-core * Three byte-identical hybrid-search.ts files deleted * Preserves composite-key semantics (path + blockId); not routed through indexer-api's reciprocalRankFusion, which keys by blockId alone and applies a top-rank bonus (deferred as separate change) Phase 4 - Unify composite index: * One createCompositeIndex in indexer-core (~225 LOC) replaces three ~275-LOC composites (MemIndex / DuckDbIndex / PGLiteIndex) * Old classes become thin factory functions (~20-40 LOC each) passing getSize closure + optional onDeleteIndex hook * Fixes O(N^2) getDocumentsBlocks bug: removes trackedBlocks map and persistence-replay dance; enumeration now unions sub-indexes once and de-dupes by composite key * Regression test in indexer-tests document-paths.suite (500 blocks must complete in <3s; O(N^2) would take 2+ seconds) Net: -1232 LOC working tree. No public API change: every @statewalker/indexer-api symbol remains exported and behaviourally identical. Tests: 575/575 passing across 6 packages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 of indexer-core-consolidation: extract the persistence-backed
Indexer factory pattern into @statewalker/indexer-core, and remove all
transitional compatibility wrappers.
New: createPersistenceBackedIndexer<F, V>
6 hooks (createFts, serializeFts, deserializeFts, createVec,
serializeVec, deserializeVec) + optional persistence. Generic over
concrete sub-index types so backends keep their instance shapes.
Preserves the wire format byte-for-byte: __manifest__ /
<name>/__config__ / <name>/fts / <name>/vec.
Callers collapse:
* flexsearch-indexer.ts: 243 -> 20 LOC
* minisearch-indexer.ts: 243 -> 20 LOC
Compatibility wrappers removed (no transitional layer kept):
* mem-index.ts (MemIndex function) deleted
* duckdb-index.ts (DuckDbIndex function) deleted
* pglite-index.ts (PGLiteIndex function) deleted
* SQL factories now call createCompositeIndex directly via a local
buildIndex closure shared between createIndex and getIndex
Barrel cleanup (indexer-mem):
* Only MemVectorIndex remains exported; dead re-exports of MemIndex,
mergeByRRF, mergeByWeights, bruteForceSearch, cosineSimilarity
dropped (no external consumers)
* Stale README rewritten to describe actual exports (closes audit
task 10.2)
Net: -494 LOC. No public API change. Tests: 575/575 passing across 6
packages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… (Phases 6-7)
Extract the shared SQL scaffolding from indexer-duckdb and indexer-pglite
into indexer-core. Both backends now differ only in their dialect objects
and a minimal SqlDb adapter.
New in @statewalker/indexer-core:
* sql-db.ts (8 LOC) - SqlDb: minimal normalised async SQL client shape
* create-sql-fts-retriever.ts (237 LOC) - shared FullTextIndex CRUD
+ resolveDocId + path-filter SQL, driven by a SqlFtsDialect
* create-sql-vector-retriever.ts (267 LOC) - shared EmbeddingIndex
CRUD + HNSW DDL + embedding bind/cast/decode, driven by a
SqlVectorDialect
* create-sql-backed-indexer.ts (~200 LOC) - shared Indexer shell:
__indexer_manifest table, extension init, per-index DDL/drop,
composite assembly with dialect-specific unionAliasSuffix
New in each backend:
* dialect.ts (DuckDB: 146 LOC, PGlite: 161 LOC) - the pieces that
genuinely differ: DDL, search SQL, embedding-literal strategy,
embedding decode, extension init. Exports an aggregated
duckdbDialect / pgliteDialect for createSqlBackedIndexer.
* wrapDbAsSqlDb() adapter normalising each driver's result shape
(db-api returns bare array; PGlite returns { rows }).
Sub-index / factory shrink:
* duckdb-full-text-index.ts: 252 -> 24 LOC
* duckdb-vector-index.ts: 243 -> 24 LOC
* duckdb-indexer.ts: 228 -> 15 LOC
* pglite-full-text-index.ts: 275 -> 24 LOC
* pglite-vector-index.ts: 245 -> 24 LOC
* pglite-indexer.ts: 220 -> 24 LOC
Absorbs deferred task 2.4 (resolveDocId extraction) which was held for
Phase 6 because the two backends' driver query methods return results
in different shapes - now normalised via SqlDb.
LIKE-based DuckDB FTS stays in the dialect for now; Phase 8 replaces
it with BM25 via the official `fts` community extension.
Net: -1426 LOC from backend files, -494 LOC effective across the
workspace. No public API change. Tests: 575/575 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 8 - Real DuckDB full-text search via the `fts` extension:
* extensionInit now loads `fts` in addition to `vss`
* FTS table gains a virtual `fts_id` column - required because
create_fts_index takes a single-column identifier; ours is the
concatenation of (doc_id, block_id) via a VIRTUAL GENERATED column
* New SqlFtsDialect.rebuild hook: the DuckDB dialect implements
PRAGMA create_fts_index(stemmer=<lang>, stopwords='none',
strip_accents=1, lower=1, overwrite=1) with a Snowball-stemmer
name derived from the FullTextIndexInfo.language via
DUCKDB_STEMMER_MAP (en→english, fr→french, ...)
* createSqlFtsRetriever tracks a dirty flag: set on every
addDocument/deleteDocuments, forced true on first open, checked
lazily before search and on flush. Dialects without a rebuild
hook (PGlite) bypass this state entirely - tsvector is
maintained automatically by the DB.
* Search SQL: SELECT d.path, b.block_id, b.content,
fts_main_<table>.match_bm25(b.fts_id, $1) AS score FROM ...
WHERE ... IS NOT NULL ORDER BY score DESC LIMIT $k
* README updated to document the fts + vss runtime extensions.
* New indexer-tests cases (cross-backend):
- multi-term rank: "alpha beta" doc ranks above "alpha" alone
- flush + search preserves content (exercises rebuild on flush)
Phase 9 - HNSW cosine metric:
* DuckDB HNSW DDL now emits `WITH (metric = 'cosine')`. Without
this, array_cosine_distance queries silently fell back to
sequential scan (vss default metric is l2sq).
* Parameter-bound embeddings attempted but reverted: the
@statewalker/db-duckdb-node driver rejects JS arrays passed via
prepared.bind() - 38 tests failed. Kept the locale-independent
string-literal path (documented at each dialect).
* PGlite HNSW already declared vector_cosine_ops; verified
unchanged.
Phase 10 - Cleanups:
* indexer-api README corrected (removed references to
non-existent helpers isCollectionPrefix et al.)
* biome check --write --unsafe across touched packages
Tests: 585/585 passing across 6 packages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-refactor READMEs documented APIs that never existed (e.g.
createMemIndex, idx.add/idx.search, chunkByTokens/chunkByParagraph)
and used @repo/ package names that don't match the published
@statewalker/ namespace. This pass rewrites every README to reflect
what the code actually exports.
Sub-repo root:
* Add @statewalker/indexer-core to the package table with a
Published=no marker; restate each backend's actual
implementation (BM25 via `fts` extension for DuckDB, HNSW
cosine everywhere, etc.)
Per-package:
* indexer-api - s/@repo\//@statewalker\//g (7 replacements);
content already accurate
* indexer-chunker - actual exports (chunkMarkdown, scanBreakPoints,
findBestCutoff, findCodeFences/isInsideCodeFence); drop the
fictional chunkByTokens/chunkByParagraph
* indexer-core - document the full post-refactor surface:
createCompositeIndex, mergeByRRF/mergeByWeights/mergeHybrid,
createPersistenceBackedIndexer, createSqlBackedIndexer,
createSqlFtsRetriever, createSqlVectorRetriever, SqlDb,
SqlBackedDialect, plus pure helpers
* indexer-mem-flexsearch / -minisearch - real API is
createXxxIndexer({persistence?}) returning Indexer with the
standard indexer.createIndex(...).index.addDocument(...)/
index.search(...) flow; document the persistence wire format
* indexer-pglite - real function is createPGLiteIndexer
(was: createPgliteIndexer); document the `vector` extension
load, the SQL shape (idx_<prefix>_docs/_fts/_vec with GIN +
HNSW cosine), the owns-db-if-none-passed semantics
* indexer-tests - real export is runIndexerTestSuite
(was: runIndexerSuite); add the fixture-loader exports
indexer-duckdb/README.md and indexer-mem/README.md were already
refreshed in Phases 8 and 5 respectively.
No code changes; tests remain 585/585.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the two integration tests that exercise the indexer-api helpers against a real flexsearch backend into indexer-mem-flexsearch/tests/ where they belong. Drop the indexer-mem-flexsearch devDependency from indexer-api so neither the direct cycle nor the longer indexer-api → mem-flexsearch → indexer-tests → indexer-api loop remains. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…exer-search strategy stack - @statewalker/indexer-api becomes pure contract (types/interfaces, zero runtime) - new @statewalker/indexer-search owns SearchPipeline, SemanticIndex, query parser, intent, reranker blending, mocks - consolidate two RRF implementations into @statewalker/indexer-core; mergeByRRF delegates to reciprocalRankFusion - defaultMultiSearch renamed to fanOutSearch and moved to indexer-core (backend-internal) - remove indexDocuments helper (callers use SemanticIndex.addDocuments) - remove RerankResult type alias (RerankerFn now returns ScoredItem[]) - remove indexer-tests/multi-search.suite.ts; semantic-index suite moves SemanticIndex import to indexer-search - update README for both packages and reorganise indexer-api/src into contract/ subdirectory Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the inline `import("@statewalker/indexer-api").DocumentPath`
type casts with a module-level type-only import. Same behavior; a few
unrelated formatting tweaks come along (SqlDb.query and the vector
search row generic).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…xer-search
Five conflict-free cleanups identified by /grill-complexity on 2026-05-23
(see umbrella notes/2026-05/2026-05-23/grill-complexity-statewalker-indexer-packages.md):
R1 — drop dead exports from @statewalker/indexer-core:
- delete fan-out-search.ts (95 LOC, docstring already admitted "not part
of any consumer-facing public surface")
- drop mergeHybrid wrapper (createCompositeIndex inlines the RRF/weights
branch directly)
- drop buildRrfTrace + RRFTrace/RRFContribution (no caller)
R2 — delete 4 unused SQL sub-index factory files in indexer-duckdb /
indexer-pglite that were never re-exported from any barrel and never
imported (~96 LOC).
R3 — relocate QMD-port utilities (parseStructuredQuery, validateLexQuery,
validateSemanticQuery, extractIntentTerms, selectBestChunk) from the
top-level @statewalker/indexer-search barrel into a secondary ./utils
sub-export. Marks them clearly as "unwired primitives", not active
orchestration surface.
R4 — collapse SemanticIndex (119 LOC class, five pass-through methods)
to two free functions: embedAndAdd, embedAndSearch. Tests rewritten.
B5 — demote @statewalker/indexer-search to "private": true (no umbrella
consumer imports it; published-package shape unearned).
Verified: all 525 indexer tests pass; 41 content-pipeline + 10
content-cli tests pass; typecheck clean except pre-existing
TextEncoder/TextDecoder baseline; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…adversarial inputs Each fix lands red/green: failing test first, then the minimum change to make it pass. Highlights: - path-prefix: boundary-aware match (/foo no longer matches /foobar) - compositeKey: length-prefixed segments so NUL/delimiter can't collide keys - merge: weighted blend preserves order when all input scores are equal - mergeByRRF / SearchPipeline: stop feeding blockId as candidate text - persistence-backed indexer: init failure no longer marks indexer initialised - FlexSearch/MiniSearch deserialize: throw on unsupported version - MemVectorIndex Arrow roundtrip: metadata preserved - SQL retrievers: LIKE wildcards in path prefixes are now escaped, and prefix matching respects path-component boundaries (new sql-path-prefix helper used by every retriever and dialect) - SQL retrievers: addDocument is a single INSERT … ON CONFLICT DO UPDATE - PGlite dialect: language is whitelisted (no SQL injection via info.language) and FTS no longer strips non-ASCII (Russian/CJK/etc. reach the engine) - cosineSimilarity: throws on dimension mismatch instead of silent zero-pad Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eline error surface
Implements the six deferred items from the 2026-05-23 adversarial review,
red/green-driven by per-backend integration suites and unit tests.
D1 — orphaned docs rows: createCompositeIndex grows an `onAfterDelete` hook;
createSqlBackedIndexer wires it to a correlated NOT EXISTS reclamation
against the per-index fts/vec tables. New docs-reclamation.suite runs
against pglite (87→91 tests) and duckdb (166→170 tests).
D2 — atomic createIndex: SqlBackedDialect grows `supportsDDLInTransaction`;
pglite sets it true (BEGIN/COMMIT/ROLLBACK wrapping), duckdb leaves it
false (compensating cleanup via dropIndexTables + manifest DELETE). The
full sequence is restructured so in-memory maps update only after SQL
succeeds. New create-index-atomicity.suite injects failures and asserts
the indexer recovers to a consistent state.
D3 — getIndex restore: SQL backend now calls fts.init() / vec.init() in the
manifest-cache-miss path so externally-dropped tables get recreated
before the index is returned. Regression test in pglite.
D4 — serialised mutations: new createSerialiser() helper feeds an in-memory
mutex that wraps createIndex / deleteIndex in both factories. Reads stay
unwrapped (snapshot-consistent against the in-memory maps). Unit test
against createPersistenceBackedIndexer + integration test on pglite
verify concurrent overwrite produces exactly one live index.
D5 — chunker fence boundary: chunk-markdown.ts now treats cutoff bounds the
same way isInsideCodeFence does (inclusive), advances past the closing
fence line (scanning to the next \n + 1), and re-scans so back-to-back
fences cannot trap the cutoff in the next one. Three boundary tests.
D6 — SearchPipeline onError: PipelineConfig grows
`onError?: (stage, error) => void`. Expander/rerank/citation stages
call it on failure before degrading to the documented fallback. When
omitted, the silent-fallback behaviour is preserved.
Misc: PGlite hookTimeout/testTimeout bumped to 30s (WASM startup cost adds
up across 92 fresh-PGlite instances per suite run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a dirty-bit + cached-output to MemVectorIndex and FlexSearchFullTextIndex. addDocument/deleteDocuments set dirty; the serializers (serializeToArrow / serialize) return the cached bytes/JSON while clean, and the deserializers prime the cache from the input so a just-loaded index also short-circuits its first serialize. Makes no-op re-syncs cheap: the persistence layer still calls the serializer each flush, but the heavy Arrow/FlexSearch encoding only runs when the index actually changed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mkotelnikov
added a commit
that referenced
this pull request
May 28, 2026
Feat/indexer core consolidation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.