Prototype smart search for Indonesian, with word lists inside the Bible version

[yukuku]

Why

Search matches letters, and for Indonesian that fails in both directions:

• It misses. kasih can't find mengasihi, because the prefix swallows the k.
• It over-matches. berkat returns every berkata.

This prototype widens each plain query word to its whole word family and matches whole words. On Terjemahan Baru, in verses:

Query	Letter search	Smart search
sembuhkan sakit	6	48
beriman	14	225
berkat	3,059	279

Word lists live in the Bible version

The word families are part of the version's own data, like xrefs and footnotes.

Where they're stored

• In a .yes file, as a new lexicon section.
• In the internal version, as {prefix}_lexicon_bt.bt.
• Both use the same Bintex format and are read by LexiconSection (AlkitabYes2).
• Version.loadLexicon() exposes them, implemented by Yes2Reader and InternalReader.

How forms are encoded (data format version 1)

• The section holds start rules, end rules, a pieces table, then the families. Each form is a sequence of tokens, with no separator characters:

  Token	Meaning
  0	the root as is
  1	the root with its start rewritten by the start rules
  2	a literal, followed by an autostring
  3	the root with its end rewritten by the end rules
  4, 5	reserved, rejected by the reader
  n ≥ 6	entry n − 6 of the pieces table

• Any text can sit between roots: mereka-rekakan is me, root, -, root, kan.

• End rules cover English spelling changes (love → loving, carry → carried) the same way start rules cover Indonesian nasal prefixes.

• For TB (2,942 families, 15,373 forms) that is 78 KiB as the internal file, or 49 KB inside the .yes after Snappy.

• The spec is in docs/binary-formats.md.

.yet and the converters

• .yet gains lexicon lines: the root, then every form spelled out, tab-separated. No notation, no rule table.
• LexiconCompiler, called by the section writer, decides the encoding. It adopts rewrite rules one at a time, each time the one that makes the output smallest, then splits every form into root tokens and text. Text runs used at least twice become shared pieces, and the rest are literals.
• For TB it picks t → "", s → y, k → g, p → m with men as a shared piece, not the textbook k → ng. The file is the same size as with the hand-written table.
• YetFileInput, YetToYes2 and YetToInternal carry the families over and print what the compiler chose. CopyProprietaryAssetsTask now packages *_lexicon_bt.bt.

Related PRs

• TB's internal file: yukuku/androidbible-proprietary#1
• The lexicon lines in in-tb.yet: yukuku/alkitab-sources#3

An Indonesian version without a lexicon gets rules-only families derived on the device.

How a query term is resolved

Checked in this order:

1. The version's word list.
2. The version's vocabulary: a word with no relatives matches only itself.
3. Affix peeling, for words the text never uses, e.g. sembuhkan → sembuh.
4. The classic letter match.

+word and quoted phrases are unchanged.

Diagnostics (visible to the user, on by default in the prototype)

Panel above the results

• how each word was resolved, and the peel trail;
• every form searched, with counts;
• forms the letters can never reach are highlighted, and forms absent from this version are faded;
• timings;
• gained/dropped verses compared with the letter search, with All / New / Dropped filters.

Result rows

• NEW / DROPPED badges, plus a via … line naming the forms that matched.

Search Lab (search menu)

• the switches and the word-list mode: automatic / built-in only / rules only;
• the selected list and vocabulary for any version;
• a "try a word" box comparing the built-in list with rules only;
• a self-check over known-tricky queries.

Also: two experimental settings (Smart search (Indonesian), Smart search diagnostics), and rewritten search tips (EN/ID).

Testing

• ./gradlew testPlainDebugUnitTest testPlainReleaseUnitTest :AlkitabYes2:testDebugUnitTest assemblePlainDebug: all 1,738 tests passing.
  • 855 debug unit tests.
  • 28 AlkitabYes2 tests, including 11 in LexiconSectionTest: rule matching at both ends, the rules found for Indonesian and English samples, text between roots, pieces vs literals, reserved tokens, round trips.
• TerjemahanBaruSmartSearchTest checks the table above against the real TB .yet. It also checks that the generated in-tb.yes section and tb_lexicon_bt.bt decode to the same families. It needs ALKITAB_TB_YET (plus optionally ALKITAB_TB_YES / ALKITAB_PROPRIETARY_DIR) and is skipped otherwise; locally it passes 9/9.
• The converters were compiled and run locally on in-tb.yet. Every regenerated tb_* internal file is byte-identical to the one in the proprietary overlay, apart from the new lexicon file.
• SmartSearchSnapshotTest renders the panel and Lab in EN/ID to Alkitab/build/snapshots/smart-search/.
• Not run on a device or emulator. The sandbox has no KVM.

Open question

The TB word list keeps kasihan as its own family, so kasih drops 145 verses, mostly belas kasihan. The Dropped filter shows them.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xk4Y839XNeb9v1wkQDpQWm

[github-actions]

📱 Preview builds

Signed release builds of 5ec5ad0 — version 5.0.0-dev.121 (23848990) — from this run.

Flavor	Application ID	APK
Alkitab	yuku.alkitab	Download (7.7 MB)
Quick Bible	yuku.alkitab.kjv	Download (7.5 MB)
Sabda Alkitab	org.sabda.alkitab	Download (7.7 MB)

Or open https://182910cb-alkitab-pr.yukuku.workers.dev on an Android device.

These share their application IDs and signature with the Play Store builds, so installing one replaces the corresponding installed app (data is kept).

This comment tracks the latest build for this PR; earlier builds keep their own URLs.