feat(search): support CJK queries in ticket and article search

[katsushi2441]

Problem

Search drops every Japanese, Chinese and Korean character before it reaches the backend.

Search.clean_query() (helpdesk/search.py) and sanitize_query() (helpdesk/api/article.py) both strip anything outside [a-zA-Z0-9\s]:

unsafe_chars = re.compile(r"[^a-zA-Z0-9\s]")

So a CJK-only query such as パスワード becomes an empty string and the search returns nothing. Sites running in ja / zh / ko cannot search their own tickets or knowledge base at all. The repo ships ko.po and zh.po, so these users exist today.

A second, smaller issue: when the first pass finds too few results, api/article.py falls back to TextBlob noun phrases. Its corpus is English-only, so for a CJK query the fallback replaces valid results with nothing.

What this does

• helpdesk/search_i18n.py (new) — NFKC normalisation that keeps letters and digits in any script, CJK detection, and n-gram helpers. Pure functions, no Frappe imports.
• Index a cjk_terms field (2- and 3-grams) for both the Redis and the SQLite backends, so substring matching works without pulling in a CJK tokenizer (FTS5's default tokenizer does not split CJK text).
• Expand CJK runs in the query into 3-grams at search time. Latin terms pass through unchanged, so existing behaviour for English queries is preserved.
• Skip the TextBlob fallback when the query contains CJK.
• Bump the SQLite index filename (helpdesk_search_v2.db) so existing sites rebuild with the new column.
• Unit tests for normalisation, detection, n-grams and query expansion.

Verification

helpdesk/test_search_i18n.py passes (5 tests). One test builds a real FTS5 table and asserts that パスワード matches a document titled パスワード変更の手順 and does not match an unrelated document.

Example of the query rewriting:

input	normalised	expanded
ﾊﾟｽﾜｰﾄﾞ変更！	パスワード変更	パスワ スワー ワード ード変 ド変更
VPN接続エラー	vpn接続エラー	vpn 接続エ 続エラ エラー

Notes

• Existing English/Latin queries take the same code path as before; only the CJK branch is new.
• The cjk_terms field is stripped from SQLite results before they are returned.
• I am happy to adjust the approach (for example a custom FTS5 tokenizer instead of n-grams) if you prefer a different direction.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

_{Reviews (2): Last reviewed commit: "fix(search): rebuild the Redis index whe..." | Re-trigger Greptile}

[katsushi2441]

Thanks — the Redis migration gap was a real one, and I have pushed a fix in 7a72cfc.

The problem: Search.index_exists() only compared num_docs against the record count. That number stays correct across schema changes, so an index built before cjk_terms existed looked valid, was never rebuilt by build_index_if_not_exists(), and could not serve the new CJK query path. The SQLite backend was already covered because its index filename changed; Redis keeps the same index name, so nothing forced a rebuild there.

The fix: index_exists() now also compares the declared schema against the fields reported by FT.INFO and treats a missing field as "index absent", so the existing rebuild path takes over.

existing = indexed_field_names(ftinfo.get("attributes"))
if existing and not self.index_fields() <= existing:
    return self._index_exists  # False -> rebuild

Two deliberate choices:

• This is generic rather than a one-off cjk_terms check, so any future field addition triggers the same rebuild instead of silently serving a stale index.
• If the attribute shape cannot be parsed, indexed_field_names() returns an empty set and the check is skipped. FT.INFO reports attributes as flat lists on some versions and as mappings on others, and I would rather fall back to the previous behaviour than force a rebuild on an unrecognised shape.

indexed_field_names() lives in search_i18n.py so the tests stay runnable with plain python -m pytest, like the rest of this PR. Tests now cover flat lists, byte strings, mappings, the legacy no-identifier shape, the stale-index case and the unknown shape. 10 tests pass.

Happy to adjust if you would rather bump the Redis index name instead — that is simpler, but it loses the ability to notice future schema drift.