Multilingual glossary, LLM translation, and reading-level support - #1211
Merged
Merged
Conversation
… pipeline Checkpoint before the all-locales override run. Translation backend & pipeline: - Add Claude (Anthropic) translator alongside Google, selectable via WORDPLAY_TRANSLATOR; preserves Wordplay markup/code, localizes embedded examples, and uses a reading-level + glossary system prompt. - Translate the glossary words first per locale, then everything else with the localized glossary supplied, so bare key terms localize to the target word. - Never ship fake English: retry/split truncated and malformed chunks, and mark genuine failures unwritten ($?) instead of English marked machine-translated. - Translate doc arrays atomically; sanitize machine-translated names into valid identifiers (fold spaces/underscores to camelCase) in both the locale and example-localization paths. - Route every locale/tutorial/source write through a single Prettier-formatting, write-if-changed helper; fix tutorial $~ marker accumulation; add interim progress logging. Glossary (#780) & docs: - Unified @term glossary references (migrated from $term), glossary block in locales, Guide glossary views, and a getConceptName resolver (breaking the ExceptionValue import cycle by avoiding the nodes graph in the locale layer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Override dry run for zh-CN (CJK probe): validates names stay valid for a caseless, space-less script, glossary-first ordering, split-retry recovery, and prose localization with concept-links/template-inputs preserved. Corrected three propagated glossary word-choice errors surgically: - value: 价值 (worth) → 值 (computed value), incl. 147 prose occurrences - name: 姓名 (a person's full name) → 名字 (identifier name), incl. 109 prose - act: 表演 (performance) → 幕 (theatrical act), glossary word only — prose 表演 left intact as the distinct "performance" concept Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Override dry run for ar-SA (RTL/bidi probe): validates that @Concept/$name tokens survive intact inside right-to-left Arabic prose, names stay valid for Arabic script (0 name warnings), glossary-first ordering, and clean markers ($?=0, no accumulation). Core glossary terms verified (قيمة/نوع/اسم); deeper Arabic prose quality needs a native review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
batch.ts spawns one self-scoped `start.ts <command> <locale>` child process per locale, a bounded number at a time (default 4), to parallelize the otherwise serial all-locales translation sweep (~28 locales × 15-40 min each). Each child loads and writes only its own locale, so there's no shared state and one failure can't abort the others. - Gated to translating: `override`/`translate` only (verify/fix/ci stay serial). - --jobs N / JOBS env for concurrency; positional locales, default all-but-en-US. - Live [locale]-prefixed output + end summary (✓/✗, elapsed, non-zero on any fail). - Pure helpers (parseBatchArgs/resolveLocales/runPool) unit-tested. - npm: locales-translate-parallel, locales-override-parallel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etadata
Make emoji generation part of the translation run, and remove the one-off
LocaleToCLDR map by deriving CLDR annotation codes from canonical language
metadata.
- start.ts now generates a locale's {locale}-emojis.json in-process during a
translate/override run (best-effort; a CLDR/network failure warns and the run
continues), so a new/updated locale gets emojis without a separate step.
- Convert scripts/generate-emojis.mjs → src/util/verify-locales/generateEmojis.ts:
a typed module that imports SupportedLocales + getCLDRCandidates directly
(no regex-parsing, no hardcoded map), writes via writeFormatted (Prettier-clean,
write-if-changed), and exports generateEmojisForLocale + a CLI entry.
- LanguageCode.ts: add `cldr` (base override, tl→fil) and `cldrByRegion` (script
overrides, zh→zh_Hant for TW/HK/MO) to LanguageMetadata, plus getCLDRCandidates
that derives the per-locale candidate list. This also fixes locales the hand
map had missed/mistyped (sv-SE, bn-BD, id-ID, ro-RO, pt-PT) which silently fell
back to English emoji names.
- Enable Nepali (ne-NP) in DraftLocales (#1204/#1205).
- npm: locales-emojis → tsx; delete the old .mjs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…locales Previously TranslatedTutorialModes = [complete], so the quick tutorial was never created, translated, or required for non-en-US locales — it fell back to English in production. Include all TutorialModes so translate/override create and fill the quick tutorial per locale and CI requires it, eliminating the English fallback. Note: this gates CI on every locale having both tutorials translated; the quick tutorials are filled by the locale translation sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…utput, how-to code Four robustness/quality fixes to the offline translator, surfaced generating Nepali: - Reconcile chunk translations by echoed index instead of a strict positional array. A miscount (dropped/merged item) now isolates to the missing index — one targeted retry, then $? — instead of failing the whole batch and recursively splitting it. This eliminates the O(n log n) split-cascade that ground a single tutorial chunk for ~20 min / ~1000 calls; ne-NP re-ran with 0 cascades. - ConceptLink.isValid: a glossary term that parses as a how-to keyword (e.g. @how → "how-to") now validates via the glossary, not only as a how-to id. - Translator prompt: instruct full translation into the target language's native script, avoiding English loanwords/transliteration unless no equivalent exists (verbatim @Concept/$name refs still excepted). - verifyHowTo: localize embedded \code\ examples in how-tos (names/text within), matching the tutorial/locale paths — no more English code in localized prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First full Claude translation of Nepali: locale strings, complete + quick tutorials, 27 how-tos, and CLDR-derived emoji names. Validated: $?=0 across locale and both tutorials, no marker accumulation. Glossary words and embedded how-to code came back in English (Claude's tendency for Nepali technical vocabulary, not a pipeline defect) and are left for the human Nepali translator to refine. (#1204, #1205) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nescaping' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
A translate/override run does five kinds of work per locale (locale strings, complete tutorial, quick tutorial, how-tos, emoji). Previously it was all-or-nothing per locale, forcing wasteful re-generation. Add a +/- flag syntax to scope a run by content type and narrow within a type. (no flags) all five categories (unchanged default) -<category> … exclude whole categories +<category> … include only these categories +<category>:<spec> narrowed include (repeat to add more) Specifiers: locale:<path-prefix>, tutorial/quick:<act>[/<scene>] (1-based), howto:<id>. Mixing +/-, a specifier on an exclude flag or on emoji, an unknown category, or a malformed specifier are usage errors. - New contentCategories.ts (pure parser + Selection) with tests. - start.ts parses the selection + focal locale, gates/narrows each step; verification always runs, only the translation pass is scoped. - verifyLocale/verifyTutorial/verifyHowTo take optional narrowing filters. - batch.ts parses + forwards the flags verbatim to each child run. - Log.exit is now typed `never`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploy-blocking verification fixes: - node.Mention.doc (27 locales) + node.DocumentedExpression.doc (gu-IN, mr-IN): restored the \¶@program¶\ / \¶$name¶\ literal-symbol code escaping that migration/translation had broken (lost ¶, swapped, or translated the word inside the code), which produced invalid "@" tokens. The code examples are language-invariant, so they're copied from en-US while keeping the translated prose. - ne-NP output.Form.names: was English "Form" (collided with en-US's global name); set to स्वरूप ($~, for human review). Also includes the machine translations produced by the parallel translate run (filled $? glossary/UI strings + new quick tutorials per locale) so the locales pass CI. Emoji regeneration is intentionally excluded pending an annotation regression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
codes.txt carries only Unicode metadata (general category, script) — not
character names — so the fallback for characters CLDR doesn't annotate
(ASCII digits, punctuation) was emitting the category code as the name
("0030" → ["nd"] instead of ["digit zero"]). Fall back to the English
name in en-US-emojis.json (the canonical name source) before the category.
Regeneration is now idempotent against the good committed files. Also
regenerates ne-NP (whose committed emojis carried the bug, since it was
added via the in-pipeline generator) and sv-SE (which had a corrupted
"↑↑↑" entry), and adds emoji files for bn-BD, id-ID, pt-PT, ro-RO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ydev/wordplay into llm-translation-overhaul
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #780. Closes #460.
Overhauls Wordplay's localization infrastructure: a unified multilingual glossary of code terms (#780), reading-level support for localized text (#460), an LLM (Claude) machine-translation backend replacing Google v2, and a hardened, parallelized translation pipeline with tooling. Also enables a new locale (Nepali) and ships Claude-translated es-MX, zh-CN, and ar-SA.
Multilingual glossary of code terms (#780)
@termreferences. Migrated glossary references from the old$term(Mention) syntax to the@termconcept-link family (migrateGlossaryRefs), so$is reserved for template-input substitution and@covers all documented-thing references (concepts + terms).@termresolves viaConceptLink→TermRef→TermView.glossaryblock in every locale (GlossaryTexts.ts): each term has a localizedword+definition, so the same concept's term can be compared across languages.GlossaryView(browse/compare terms),GlossaryEntry, andTermView; glossary entries also surface in global search so a term match is recognizable.getConceptNameresolves a term id to the localized concept name from canonical metadata (replacing duplicated glossary entries), with no nodes-graph dependency (also broke anExceptionValueimport cycle).glossaryScan+LocalizationQuality) offers one-click@termfixes;linkGlossarytooling annotates the first mention of each term in docs and tutorial dialogue.Reading-level support (#460)
readingLevel.ts'sPLAIN_LANGUAGE_GUIDANCEis part of the translator's system prompt, so machine translations target young, multilingual learners.LocalizationQualitycalls theanalyzeLocalizationcloud function, which flags overly complex strings with a specific WCAG plain-language note (which principle to fix).LLM (Claude) machine translation
Translatorinterface withClaudeTranslator(Anthropic) alongsideGoogleTranslator, selected explicitly viaWORDPLAY_TRANSLATOR(no silent default). Claude has no enumeration endpoint, so it covers the full offered locale set.$?) — which fails the locale gate loudly and is retried next run — rather than emitting English marked as a machine translation.protect.tskeeps\code\,@Concept, and$nameintact; embedded\code\examples are localized as valid, conflict-free programs (translateProjectContent);docarrays translate atomically; machine-translated names are sanitized into valid identifiers (toValidName— folds spaces/underscores to camelCase).getLLMTranslationscloud function.Pipeline hardening & tooling
writeFormattedroutes every locale/tutorial/source write through Prettier with write-if-changed, so generated files stay consistent and re-runs don't churn git.batch.ts(npm run locales-override-parallel/locales-translate-parallel) runs N self-scoped per-locale processes concurrently (default 4,--jobs), with per-locale isolation, live[locale]-prefixed logs, and a ✓/✗ summary.{locale}-emojis.jsonautomatically (no separate step). CLDR annotation codes were unified into the canonicalLanguagesmetadata (getCLDRCandidates, withcldr/cldrByRegionfields), removing the one-offLocaleToCLDRmap and fixing several locales the hand-map had missed or mistyped (e.g.sv-SE,bn-BD,id-ID,ro-RO,pt-PT) that previously fell back to English emoji names.npm run locales-translate <locale>), now including emojis.$~marker-accumulation bug (markers compounding across override runs); added interim per-chunk/per-example progress logging for the long runs.Locales
ne-NP) enabled in draft locales (Add Nepali (ne-NP) localization #1204, Begin Nepali localization #1205).es-MX,zh-CN,ar-SA. zh-CN included glossary word-choice fixes (value/name/act) propagated through the prose.Validation
npm run check:nowclean (strict TS, noany/as); 2951 unit tests pass.@Concept/$nametokens survive intact, and failures surface as$?rather than silent English.🤖 Generated with Claude Code