Release canonical name normalization 0.4.0

README.md

@@ -195,7 +195,9 @@ if result.success:
 result = detector.normalize_name("John Smith")
 if not result.success:
     print(f"Error: {result.error_message}")
+    print(f"Canonical person name: {result.canonical_name.text}")
     # Expected Output: Error: name not recognised as Chinese
+    # Canonical person name: John Smith
 
 # --- Example 5: Japanese name in Chinese characters (ML-enhanced detection) ---
 result = detector.normalize_name("山田太郎")
@@ -251,6 +253,76 @@ When you call `normalize_name`, you get a `ParseResult` with helpful structured
 - `parsed_original_order`: A `ParsedName` with the same semantic `surname` and
   `given_name` labels as `parsed`, plus an `order` list that records how those
   components appeared in the input.
+- `canonical_name`: an all-person canonical representation. This is populated
+  for Chinese and non-Chinese people, while the legacy `success`, `result`, and
+  `parsed` fields remain Chinese-recognition fields.
+
+### Canonical names for all people
+
+`canonical_name.text` is the fully normalized display form. Its `normalized`
+components expose `given_name`, `middle_name`, `surname`, and `suffix`, plus
+immutable token tuples and their display order. Name dashes and apostrophes are
+standardized to ASCII `-` and `'`; obvious titles and credentials are removed;
+and true generational suffixes are kept in the suffix field.
+
+Periods are treated by role and shape rather than removed globally. Known
+leading titles and trailing credentials are consumed; generational suffixes are
+canonicalized; pure dotted initial clusters are uppercased while preserving the
+source dot positions; transliteration abbreviations such as ``M.Yu.`` retain
+their mixed casing; and a terminal full stop on an ordinary word is removed as
+sentence punctuation.
+
+```python
+western = detector.normalize_name("Dr. Ana–Maria O’Neill PhD")
+assert not western.success  # unchanged: not recognized as Chinese
+assert western.parsed is None
+assert western.canonical_name.text == "Ana-Maria O'Neill"
+assert western.canonical_name.normalized.given_name == "Ana-Maria"
+assert western.canonical_name.normalized.surname == "O'Neill"
+
+suffixed = detector.normalize_person_name("Steve Blando IV")
+assert suffixed.text == "Steve Blando IV"
+assert suffixed.normalized.suffix == "IV"
+
+repaired = detector.normalize_person_name_components(
+    first_name="dr steve",
+    middle_name="marsh",
+    last_name="phd",
+)
+assert repaired.text == "Steve Marsh"
+assert repaired.normalized.given_name == "Steve"
+assert repaired.normalized.middle_name == ""
+assert repaired.normalized.surname == "Marsh"
+```
+
+For an undelimited raw name, token order alone cannot always distinguish middle
+names from multi-token family names. Passing structured components makes the
+caller's first/middle/last assignment the default normalized contract. Cultural
+convention alone does not move a source middle token into the given or family
+field. The same conservative East Asian router used for raw names may reinterpret
+a structured source sequence when its directional evidence is decisive; the
+structured ``canonical_name.source`` still retains the caller's original roles,
+order, and token lineage. Otherwise, roles are re-inferred only when mechanical
+cleanup, such as removing a title or credential, empties a required boundary.
+
+Raw parsing preserves visible order by default. A separate conservative router
+assigns semantic family-first components only for evidence combinations that
+held the existing non-Chinese benchmark constant: three-syllable compact
+Hangul; Japanese native text supported by the Chinese/Japanese classifier and
+component dictionaries; strict Korean romanized shapes; diacritic-bearing
+Vietnamese with a supported initial surname; and two-token Japanese
+romanizations whose surname/given dictionaries support only the family-first
+direction. Ambiguous or unsupported names retain the generic input-order
+normalization.
+
+The East Asian component assets contain no complete-person exceptions. Their
+sources, hashes, licenses, and regeneration command are documented in
+`sinonym/data/EAST_ASIAN_NAME_LEXICONS.md` and
+`scripts/build_east_asian_name_lexicons.py`.
+
+TIMO clients can opt into `sinonym_v2` or `sinonym_routing_v2` to receive the
+same nested canonical payload. The existing `sinonym_v1` and
+`sinonym_routing_v1` response schemas remain unchanged.
 
 Notes:
 - The tokens in `parsed` and `parsed_original_order` are the same normalized tokens; only the conceptual ordering differs via the `order` list.

docs/canonical_name_evaluation.md

@@ -0,0 +1,115 @@
+# Canonical name normalization evaluation
+
+## Result
+
+The final rules were evaluated on 1,000 manually reviewed, real non-Chinese
+author names:
+
+| Measure | Exact | Accuracy |
+|---|---:|---:|
+| Person outcome | 1,000 / 1,000 | 100% |
+| Canonical display text | 1,000 / 1,000 | 100% |
+| First/given component | 1,000 / 1,000 | 100% |
+| Suffix component | 1,000 / 1,000 | 100% |
+| Middle component | 982 / 1,000 | 98.2% |
+| Last/surname component | 982 / 1,000 | 98.2% |
+| All five fields together | 982 / 1,000 | 98.2% |
+
+The 18 component-only differences are reported as semantic-boundary
+differences, separately from display accuracy. Raw undelimited strings cannot
+identify those middle/surname boundaries reliably. When callers provide
+structured first/middle/last fields, those source roles are authoritative by
+default and are repaired only after mechanical cleanup empties a boundary.
+
+The originally frozen 200-name holdout scores 196/200 (98.0%) on semantic
+components and 200/200 on canonical display. The four component differences
+remain visible under the source-preserving policy rather than being closed by
+authority-specific family-span rules.
+
+## Data and manual review
+
+The sampling pool combined:
+
+- 978 selected names from the internal S2 scholarly-author Parquet. The pinned
+  source file contains 41,551,414 rows and has SHA-256
+  `E4050B4832DAAF2F0B8464F0B78E6CA0D35412897BDDA5AF148F7BACBEE9D363`.
+- 22 selected names from the repository's ACL 2025 author list.
+
+Candidate sampling was deterministic, source-balanced, deduplicated, and
+stratified by visible name shape. The final 1,000 contain:
+
+| Shape | Count |
+|---|---:|
+| Ordinary | 498 |
+| Initials | 130 |
+| Non-ASCII Latin | 94 |
+| Hyphen | 69 |
+| Apostrophe | 57 |
+| Family particle | 42 |
+| Comma form | 40 |
+| Suffix | 39 |
+| Four-plus tokens | 31 |
+
+Every initial output was assigned manually from the raw string under the
+written rubric. Parser, model, and normalizer outputs were not shown to or used
+by that initial labeling step. Ambiguous, corrupt, Chinese, and non-person rows
+were excluded from gold rather than guessed. Review produced 1,053 usable gold
+rows; a salted hash selected 1,000, then a separate salted hash fixed an 800/200
+development/holdout split. The remaining 53 rows were a development reserve.
+
+After the final evaluation exposed 23 ambiguous middle/surname boundaries, the
+user explicitly requested a web-source adjudication of every mismatch. Six
+manual labels were corrected from structured institutional or bibliographic
+authority records. The remaining 18 boundaries are retained as semantic
+benchmark differences; they are not treated as sufficient evidence to override
+structured source roles. Each changed review row records its external source
+URLs, and the full adjudication ledger is retained in ignored scratch artifacts.
+
+The selected JSONL has SHA-256
+`B367254A7282DA3B30A37B478A679202E5C65D465541A3A0E772AA649D30B8DE`.
+It contains 977 high-confidence and 23 medium-confidence reviews.
+
+Raw S2-derived names and manual review files remain under ignored
+`scratch/canonical_names/`; they are not committed because they contain
+internal personal data with upstream redistribution constraints. The ACL
+source's redistribution terms are also not documented locally.
+
+## Evaluation protocol
+
+Rules were iterated only against the 800-name development split and the
+53-name reserve. The holdout was evaluated once after the tuning rules were
+frozen. A later independent code review found specification-level edge cases
+not represented in the gold packet; those fixes were developed only against
+synthetic regression strings, without consulting holdout labels. The full
+packet was then rerun as a regression check and the reported holdout score was
+unchanged. Exact match requires the person outcome, canonical text, first,
+middle, last, and suffix to all agree with the manual record.
+
+Reproduction commands, when the ignored source/review artifacts are present:
+
+```powershell
+uv run python scratch\canonical_names\assemble_gold.py `
+  --acl-packet scratch\canonical_names\packets\acl_pilot_60.jsonl `
+  --s2-packet scratch\canonical_names\packets\s2_candidates_1300.jsonl `
+  --reviews-dir scratch\canonical_names\reviews `
+  --output scratch\canonical_names\gold\canonical_non_chinese_gold_1000.jsonl `
+  --reserve-output scratch\canonical_names\gold\canonical_non_chinese_reserve.jsonl `
+  --manifest scratch\canonical_names\gold\manifest.json `
+  --size 1000 --holdout-size 200
+
+uv run python scratch\canonical_names\evaluate_gold.py `
+  --input scratch\canonical_names\gold\canonical_non_chinese_gold_1000.jsonl `
+  --split all --limit 1000 `
+  --report scratch\canonical_names\metrics\all_final.json `
+  --mismatches scratch\canonical_names\metrics\all_final_mismatches.jsonl
+```
+
+## Boundary policy
+
+An undelimited raw string still cannot identify every middle/surname boundary
+from shape alone. The normalizer therefore keeps the conservative final-token
+surname rule, family-particle rules, and initial-shape rules as its default.
+It does not use person-specific or authority-specific family-span exceptions.
+Structured input preserves the supplied first/middle/last roles unless
+mechanical cleanup empties a required boundary. Semantic boundary evaluation is
+reported separately rather than changing that storage contract.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "sinonym"
-version = "0.3.1"
+version = "0.4.0"
 description = "Chinese Name Detection and Normalization Module"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -67,6 +67,7 @@ include = [
     "scripts/verify_multiprocess.py",
     "scripts/generate_surname_romanizations.py",
     "scripts/generate_name_statistics.py",
+    "scripts/build_east_asian_name_lexicons.py",
     "scripts/train_ml_classifier_for_chinese_vs_japanese.py",
     "README.md",
     "pyproject.toml",