Fix IndexError on name parts that are all combining marks (0.3.1)#20
Merged
Conversation
capitalize_name_part strips Unicode Mn characters (combining marks,
variation selectors) after NFD normalization. A part composed entirely
of Mn chars (e.g. a lone combining acute, or an ideographic variation
selector like U+E0100 embedded in a CJK name) became empty, and the
capitalizer then indexed [0] into the empty string, raising IndexError
and crashing normalize_name. Return the empty result instead.
Seen in production on paper_id 274957019 ('彬人 樽󠄀井', with U+E0100).
Bump 0.3.0 -> 0.3.1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
capitalize_name_part strips Unicode Mn characters (combining marks, variation selectors) after NFD normalization. When a name part is composed entirely of Mn characters, stripping leaves an empty string, and the code then indexed [0] into it — raising IndexError and crashing normalize_name.
Fix: guard the empty case in _normalize_and_capitalize_single_part (utils/string_manipulation.py) and return the empty result instead of indexing. One-line behavioral guard; real names are unaffected.
Why it matters
Hit in a production backfill classifying every corpus author name. Example: paper_id 274957019, name
彬人 樽󠄀井, which carries an ideographic variation selector U+E0100 (category Mn) between 樽 and 井. The whole batch/pool aborts on the raised exception, so a single pathological name blocks otherwise-clean data.Repro (before this change)
Tests
Added two regression tests in tests/test_bug_report_fixes.py — a unit test on capitalize_name_part (lone combining mark and U+E0100 both return "", while josé→Jose and ou-yang→Ou-Yang are unchanged) and an integration test that normalize_name on the U+E0100 name no longer raises. Full file passes (42), ruff clean.
Note
Verified this crash still reproduces on the 0.4.0 branch (PR #19) at the same site (string_manipulation.py:593), so this guard is worth carrying forward there too. Separately, the embedded variation selector also defeats 0.4.0's Japanese-lexicon match (a distinct classification issue, not addressed here — that belongs in preprocessing).
Bumps 0.3.0 -> 0.3.1.