feat: Suffix-stripping CMU fallback via lookup libraries for OOV stress recovery

[craigtrim]

Overview

phones_for_word() in pystylometry/prosody/pronouncing.py returns [] for any word not in the CMU Pronouncing Dictionary (~134K entries). Every downstream prosodic analysis — stress patterns, beat detection, metrical foot estimation — silently loses that word. The cmu_coverage field reports the gap but does nothing to close it.

The infrastructure to close it already exists in pystylometry's dependency tree.

---

What We Have

Three lookup libraries — all already optional dependencies of pystylometry under the lexical extras group — ship with pre-computed derivational suffix data, built offline by the morphroot compiler:

Library	Version	Corpus	Coverage	Suffix API
gngram-lookup	1.4.0	Google Ngrams	5M words	get_suffixes(word) → list[str] | None
bnc-lookup	1.5.0	BNC	669K words	get_suffixes(word) → list[str] | None
wordnet-lookup	1.3.4	WordNet	88K words	get_suffixes(word) → list[str] | None

All three return suffixes in innermost-first order (e.g., "beautifully" → ["ful", "ly"]). All three handle allomorphic restoration internally (y-drop, consonant doubling, e-elision). All three return [] for monomorphemic words and None for unknown words.

morphroot is not needed at runtime. It is the build-time compiler that pre-computes suffix data into static hash-bucketed modules shipped inside each lookup package. The runtime path is a single O(1) function call.

---

What Needs to Change

Add a suffix-stripping cascade behind phones_for_word() in pronouncing.py. When CMU exact lookup misses:

Step 1 — Suffix decomposition

Call get_suffixes(word) from available lookup libraries (try gngram first for breadth, fall back to BNC, then WordNet). If suffixes are found, progressively strip them from the surface form to recover a stem.

Step 2 — Stem lookup

Try the recovered stem in CMU. If the stem itself is OOV, strip the next suffix and retry. Continue until a CMU hit or no suffixes remain.

Step 3 — Stress reconstruction

The CMU entry for the stem gives us the root stress pattern. For many English derivational suffixes, stress behavior is well-documented (Chomsky & Halle, 1968):

Suffix	Stress effect	Example
-ness, -less, -ful, -ment, -ship	Stress-neutral (no shift)	happy [1,0] → happiness [1,0,0]
-tion, -sion, -ity	Stress shifts to preceding syllable	educáte → educátion
-ic, -ical	Stress shifts to suffix	history → históric
-ous, -al, -ive, -ent	Stress-neutral or minor shift	danger → dangerous
-ly	Stress-neutral	quick → quickly

For the initial implementation, stress-neutral suffixes (the majority) can simply append unstressed syllables (0) to the root pattern. Stress-shifting suffixes can be handled conservatively — return the root pattern without attempting reconstruction, still better than returning [].

Step 4 — Source tagging (optional)

Add a phones_source field to results so downstream consumers (like beat detection's cmu_coverage) can distinguish:

• "cmu" — exact dictionary hit
• "suffix_cascade" — recovered via suffix stripping
• None — true OOV, no recovery possible

---

Why This Matters for Stylometry

The OOV problem is not random

Words that fall outside CMU are disproportionately author-characteristic vocabulary: neologisms, domain jargon, compound coinages, deliberately unusual word choices. These are exactly the words that carry stylometric signal. Dropping them from prosodic analysis systematically biases results toward common vocabulary — the least discriminating part of an author's language.

The impact on beat detection (#76)

Beat detection classifies phrase-level stress shapes (isocolon vs. climactic vs. anti-climactic). Each OOV word in a unit is a hole in the stress sequence. A unit with 50% OOV words produces an unreliable stress shape, even if the known words are analyzed perfectly. Recovering stress patterns for even the stress-neutral-suffix cases (the majority of English derivational morphology) meaningfully improves beat shape reliability.

The impact on AI-tell detection (#69)

LLM-generated text tends to use common vocabulary (high CMU coverage). Human expert writing uses more specialized vocabulary (lower CMU coverage). If we can't analyze the prosody of the specialized vocabulary, we lose a discrimination axis — the very words that distinguish human from AI are the ones we're dropping.

---

Linguistic Background

English derivational morphology has a well-studied property: most suffixes are stress-neutral (they attach without changing the stress pattern of the base). This was formalized in Chomsky & Halle's The Sound Pattern of English (1968) and refined by subsequent work on lexical phonology (Kiparsky, 1982; Mohanan, 1986).

The stress-neutral class includes the most productive English suffixes: -ness, -less, -ful, -ly, -ment, -er (agentive), -ing, -ed. These account for the large majority of derived forms in running text. The stress-shifting class (-tion, -ity, -ic) is smaller and more predictable — stress moves to a fixed position relative to the suffix.

This means that for most OOV derived forms, the stress pattern of the root (if recoverable via CMU) is a reliable proxy for the stress pattern of the full word, with unstressed syllables appended for the suffix material.

---

Implementation Path

1. Add optional imports for lookup libraries in pronouncing.py (same pattern used in lexical/hapax.py)
2. Add _suffix_cascade_lookup(word) that tries suffix stripping → CMU stem lookup
3. Wire it into phones_for_word() as a fallback before returning []
4. Optionally expose phones_source metadata for downstream confidence weighting
5. Unit tests with known OOV-but-decomposable words (unhappiness, beautifully, nationalization)

No new dependencies — uses libraries already in the optional dependency tree.

---

Scope

This issue covers the suffix-stripping fallback only (Tier 2 of the G2P cascade described in #81). It does not cover:

• Tier 3 suffix/pattern rules for jargon and acronyms
• Tier 4 neural G2P fallback
• Any changes to morphroot itself

---

Related Issues

• feat: Extend ARPAbet coverage to out-of-vocabulary words via G2P cascade #81 — G2P cascade design (this issue implements Tier 2 of that cascade)
• Feature: Beat Detection — Phrase-Level Stress Shape Analysis #76 — Beat detection (primary downstream consumer — cmu_coverage improves)
• Feature: AI Stylistic Tell Detection — Tricolon, Stacked Sentences, and AI Vocabulary Fingerprints #69 — AI stylistic tell detection (better prosodic coverage of human-characteristic vocabulary)
• Replace pronouncing with direct cmudict access #60 — Replace pronouncing with direct cmudict access (completed — established the pronouncing.py abstraction layer this builds on)
• CMU dictionary edge case: syllable_count returns 0 for some entries #63 — CMU dictionary edge case: syllable_count returns 0 (related CMU coverage gap)

References

• Chomsky, N. & Halle, M. (1968). The Sound Pattern of English. Harper & Row.
• Kiparsky, P. (1982). Lexical morphology and phonology. In I. S. Yang (Ed.), Linguistics in the Morning Calm, 3-91. Hanshin.
• Mohanan, K. P. (1986). The Theory of Lexical Phonology. Reidel.
• Weide, R. L. (1998). The CMU Pronouncing Dictionary, release 0.6d. Carnegie Mellon University.