Port AAP2 dictionary lookup rich-comment behavior from TypeDuck alpha

[graphemecluster]

Background

Yune currently documents and implements the TypeDuck dictionary lookup behavior up to:

• TypeDuck-HK/rime-dictionary-lookup-filter commit 3e4605c4fae99f068df2edb85aaeab5a97752795
• TypeDuck-Web behavior/feel target commit 03f9afd2cf6ca75653197f2193f24d1cd0adbd83

Since then, the TypeDuck alpha branch moved the dictionary lookup payload to the AAP2 row shape and changed how rich comment rows are expanded, ordered, deduplicated, and displayed.

This is testable against:

• commits: https://github.com/TypeDuck-HK/TypeDuck-Web/commits/aap2-alpha
• temporary behavior/feel target: https://typeduck-hk.github.io/TypeDuck-Web/aap2-alpha/

That alpha URL temporarily replaces https://www.typeduck.hk/web/ as the behavior/feel target. Everything in the alpha version will go to the main link upon official release.

Important: the TypeDuck-Web schema submodule/checkout must also be moved to its aap2-alpha branch at the same time. The AAP2 comment row format depends on matching schema dictionary rows; mixing the new web/plugin behavior with the old schema data will break testing.

Relevant TypeDuck changes

Useful comparison commands:

git -C third_party/typeduck-web/source log --oneline 03f9afd2cf6ca75653197f2193f24d1cd0adbd83..aap2-alpha -- \
  src/CandidateInfo.ts src/Candidate.tsx src/DictionaryPanel.tsx src/consts.ts librime schema

git -C third_party/typeduck-web/source diff 03f9afd2cf6ca75653197f2193f24d1cd0adbd83..aap2-alpha -- \
  src/CandidateInfo.ts src/Candidate.tsx src/DictionaryPanel.tsx src/consts.ts

git -C third_party/typeduck-web/source/librime/plugins/rime-dictionary-lookup-filter \
  log --oneline 3e4605c4fae99f068df2edb85aaeab5a97752795..HEAD

git -C third_party/typeduck-web/source/librime/plugins/rime-dictionary-lookup-filter \
  diff 3e4605c4fae99f068df2edb85aaeab5a97752795..HEAD -- src/DictionaryLookupFilter.cpp

git -C third_party/typeduck-web/source/schema branch --show-current

Observed relevant commits:

• TypeDuck-Web 23a7c7e - Migrate to aap2 format and make frontend display related entries (canonical, components)
• TypeDuck-Web 245ab72 - Deduplicate dictionary entries and fix unpredictable appearance condition on candidate panel
• TypeDuck-Web db21054 - Fix missing highlight background under a cell for candidates not inside dictionary
• plugin repo 4d0e7b3 - Lookup related entries (canonical, components) using new aap2 format
• plugin repo 3671814 - Deduplicate dictionary entries and fix unpredictable appearance condition on candidate panel

The palette/CI changes on aap2-alpha are not engine-porting requirements, except that the alpha page is now the visual/behavior oracle for candidate/dictionary panel feel.

New rich comment row contract

The raw AAP2 dictionary lookup entry payload is now:

jyutping,displayed_honzi,displayed_jyutping,canonical_honzi,canonical_jyutping,components_honzi,components_jyutping,pron_order,pron_label,lit_col_reading,pos,register,label,written_form,vernacular_form,collocation,eng,hin,urd,nep,ind

match_input_buffer and the lookup honzi are not in the raw dictionary row. The filter prepends them manually.

The emitted comment row after \f\r is now:

match_input_buffer,honzi,jyutping,canonical_honzi,canonical_jyutping,components_honzi,components_jyutping,pron_label,lit_col_reading,pos,register,label,written_form,vernacular_form,collocation,eng,hin,urd,nep,ind

Important differences from the old v1.1.2/yune fixture shape:

• displayed_honzi / displayed_jyutping are resolved inside the plugin and not emitted as separate columns.
• honzi = displayed_honzi || lookup_honzi.
• jyutping = displayed_jyutping || raw jyutping.
• pron_order is not emitted.
• normalized_form was removed.
• definition column order is now eng, hin, urd, nep, ind.
• pipe (|) is now used inside several metadata fields, including POS/register/label and component lists.

The composition pseudo-row must put composition in the label column:

phrase->set_comment(prefix + "\f\r1," + cand->text() + "," + result +
                    ",,,,,,,,,composition,,,,,,,," + entries);

After stripping \r1,, this means downstream parsers should treat label as the composition marker, not the old field index.

Plugin behavior to port

The current C++ implementation is concentrated in src/DictionaryLookupFilter.cpp in the plugin repo. The key helper structure is:

struct LookupLine {
    string lookupHonzi;
    string rawLine;
    vector<string> columns;
    string displayHonzi;
    string displayJyutping;
    string tail;
};

struct EmittedLine {
    string commentLine;
    string dedupeKey;
};

Parsing deliberately only reads the leading columns needed for routing:

const vector<string> columns = LeadingColumns(rawLine, 8);
const bool match = pronunciations.find(columns[0]) != pronunciations.end();
(match ? matchedLines : remainingLines).insert(
    {PronOrder(columns), MakeLookupLine(honzi, rawLine, columns)});

No full CSV parser is needed for routing because the first 10 AAP2 columns are guaranteed not to contain ".

Output line construction:

displayHonzi = columns[1].empty() ? lookupHonzi : columns[1];
displayJyutping = columns[2].empty() ? columns[0] : columns[2];

// tail keeps canonical/components, then skips pron_order:
// columns 3..6 + raw columns from index 8 onward

Canonical redirect behavior:

canonical_honzi = canonical_honzi || lookup_honzi;
canonical_jyutping = canonical_jyutping || row_jyutping;

If both canonical columns are empty, the row is canonical.

Recursive related-entry lookup:

• For each emitted main line, append related rows immediately after that line.
• First append canonical redirects, if present.
• Then append components from:

zip(components_honzi.split("|"), components_jyutping.split("|"))

• Inner recursive lookups keep only rows whose pronunciation matches the recursive lookup pronunciation.
• Use an active lookup set keyed by honzi + "\f" + normalized_jyutping to avoid recursion loops.
• Normalize lookup jyutping by removing ; and spaces, then split pronunciations by \f.

match_input_buffer / row inclusion rules

The current plugin documents the rules this way:

// Emission rules:
// - Direct pronunciation matches and their component entries use
//   match_input_buffer=1, so they appear in both candidate selection and
//   dictionary panels.
// - Canonical redirects use match_input_buffer=0 even when collected while
//   building the 1 group. Empty canonical columns mean the row is already
//   canonical.
// - If there are no direct matches for a non-sentence lookup, unmatched
//   dictionary rows and their related entries are used as the fallback 1 group.
// - If direct matches exist, unmatched canonical rows are kept with 0;
//   unmatched noncanonical rows are kept only through their 0 canonical
//   redirects.
// - Deduplication ignores match_input_buffer and pronOrder, keeps the last
//   collected position within each group, and lets the 1 group win over the
//   0 group.
// - pronOrder is not emitted; it only sorts within one honzi lookup.
//   Related canonical/component lookups keep discovery order across honzi.

More explicitly:

• match_input_buffer=1: candidate selection panel + dictionary panel.
• match_input_buffer=0: dictionary panel only.
• Direct matched rows use 1.
• Components of direct matched rows use 1.
• Canonical redirects use 0, even when the source row is in the first group.
• If there are no direct matches and this is not a sentence lookup, unmatched word rows become the fallback 1 group.
• If direct matches exist, unmatched canonical rows are kept as 0; unmatched noncanonical rows are dropped except for possible canonical redirects.
• After the first group is emitted, matching dedupe keys in the second group are suppressed.

This fixes cases like homographs with non-input pronunciations. For example, AAP2 rows for 處 include cyu2, cyu3, cyu5 -> cyu3, syu2, and syu3; when typing cyu3, cyu2 should still be present as a dictionary-only canonical row, and syu* rows should be included according to the canonical/noncanonical rules instead of disappearing from the raw rows.

Deduplication details

Dedupe key:

display_honzi + "," + display_jyutping + "," + output_tail_without_pron_order

Do not include:

• match_input_buffer
• pron_order

Group behavior:

• Divide rows into group 1 (candidateAndDictionaryRows) and group 0 (dictionaryOnlyRows).
• Within each group, keep the row at the position where the duplicate was collected for the last time.
• Deduplicate across groups too: if a key was inserted in group 1, do not insert it again in group 0.

Ordering:

• pron_order only sorts rows within the same honzi lookup.
• Do not reorder across different honzi values; related canonical/component lookups should keep discovery/insertion order.

TypeDuck-Web behavior target note

Yune keeps a pinned TypeDuck-Web checkout at:

third_party/typeduck-web/source

Per the yune docs, this is an integration/behavior/feel target and harness source. The engine does not include TypeDuck-Web frontend code, and the port should not move React parser/display behavior into yune-core.

The engine-side obligation is the raw RimeCandidate.comment/candidate comment byte contract above. The TypeDuck-Web parser/display behavior below is useful only for interpreting the behavior target and for updating yune's TypeDuck-Web integration expectations if that harness is refreshed.

For reference, TypeDuck-Web CandidateInfo.ts now parses the emitted AAP2 row shape:

const [
  matchInputBuffer, honzi, jyutping, canonicalHonzi, canonicalJyutping,
  _componentsHonzi, _componentsJyutping, pronLabel, litColReading,
  partOfSpeech, register, label, written, vernacular, collocation,
  eng, hin, urd, nep, ind
] = parseCSV(value);

Behavior target changes visible through TypeDuck-Web:

• Drop normalizedForm.
• Drop displayedHonzi, displayedJyutping, and pronOrder from the emitted row contract.
• canonicalReference is present if either canonical_honzi or canonical_jyutping is non-empty.
• If canonicalReference exists, show dim redirect text:

→{canonicalHonzi}({canonicalJyutping})

or just whichever side exists.

• Do not render a main definition or other-language rows for canonical redirects.
• Candidate selection fallback labels also show the redirect before falling back to labels.
• POS/register/labels split on |.
• lit_col_reading can encode a direction and related readings; TypeDuck-Web renders the related readings with Jyutping spacing.
• Written/vernacular/collocation values split display lines with |.

Do not treat TypeDuck-Web display-label maps, such as pron_label, register, POS, or label translations, as engine requirements. Those mappings change as the dictionary/UI is maintained and are frontend behavior-target details.

Yune implementation areas likely affected

Current yune code still appears shaped around the old payload:

• crates/yune-core/src/filter/mod.rs
  • DictionaryLookupFilter::comment_for_candidate
  • sentence_lookup_records
  • composition_lookup_record
  • append_dictionary_lookup_record
• crates/yune-core/src/dictionary/source.rs
  • DictionaryLookupRecord
  • parse_typeduck_lookup_dict_yaml
  • collect_dictionary_lookup_records
• crates/yune-core/src/engine.rs
  • primary_dictionary_lookup_records
  • currently checks fields.get(7) == "composition" after stripping \r1,; with the AAP2 emitted row, label is later in the row and should be checked by name/index after updating the schema.
• crates/yune-core/tests/cantonese_parity.rs
  • existing source-row fixtures and helper assertions are still good places to add AAP2 cases.

The current yune parse_typeduck_lookup_dict_yaml stores code-first lookup dictionary payloads as:

DictionaryLookupRecord {
    code: normalize_table_code(code),
    fields: vec![text.to_owned(), payload.to_owned()],
}

That is not enough for AAP2 related-entry behavior. The Rust representation probably needs to keep parsed AAP2 columns, lookup honzi, display honzi/jyutping fallback, canonical fields, component fields, pron_order for same-honzi sorting, and an emitted tail with pron_order removed.

Suggested yune porting checklist

• Add an AAP2 lookup-record representation for TypeDuck dictionary lookup rows.
• Parse the new raw payload shape without full CSV overengineering; routing only needs the leading columns.
• Emit the new row shape and remove pron_order from output.
• Move displayed-honzi/displayed-jyutping fallback into the filter implementation.
• Implement recursive canonical redirects with fallback (canonical_honzi || honzi, canonical_jyutping || jyutping).
• Implement recursive component lookup over zipped components_honzi|components_jyutping.
• Ensure inner lookups only keep pronunciation-matching rows.
• Implement the group 1 / group 0 inclusion rules above, including canonical redirects as 0 even if collected while building group 1.
• Deduplicate by output tail after displayed honzi/jyutping, excluding match_input_buffer and pron_order; keep last collected position within group; suppress group 0 duplicates already emitted in group 1.
• Update composition pseudo-row so composition is in the label column.
• Update any parser that detects composition rows by positional index.
• Add fixtures for canonical redirect and component recursion, plus homograph non-input rows such as 處 (cyu* / syu*).
• If refreshing the vendored TypeDuck-Web harness under third_party/typeduck-web/source, update its integration expectations separately from the engine-byte port.

Suggested acceptance tests

Engine/comment-byte tests:

• A candidate with direct pronunciation matches emits \f\r1,... rows for direct matches.
• Related canonical rows appear immediately after their parent and use match_input_buffer=0.
• Component rows appear immediately after their parent/redirect chain and use match_input_buffer=1 when they are part of the first group.
• If there are direct matches, unmatched canonical rows remain as 0; unmatched noncanonical rows do not appear unless reached through a canonical redirect.
• If there are no direct matches for a non-sentence word lookup, unmatched rows become the fallback 1 group.
• Dedupe ignores match_input_buffer and pron_order, and group 1 wins over group 0.
• pron_order still sorts rows for the same honzi but does not reorder rows across different honzi.
• Composition row has composition in the emitted label column.

Behavior/feel checks, if refreshing yune's TypeDuck-Web harness:

• Use https://typeduck-hk.github.io/TypeDuck-Web/aap2-alpha/ as the current behavior/feel oracle.
• Canonical redirects render as dim →... text instead of definitions.
• Other-language rows are hidden for canonical redirect entries.
• Candidate selection highlight covers every cell, including candidates without dictionary entries.

These checks are not yune-core engine requirements by themselves; they verify that yune's TypeDuck-Web integration still presents the corrected engine comments like the current TypeDuck alpha.

Generated by Codex