Repository navigation
Tier 1+2 optimizations: arena index, array DP, span probing - #9
Merged
Merged
Conversation
- WordDictionary: the character-level prefix index is now arena- allocated (struct nodes referenced by index, sibling-chain children upgrading to per-node dictionaries at high branching factors), and dict.txt is parsed with spans instead of Split/int.Parse. - GetDag returns dense WordEnd[][] rows built through one reusable scratch list; Calc fills int[]/double[] route arrays instead of a Dictionary of Pair allocations. - FinalSeg Viterbi: the B/M/E/S hidden states use flat double[]/int[] dynamic programming tables - no per-character dictionary or Node chain allocations. - CutForSearch/Tokenize: gram probes go through ContainsWord span overloads, allocating only accepted grams. Cold start improved from ~750ms to ~680ms (index arena costs less than the previous node graph, span parsing is cheaper); upstream is ~480ms - the remaining gap is the string Trie that still backs the public surface (see WordDictionary).
Member
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
落实此前排期的第一、二梯队优化。行为保持逐字节一致（123 个测试含逐字比对全过，NativeAOT 冒烟通过）。
变更
第一梯队（冷启动）
WordDictIndex：字符前缀索引改为 arena 分配——结构体节点存入大数组、以索引互指，子节点用兄弟链（低分支）+ 阈值升级字典（高分支），构建期不再有 55 万次逐节点堆分配。LoadDict改 Span 解析：ReadOnlySpan切分 +int.TryParse(span)，替代每行Split(' ')的数组与小字符串分配；坏行行为与原实现一致（无空格 Debug.Fail 跳过，坏词频 FormatException 中止加载）。第二梯队（运行时）
GetDag返回稠密WordEnd[][]（单个复用的 scratch list 填充），Calc填充int[]/double[]路由数组——彻底移除每句 n 个List+ n+1 个Pair分配与字典开销。double[n*4]+int[n*4]平坦表——每字符零字典、零 Node 链分配；平局裁决（<=取后者）与原实现逐一对齐。CutForSearch/Tokenize：gram 探查改走ContainsWord(ReadOnlySpan<char>)，仅在命中时分配字符串。Benchmark（i9-10900KF / .NET 10.0.9 / BDN ShortRun，vs 上游 0.42.2）
全部 12 个场景 1.28x~2.89x；分配普遍下降 25%~67%（如 medium 精确+HMM 872→364 KB）。冷启动 ~680ms（arena+Span 解析较上轮 ~750ms 改善；剩余差距是支撑公开 API 的字符串 Trie，优化路径见代码注释）。