v3 collapse closures + tokenizers.Encoding offset API - #87
Closed
hallerite wants to merge 1 commit into
Closed
MacroscopeApp / Macroscope - Approvability Check
completed
Jun 26, 2026 in 1m 41s
Not eligible (click on check for details)
- Eligibility
- Correctness
Note
More information about how approvability works can be found in our Help Center.
Details
Eligibility
This PR changes core tokenization functionality by switching from HuggingFace's tokenizer API to the underlying Rust tokenizers library directly, adds segment collapsing optimization logic across multiple renderers, and introduces a new dependency. These runtime behavior changes to core functionality warrant human review.
Notes:
No code owners found in target merge base 082836b
Loading