feat: support offsetless BYO tokenizers

[hallerite]

Stacked on #70; review and merge after that PR.

Why

Removing Transformers from the base install should allow a genuinely minimal BYO tokenizer. Character offsets are only needed for precise body/scaffold attribution, not for producing correct token IDs.

What changed

• Split the structural Tokenizer and OffsetTokenizer capabilities.
• Preserve a single joined BPE pass for offsetless tokenizers and return is_content=[] instead of rejecting them.
• Keep offset-aware behavior unchanged.
• Return sampled_mask=[] for PrimeQwen when its assistant sampled/scaffold split cannot be recovered without offsets.
• Update the README and cover the offsetless contract across the renderer matrix.
• Preserve Hy3 and Laguna 2.1 system-message indices without splitting the joined BPE pass.

Verification

• Full suite: 3813 passed, 179 skipped, 1 xfailed
• Renderer-wide offsetless matrix: 153 passed
• Focused attribution and multimodal suite: 774 passed, 19 skipped
• Ruff passes.
• Ty completes successfully with the branch's existing advisory diagnostics.

Note

Support offsetless BYO tokenizers and move transformers to optional extras

• Introduces Tokenizer, OffsetTokenizer, and ChatTemplateTokenizer protocols in base.py; all renderer constructors now accept any conforming tokenizer instead of transformers.PreTrainedTokenizer.
• attribute_text_segments no longer raises when a tokenizer lacks offset support. It returns token IDs with has_content_attribution=False; renderers then emit empty is_content masks via _content_mask_or_empty.
• Moves transformers>=4.50.0 and pillow>=12.2.0 into optional extras (renderers[transformers], renderers[multimodal]) in pyproject.toml. Calls that need transformers now raise clear ImportError messages via _require_transformers.
• Removes RendererPool, create_renderer_pool, and _maybe_offload; generate() in client.py now accepts only Renderer.
• Risk: RendererPool and create_renderer_pool are removed from base.py and init.py — any out-of-tree imports of these names will fail. Passing a RendererPool to renderers.client.generate now raises a type error. Base installs no longer include transformers; consumers relying on auto-resolve or load_tokenizer must install the transformers extra.

^{Macroscope summarized ad0326c.}

---

Note

Medium Risk
Wide renderer surface change: token IDs stay aligned but loss-mask consumers lose is_content/sampled_mask on offsetless tokenizers, and decode-inferred message attribution on Hy3/Laguna may mis-label edge cases.

Overview
Hand-coded renderers no longer require a fast tokenizer with return_offsets_mapping. The base Tokenizer protocol covers encode / decode / special-token IDs; OffsetTokenizer is the optional add-on for character-offset attribution.

When offsets are missing, renderers still run a single joined BPE pass (same token_ids as before) but expose is_content=[] via _content_mask_or_empty, instead of raising at construction or returning misleading body/scaffold masks. attribute_text_segments now returns AttributedTextSegments with has_content_attribution so callers know when bools are exact vs placeholders. PrimeQwen3Renderer also clears sampled_mask when assistant sampled/scaffold splits cannot be recovered without offsets.

Hy3 and Laguna 2.1 add _infer_offsets_from_decode (and conservative fallbacks) so joined system/tool headers still get sensible message_indices without splitting the BPE pass; that path does not re-enable is_content.

Docs (README, pyproject) and exports (OffsetTokenizer) reflect the softer BYO contract. New tests/test_offsetless_tokenizers.py matrix-tests render + bridge parity across the renderer set.

^{Reviewed by Cursor Bugbot for commit ad0326c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 17822f8. Configure here.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds offsetless BYO-tokenizer support through shared runtime helpers and changes attribution and sampling metadata across many production renderers. Its decoder-based fallbacks and broad cross-renderer behavior make the change larger than a small isolated compatibility tweak.

Notes:

• Macroscope's correctness review did not run, so approvability was decided on eligibility alone.

No code changes detected at ad0326c. Prior analysis still applies.

^{You can add or adjust custom eligibility rules. Learn more.}

The merge-base changed after approval.