Skip to content

Add typed KV cache configuration and runtime reporting#453

Open
aleroot wants to merge 2 commits into
ml-explore:mainfrom
aleroot:lampo-turboquant-integration
Open

Add typed KV cache configuration and runtime reporting#453
aleroot wants to merge 2 commits into
ml-explore:mainfrom
aleroot:lampo-turboquant-integration

Conversation

@aleroot

@aleroot aleroot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Proposed changes

This PR is a follow-up to #232, which introduced TurboQuant KV-cache compression, Metal kernels, prompt-cache persistence, boundary-layer protection, and the initial kvScheme integration.

The underlying TurboQuant implementation is unchanged. This PR focuses on making KV-cache configuration safe, extensible, and consistently applied across the library.

Previously, configuration was spread across maxKVSize, kvBits, kvGroupSize, quantizedKVStart, and the stringly typed kvScheme. This caused several problems:

  • Unknown scheme strings could be silently ignored.
  • MTP speculative decoding did not forward kvScheme.
  • Wired-memory measurement did not measure the requested TurboQuant configuration.
  • maxKVSize could create an entirely rotating cache topology, making compression an all-FP16 no-op without exposing that result to callers.
  • Cache traversal, conversion, and configuration forwarding were repeated across generation paths.
  • Applications could not reliably report whether compression was active, pending, or skipped.

This PR introduces:

  • A typed KVCacheConfiguration covering capacity, compression strategy, and compatibility policy.
  • An opaque strategy representation, allowing future cache implementations to be added without exposing an exhaustive public enum.
  • Typed affine and TurboQuant configurations, including quality-first, balanced, and memory-first presets.
  • An immutable resolved cache plan shared by autoregressive, classic speculative, MTP, guided-generation, and wired-memory paths.
  • Centralized traversal and classification of nested and hybrid cache topologies.
  • Explicit compatibility policies for partially supported, mixed, and all-rotating cache configurations.
  • Fail-closed behavior for typed compression requests that would otherwise affect no layers.
  • Effective per-layer runtime reporting, including active, pending, skipped, rotating, and recurrent cache state.
  • A compatibility adapter for the existing scalar and kvScheme parameters.
  • Consistent typed capacity handling across the default, Falcon-H1, Qwen3.5, Pixtral, and Mistral3 cache factories.
  • Updated KV-cache documentation and configuration tests.

A bounded compressed rotating cache is intentionally not introduced here. Rotating layers remain FP16, as in #232. Instead, callers can now explicitly allow partial compression or reject configurations where compression would not engage.

Checklist

Put an x in the boxes that apply.

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

@aleroot
aleroot force-pushed the lampo-turboquant-integration branch 2 times, most recently from b868cbb to 28f7344 Compare July 22, 2026 16:21
Centralize cache strategy resolution, compatibility validation, and nested cache traversal behind a resolved generation plan. Wire the typed configuration through autoregressive, speculative, MTP, guided, and wired-memory paths while preserving legacy parameter behavior.
@aleroot
aleroot force-pushed the lampo-turboquant-integration branch from 0ce1218 to 2419b1c Compare July 22, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant