Skip to content

feat[bc]: move prompt cache reconciliation into llm-llamacpp - #4570

Open
gianni-cor wants to merge 6 commits into
mainfrom
codex/tmp-addon-full-prompt-cache
Open

gianni-cor wants to merge 6 commits into
mainfrom
codex/tmp-addon-full-prompt-cache

Conversation

@gianni-cor

@gianni-cor gianni-cor commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • Cached turns currently depend on SDK-side delta-message and tool bookkeeping, so a stale caller view can silently append the wrong history.
  • Warm tool turns can restore KV without receiving tools, leaving the chat-template grammar unarmed after the first turn.

📝 How does it solve it?

  • Treats messages and tools on every cached addon request as the complete authoritative history, renders once, and reconciles the resulting token/media ledger by longest common prefix.
  • Persists the versioned ledger inside the existing sequence-state file, uses exact trimming for attention models, and uses bounded process-local checkpoints for recurrent and hybrid divergence.
  • Keeps generated reasoning resident until the next full render naturally excludes it, while preserving transactional request rollback for cancellation and failures.
  • Removes the obsolete eager reasoning compactor, reasoning-specific recovery paths, remove_thinking_from_context, and RuntimeStats.thinkingBlockDiscards.
  • Updates addon documentation, examples, lifecycle tests, and bumps @qvac/llm-llamacpp to 0.54.0.

🧪 How was it tested?

  • npm run build:ts
  • npm run typecheck
  • npm run lint:ts
  • npm run test:types:consumer
  • Native C++ build on qvac-dev-linux-x64 with physical GPU1.
  • 122/122 cache-ledger, runtime-stat, snapshot, reasoning, and template tests passed.
  • 6/6 model-backed tool-grammar, rollback, recurrent, continuous-batching, and multimodal reconciliation tests passed.
  • 3/3 restored-cache corruption and rollback tests passed.
  • 33/33 cache-management lifecycle tests passed after migration to committed prefill and authoritative full-history inputs.

💥 Breaking Changes

BEFORE:

const prompt = firstTurn ? [...tools, ...newMessages] : newMessages

await model.run(prompt, {
  cacheKey,
  generationParams: { remove_thinking_from_context: true }
})

AFTER:

const prompt = [systemMessage, ...tools, ...fullConversation]

await model.run(prompt, { cacheKey })

Cached callers must resend the full conversation and complete tool list on every turn. Cache files without the embedded ledger are treated as cold misses. Current SDK releases remain incompatible until the SDK follow-up adopts this contract.

🔌 API Changes

// Removed from generation parameters:
remove_thinking_from_context

// Removed from runtime statistics:
thinkingBlockDiscards

@github-actions

Copy link
Copy Markdown
Contributor

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

  • ./docs/website
  • ./packages/fabric/test/integration
  • ./packages/llm-llamacpp/benchmarks/server
  • ./packages/llm-llamacpp/benchmarks/performance
  • ./packages/inference-addon-cpp/mobile
  • ./packages/asr-ggml/benchmarks/server
  • ./packages/embed-llamacpp/benchmarks/server
  • ./packages/embed-llamacpp/benchmarks/performance
  • ./packages/sdk/e2e
  • ./packages/vla-ggml/sim/server
  • ./.github/actions/release-merge-guard

@github-actions

Copy link
Copy Markdown
Contributor

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

@gianni-cor gianni-cor added run-cpp-addon-tests CI: run C++ addon tests (requires verified) run-desktop-addon-tests CI: run desktop integration tests (requires verified) labels Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-cpp-addon-tests CI: run C++ addon tests (requires verified) run-desktop-addon-tests CI: run desktop integration tests (requires verified)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant