Skip to content
Abdur-Rafay-ARPublic

About

Transparent, cost-aware caching for LLM API responses. Cut spend and latency on repeated OpenAI, Anthropic, and Gemini calls without changing your code.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

ai-cache

Transparent, cost-aware caching for LLM API responses.

Add one line. Every repeated call your process makes to OpenAI, Anthropic, or Google Gemini is served from a local cache in microseconds instead of being re-billed and re-waited-on.

CI PyPI Python License: MIT

import ai_cache
ai_cache.enable(ttl="24h")

# Nothing below this line is cache-aware.
from openai import OpenAI
client = OpenAI()
client.chat.completions.create(model="gpt-4o", messages=[...])   # billed
client.chat.completions.create(model="gpt-4o", messages=[...])   # free, ~0.2 ms

Then find out what it bought you:

$ ai-cache stats

  Backend       sqlite
  Entries       1,284
  Payload size  14.2 MB

  PROVIDER   MODEL              ENTRIES  HITS    SIZE     SAVED
  ---------  -----------------  -------  ------  -------  --------
  openai     gpt-4o             612      4,201   8.1 MB   $94.22
  anthropic  claude-sonnet-4    431      2,118   4.9 MB   $61.07

  6,319 hits served, $155.29 saved

Why this exists

Most LLM workloads repeat themselves far more than their authors expect. Test suites replay the same prompts on every run. Prompt engineering iterates on one variable while everything else stays fixed. Evaluation harnesses sweep the same dataset. Production traffic has a long head of common questions. Every one of those repeats is billed again and waited on again.

ai-cache sits underneath your provider SDK and returns the stored response when the same request comes back. Your code does not change, your control flow does not change, and the response object you get on a hit is the same type with the same fields as the one you got on a miss.

What makes it more than a dictionary:

  • Keys that are actually correct. Volatile parameters (timeout, extra_headers, user, request IDs, callbacks) are excluded, so identical requests do not miss on transport noise. Response-shaping parameters (seed, tools, response_format) are included, so different requests never collide. Pydantic models, enums, dataclasses, and SDK sentinels all normalize to the same key.
  • Real streaming support. Streamed responses are recorded chunk by chunk and replayed on a hit, with time-to-first-token unchanged. An interrupted stream is never stored as if it were complete.
  • Cost accounting. Hits are denominated in tokens and dollars using a built-in pricing table, so "did this help?" has a number.
  • Pluggable storage. SQLite by default; memory, Redis, or a tiered combination when you need cross-process sharing.
  • Inspectable. A CLI to list, show, explain, prune, export, and import.

Installation

pip install ai-cache

The library itself has zero runtime dependencies - it runs on the standard library alone, so it never pulls a provider SDK into your environment or fights with the versions you already pin. CI enforces this.

Optional extras:

pip install "ai-cache[redis]"      # Redis backend for cross-process sharing
pip install "ai-cache[semantic]"   # local embedding model for near-match caching

Requires Python 3.8+.


Quick start

Cache everything

import ai_cache

ai_cache.enable()                     # ~/.ai-cache, no expiry
ai_cache.enable(ttl="24h")            # entries expire after a day
ai_cache.enable(cache_dir="./cache")  # keep it with the project

Works with OpenAI, Anthropic, and Google Gemini - sync and async, streaming and not. Any OpenAI-compatible endpoint (Azure OpenAI, Together, Groq, OpenRouter, vLLM, Ollama) routes through the same classes and is cached automatically.

Control it per block

with ai_cache.bypass():        # neither read nor write
    fresh = client.chat.completions.create(...)

with ai_cache.refresh():       # ignore what is stored, but overwrite it
    updated = client.chat.completions.create(...)

with ai_cache.scope(namespace="experiment-7", tags=["sweep"], ttl="1h"):
    run_evaluation()

ai_cache.invalidate(tag="sweep")

These are thread-local: bypassing in one worker does not disable caching for the rest of a pool.

Cache whole functions

Interceptors see SDK calls. @cached sees whatever you point it at - chains, agent steps, retrieval pipelines, providers with no interceptor - and caches prompt construction and parsing along with the API call.

@ai_cache.cached(ttl="1h", ignore=["client"])
def classify(text: str, client=None) -> str:
    response = client.chat.completions.create(...)
    return parse(response)

Arguments are bound through the signature, so classify("x") and classify(text="x") share a key. It is a transparent pass-through when caching is disabled, so decorated code runs unchanged in environments that never call enable().


Configuration

Every option is available to enable(), and every one has an AI_CACHE_* environment variable - so caching can be configured for a CI job without touching application code.

Option Env var Default What it does
backend AI_CACHE_BACKEND sqlite sqlite, memory, redis, tiered, null
cache_dir AI_CACHE_DIR ~/.ai-cache Where the database lives
ttl AI_CACHE_TTL None Entry lifetime: "30m", "24h", "7d"
namespace AI_CACHE_NAMESPACE default Partition key; namespaces never collide
max_entries AI_CACHE_MAX_ENTRIES None Evict above this count
max_size_bytes AI_CACHE_MAX_SIZE None Evict above this size: "500MB"
eviction AI_CACHE_EVICTION lru lru, lfu, fifo, none
read_only AI_CACHE_READ_ONLY false Serve hits, never write
write_only AI_CACHE_WRITE_ONLY false Populate without serving
cache_non_deterministic - true Cache calls that use sampling
cache_streaming AI_CACHE_STREAMING true Record and replay streams
semantic AI_CACHE_SEMANTIC false Match similar prompts
similarity_threshold AI_CACHE_SIMILARITY 0.95 Minimum cosine similarity
providers AI_CACHE_PROVIDERS all Restrict interception
verbose AI_CACHE_VERBOSE false Log every cache decision
# Turn caching on for a whole CI job without editing a line of Python
export AI_CACHE_ENABLED=true AI_CACHE_TTL=7d AI_CACHE_READ_ONLY=true

The CLI

A cache you cannot inspect is a cache you cannot trust. ai-cache reads the same store the library writes, and never patches an SDK - so running it cannot perturb a program using the cache concurrently.

$ ai-cache stats                     # hit counts and savings by model
$ ai-cache ls --model gpt-4o         # what is stored, with prompt excerpts
$ ai-cache show 79fe3e28             # one entry in full (key prefixes work)
$ ai-cache explain request.json      # why a request keyed the way it did
$ ai-cache prune --older-than 7d     # reclaim space
$ ai-cache invalidate --tag nightly  # targeted deletion
$ ai-cache export -o cache.jsonl     # move a warmed cache between machines

explain is the answer to "why did these two calls not share an entry?"

$ ai-cache explain request.json

  Cache key
  79fe3e2806a6a02f76ab0cbdaaa74f724f50f279a315a95236ac01c27d74c945

  Parameters that determine the key
  messages = [{"role": "user", "content": "question 1"}]
  temperature = 0

  Ignored (cannot change the response body)
  timeout, extra_headers

  HIT: stored 4h ago

Recipes

Deterministic, offline, free tests

Record once, replay forever. Commit the cassette; run CI with read_only=True so a test that reaches an unrecorded prompt fails loudly instead of silently spending money.

ai_cache.enable(
    cache_dir="tests/fixtures/cassette",
    read_only=os.environ.get("RECORD") != "1",
    namespace="tests",
)
RECORD=1 pytest    # once, with API keys
pytest             # thereafter: offline, deterministic, free

Shared cache across a fleet

ai_cache.enable(
    backend="tiered",                  # in-process tier in front of Redis
    redis_url=os.environ["REDIS_URL"],
    namespace=f"prod-{PROMPT_VERSION}",  # bump on every prompt change
    ttl="12h",
    max_size_bytes="2GB",
)

Namespacing by prompt version matters: without it, shipping a new template serves answers generated by the old one.

Metrics

ai_cache.on_event(lambda e: statsd.increment(f"llm.cache.{e.status}"))
print(ai_cache.get_stats(prometheus=True))

A listener that raises is logged and skipped - a broken metrics pipeline never breaks user traffic.

Semantic near-match caching

Off by default, and deliberately strict when enabled. Serving a stored answer for a similar prompt is a behavioral change you should opt into consciously.

ai_cache.enable(semantic=True, similarity_threshold=0.95)

Guardrails: candidates are restricted to the same provider and model, the similarity score is reported on every semantic hit so you can log or reject it, and an embedding failure degrades to exact matching rather than raising. The bundled embedder is local and dependency-free; pass embedder= for a real embedding model when you need synonym matching.


How it works

Interception. enable() patches the resource classes where provider SDKs define their methods (openai.resources.chat.completions.Completions.create, anthropic.resources.messages.Messages.create, and so on). Patching the class rather than the instance means every client is covered - including ones constructed before enable() was called, and ones built inside libraries like LangChain or LlamaIndex. disable() restores the exact prior state.

Keying. Requests are canonicalized before hashing: Pydantic models, dataclasses, enums, sets, and bytes reduce to stable JSON; volatile parameters are dropped; SDK "not supplied" sentinels collapse to absent. A key version is mixed into every hash, so a future change to these rules invalidates old entries rather than serving mismatched payloads.

Storage. SQLite with WAL journaling (readers do not block on a concurrent writer), one connection per thread, and eviction pushed down into SQL so large caches do not pull their working set into Python memory.

Failure. Every cache failure degrades to a miss. A corrupt entry, a full disk, an unreachable Redis, a request that cannot be keyed - all of them fall through to the real provider call and log. A cache added to make an application faster must never be able to take it down.


Design decisions worth knowing

Sampled requests are cached by default. If temperature > 0, the provider may legitimately return something different each time, and caching hides that. This is the right default for the dominant use cases (testing, evaluation, prompt iteration, repeated production questions), but it is a real trade-off. Set cache_non_deterministic=False when callers depend on genuine variety; declining to cache then logs which sampling parameters caused it.

Cost figures are estimates. They come from a built-in list-price table and your stored token counts. Negotiated rates, provider-side prompt caching, and batch discounts are not reflected. Models with no known rate count as zero and are reported separately under models_without_pricing, so the gap is visible rather than quietly understating savings. Override rates with ai_cache.pricing.register() or AI_CACHE_PRICING_FILE.

Prompts are stored in plain text. The cache is a local file containing your requests and responses. It is not encrypted. On a shared machine, set collect_prompt_excerpts=False and treat ~/.ai-cache with the same care as any other file holding user data. See SECURITY.md.

Semantic matching can be wrong. A high similarity score is not a guarantee of interchangeability. Start at the 0.95 default, log event.similarity, and lower it only with evidence from your own prompts.


API reference

Lifecycle - enable(**config), disable(), is_enabled(), configure(**changes)

Scopes - bypass(), refresh(), scope(namespace, tags, ttl), @cached(...)

Management - clear(namespace), invalidate(provider, model, tag, older_than), maintain(), entries(...)

Introspection - get_stats(prometheus=False), on_event(listener), active_providers(), get_engine()

Errors - all derive from AICacheError: ConfigurationError, BackendError, BackendUnavailableError, SerializationError, NotEnabledError

Full docstrings are on every public function; help(ai_cache.enable) is authoritative.


Upgrading from 0.1.x

Version 0.2 is a rewrite. enable(), disable(), clear(), get_stats(), is_enabled(), and invalidate() keep their signatures, so most code needs no changes.

  • Your existing cache file will be emptied on first use. v0.1 entries were keyed by rules this version no longer uses; keeping them would serve responses that do not correspond to the request that produced them. The table is dropped rather than migrated.
  • Anthropic and Gemini caching worked in neither 0.1 release. The Anthropic interceptor read an attribute that does not exist and swallowed the error; Gemini keyed off repr() strings containing memory addresses, which never repeat. If your hit rate was mysteriously zero, this is why.
  • ai_cache.cache_engine.CacheEngine still imports, with a DeprecationWarning. Its old get/set methods are not reimplemented - lookup() returns a LookupResult where get() returned a bare payload. It will be removed in 1.0.
  • get_stats() now returns hit_rate as a float (0.87), not a string ("87.0%").

See CHANGELOG.md.


Contributing

Contributions are welcome - see CONTRIBUTING.md for setup, testing, and the architecture tour. Good first issues are labelled good first issue.

Adding a provider takes about thirty lines and does not require modifying this package:

from ai_cache.interceptors import BaseInterceptor, register_interceptor

class CohereInterceptor(BaseInterceptor):
    provider = "cohere"
    module_name = "cohere"

    def install(self) -> bool:
        import cohere
        return self.patches.apply(
            cohere.Client, "chat", self.wrap_sync(cohere.Client.chat)
        )

    def extract_model(self, args, kwargs, instance=None) -> str:
        return kwargs.get("model", "unknown")

register_interceptor(CohereInterceptor)

License

MIT - see LICENSE.

About

Transparent, cost-aware caching for LLM API responses. Cut spend and latency on repeated OpenAI, Anthropic, and Gemini calls without changing your code.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages