Transparent, cost-aware caching for LLM API responses.
Add one line. Every repeated call your process makes to OpenAI, Anthropic, or Google Gemini is served from a local cache in microseconds instead of being re-billed and re-waited-on.
import ai_cache
ai_cache.enable(ttl="24h")
# Nothing below this line is cache-aware.
from openai import OpenAI
client = OpenAI()
client.chat.completions.create(model="gpt-4o", messages=[...]) # billed
client.chat.completions.create(model="gpt-4o", messages=[...]) # free, ~0.2 msThen find out what it bought you:
$ ai-cache stats
Backend sqlite
Entries 1,284
Payload size 14.2 MB
PROVIDER MODEL ENTRIES HITS SIZE SAVED
--------- ----------------- ------- ------ ------- --------
openai gpt-4o 612 4,201 8.1 MB $94.22
anthropic claude-sonnet-4 431 2,118 4.9 MB $61.07
6,319 hits served, $155.29 savedMost LLM workloads repeat themselves far more than their authors expect. Test suites replay the same prompts on every run. Prompt engineering iterates on one variable while everything else stays fixed. Evaluation harnesses sweep the same dataset. Production traffic has a long head of common questions. Every one of those repeats is billed again and waited on again.
ai-cache sits underneath your provider SDK and returns the stored response when the same request comes back. Your code does not change, your control flow does not change, and the response object you get on a hit is the same type with the same fields as the one you got on a miss.
What makes it more than a dictionary:
- Keys that are actually correct. Volatile parameters (
timeout,extra_headers,user, request IDs, callbacks) are excluded, so identical requests do not miss on transport noise. Response-shaping parameters (seed,tools,response_format) are included, so different requests never collide. Pydantic models, enums, dataclasses, and SDK sentinels all normalize to the same key. - Real streaming support. Streamed responses are recorded chunk by chunk and replayed on a hit, with time-to-first-token unchanged. An interrupted stream is never stored as if it were complete.
- Cost accounting. Hits are denominated in tokens and dollars using a built-in pricing table, so "did this help?" has a number.
- Pluggable storage. SQLite by default; memory, Redis, or a tiered combination when you need cross-process sharing.
- Inspectable. A CLI to list, show, explain, prune, export, and import.
pip install ai-cacheThe library itself has zero runtime dependencies - it runs on the standard library alone, so it never pulls a provider SDK into your environment or fights with the versions you already pin. CI enforces this.
Optional extras:
pip install "ai-cache[redis]" # Redis backend for cross-process sharing
pip install "ai-cache[semantic]" # local embedding model for near-match cachingRequires Python 3.8+.
import ai_cache
ai_cache.enable() # ~/.ai-cache, no expiry
ai_cache.enable(ttl="24h") # entries expire after a day
ai_cache.enable(cache_dir="./cache") # keep it with the projectWorks with OpenAI, Anthropic, and Google Gemini - sync and async, streaming and not. Any OpenAI-compatible endpoint (Azure OpenAI, Together, Groq, OpenRouter, vLLM, Ollama) routes through the same classes and is cached automatically.
with ai_cache.bypass(): # neither read nor write
fresh = client.chat.completions.create(...)
with ai_cache.refresh(): # ignore what is stored, but overwrite it
updated = client.chat.completions.create(...)
with ai_cache.scope(namespace="experiment-7", tags=["sweep"], ttl="1h"):
run_evaluation()
ai_cache.invalidate(tag="sweep")These are thread-local: bypassing in one worker does not disable caching for the rest of a pool.
Interceptors see SDK calls. @cached sees whatever you point it at - chains, agent steps, retrieval pipelines, providers with no interceptor - and caches prompt construction and parsing along with the API call.
@ai_cache.cached(ttl="1h", ignore=["client"])
def classify(text: str, client=None) -> str:
response = client.chat.completions.create(...)
return parse(response)Arguments are bound through the signature, so classify("x") and classify(text="x") share a key. It is a transparent pass-through when caching is disabled, so decorated code runs unchanged in environments that never call enable().
Every option is available to enable(), and every one has an AI_CACHE_* environment variable - so caching can be configured for a CI job without touching application code.
| Option | Env var | Default | What it does |
|---|---|---|---|
backend |
AI_CACHE_BACKEND |
sqlite |
sqlite, memory, redis, tiered, null |
cache_dir |
AI_CACHE_DIR |
~/.ai-cache |
Where the database lives |
ttl |
AI_CACHE_TTL |
None |
Entry lifetime: "30m", "24h", "7d" |
namespace |
AI_CACHE_NAMESPACE |
default |
Partition key; namespaces never collide |
max_entries |
AI_CACHE_MAX_ENTRIES |
None |
Evict above this count |
max_size_bytes |
AI_CACHE_MAX_SIZE |
None |
Evict above this size: "500MB" |
eviction |
AI_CACHE_EVICTION |
lru |
lru, lfu, fifo, none |
read_only |
AI_CACHE_READ_ONLY |
false |
Serve hits, never write |
write_only |
AI_CACHE_WRITE_ONLY |
false |
Populate without serving |
cache_non_deterministic |
- | true |
Cache calls that use sampling |
cache_streaming |
AI_CACHE_STREAMING |
true |
Record and replay streams |
semantic |
AI_CACHE_SEMANTIC |
false |
Match similar prompts |
similarity_threshold |
AI_CACHE_SIMILARITY |
0.95 |
Minimum cosine similarity |
providers |
AI_CACHE_PROVIDERS |
all | Restrict interception |
verbose |
AI_CACHE_VERBOSE |
false |
Log every cache decision |
# Turn caching on for a whole CI job without editing a line of Python
export AI_CACHE_ENABLED=true AI_CACHE_TTL=7d AI_CACHE_READ_ONLY=trueA cache you cannot inspect is a cache you cannot trust. ai-cache reads the same store the library writes, and never patches an SDK - so running it cannot perturb a program using the cache concurrently.
$ ai-cache stats # hit counts and savings by model
$ ai-cache ls --model gpt-4o # what is stored, with prompt excerpts
$ ai-cache show 79fe3e28 # one entry in full (key prefixes work)
$ ai-cache explain request.json # why a request keyed the way it did
$ ai-cache prune --older-than 7d # reclaim space
$ ai-cache invalidate --tag nightly # targeted deletion
$ ai-cache export -o cache.jsonl # move a warmed cache between machinesexplain is the answer to "why did these two calls not share an entry?"
$ ai-cache explain request.json
Cache key
79fe3e2806a6a02f76ab0cbdaaa74f724f50f279a315a95236ac01c27d74c945
Parameters that determine the key
messages = [{"role": "user", "content": "question 1"}]
temperature = 0
Ignored (cannot change the response body)
timeout, extra_headers
HIT: stored 4h agoRecord once, replay forever. Commit the cassette; run CI with read_only=True so a test that reaches an unrecorded prompt fails loudly instead of silently spending money.
ai_cache.enable(
cache_dir="tests/fixtures/cassette",
read_only=os.environ.get("RECORD") != "1",
namespace="tests",
)RECORD=1 pytest # once, with API keys
pytest # thereafter: offline, deterministic, freeai_cache.enable(
backend="tiered", # in-process tier in front of Redis
redis_url=os.environ["REDIS_URL"],
namespace=f"prod-{PROMPT_VERSION}", # bump on every prompt change
ttl="12h",
max_size_bytes="2GB",
)Namespacing by prompt version matters: without it, shipping a new template serves answers generated by the old one.
ai_cache.on_event(lambda e: statsd.increment(f"llm.cache.{e.status}"))
print(ai_cache.get_stats(prometheus=True))A listener that raises is logged and skipped - a broken metrics pipeline never breaks user traffic.
Off by default, and deliberately strict when enabled. Serving a stored answer for a similar prompt is a behavioral change you should opt into consciously.
ai_cache.enable(semantic=True, similarity_threshold=0.95)Guardrails: candidates are restricted to the same provider and model, the similarity score is reported on every semantic hit so you can log or reject it, and an embedding failure degrades to exact matching rather than raising. The bundled embedder is local and dependency-free; pass embedder= for a real embedding model when you need synonym matching.
Interception. enable() patches the resource classes where provider SDKs define their methods (openai.resources.chat.completions.Completions.create, anthropic.resources.messages.Messages.create, and so on). Patching the class rather than the instance means every client is covered - including ones constructed before enable() was called, and ones built inside libraries like LangChain or LlamaIndex. disable() restores the exact prior state.
Keying. Requests are canonicalized before hashing: Pydantic models, dataclasses, enums, sets, and bytes reduce to stable JSON; volatile parameters are dropped; SDK "not supplied" sentinels collapse to absent. A key version is mixed into every hash, so a future change to these rules invalidates old entries rather than serving mismatched payloads.
Storage. SQLite with WAL journaling (readers do not block on a concurrent writer), one connection per thread, and eviction pushed down into SQL so large caches do not pull their working set into Python memory.
Failure. Every cache failure degrades to a miss. A corrupt entry, a full disk, an unreachable Redis, a request that cannot be keyed - all of them fall through to the real provider call and log. A cache added to make an application faster must never be able to take it down.
Sampled requests are cached by default. If temperature > 0, the provider may legitimately return something different each time, and caching hides that. This is the right default for the dominant use cases (testing, evaluation, prompt iteration, repeated production questions), but it is a real trade-off. Set cache_non_deterministic=False when callers depend on genuine variety; declining to cache then logs which sampling parameters caused it.
Cost figures are estimates. They come from a built-in list-price table and your stored token counts. Negotiated rates, provider-side prompt caching, and batch discounts are not reflected. Models with no known rate count as zero and are reported separately under models_without_pricing, so the gap is visible rather than quietly understating savings. Override rates with ai_cache.pricing.register() or AI_CACHE_PRICING_FILE.
Prompts are stored in plain text. The cache is a local file containing your requests and responses. It is not encrypted. On a shared machine, set collect_prompt_excerpts=False and treat ~/.ai-cache with the same care as any other file holding user data. See SECURITY.md.
Semantic matching can be wrong. A high similarity score is not a guarantee of interchangeability. Start at the 0.95 default, log event.similarity, and lower it only with evidence from your own prompts.
Lifecycle - enable(**config), disable(), is_enabled(), configure(**changes)
Scopes - bypass(), refresh(), scope(namespace, tags, ttl), @cached(...)
Management - clear(namespace), invalidate(provider, model, tag, older_than), maintain(), entries(...)
Introspection - get_stats(prometheus=False), on_event(listener), active_providers(), get_engine()
Errors - all derive from AICacheError: ConfigurationError, BackendError, BackendUnavailableError, SerializationError, NotEnabledError
Full docstrings are on every public function; help(ai_cache.enable) is authoritative.
Version 0.2 is a rewrite. enable(), disable(), clear(), get_stats(), is_enabled(), and invalidate() keep their signatures, so most code needs no changes.
- Your existing cache file will be emptied on first use. v0.1 entries were keyed by rules this version no longer uses; keeping them would serve responses that do not correspond to the request that produced them. The table is dropped rather than migrated.
- Anthropic and Gemini caching worked in neither 0.1 release. The Anthropic interceptor read an attribute that does not exist and swallowed the error; Gemini keyed off
repr()strings containing memory addresses, which never repeat. If your hit rate was mysteriously zero, this is why. ai_cache.cache_engine.CacheEnginestill imports, with aDeprecationWarning. Its oldget/setmethods are not reimplemented -lookup()returns aLookupResultwhereget()returned a bare payload. It will be removed in 1.0.get_stats()now returnshit_rateas a float (0.87), not a string ("87.0%").
See CHANGELOG.md.
Contributions are welcome - see CONTRIBUTING.md for setup, testing, and the architecture tour. Good first issues are labelled good first issue.
Adding a provider takes about thirty lines and does not require modifying this package:
from ai_cache.interceptors import BaseInterceptor, register_interceptor
class CohereInterceptor(BaseInterceptor):
provider = "cohere"
module_name = "cohere"
def install(self) -> bool:
import cohere
return self.patches.apply(
cohere.Client, "chat", self.wrap_sync(cohere.Client.chat)
)
def extract_model(self, args, kwargs, instance=None) -> str:
return kwargs.get("model", "unknown")
register_interceptor(CohereInterceptor)MIT - see LICENSE.