ollama-embedding-cache.md

Ollama Embedding Cache

Feature: SHA256-based LRU embedding cache for OllamaEmbeddingClient Version: 1.0.0 Status: Production Ready ✅ Performance: 3800× speedup, 100% latency reduction for cached content

---

Overview

The Ollama Embedding Cache eliminates redundant API calls to the Ollama embedding service by caching embeddings using SHA256-based content hashing. This provides dramatic performance improvements for repeated content.

Key Metrics

Metric	Value
Cache Hit Latency	~0.03ms
Cache Miss Latency	~97ms
Speedup Factor	3800×
Latency Reduction	100% (for cached content)
Hit Rate (realistic workload)	78%+

Architecture

User Request
    ↓
generate_embedding(text)
    ↓
SHA256 Hash Generation
    ↓
┌─────────────────┬─────────────────────┐
│ CACHE HIT       │ CACHE MISS          │
├─────────────────┼─────────────────────┤
│ Return cached   │ Call Ollama API     │
│ embedding       │ (~97ms)             │
│ (~0.03ms)       │                     │
│                 │ Store in cache      │
│                 │ (LRU eviction)      │
└─────────────────┴─────────────────────┘
    ↓
Return embedding

---

Features

1. SHA256-Based Cache Keys

• Content Hashing: hashlib.sha256(content.encode()).hexdigest()
• Deterministic: Same content always produces same cache key
• Collision-Resistant: SHA256 ensures uniqueness
• Variable Length Support: Handles arbitrary content length

2. LRU Eviction Policy

• Max Size: Configurable (default: 1000 entries)
• Eviction: Least Recently Used when cache full
• Access Tracking: OrderedDict maintains insertion order
• Update on Access: Move to end on cache hit (most recent)

3. Thread Safety

• Lock Protection: threading.Lock for all cache operations
• Concurrent Access: Safe for multi-threaded environments
• Atomic Operations: Get/put/evict are atomic

4. Performance Metrics

• Hit/Miss Tracking: Real-time hit/miss counters
• Eviction Tracking: Count of evicted entries
• Hit Rate Calculation: (hits / (hits + misses)) * 100
• Cache Stats API: get_cache_stats() for monitoring

---

Configuration

Environment Variables (.env.devstream)

# Enable SHA256-based LRU embedding cache
# Reduces redundant Ollama API calls for duplicate content
DEVSTREAM_EMBEDDING_CACHE_ENABLED=true

# Maximum number of cached embeddings (LRU eviction when full)
# Default: 1000 entries (~10MB memory for 300-dim embeddings)
DEVSTREAM_EMBEDDING_CACHE_SIZE=1000

Python API

from ollama_client import OllamaEmbeddingClient

# Initialize with cache enabled (default)
client = OllamaEmbeddingClient()

# Initialize with cache disabled
client = OllamaEmbeddingClient()
client.cache_enabled = False

# Initialize with custom cache size
client = OllamaEmbeddingClient()
client.cache_max_size = 500

---

Usage

Basic Usage

from ollama_client import OllamaEmbeddingClient

client = OllamaEmbeddingClient()

# First call - cache miss (~97ms)
embedding1 = client.generate_embedding("Python is a programming language")

# Second call - cache hit (~0.03ms)
embedding2 = client.generate_embedding("Python is a programming language")

# embeddings are identical
assert embedding1 == embedding2

Cache Statistics

# Get cache performance metrics
stats = client.get_cache_stats()

print(f"Cache size: {stats['size']}/{stats['max_size']}")
print(f"Hits: {stats['hits']}")
print(f"Misses: {stats['misses']}")
print(f"Evictions: {stats['evictions']}")
print(f"Hit rate: {stats['hit_rate']:.1f}%")

Cache Management

# Clear cache (reset all metrics)
client.clear_cache()

# Disable cache temporarily
client.cache_enabled = False
embedding = client.generate_embedding("text")  # Always calls API

# Re-enable cache
client.cache_enabled = True

---

Performance Benchmarks

Test Environment

• Hardware: Apple M2 Pro
• Python: 3.11.13
• Ollama Model: embeddinggemma:300m
• Embedding Dimension: 768

Results

1. Cache Miss Latency (Ollama API Call)

Test	Latency
Text 1	106.11ms
Text 2	89.11ms
Text 3	98.18ms
Text 4	96.40ms
Text 5	94.34ms
Average	96.83ms

2. Cache Hit Latency (Memory Lookup)

Test	Latency
Text 1	0.05ms
Text 2	0.02ms
Text 3	0.03ms
Text 4	0.02ms
Text 5	0.01ms
Average	0.03ms

3. Realistic Workload (80% cache hit rate)

• Total Requests: 50
• Total Time: 894.11ms
• Average Latency: 17.88ms
• p95 Latency: 91.21ms
• Cache Hit Rate: 78.4%

Performance Summary

Metric	Value
Cache Miss Avg	96.83ms
Cache Hit Avg	0.03ms
Speedup	3809.8×
Latency Reduction	100.0%

Verdict: ✅ EXCELLENT - Recommended for production use

---

Implementation Details

Cache Key Generation

def _generate_cache_key(self, text: str) -> str:
    """
    Generate SHA256-based cache key for text content.

    Args:
        text: Text content to hash

    Returns:
        SHA256 hash as hexadecimal string (64 characters)
    """
    return hashlib.sha256(text.encode('utf-8')).hexdigest()

Cache Get (Thread-Safe)

def _cache_get(self, cache_key: str) -> Optional[List[float]]:
    """
    Retrieve embedding from cache (thread-safe).

    LRU Behavior: Move accessed item to end (most recently used).
    """
    if not self.cache_enabled:
        return None

    with self._cache_lock:
        if cache_key in self._embedding_cache:
            # Move to end (most recently used)
            self._embedding_cache.move_to_end(cache_key)
            self._cache_hits += 1
            return self._embedding_cache[cache_key]

        self._cache_misses += 1
        return None

Cache Put (LRU Eviction)

def _cache_put(self, cache_key: str, embedding: List[float]) -> None:
    """
    Store embedding in cache with LRU eviction (thread-safe).

    Eviction: When cache full, remove first item (least recently used).
    """
    if not self.cache_enabled:
        return

    with self._cache_lock:
        # Check if cache is full
        if cache_key not in self._embedding_cache and \
           len(self._embedding_cache) >= self.cache_max_size:
            # Evict least recently used (first item)
            evicted_key, _ = self._embedding_cache.popitem(last=False)
            self._cache_evictions += 1

        # Add to cache (or update if exists)
        self._embedding_cache[cache_key] = embedding
        # Move to end (most recently used)
        self._embedding_cache.move_to_end(cache_key)

---

Testing

Unit Tests

Location: tests/unit/test_ollama_cache.py

# Run unit tests
.devstream/bin/python -m pytest tests/unit/test_ollama_cache.py -v

Test Coverage:

• ✅ SHA256 cache key generation (deterministic, unique, format)
• ✅ Cache operations (get, put, miss, hit, disabled)
• ✅ LRU eviction (when full, multiple evictions, access order)
• ✅ Cache statistics (initial, after operations, clear)
• ✅ Thread safety (concurrent access, concurrent eviction)
• ✅ End-to-end integration (cache hit, disabled, performance)

Results: 19/19 tests passed ✅

Performance Benchmark

Location: tests/benchmarks/benchmark_ollama_cache.py

# Run benchmark
.devstream/bin/python tests/benchmarks/benchmark_ollama_cache.py

Benchmark Coverage:

1. Cache miss latency (Ollama API call)
2. Cache hit latency (memory lookup)
3. Performance comparison (speedup, latency reduction)
4. Realistic workload (80% hit rate)
5. Cache statistics (hits, misses, evictions, hit rate)

---

Memory Usage

Calculation

• Embedding Dimension: 768 (embeddinggemma:300m)
• Float Size: 8 bytes (Python float)
• Embedding Size: 768 × 8 = 6,144 bytes (~6KB)
• Cache Max Size: 1000 entries
• Total Memory: 1000 × 6KB = ~6MB

Overhead

• OrderedDict Overhead: ~100 bytes per entry
• SHA256 Cache Keys: 64 bytes per entry
• Total Overhead: ~164 bytes per entry = ~164KB for 1000 entries

Total Memory Usage: ~6MB (embeddings) + ~164KB (overhead) = ~6.2MB

---

Best Practices

When to Enable Cache

✅ Enable cache when:

• Processing duplicate or similar content
• High read-to-write ratio (many reads, few writes)
• Latency-sensitive applications
• Production deployments with repeated queries

❌ Disable cache when:

• Processing unique content every time
• Memory-constrained environments
• Testing/debugging (to force fresh API calls)

Cache Size Tuning

Workload	Recommended Size
Low Volume (<100 unique texts/day)	100 entries
Medium Volume (100-1000 unique texts/day)	500 entries
High Volume (1000+ unique texts/day)	1000-2000 entries

Formula: cache_size = unique_texts_per_day * 0.5

Monitoring

# Log cache stats periodically
import logging

stats = client.get_cache_stats()
logging.info(
    "Ollama cache stats",
    hit_rate=stats['hit_rate'],
    size=stats['size'],
    evictions=stats['evictions']
)

# Alert if hit rate drops below threshold
if stats['hit_rate'] < 50.0:
    logging.warning("Cache hit rate low", hit_rate=stats['hit_rate'])

---

Troubleshooting

Low Hit Rate

Symptom: Hit rate <30% in realistic workload

Possible Causes:

1. Content is mostly unique (not repeated)
2. Cache size too small (frequent evictions)
3. Content variations (whitespace, capitalization)

Solutions:

1. Increase cache size: DEVSTREAM_EMBEDDING_CACHE_SIZE=2000
2. Normalize content before embedding (trim, lowercase)
3. Monitor eviction count - increase size if evictions > 10% of misses

High Memory Usage

Symptom: Process memory usage growing over time

Possible Causes:

1. Cache size too large
2. Large embedding dimensions
3. Memory leak (unlikely with LRU eviction)

Solutions:

1. Reduce cache size: DEVSTREAM_EMBEDDING_CACHE_SIZE=500
2. Clear cache periodically: client.clear_cache() (resets metrics)
3. Monitor cache size: stats['size'] should be ≤ cache_max_size

Cache Disabled Unexpectedly

Symptom: All requests show as cache misses

Possible Causes:

1. DEVSTREAM_EMBEDDING_CACHE_ENABLED=false in config
2. Cache explicitly disabled: client.cache_enabled = False

Solutions:

1. Check .env.devstream: DEVSTREAM_EMBEDDING_CACHE_ENABLED=true
2. Verify in code: client.cache_enabled should be True

---

Migration Guide

Existing Code (No Cache)

from ollama_client import OllamaEmbeddingClient

client = OllamaEmbeddingClient()
embedding = client.generate_embedding("text")

New Code (With Cache)

No changes required! Cache is enabled by default and works transparently.

from ollama_client import OllamaEmbeddingClient

# Cache automatically enabled
client = OllamaEmbeddingClient()

# First call - cache miss
embedding1 = client.generate_embedding("text")

# Second call - cache hit (transparent speedup)
embedding2 = client.generate_embedding("text")

Opt-Out (Disable Cache)

# Disable via environment variable
# .env.devstream: DEVSTREAM_EMBEDDING_CACHE_ENABLED=false

# Or disable in code
client = OllamaEmbeddingClient()
client.cache_enabled = False

---

Future Enhancements

Planned Features

1. Persistent Cache (v1.1.0)

   • Store cache to disk (SQLite or Redis)
   • Survive process restarts
   • Share cache across sessions

2. Cache Warming (v1.2.0)

   • Pre-populate cache with common queries
   • Reduce cold-start latency

3. Adaptive Cache Size (v1.3.0)

   • Dynamically adjust cache size based on hit rate
   • Auto-tune for optimal performance/memory trade-off

4. Cache Analytics (v1.4.0)

   • Track cache performance over time
   • Generate reports (hit rate trends, eviction patterns)

Research Areas

• Semantic Similarity Caching: Cache near-duplicate content
• Compressed Cache: Store embeddings in compressed format
• Distributed Cache: Share cache across multiple processes/machines

---

References

Internal Documentation

• Implementation: .claude/hooks/devstream/utils/ollama_client.py
• Unit Tests: tests/unit/test_ollama_cache.py
• Benchmark: tests/benchmarks/benchmark_ollama_cache.py
• Configuration: .env.devstream

External Resources

• LRU Cache Pattern
• SHA256 Hashing
• Python OrderedDict
• Python Threading

---

Document Version: 1.0.0 Last Updated: 2025-10-01 Status: Production Ready ✅ Maintainer: DevStream Team