agenticstack-cache

agenticstack-cache

LLM response cache for AgenticStack: serve repeated prompts from a local store instead of paying for the same completion twice.

Why a provider wrapper, not a hook plugin

AgenticStack's hook points (BEFORE_LLM_CALL / AFTER_LLM_CALL) can observe and transform requests and responses, but they cannot skip the provider call — a filter chain always falls through to the actual generate(). A cache's whole job is to short-circuit that call, so the cache is implemented as a provider wrapper: CachedProvider is itself an LLMProvider that wraps any other provider and answers hits locally. It drops in anywhere a provider does, including Agent(provider=CachedProvider(inner)).

Install

pip install agenticstack-cache

Use

from agenticstack_cache import CachedProvider, InMemoryCache, SQLiteCache

# In-memory LRU (per-process)
provider = CachedProvider(inner, cache=InMemoryCache(max_entries=512))

# Persistent across processes (stdlib sqlite3)
provider = CachedProvider(inner, cache=SQLiteCache("cache.db", ttl=3600))

response = await provider.generate(messages)   # miss -> calls inner
response = await provider.generate(messages)   # hit  -> no inner call

provider.hits, provider.misses  # -> 1, 1
provider.stats                  # -> {"hits": 1, "misses": 1}

With an agent:

from agenticstack import Agent

agent = Agent(name="Helper", provider=CachedProvider(inner))

Cache key

sha256 over the request's semantic identity:

• inner provider's model name
• message (role, content) pairs
• tool names offered
• sorted extra kwargs

Anything that changes the request (different history, different temperature kwarg, different tools) is a different key.

What gets cached

Only pure-text responses. Responses carrying tool_calls are passed through uncached — tool-call turns are stateful (their IDs must pair with fresh tool-result messages), so replaying them would corrupt conversations. Cached entries store content, model, and finish_reason only; a hit reconstructs an LLMResponse with zero token usage (no tokens were spent) and raw_response["cached"] = True.

Backends

ttl=None (the default) means entries never expire. Expired entries read as misses and are purged on access.

Plugin form

CachePlugin participates in the plugin runtime (manifest, lifecycle) but registers no capabilities — a cache must wrap a specific provider instance, so the primary API is the explicit wrapper above.

Permissions

filesystem — the SQLite backend writes a local database file.