The economics of semantic caching, measured

We ran an experiment across 12 production customers for 30 days: identical routing rules, semantic cache enabled on half the traffic.

The headline result: 41% median cost reduction across chat-heavy workloads, and 7% median cost reduction on structured extraction workloads.

Chat wins big because near-duplicate prompts are common in support and RAG applications. Extraction wins less because prompt-uniqueness is high and cache invalidation is aggressive.

More detail in the full write-up.