Notes from Terry Crowley on Caching

This is a summary of Cache Stories by Terry Crowley, where he explains how to cache effectively in complex, long-lived systems operating at scale:

• Don’t cache information redundantly at multiple layers of the system. If you do, the lower layer should allow its cache to be bypassed, similar to how databases tell the kernel to not cache (since the database does it better).

• Caches should be instrumented fully so that their cost and benefit are understood. When the Windows team did this, they found that some caches had a 0% hit rate. Not low, but 0%. These caches did make sense when they were introduced, but later, a higher layer introduced its own cache, and cache hits were satisfied from the higher layer, resulting in the lower layer having a 0% hit rate. So when you implement a cache, fully instrument it so that if a later refactoring renders it ineffective, you’ll know.

• When an engineer gets assigned a performance bug, he sometimes adds a cache to fix it. But in the process, he slows down the overall system across other use cases.

• Space is sometimes at a tradeoff with time. In other cases, reducing memory use improves locality of reference and thus speed.