Lecture III - Caching

Programming: Everyday Decision-Making Algorithms

Author
Affiliation

Dr. Nils Roemer

Kühne Logistics University Hamburg - Winter 2025

Learning Objectives

By the end of this lecture, you will be able to:

  • Explain the fundamental concepts of caching and its importance
  • Compare different cache replacement strategies
  • Identify caching principles in everyday life
  • Apply caching concepts to personal productivity
  • Understand the relationship between caching and attention management

Introduction

A Quick Question First

Question: How many of you have a messy desktop right now?

Raise your hand!

Today’s lecture will explain why that matters more than you think…

Starting with an everyday decision

  • We have a problem: Our cupboard.
  • It’s time to put things in order.

Shelf unit with empty upper shelves and a heap of clothes spilling out of the bottom compartment: plenty of storage, but no order.

Question: What could we do?

  • Better organization
  • Clearing out things we no longer need
  • Now we have two problems:
    • Storing?
    • Clearing out?

Open cupboard with clothes neatly folded and sorted into many small compartments, the tidy counterpart to the heap from before.

Two Approaches to Storage

  • Better Organization:
    • Subdivide storage
    • Efficient sorting
  • More Space:
    • Increase capacity
  • Question: Which approach is better?

Cupboard subdivided into many small compartments, each holding a neat stack of folded clothes: better organization within the same space.

Even best organization has limits

  • Organization helps, but takes time.
  • More space helps, but has limits.
  • Every storage has a finite capacity.

Neatly organized cupboard in which every compartment is already filled with folded clothes: even a tidy storage has no room left at some point.

Question: What if storage is full?

Large, still empty walk-in wardrobe with long rows of shelves, drawers and clothes rails: far more capacity than a single cupboard.

We could increase the capacity

But…

  • Increasing capacity = costly
  • Trade-off: Size vs. Speed
  • Larger = slower to search
  • Sooner or later: Still fills up

Walk-in wardrobe with every shelf and rail full and clothes and shoes piled on the floor: the larger storage has filled up as well.

Question: What else faces this?

  • Cupboards
  • Computers
  • Email inbox
  • Smartphones
  • Warehouses
  • Our brain?!

Question: Impact of full storage?

  • Access speed drops
  • Processing time up
  • Performance down

Overfull walk-in wardrobe with packed shelves and heaps of clothes and shoes covering the floor, where finding a single item means a long search.

Clearing out

Why Clearing Out Matters

  • True for cupboards, computers, brains…
  • But what stays and what goes?

Learning from Computer Science

The evolution of computer memory

  • 1950s: Computer science faced the same problem
  • Processors got faster (Moore’s Law)
  • Memory demands grew
  • But memory speed couldn’t keep up

→ The Memory Wall

The Bottleneck

  • Modern CPUs: Billions of ops/second
  • Problem: Data isn’t available fast enough
  • Question: What’s the point of a fast CPU if it has to wait for slow memory?

→ Von Neumann Bottleneck

Cache

Cache: The Solution

A hierarchical memory system

  • L1 Cache: Tiny but ultra-fast (64-256 KB)
  • L2/L3 Cache: Larger, still fast (MB range)
  • RAM: Main workspace (8-32 GB)
  • Storage: Huge but slow (256 GB - 2 TB)

Like a library…

The Library Principle

  • Library storage (5 million books, Mass Storage)
  • Subject locations (100K books, RAM)
  • Your desk (5 borrowed books, L2)
  • Short-term memory (current page, L1)

Closer = Faster = Smaller

Tidy desk with an open laptop and a small stack of books within reach: only the few things needed right now are kept closest, like data in a fast cache.

The Trade-off: Size and Speed

Memory hierarchy as a pyramid: registers at the tip hold about 1 KB and respond in 0.3 ns, followed by L1 to L4 cache, main memory and SSD down to the disk drive at the base with several terabytes but about 3 ms access time. Each level down is bigger, each level up is faster.

Registers are 10 million times faster than the hard drive!

Why can’t we just make everything as fast as L1 cache?

Why the Trade-off?

Why not make everything ultra-fast?

  • Physical limits: Larger caches sit further from CPU, signals take longer to travel, and dense fast memory generates extreme heat that must be dissipated
  • Economic limits: SRAM costs $50,000/GB vs. HDDs at $0.02/GB
  • Solution: Multiple cache levels, each optimized for different needs

The nessecity of clearing out

  • L1 and L2 cache only contain most necessary data.
  • The same should apply to your desk.
  • Therefore, both must be cleared regularly.

Desk buried under towering stacks of books that surround the laptop and cover the floor: a workspace that was never cleared out.

Clearing out Strategies

Question: Eviction Strategies?

  • Random
  • First-In, First-Out (FIFO)
  • Least Frequently Used (LFU)
  • Least Recently Used (LRU)

How to clear up?

  • Optimal: Clairvoyance
    • Keep what you’ll need
    • Remove what you won’t
  • Question: What’s the problem?

→ Requires knowledge of the future!

Realistic Strategies

  • Least Recently Used (LRU) is the dominant strategy.
  • Evicts the least recently accessed item from the cache when space is needed.
  • Leads to much better performance on average than, for example, random eviction.
  • Question: Why do you think least recently used is the better strategy?

Why LRU Works: Temporal Locality

  • Temporal Locality: Recent use → Likely need again soon
  • Examples:
    • Books on your desk
    • Apps opened today
    • People texted this morning
  • Performance: 80-90% hit rate (vs. 50-60% random)

Recent past predicts near future

Managerial and personal insights:

  • Let go of unused things → LRU principle
  • Keep things where used → Spatial locality
  • Result: Significant productivity increase

Marie Kondo = LRU for physical objects!

Spatial Locality

Question: Can you think of examples where spatial locality is applied in your daily life?

Bedroom with worn clothes left on the bed and on the floor right beside it: things kept exactly where they are used.

Mathematically optimal

Productivity

Our Brains are Caches

  • We’ve learned how computers manage limited cache space
  • But why does this matter for humans?
  • Your brain works remarkably similar to a computer cache:
    • Limited capacity for active information
    • Fast access to recently used information
    • Must constantly decide what to keep and what to forget

Cache Vulnerabilities

  • Denial-of-Service (DoS) attacks exploit cache limitations:
    • Cache Flooding: Overload with excessive requests
    • Cache Poisoning: Insert malicious data to evict important information
  • These attacks overload a system with excessive requests or data.
  • Causing it to slow down or crash.
  • The system is forced to evict important data.

Productivity Killers

  • Overload (too much information, cache capacity exceeded)
  • Exhaustion (too long without “cache clearing”)
  • Context switching (interruption of “flow”, ~23 minutes to get back on track)
  • Distraction (Cache Flooding: constant notifications, social media)
  • Misinformation (Cache Poisoning: fake news, misleading information)

The Attention Crisis

  • Technology & Social Media: Designed to capture attention (Cache Flooding)
  • Constant Interruptions: Notifications = Forced cache evictions
  • Result: Reduced creativity, critical thinking, and productivity

Reflection: How many times did you check your phone during this lecture?

Manage Your Mental Cache

  • Limit screen time and practice “monotasking” (Reduce cache thrashing).
  • Prioritize sleep, nutrition, and mindful habits (Maintain cache performance).
  • Create protected time for deep work (Prevent cache flooding).
  • Be selective about what enters your attention (Smart cache management).

Mitigation

Mitigation

  • Distraction can hardly be avoided in today’s world but can be mitigated.
  • This is particularly important for managers.
  • This lecture is designed to raise your awareness of what you can do to keep your brain working efficiently.
  • Want to dive deeper? Read “Stolen Focus” by Johann Hari

Awareness: Levels of Attention

  • Spotlight – Immediate goals – Focus (Your L1 Cache: What are you working on RIGHT NOW?)
  • Starlight – Medium-term goals – Wishes (Your L2 Cache: What matters this week/month?)
  • Daylight – Long-term goals – Values (Your Storage: What defines your life direction?)

Practical Strategies

  • Prioritization: What deserves to be in your mental cache?
  • Structure (Schedule): Time-boxing prevents cache overflow
  • Breaks: Regular “cache clearing” prevents exhaustion
  • Enable flow: Dedicated workspace, manage notifications, clear communication
  • Meditation & exercise: Maintenance routines for optimal cache performance

Cache Hierarchy for Workspace

  • Hot Cache (On your desk): Only items for current tasks (today)
  • Warm Cache (Nearby shelf/drawer): This week’s projects (this week)
  • Cold Storage (Archive/closet): Everything else

→ The key principle: If you haven’t used it recently, move it away!

Self-Assessment

Question: Right now:

  1. How many browser tabs do you have open? ___
  2. How many notifications did you receive in the last hour? ___
  3. How many unread emails in your inbox? ___

Do you see any risks of cache flooding?

Key Takeaways

Key Takeaways: The Concepts

  • Caching is universal: It applies to computers, organizations, and human cognition
  • The fundamental trade-off: Fast storage is small, large storage is slow
  • LRU works because of temporal locality: Recent past predicts near future

Key Takeaways: Your Life

  • Your brain is a cache: Limited capacity, vulnerable to flooding and poisoning
  • Protect your attention: It’s your scarcest and most valuable resource
  • Apply caching principles to life: Keep what you use, remove what you don’t

Literature

Interesting literature to start

  • Christian, B., & Griffiths, T. (2016). Algorithms to live by: the computer science of human decisions. First international edition. New York, Henry Holt and Company.1
  • Ferguson, T.S. (1989) ‘Who solved the secretary problem?’, Statistical Science, 4(3). doi:10.1214/ss/1177012493.

Books on Programming

  • Downey, A. B. (2024). Think Python: How to think like a computer scientist (Third edition). O’Reilly. Here
  • Elter, S. (2021). Schrödinger programmiert Python: Das etwas andere Fachbuch (1. Auflage). Rheinwerk Verlag.

Think Python is a great book to start with. It’s available online for free. Schrödinger Programmiert Python is a great alternative for German students, as it is a very playful introduction to programming with lots of examples.

More Literature

For more interesting literature, take a look at the literature list of this course.

Footnotes

  1. The main inspiration for this lecture. Nils and I have read it and discussed it in depth, always wanting to translate it into a course.↩︎