How Do Streaming Platforms Serve Video in Under 50ms?

Why CDN Edge Caching Matters

When a user in Tokyo requests an image hosted on a server in Virginia, the round-trip takes 150–200ms just for the network latency. Multiply that by the 50+ assets on a typical web page, and page load times spike past 3 seconds — the threshold where 53% of mobile users abandon the page.

Streaming platforms face this problem at extreme scale. A single popular video generates millions of concurrent requests from every continent. Without edge caching, every request travels back to the origin server, saturating its bandwidth and increasing latency for all users. At Netflix-scale traffic (15% of global internet bandwidth during peak hours), this would be physically impossible without distributed caching at the edge.

CDN edge caching solves this by placing cache servers at Points of Presence (PoPs) close to users worldwide. But configuring TTLs, managing cache invalidation, and surviving origin failures requires careful engineering.

Practical note CDN caching is wonderful until invalidation becomes your deploy mechanism. Treat purges like production traffic events, because that is what they are.

The mental model: The Cache Hierarchy

CDN edge caching uses a multi-tier architecture to balance latency and origin protection:

1. Edge PoP (Point of Presence): A cache server geographically close to the user. On a cache HIT, content is served in ~5ms. Each PoP maintains a local LRU cache with configurable capacity and TTL.

2. Origin Shield: A mid-tier shared cache that sits between edge PoPs and the origin server. When an edge PoP has a cache MISS, the request goes to the origin shield (~25ms) instead of directly to the origin. This collapses redundant requests from multiple PoPs.

3. Origin Server: The authoritative source of content. Only reached when both the edge and shield miss (~120ms). The origin is the most expensive and fragile component in the chain.

Content popularity follows a Zipf distribution — a few items get most of the traffic. This skew is what makes caching effective: the top 1% of content often serves 80%+ of requests.

Decisions that matter

Parameter	Range	Low Setting Impact	High Setting Impact
TTL	5–300s	Fresh data but frequent origin hits, low cache hit ratio	High hit ratio but stale content risk, slow updates
Cache capacity per PoP	5–200 items	Frequent LRU evictions, low hit ratio	High memory cost, better hit ratio for long-tail content
Unique content items	10–500	Easy to cache everything, high hit ratio	Long-tail content causes eviction pressure
Popularity skew (Zipf)	0.1–3.0	Even traffic distribution, low cache efficiency	Concentrated traffic on few items, high hit ratio
Stale-While-Revalidate	On/Off	Expired content returns errors until refreshed	Serves stale content instantly while refreshing in background

Where it breaks: Thundering Herd After Cache Purge

The failure mode: A content update triggers a global cache purge. All edge PoPs and the origin shield simultaneously invalidate their entries. The next wave of user requests — potentially thousands per second — all miss cache and hit the origin server simultaneously. The origin CPU spikes to 100%, connections exhaust, and the server crashes.

What you will notice:

• Origin server CPU and bandwidth spike sharply after a purge command
• Cache hit ratio drops to 0% across all PoPs simultaneously
• Origin response times climb from 120ms to 5000ms+, then timeout
• CDN returns 502/503 errors as the origin becomes unreachable

Warning A global cache purge is the most dangerous operation in a CDN. It instantly removes all cached content and redirects the full traffic load to the origin. Always prefer staggered purge or soft purge instead.

What I would do:

• Staggered purge: Purge PoPs in rolling waves (region by region) instead of globally at once
• Stale-While-Revalidate (SWR): Enable SWR so expired content is served immediately while the edge fetches fresh content from origin in the background. This prevents the thundering herd entirely.
• Origin shield: Ensure the origin shield absorbs the first wave of misses. Only one request per content item reaches the origin; subsequent edge misses are served from the shield.
• Soft purge: Instead of deleting cached content, mark it as stale. SWR will serve stale content and update asynchronously.

Where it breaks: Cache Thrashing From Long-Tail Content

The failure mode: The content catalog is much larger than the edge cache capacity. Low-popularity items constantly evict high-value cached content due to LRU eviction. Even though popular items should stay cached, the flood of unique requests pushes them out.

What you will notice:

• Cache hit ratio stays below 60% despite high traffic volume
• evicted_keys counter increases rapidly on edge nodes
• Origin bandwidth usage remains high even during steady-state traffic
• Popular content shows intermittent cache misses (shouldn’t happen with proper sizing)

What I would do:

• Increase cache capacity: Size edge caches to hold at least the top 80% of requested content (based on Zipf analysis)
• Admission control: Only cache items that have been requested at least 2 times. First requests go to origin; subsequent requests serve from cache. This prevents one-hit wonders from polluting the cache.
• Segmented caching: Use separate cache zones for static assets (high TTL) and dynamic content (low TTL) to prevent cross-eviction.

Where it breaks: Origin Server Failure During TTL Expiry Wave

The failure mode: The origin server goes down (hardware failure, deployment crash, or database outage). Meanwhile, edge cache TTLs begin expiring. Without SWR enabled, expired entries are treated as invalid. Edge PoPs have no way to refresh content and start returning errors to users.

What you will notice:

• Error rate gradually increases as TTLs expire across PoPs (not instant — follows TTL expiration curve)
• Some regions still serve content (their TTLs haven’t expired yet) while others return 502
• Origin health checks fail but CDN continues routing to it
• Monitoring shows a declining cache hit ratio that correlates with time since origin failure

Hint The interactive demo’s “Origin Failure” preset demonstrates this scenario. Enable Stale-While-Revalidate before triggering the failure to see how stale content keeps users happy during outages.

What I would do:

• Enable Stale-While-Revalidate: SWR serves expired content when the origin is unreachable. Users get slightly stale data instead of errors.
• Origin failover: Configure a secondary origin server. CDN health checks detect the primary failure and route refresh requests to the backup.
• Long TTLs for static assets: Images, CSS, JS should have 24-hour+ TTLs with versioned filenames (app.v2.css). Content updates use new URLs instead of purging old ones.
• Circuit breaker on origin: If origin errors exceed a threshold, stop sending requests and serve stale content exclusively.

The tradeoff

Approach	Pros	Cons
Short TTL (15s)	Content always fresh, fast updates	High origin load, frequent revalidation traffic
Long TTL (6h)	Excellent hit ratio, minimal origin load	Stale content persists, purge needed for updates
Stale-While-Revalidate	Zero-latency refreshes, origin protection	Clients may briefly see outdated content
Origin Shield	Collapses redundant origin requests from edges	Extra hop latency on shield miss, added infrastructure cost
Cache Purge	Immediate global content update	Thundering herd risk, temporary hit ratio drop
Versioned URLs	No purge needed, instant switch	Requires build pipeline changes, more CDN storage

Hint For most web applications, start with a long TTL (1–6 hours) for static assets using versioned URLs, and a short TTL (15–60 seconds) for dynamic API responses with SWR enabled. This gives you the best of both worlds.

A practical debugging checklist

1. Check hit ratio per PoP: CDN dashboards (Cloudflare Analytics, CloudFront reports) show per-region hit/miss ratios
2. Inspect response headers: X-Cache: HIT or X-Cache: MISS tells you if the edge served the content
3. Check TTL remaining: Cache-Control: max-age=300, s-maxage=600 headers reveal TTL configuration
4. Monitor origin bandwidth: A sustained increase in origin bandwidth indicates cache inefficiency
5. Test cache warming: curl -I https://cdn.example.com/asset.js from different regions to verify edge caching
6. Verify purge propagation: After a purge, request from each region to confirm all PoPs received the invalidation

Try it yourself

Understanding CDN cache behavior requires watching requests flow through the hierarchy under different conditions. Try the interactive CDN Edge Caching simulator to experiment with:

• Normal Traffic preset to see balanced cache hit ratios across regions
• Traffic Spike to watch a single PoP warm its cache while others stay stable
• Cache Purge to observe the thundering herd effect on the origin server
• Origin Failure with SWR enabled to see how stale content keeps users happy during outages
• TTL Expiry Wave with 15-second TTLs to see frequent origin surges