Rate Limiting Simulator

Mental Model

Rate limiting is a foundational defense mechanism used to protect APIs and services from resource exhaustion, brute force attacks, and cascading failures under heavy traffic spikes.

The Token Bucket algorithm is one of the most widely used rate limiting behaviors because it is simple and handles bursts naturally. Think of the rate limiter as a bucket that holds a maximum number of tokens. Every incoming request must consume a token to be accepted. If the bucket is empty, the request is rejected immediately. Meanwhile, tokens are refilled at a constant rate over time up to the bucket’s capacity.

graph TD
    Client[Client Traffic] -->|Request| Limiter{Tokens Available?}
    Limiter -->|Yes: Consume Token| Backend[Forward to Backend]
    Limiter -->|No: Rejected| Drop[429 Too Many Requests]
    Refiller[Refill Scheduler] -->|Refill Tokens at Constant Rate| Bucket[Token Bucket]

---

Components and Why They Exist

Client Traffic Generator

Simulates realistic incoming API request loads to stress-test the rate limiter. Dispatches requests at configurable rates, mimicking normal usage, sudden bursts, or sustained DDoS-like overloads. In the demo, observe how request bubbles flow into the system and how the request rate slider dynamically increases input pressure.

API Rate Limiter (Token Bucket)

Acts as the gatekeeper protecting upstream application servers from overload. Inspects the token bucket, decrements the token count for each incoming request, and returns an error (HTTP 429) if the bucket is empty. In the demo, note the token count dropping for each accepted request and the limiter flashing red when rejecting requests.

Token Refill Scheduler

Restores the rate limiter’s capacity over time. Incrementally adds tokens back to the bucket at a constant rate (e.g. 5 tokens/second) up to the maximum capacity. In the demo, watch the token levels rise during periods of low traffic and how the refill rate slider increases recovery speed.

Dropped Traffic Sink

Represents the blocked clients receiving rate limit exceptions. Collects and counts rejected requests that did not have a token available. In the demo, notice that follow the red request particles traveling to the dropped sink when the bucket is exhausted.

Metrics Dashboard

Provides real-time visibility into rate limiting efficacy and traffic health. Graphs accepted/sec, rejected/sec, drop rates, and bucket fill states. In the demo, watch the graphs transition during burst and overload events, noting the spike in drop rate.

---

Edge Cases and Misconceptions

Burst Capacity vs. Refill Rate

A common misconception is that setting a refill rate of 10 tokens/sec means the system will never process more than 10 requests in any single second. If the bucket capacity is set to 50, a sudden burst of 50 requests hitting at the exact same millisecond will all be accepted because the bucket was full. The bucket capacity dictates the maximum burst size, while the refill rate dictates the sustained limit.

Atomic Token Refills and Lock Contention

In a high-throughput system, calculating token refills with a background cron or interval thread is inefficient and creates CPU overhead. Instead, production systems calculate refills lazily. When a request arrives, the system calculates the time elapsed since the last request, multiplies it by the refill rate, adds that many tokens (capped at capacity), and updates the timestamp. This keeps updates $O(1)$ and avoids thread contention.

Refill Rate Granularity

If refill rates are calculated in coarse windows (e.g., refilling 60 tokens every minute instead of 1 token every second), a client could consume all 60 tokens in the first second of the minute and be blocked for the remaining 59 seconds. Refill calculations should use sub-second or millisecond precision to smooth out token distribution.