How would you design a rate limiter for an API, and how do token bucket and sliding window actually differ?
A rate limiter caps how many requests a client can make in a given time window, and the two most common algorithms handle bursts differently. Token bucket gives each client a bucket that refills at a steady rate up to a max capacity, a request consumes a token, and if the bucket’s empty the request is rejected, letting clients burst up to the bucket’s full capacity as long as they’ve been idle long enough to refill it. Sliding window counts actual requests in a rolling time period (say, the last 60 seconds, recalculated continuously) rather than resetting at fixed boundaries, giving a smoother, more accurate limit than a naive fixed window.
Why a naive fixed window is broken
A fixed window (e.g., “100 requests per minute, resetting every clock minute”) lets a client send 100 requests in the last second of one window and another 100 in the first second of the next, 200 requests in two real seconds, technically within both individual limits but nowhere near the intended rate. Sliding window fixes this by continuously considering a moving time range instead of resetting at a fixed boundary.
Best practice
Store rate-limit counters in a fast shared store (Redis) so the limiter works correctly across multiple API servers, not per-instance, and use a sliding-window-counter approximation (weighting the previous window’s count) for good accuracy without the memory cost of storing every individual request timestamp.
Edge case interviewers probe for
How do you rate-limit fairly across many distributed servers without a single Redis instance becoming a bottleneck or single point of failure? This pushes toward discussing a Redis cluster with consistent hashing per client key, or accepting slightly relaxed accuracy in exchange for a local, per-server approximate limiter that syncs periodically.
Common mistake
Implementing a fixed window limiter and assuming it’s equivalent to a sliding window, missing the boundary-burst problem entirely until it shows up as unexpected traffic spikes at legitimate rate-limit-window edges.
What the interviewer is checking
Whether you know concrete algorithm tradeoffs (token bucket’s burst tolerance versus sliding window’s smoothing) rather than a single generic “we’d rate limit it,” and whether you think about distributed correctness, not just single-server logic.
Token bucket is like a movie theater giving you a stack of tickets that slowly refills over time, up to a max stack size, you can burn through several at once if you’ve saved some up, but once they’re gone you have to wait for more to trickle in. Sliding window is like a bouncer who, at every single moment, looks back exactly one hour and counts how many people actually walked in during that rolling hour, rather than just resetting the count the moment the clock hits a new hour.
The fixed-window version, resetting cleanly on the hour, has an exploit: you could sneak in 50 people in the last minute before the reset and another 50 in the first minute after, 100 people in two minutes even though the limit is “50 per hour.” The sliding window bouncer never gets fooled this way, since they’re always looking at a true rolling hour, not a reset-able fixed clock boundary.
Why interviewers ask this
Rate limiting is a real, common system design component, and this question checks whether you know actual named algorithms and their tradeoffs rather than a vague “count requests and block.”
What a strong answer signals
Naming the fixed-window boundary-burst flaw specifically, and knowing how to make the limiter work correctly across multiple distributed servers, not just a single process.
Common follow-ups
- How would you rate-limit per-user versus per-IP, and why might you need both?
- How does the leaky bucket algorithm differ from token bucket?
- How would you communicate the limit to clients (headers, retry-after)?
Advanced variation
“How would you rate limit fairly when legitimate clients share an IP behind NAT?” expects discussing layered limits (per-API-key alongside per-IP) so one heavy client behind a shared IP doesn’t throttle everyone else on that IP.
An API used a simple fixed-window counter resetting every minute, and a partner integration kept triggering false “you’re fine” checks right before hammering the API at the window boundary, effectively doubling their real request rate for a couple seconds every minute. Switching to a Redis-backed sliding window counter, weighting the previous minute’s count proportionally to how much of it overlapped the current rolling window, closed the exploit without needing to store every individual request timestamp.
# Token bucket: refills steadily, allows bursts up to capacity
class TokenBucket:
def __init__(self, capacity, refill_rate_per_sec):
self.capacity = capacity
self.tokens = capacity
self.refill_rate = refill_rate_per_sec
self.last_check = time.time()
def allow(self):
now = time.time()
elapsed = now - self.last_check
self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
self.last_check = now
if self.tokens >= 1:
self.tokens -= 1
return True
return False- 1Token bucket allows controlled bursts up to a refillable capacity.
- 2A naive fixed window lets clients double their effective rate right at the window boundary.
- 3Sliding window counts requests in a continuously moving time range, closing the boundary exploit.
- 4Rate limit state must live in a shared store across servers, not per-instance memory.
- 5Layering per-user and per-IP limits prevents one heavy client from throttling others sharing an IP.