Intermittent timeouts under load almost always mean something is queueing, not crashing outright. Start by separating the problem into three layers: is the client giving up too early (timeout configured too aggressively), is the network dropping or delaying packets between hops, or is the server itself slow to respond because a resource is saturated. Each layer has a distinct signature, and guessing without data wastes the incident.
Where to look first
Check connection pool exhaustion before anything else, it’s the most common cause in practice: if a service’s outbound connection pool to a database or downstream API is capped and every connection is busy, new requests queue and eventually time out, even though the downstream service itself is healthy. Look at queue depth and thread pool utilization on the service under load, not just CPU, since a service can time out while CPU sits at 40% if it’s blocked waiting on I/O or locks.
Best practice
Correlate timeouts with a specific percentile, not just “some requests fail.” If p50 latency is fine but p99 spikes under load, that points to contention (a shared resource, a lock, a connection pool) rather than a uniformly slow dependency, which would push the whole distribution up, not just the tail.
Edge case interviewers probe for
Ask what happens when the timeout is set shorter than the actual retry-with-backoff behavior of a client library. A client can appear to “intermittently time out” purely because of its own retry policy interacting badly with a slow but eventually-successful downstream call, the service was never actually failing, the client just gave up too early and logged it as a failure.
Common mistake
Jumping straight to “scale up the servers” before confirming where the bottleneck actually is. Adding capacity to a service that’s fine, while the real bottleneck is a shared connection pool or a downstream dependency, doesn’t fix anything and burns an incident’s worth of time.
What the interviewer is checking
Whether you have a systematic debugging method (client vs. network vs. server, percentiles not averages, resource saturation not just CPU) instead of pattern-matching to “just add more servers.”
Imagine a coffee shop with three baristas and a line that only gets long during the morning rush. If customers start complaining about waiting forever, you wouldn’t immediately hire five more baristas without first checking why the line is slow: maybe the espresso machine only has two nozzles even though there are three baristas (a shared resource, like a connection pool), maybe orders are getting written down wrong and redone (retries), or maybe there really are just too many customers for the staff you have (real capacity problem).
“Sometimes it works, sometimes it doesn’t” almost always means something has a limited number of slots, like the two nozzles, and it only becomes a problem once enough people show up at once to actually hit that limit. Find the two-nozzle espresso machine before you hire more baristas.
Why interviewers ask this
Intermittent, load-dependent failures are one of the most common real incidents, and they filter for candidates who debug systematically instead of guessing.
What a strong answer signals
That you check saturation and percentiles before reaching for “scale it up,” and that you know shared resources (pools, locks, queues) are usually the actual bottleneck.
Common follow-ups
- How would you find which specific resource is saturated?
- How do retries make this worse if not designed carefully?
- What’s the difference between a timeout and a circuit breaker?
Advanced variation
“Design a load test that would have caught this before production,” which expects discussion of realistic traffic shapes, not just constant-rate load, since bursty traffic reveals pool exhaustion that steady load never triggers.
A checkout service saw p99 latency spike to 8 seconds during flash sales while p50 stayed under 200ms. Investigation found the database connection pool was capped at 20 connections, fine for normal traffic, but during a sale, request volume tripled and requests queued waiting for a free connection, some queueing long enough to hit the client’s 5-second timeout. Raising the pool size helped short-term, but the real fix was adding a request queue with backpressure so the service degraded gracefully (returning a fast “try again” response) instead of holding connections open until they timed out.
- 1Separate client, network, and server causes before guessing at a fix.
- 2Check saturation of shared resources like connection pools before assuming you need more servers.
- 3Compare p50 to p99 latency; a widening tail under load points to contention, not uniform slowness.
- 4Client-side retry and timeout settings can manufacture apparent failures on a healthy backend.
- 5Prefer graceful degradation (backpressure, fast failure) over holding connections open until they time out.