42. p99 latency for an API suddenly jumped from 120 ms to 2 s. How do you investigate?
HardDebugging & production
Show model answer
Stabilise first: check whether a recent deploy, config or feature-flag change lines up with the spike, and roll back if so. Then narrow the scope with metrics: all endpoints or one, all regions or one, all customers or a heavy tenant?
Follow the request path with traces to find where time is spent: the database (slow queries, lock contention, missing index after a data growth step), a downstream dependency (timeouts, retries piling up), the host (CPU saturation, garbage-collection pauses, thread-pool or connection-pool exhaustion), or the network. Compare a slow trace with a normal one.
Fix the cause, add an alert on the symptom, and write a short blameless review covering detection time and prevention.
What interviewers listen for
- Correlate with recent changes
- Narrow scope with metrics
- Use traces to find the slow hop
- Check pools, GC, locks and retries