02:09 — a release goes out
Release r-0417 adds a /search route that opens more database connections per request. Verification passes: the probes hit it once and it responds.
02:14 — detection
p95 latency on the API climbs from 180 ms to 4.8 s. Error rate ticks up to 0.4%. The threshold fires within a minute.
02:14 — correlation
The agent pulls API logs for the window and finds 312 pool_timeout lines. It notes the release five minutes earlier and the new route in its diff.
02:14 — diagnosis
Pool size 10 is insufficient for the concurrency the new route creates. This is a configuration problem, not a code bug, so rolling back is not the best fix.
02:14 — repair
The agent raises the pool size to 30 and performs a rolling restart of the three API servers, one at a time, so capacity never drops to zero.
02:16 — verification
Probes pass, p95 returns to 176 ms, errors to zero. The incident is marked resolved only now.
Morning — report
You read: cause, action, before/after metrics, and a recommendation to make the pool size explicit in the repository. Total human time: the length of the report.