checkoutINC-2041
Checkout degraded — p99 latency and DB CPU
MitigateEscalateResolve
● In Progress
Sev2 · Moderate AUTO
P2 COMPUTED
Owner · Priya Nair
⚠ SLA · at risk · 51m
✦ AI Investigator14:04:00
SYMPTOM Checkout p99 above 2s; DB CPU 91%; 3.1% errors. 4 alerts correlated.
SERVICE checkout-api → rds/checkout-db
ROOT CAUSE Investigating…
FIX Not stated yet
●incident opened from 4 correlated alerts · sev2 assigned automatically · 14:04:22
Priya · 14:05
checkout is slow and the DB is hot — same thing?
PN
✦
Investigator · 14:05
Yes — same incident. Checking the run itself rather than guessing from history.
Root cause: the 1s idle timeout churns connections; the pool saturates under normal load and the DB burns CPU on handshakes.
⏺ Alert6 wants to run runbook/revert-pool-timeoutconfidence 0.92
$ kubectl set env deploy/checkout-api DB_POOL_IDLE_TIMEOUT=30s
$ kubectl rollout restart deploy/checkout-api
Yes, run it
Yes, and don't ask again for this runbook
No, tell me more
Priya · 14:06
why would 1s saturate the pool at normal traffic?
PN
✦
Investigator · 14:06
Median request gap on checkout-api is 1.4s, so most connections idle past 1s and get closed; every request re-opens one — ~40 handshakes/s on a pool sized for 6. Same pattern as INC-1877 in March.
●priya approved runbook/revert-pool-timeout · queued for execution · 14:07:02
›▍Ask the Investigator, or / for runbooks
/status/stakeholder-update/safe-to-resolve⏎ send · ⇧⏎ newline
Alerts 4grouped
checkout-api p99 · Datadogfiring
rds CPU 91% · CloudWatchfiring
queue lag · Grafanafiring
error rate 3.1% · New Relicfiring
Issues 1
ISS-887 · 4 alerts · promoted to this incident
Runbooks 1
Revert pool timeout · 3 steps · suggested
Timeline
14:04:12 alerts received
14:04:19 issue ISS-887
14:04:22 incident INC-2041
14:05:31 investigation complete
14:07:02 fix approved
Every query, result and approval lands on the timeline and the auto-drafted RCA.