CPU at 92% is a symptom. The decision an operator needs is whether the cause is a plan regression, connection storm, vacuum debt, checkpoint pressure, storage latency or legitimate demand—and what can be changed safely. Effective AWS database observability therefore joins four layers: application experience, PostgreSQL workload, database internals and cloud infrastructure.
Diagnosis chain: impact → workload → mechanism → resource
Correlation needs aligned timestamps, stable identifiers and history. A dashboard that cannot traverse this chain is visualization, not diagnosis.
Define service objectives before alarms
Start with availability and latency SLOs for critical transactions, then define warning and critical thresholds based on an error budget. Infrastructure alarms without user impact create noise; application alarms without database context slow diagnosis. Every page should identify an owner, a runbook, expected evidence and an escalation path.
Translate business promises into measurable SLIs
For an availability SLI, define a good event precisely: for example, a checkout transaction that commits successfully within 800 ms and is not a duplicate. SLI = good events / valid events. Exclude only conditions named in the contract; silently removing timeouts makes the SLI meaningless. Use latency distributions for important journeys because averages hide the tail that users experience.
Burn-rate alerts are often more actionable than static thresholds. A fast burn over a short window catches severe outages; a slower burn over a long window catches persistent degradation. Database saturation becomes paging evidence when it consumes the service error budget or approaches a known failure boundary.
Collect the four layers
| Layer | Essential signals | Question answered |
|---|---|---|
| Application | Request rate, errors, p95/p99, traces, pool wait | Are users affected? |
| Queries | Fingerprint, calls, total/mean time, rows, temp I/O, plans | Which workload changed? |
| PostgreSQL | Sessions, waits, locks, WAL, checkpoints, vacuum, bloat, replication | Which engine mechanism is limiting progress? |
| AWS | CPU, memory, IOPS, throughput, queue, latency, storage, events | Which resource or platform event contributes? |
CloudWatch supplies RDS metrics and events. Enhanced Monitoring adds operating-system detail. AWS database performance tooling helps analyze database load and waits. Inside PostgreSQL, enable and size pg_stat_statements deliberately, turn on track_io_timing after measuring overhead, and export logs with a retention policy. Sanitize SQL samples and bind values according to data-classification requirements.
Use identifiers that survive every layer
Carry a trace/request ID through the application and logs. Normalize SQL into a stable fingerprint so literals do not create millions of “different” queries. Record database/instance identity, role, application name, deployment version and plan hash/version. Align clocks and collection intervals. Without these keys, operators can see simultaneous spikes but cannot prove they describe the same causal event.
Control observability overhead and cardinality
Collection competes with production. Prefer deltas from cumulative views, cap SQL sample size, bound retention and avoid labels such as raw query text, user ID or request ID in time-series dimensions. High-cardinality detail belongs in logs/traces with controlled indexing, not in every metric label. Test the CPU, I/O, storage and network cost of collection during peak load and define degraded collection behavior.
Data handling: SQL text, bind values, application names and error messages may contain personal or confidential data. Apply redaction, encryption, access separation and retention before exporting them from the database boundary.
Alert on risk, not every movement
- Availability: failed connections, unhealthy writer, failover/reboot events and backup failure.
- Saturation: sustained CPU, low memory, connection budget, storage headroom, queue and latency.
- Workload: p95 query regression, new high-load fingerprint, blocked sessions and long transactions.
- PostgreSQL health: vacuum debt, transaction ID risk, temp-file spikes, requested checkpoints and WAL retention.
- Replication: replay lag, slot backlog, inactive consumers and recovery conflicts.
Use multi-window thresholds: a fast alarm catches severe failures; a slower one confirms sustained degradation. Route warnings to a queue and pages only to a human who can act. Deduplicate downstream symptoms when one database event explains them.
| Bad alert | Actionable alert |
|---|---|
| CPU > 80% once | Sustained CPU saturation + growing runnable/query queue + SLO burn |
| Connections > 100 | Usable connection budget < 15% and pool wait rising for 10 minutes |
| Replica lag > 30 s | Lag is increasing, recovery/read SLA at risk, cause dimension attached |
| Long query found | New/regressed fingerprint dominates database time and affects a critical journey |
Preserve incident evidence
At incident start, capture active sessions, waits, blocking tree, top query deltas, plan versions, database parameters, deployment events and relevant AWS metrics. Cumulative counters need deltas; snapshots need timestamps. Avoid running heavy diagnostic queries repeatedly on an already saturated primary.
Follow a low-risk triage order
- Confirm user impact and blast radius: one endpoint, one tenant, one database or the fleet.
- Check changes: deploy, parameter, failover, maintenance, secret rotation or traffic shift.
- Classify the dominant wait/resource and identify blockers before killing sessions.
- Compare current query mix and plans with the last healthy baseline.
- Choose the smallest reversible mitigation, observe recovery and preserve evidence.
Restarting the database can erase the state needed for root cause and amplify an outage through reconnection storms. Cancellation, traffic shaping, disabling a bad feature or adding temporary capacity may be safer, but each requires a pre-approved owner and rollback.
Turn every incident into a detection improvement
A post-incident review should record user impact, timeline, trigger, contributing conditions, why controls did not prevent it, why monitoring did or did not detect it, and measurable actions. Validate alarms with game days: fail a connection, fill a test volume threshold, block a transaction, rotate a secret and execute a controlled failover.
A useful dashboard tells a story: user latency rose, database load shifted to one fingerprint, its plan changed, physical reads increased and storage latency followed. Five unrelated green/red panels cannot provide that causal chain.
How PG Monitoring turns AWS telemetry into PostgreSQL diagnosis
PG Monitoring preserves the database context that generic infrastructure metrics lack: normalized query history, plan changes, waits, blocking relationships, vacuum/table health, replication, incidents and capacity trends. It lets an operator move from an AWS symptom to the exact workload change and compare it with a healthy baseline. Automated anomaly and regression detection reduces the delay between “something moved” and “this query or engine mechanism caused the risk.”
Practical outcome: fewer isolated alerts, faster root cause and an evidence trail for post-incident improvement. Connect PG Monitoring to your AWS PostgreSQL environment and turn CloudWatch signals into a repeatable database-diagnosis workflow.