Helena Demo

Worker OOM every ~3 hours. Traced to a growing Map<userId, PendingBatch> that was never pruned when batches flushed. Added TTL. Deployed. Worker stable for 24h+ afterward.

PagerDuty webhook endpoint TLS cert expired without renewal. Cert renewed via ACME. Added a monitoring rule for cert expiry <14 days.

CustomerReport query holding locks on transactions table. Killed the session. Long-term: moved reporting to read replica.

Notification test Labels: {"alertname":"TestAlert","instance":"Grafana"}

Alert on api-gateway. Cause traced to a bad deploy of auth-service that returned 500 on missing header instead of 401. Rolled back. Followed up with PR to fix the header path.

ECR credentials rotated last night but the k8s pull secret was not updated. Fixed by refreshing the imagePullSecret. Deployment rolled forward.

Repo: acme/orders-processor Author: sarah Files changed: 2 Security patch for CVE-2024-XXXX. No behavior changes.

Pool at 500/500. Long-running analytics query holding connections. Killed the offending session, added connection timeout. Set alerts for pool > 80%.