Skip to main content

Incident Response Runbook

General incident detection, triage, and resolution playbook for Cruvero production environments. Covers Temporal, LLM provider, Postgres, and runaway workflow scenarios.

Trigger Conditions​

Open an incident when any of the following occur:

  • Health checks return unhealthy for critical components.
  • Error rate spikes above normal baseline.
  • Workflow backlog and latency rise rapidly.
  • Cost or token usage anomalies indicate runaway behavior.

Detection and Triage​

  1. Confirm alert and incident scope (tenants, regions, components).
  2. Pull health snapshots:
    • curl -fsS http://<ui>/api/health
    • curl -fsS http://<ui>/api/health/detail
  3. Classify primary failing component:
    • Temporal
    • LLM provider/failover
    • Postgres
    • Runaway workflow/agent
  4. Freeze high-risk operations if needed (bulk writes/imports).

Component Procedures​

Temporal Down​

  1. Check Temporal frontend/matching/history status.
  2. Confirm network path from worker namespace to Temporal.
  3. Restart worker pods only after Temporal health is restored.
  4. Verify workers reconnect and queue depth begins to drain.

Verification:

  • readyz on workers succeeds.
  • Workflow schedule-to-start latency returns toward SLO.

LLM Provider Down​

  1. Confirm provider errors (429/503/timeouts) from logs/metrics.
  2. Verify failover chain switched providers.
  3. Check failover audit/metric events for expected provider transition.
  4. If no failover occurred, temporarily force provider switch via config and restart workers.

Verification:

  • New requests succeed via secondary provider.
  • /api/health/detail shows degraded primary, healthy active provider.

Postgres Down​

  1. Check primary DB health and replication state.
  2. Trigger managed failover or Patroni promotion procedure.
  3. Update connection endpoint if required.
  4. Restart affected workloads after DB write availability recovers.

Verification:

  • SELECT 1 and critical table read/write checks pass.
  • No sustained increase in DB connection failures.

Runaway Agent / Workflow​

  1. Identify high-cost or looping workflow IDs.
  2. Stop offending run(s):
    • go run ./cmd/control --workflow-id <id> --action pause
    • or terminate via Temporal CLI/UI policy.
  3. Inspect quota usage and audit trail for runaway cause.
  4. Apply tenant policy constraints (tool/model/quotas) before resuming traffic.

Verification:

  • Cost growth normalizes.
  • Queue depth and token usage stabilize.

Post-Incident Actions​

  1. Export relevant audit events for incident window.
  2. Complete timeline and root cause analysis (RCA).
  3. Define preventive controls (alerts, guardrails, config defaults).
  4. Track follow-up items with owners and due dates.

Escalation Path​

  • Primary on-call -> Incident Commander -> Platform/SRE.
  • Add Security lead for data exposure/credential concerns.
  • Add Database owner for failover/PITR decisions.
  • Add Product/Application owner for policy/workflow regressions.