Skip to main content

Security Incident Runbook

Security-specific incident containment and forensics playbook. Covers prompt injection, data exfiltration, credential exposure, and cross-tenant breach scenarios.

Trigger Conditions​

Run this playbook when any of the following are detected:

  • prompt injection attempts with possible tool execution,
  • data exfiltration attempts,
  • credential exposure in logs/output,
  • suspected cross-tenant data access.

Immediate Containment​

  1. Declare security incident and assign incident commander.
  2. Scope impact (tenant(s), workflow IDs, time window, affected systems).
  3. Isolate active threat path:
    • pause/terminate affected workflows,
    • disable affected tenant if cross-tenant risk exists,
    • block suspicious MCP/tool endpoints.

Scenario Procedures​

Prompt Injection Detected​

  1. Inspect audit trail for injected payload and resulting actions.
  2. Confirm whether tool execution actually occurred.
  3. If execution occurred, inspect side effects and outbound calls.
  4. Tighten sanitization/output filters and tool allowlists before resuming.

Verification:

  • No ongoing malicious prompts executing tools.
  • Updated policies prevent replay of same payload.

Data Exfiltration Attempt​

  1. Review network egress and tool call outputs.
  2. Confirm whether sensitive data left the boundary.
  3. Restrict egress/network policy and block abusive endpoints.
  4. Preserve forensic artifacts (audit exports, traces, logs).

Verification:

  • Suspicious egress no longer present.
  • Output filters redact/deny sensitive payloads as expected.

Credential Exposure​

  1. Identify exposed credential(s) and blast radius.
  2. Rotate credentials immediately (LLM provider keys, DB creds, API tokens).
  3. Revoke old credentials and verify no residual access.
  4. Audit historical access with compromised key window.

Verification:

  • New credentials active and old credentials invalid.
  • No further auth using compromised credentials.

Cross-Tenant Breach Suspected​

  1. Isolate affected tenant(s) and disable write paths if needed.
  2. Validate RLS/session tenant binding and tenant middleware behavior.
  3. Check audit chain continuity for impacted tenants.
  4. Run focused regression checks for tenant-scoped queries.

Verification:

  • No cross-tenant rows visible from tenant-scoped contexts.
  • RLS enforcement checks pass for affected tables.

Post-Incident Actions​

  1. Complete incident report with timeline, impact, and root cause.
  2. Notify stakeholders per policy/compliance requirements.
  3. Add preventive controls:
    • stronger allowlists,
    • stricter network policy,
    • improved anomaly alerts,
    • regression tests for discovered gap.

Escalation Path​

  • On-call engineer -> Security lead -> Incident commander.
  • Engage legal/compliance for confirmed data exposure.
  • Engage platform/database owners for infrastructure-level containment.