Operational Runbooks¶
Incident response procedures for AppControl production environments.
Runbook 1: Agent Not Connecting¶
Symptoms: Agent shows as UNREACHABLE in the dashboard. No heartbeat events.
Diagnosis:
# 1. Check agent process on the host
systemctl status appcontrol-agent
journalctl -u appcontrol-agent --since "30 min ago"
# 2. Check network connectivity to gateway
curl -k https://gateway.your-domain.com:4443/health
# 3. Check TLS certificate validity
openssl x509 -in /etc/appcontrol/agent.crt -noout -dates
openssl verify -CAfile /etc/appcontrol/ca.crt /etc/appcontrol/agent.crt
# 4. Check gateway logs for rejected connections
kubectl logs -n appcontrol -l app.kubernetes.io/component=gateway --tail=50
# 5. Check backend for agent registration
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s localhost:3000/api/v1/agents | jq '.[] | select(.hostname=="the-host")'
Resolution:
| Cause | Fix |
|---|---|
| Agent process down | systemctl restart appcontrol-agent |
| Certificate expired | Regenerate cert signed by CA, restart agent |
| Network firewall | Allow TCP 4443 from agent to gateway |
| Gateway down | Check gateway pods: kubectl get pods -n appcontrol -l app.kubernetes.io/component=gateway |
| DNS resolution | Verify agent can resolve gateway hostname |
Runbook 2: Backend High CPU / Slow Responses¶
Symptoms: API latency > 2s, Prometheus http_request_duration_seconds P95 elevated.
Diagnosis:
# 1. Check pod resource usage
kubectl top pods -n appcontrol -l app.kubernetes.io/component=backend
# 2. Check database connection pool
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s localhost:3000/metrics | grep db_pool
# 3. Check PostgreSQL for slow queries
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
psql -U appcontrol -c "SELECT pid, now()-query_start AS duration, query FROM pg_stat_activity WHERE state='active' AND query_start < now() - interval '5 seconds' ORDER BY duration DESC;"
# 4. Check for table bloat (check_events grows fast)
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
psql -U appcontrol -c "SELECT relname, pg_size_pretty(pg_total_relation_size(oid)) FROM pg_class WHERE relname LIKE 'check_events%' ORDER BY pg_total_relation_size(oid) DESC LIMIT 10;"
# 5. Check active operations (stuck locks?)
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s localhost:3000/metrics | grep http_requests
Resolution:
| Cause | Fix |
|---|---|
| Too many concurrent operations | Check operation lock conflicts in logs |
| Database slow queries | Add missing indexes, run ANALYZE |
| check_events table too large | Reduce RETENTION_CHECK_EVENTS_DAYS, drop old partitions |
| Connection pool exhausted | Increase DATABASE_MAX_CONNECTIONS env var |
| Memory pressure | Increase backend resources.limits.memory |
Runbook 3: Database Partition Overflow¶
Symptoms: INSERT errors on check_events, logs show "no partition for value".
Diagnosis:
# Check existing partitions
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
psql -U appcontrol -c "SELECT tablename FROM pg_tables WHERE tablename LIKE 'check_events_y%' ORDER BY tablename;"
# Check if current month partition exists
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
psql -U appcontrol -c "SELECT tablename FROM pg_tables WHERE tablename = 'check_events_y2026m02';"
Resolution:
# Create missing partition manually
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
psql -U appcontrol -c "CREATE TABLE IF NOT EXISTS check_events_y2026m02 PARTITION OF check_events FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');"
# Also create next month
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
psql -U appcontrol -c "CREATE TABLE IF NOT EXISTS check_events_y2026m03 PARTITION OF check_events FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');"
# Backend auto-creates partitions daily. If this keeps happening,
# check the partition maintenance task in backend logs:
kubectl logs -n appcontrol deploy/appcontrol-backend | grep "Partition maintenance"
Runbook 4: Token Revocation for Compromised Account¶
Symptoms: Suspected unauthorized access via a compromised JWT token.
Actions:
# 1. Immediately revoke all tokens for the user via API
# (requires admin API key or break-glass access)
curl -X POST https://appcontrol.your-domain.com/api/v1/auth/revoke-all \
-H "Authorization: Bearer ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"user_id": "UUID_OF_COMPROMISED_USER"}'
# 2. Verify the revocation entry exists in PostgreSQL
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s localhost:3000/api/v1/auth/revoked-tokens | jq
# 3. If revocation is not working, rotate the JWT_SECRET to invalidate ALL tokens
kubectl create secret generic appcontrol-jwt \
--namespace appcontrol \
--from-literal=jwt-secret="$(openssl rand -base64 64)" \
--dry-run=client -o yaml | kubectl apply -f -
# 4. Restart backend to pick up new secret
kubectl rollout restart -n appcontrol deployment/appcontrol-backend
# 5. Check audit log for the compromised user's actions
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s "localhost:3000/api/v1/audit?user_id=UUID_OF_COMPROMISED_USER&limit=100"
Runbook 5: Token revocation storage failure¶
Check application database connectivity (see Runbook 2). Revoked token fingerprints
are stored in revoked_tokens. Authentication must refuse a request when the
revocation check fails; do not disable this check to restore access.
Runbook 6: DR Switchover Stuck in Phase¶
Symptoms: Switchover operation started but stuck (e.g., in DRAINING phase for > 10 minutes).
Diagnosis:
# 1. Check switchover status
curl -s https://appcontrol.your-domain.com/api/v1/apps/APP_ID/switchover/status \
-H "Authorization: Bearer TOKEN" | jq
# 2. Check switchover_log for the latest entries
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s "localhost:3000/api/v1/apps/APP_ID/switchover/log" | jq
# 3. Check if agents on DR site are connected
kubectl exec -n appcontrol deploy/appcontrol-backend -- \
curl -s "localhost:3000/api/v1/agents?zone=dr-site" | jq
Resolution:
# Force-cancel the stuck switchover (requires manage permission)
curl -X POST https://appcontrol.your-domain.com/api/v1/apps/APP_ID/switchover/cancel \
-H "Authorization: Bearer TOKEN"
# If the cancel itself is stuck, check for operation locks
kubectl logs -n appcontrol deploy/appcontrol-backend | grep "operation_lock"