Production Deployment Guide¶
This guide covers deploying AppControl in a production Kubernetes environment.
Reconstruction deployment boundary — 12 September 2026
The current reconstruction pilot uses one backend, strategy.type: Recreate and no backend autoscaler. Cross-backend command/event routing and durable operation resume are not implemented. This topology has a planned control-plane interruption during backend updates and does not claim backend HA. Complete the client's hardening checks and reconstruction qualification before critical use; a Helm install alone does not qualify production recovery.
Prerequisites¶
| Component | Version | Notes |
|---|---|---|
| Kubernetes | 1.26+ | EKS, GKE, AKS, or OpenShift 4.12+ |
| Helm | 3.12+ | |
| PostgreSQL | 16+ | Managed (RDS, CloudSQL, Azure DB) recommended |
| cert-manager | 1.12+ | For TLS certificate management |
| kubectl | 1.26+ | Matching cluster version |
Architecture Overview¶
┌──────────────┐
│ Ingress / │
│ Load Balancer│
└──────┬───────┘
│
┌────────────┼────────────┐
│ │ │
┌────▼───┐ ┌────▼───┐ ┌────▼────┐
│Frontend│ │Backend │ │ Gateway │
│(nginx) │ │(Axum) │ │ (Axum) │
│ x2 │ │ x1 │ │ x2 │
└────────┘ └──┬──┬──┘ └────┬────┘
│ │ │
┌─────────┘ └────┐ │ Agents connect
│ │ │ via mTLS
┌────▼─────┐ │
│PostgreSQL │ ▼
│ (HA) │ 🖥 Agents
│ REQUIRED │
└───────────┘
Note: Only the Backend connects to PostgreSQL.
Gateways and Agents have no database dependency.
Step 1: Namespace and Secrets¶
# Create namespace
kubectl create namespace appcontrol
# Create database secret (use your managed DB credentials)
kubectl create secret generic appcontrol-db \
--namespace appcontrol \
--from-literal=database-url="postgres://appcontrol:PASSWORD@your-rds-host:5432/appcontrol?sslmode=require"
# Create JWT signing secret
kubectl create secret generic appcontrol-jwt \
--namespace appcontrol \
--from-literal=jwt-secret="$(openssl rand -base64 64)"
Step 2: TLS Certificates¶
Option A: cert-manager (recommended)¶
# issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: appcontrol-ca
spec:
selfSigned: {}
---
# For agent mTLS: create a CA for signing agent certificates
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: appcontrol-ca-cert
namespace: appcontrol
spec:
isCA: true
commonName: appcontrol-ca
secretName: appcontrol-ca-secret
issuerRef:
name: appcontrol-ca
kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: appcontrol-agent-issuer
namespace: appcontrol
spec:
ca:
secretName: appcontrol-ca-secret
---
# Gateway server certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: appcontrol-gateway-cert
namespace: appcontrol
spec:
secretName: appcontrol-gateway-tls
issuerRef:
name: appcontrol-agent-issuer
commonName: appcontrol-gateway
dnsNames:
- appcontrol-gateway
- appcontrol-gateway.appcontrol.svc.cluster.local
- gateway.your-domain.com
Option B: Manual certificates¶
# Generate CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -days 3650 \
-subj "/CN=AppControl CA"
# Generate gateway certificate
openssl genrsa -out gateway.key 2048
openssl req -new -key gateway.key -out gateway.csr \
-subj "/CN=appcontrol-gateway"
openssl x509 -req -in gateway.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out gateway.crt -days 365
# Generate agent certificate (one per agent)
openssl genrsa -out agent-01.key 2048
openssl req -new -key agent-01.key -out agent-01.csr \
-subj "/CN=agent-01"
openssl x509 -req -in agent-01.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out agent-01.crt -days 365
# Create Kubernetes secrets
kubectl create secret generic appcontrol-gateway-tls \
--namespace appcontrol \
--from-file=tls.crt=gateway.crt \
--from-file=tls.key=gateway.key \
--from-file=ca.crt=ca.crt
Step 3: Database Setup¶
Managed PostgreSQL (recommended)¶
Create a PostgreSQL 16 instance in your cloud provider:
- AWS RDS:
db.r6g.large, Multi-AZ, automated backups - GCP CloudSQL:
db-custom-2-7680, HA with failover replica - Azure Database:
GP_Gen5_2, Zone-redundant HA
Key settings:
max_connections = 200
shared_buffers = 2GB
effective_cache_size = 6GB
work_mem = 16MB
maintenance_work_mem = 512MB
wal_level = replica # For PITR
archive_mode = on
archive_command = 'your-wal-archive-command'
In-cluster PostgreSQL (dev/staging only)¶
The Helm chart includes a PostgreSQL StatefulSet with PVC. Set postgresql.enabled: true in values.
Step 4: Helm Values for Production¶
Create production-values.yaml:
global:
imageRegistry: "your-registry.example.com/"
imagePullSecrets:
- name: regcred
backend:
replicaCount: 1
strategy:
type: Recreate
image:
tag: "v0.1.0"
resources:
requests:
cpu: "1"
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
env:
RUST_LOG: "info"
APP_ENV: "production"
HA_MODE: "false" # Rate-limit setting; does not enable backend HA
SHUTDOWN_TIMEOUT_SECS: "25"
CORS_ORIGINS: "https://appcontrol.your-domain.com"
database:
existingSecret: "appcontrol-db"
secretKey: "database-url"
jwt:
existingSecret: "appcontrol-jwt"
secretKey: "jwt-secret"
frontend:
replicaCount: 2
image:
tag: "v0.1.0"
gateway:
replicaCount: 2
image:
tag: "v0.1.0"
resources:
requests:
cpu: 500m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
# Disable in-cluster databases for production
postgresql:
enabled: false
podDisruptionBudget:
enabled: true
backend:
minAvailable: 1
gateway:
minAvailable: 1
frontend:
minAvailable: 1
ingress:
enabled: true
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
hosts:
- host: appcontrol.your-domain.com
paths:
- path: /api
pathType: Prefix
- path: /ws
pathType: Prefix
- path: /
pathType: Prefix
tls:
- secretName: appcontrol-tls
hosts:
- appcontrol.your-domain.com
networkPolicy:
enabled: true
Step 6: Deploy¶
Render the chart and check the backend replica count, Recreate strategy, explicit environment settings and secret references before installation. The chart has no HPA; disable any external autoscaler targeting the backend. The production authentication configuration must be supplied and verified as described in Hardening; setting APP_ENV does not configure an identity provider.
# Inspect locally before applying; protect rendered files if your values contain secrets
helm template appcontrol ./helm/appcontrol \
--namespace appcontrol \
--values production-values.yaml
# Add the Helm chart
helm install appcontrol ./helm/appcontrol \
--namespace appcontrol \
--values production-values.yaml
# Verify deployment
kubectl get pods -n appcontrol
kubectl get pdb -n appcontrol
# Check backend health
kubectl exec -n appcontrol deploy/appcontrol-backend -- curl -s localhost:3000/health
kubectl exec -n appcontrol deploy/appcontrol-backend -- curl -s localhost:3000/ready
Step 7: Monitoring Setup¶
Prometheus¶
The backend exposes metrics at /metrics (Prometheus exposition format):
# prometheus-scrape-config.yaml
scrape_configs:
- job_name: appcontrol-backend
kubernetes_sd_configs:
- role: pod
namespaces:
names: [appcontrol]
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
regex: backend
action: keep
Key metrics:
- http_requests_total — Request count by method and status
- http_request_duration_seconds — Request latency histogram
- ws_connections_active — Active WebSocket connections
- agents_connected — Number of connected agents
- state_transitions_total — FSM state changes
- db_pool_connections — Database pool utilization
Grafana Dashboard¶
Create a Grafana dashboard with panels for:
- Request rate and error rate (from http_requests_total)
- P50/P95/P99 latency (from http_request_duration_seconds)
- Agent connection count over time
- Database pool saturation
- State transition rate
Backup Strategy¶
PostgreSQL PITR (Point-in-Time Recovery)¶
For managed databases, enable automated backups:
- AWS RDS: Automated backups with 35-day retention, enable PITR
- GCP CloudSQL: Automated daily backups, enable PITR with WAL
- Azure DB: Geo-redundant backup with 35-day retention
For in-cluster PostgreSQL:
# Manual backup
kubectl exec -n appcontrol appcontrol-postgresql-0 -- \
pg_dump -U appcontrol -Fc appcontrol > backup_$(date +%Y%m%d).dump
# Restore
kubectl exec -i -n appcontrol appcontrol-postgresql-0 -- \
pg_restore -U appcontrol -d appcontrol --clean < backup_20260223.dump
Configuration Snapshots¶
AppControl automatically stores config snapshots (before/after JSONB) in config_versions. No manual backup needed for configuration.
Append-Only Event Tables¶
Tables action_log, state_transitions, check_events, switchover_log are append-only. They are automatically partitioned by month. Old partitions are dropped by the data retention policy (configurable via RETENTION_CHECK_EVENTS_DAYS).
Upgrade Procedure¶
Recommended upgrade order¶
Upgrade components in this order to avoid protocol mismatches:
- Backend — handles database migrations and API changes
- Frontend — depends on updated API
- Gateways — relay upgrades require verified alternate endpoints and command reconciliation
- Agents — depend on gateway + backend being ready
Backend, Frontend & Gateway (planned update)¶
Schedule a maintenance window and stop new operation submissions. Wait for active reconstructions/switchovers to reach their intended terminal state, and reconcile any command/provider job with an unknown result. Do not upgrade during an active or unresolved operation. Back up the database and required configuration, and verify image/database compatibility before proceeding.
The backend uses Recreate: Kubernetes terminates the old backend before bringing up the new one for this Deployment update. Control-plane access is interrupted. A PDB does not preserve active operations or make this update zero-downtime. With one backend and minAvailable: 1, voluntary node eviction can also be blocked; plan node maintenance explicitly.
# 1. Update image tags in production-values.yaml
# 2. Render and verify one backend, Recreate and no backend HPA
helm template appcontrol ./helm/appcontrol \
--namespace appcontrol \
--values production-values.yaml
# 3. Upgrade during the maintenance window; backend access will be interrupted
helm upgrade appcontrol ./helm/appcontrol \
--namespace appcontrol \
--values production-values.yaml
# 4. Monitor rollout
kubectl rollout status -n appcontrol deployment/appcontrol-backend
kubectl rollout status -n appcontrol deployment/appcontrol-gateway
kubectl rollout status -n appcontrol deployment/appcontrol-frontend
# 5. Verify readiness, then fresh agent/component checks and operation histories
kubectl exec -n appcontrol deploy/appcontrol-backend -- curl -s localhost:3000/ready
Gateway redundancy and configured gateway.urls allow agents to reconnect to another reachable gateway. Verify this behavior on the actual endpoints and certificates; a command in flight can still require reconciliation. Gateway/frontend replica counts do not remove the backend interruption or qualify multi-backend reconstruction. A PDB constrains supported voluntary evictions; it is not a blanket guarantee of service continuity during Deployment updates.
Agent binary upgrades¶
Agents run outside Kubernetes (on monitored servers) and require a separate upgrade path. Three options are available, detailed in the Agent Installation Guide — Upgrading Agents:
| Method | Best for | Integrity check | Rollback | Centralized tracking |
|---|---|---|---|---|
| Managed update (API) | Air-gapped or production | Yes (SHA-256) | Automatic | Yes (agent_update_tasks) |
| Direct download | Connected agents | Yes (SHA-256) | Automatic | Yes |
| Manual replacement | Emergency or dev | No | Manual | No |
Recommended approach for production:
- Upload the new agent binary to the backend via API.
- Push the update to 2–3 test agents. Verify health checks resume and the correct version appears in the Agent Management UI.
- Batch-upgrade remaining agents using the batch update API.
- Monitor progress via
GET /api/v1/admin/agent-update-tasks.
Gateway High Availability¶
Gateways are stateless WebSocket relays with no database dependency. High availability is achieved through agent-driven failover, not server-side clustering.
Agent-side failover configuration (in agent.yaml):
gateway:
urls:
- "wss://gw-primary.prod:4443/ws"
- "wss://gw-secondary.prod:4443/ws"
failover_strategy: "ordered" # or "round-robin"
primary_retry_secs: 300 # Retry primary every 5 minutes
Gateway maintenance workflow:
- Suspend the gateway in the UI or API (
is_active = false). - Connected agents detect the disconnection and failover to the next gateway.
- Perform maintenance or upgrade on the suspended gateway.
- Re-enable the gateway (
is_active = true). - Agents with
failover_strategy: "ordered"will return to the primary gateway withinprimary_retry_secs.
Rollback¶
Backend, Frontend & Gateway¶
Use the same maintenance and reconciliation procedure as an upgrade. Select a tested previous image version compatible with the current database; Helm does not reverse database migrations or restore application data. Preserve the current single-backend topology instead of blindly restoring a release whose values used multiple backends or rolling updates.
# Set the tested previous image tags in the current production-values.yaml
# Keep backend.replicaCount=1 and backend.strategy.type=Recreate
helm template appcontrol ./helm/appcontrol \
--namespace appcontrol --values production-values.yaml
helm upgrade appcontrol ./helm/appcontrol \
--namespace appcontrol --values production-values.yaml
# Verify
kubectl rollout status -n appcontrol deployment/appcontrol-backend
After restart, verify new checks and reconcile affected operation records. A stored operation does not automatically continue from its previous step.
Agents¶
If a managed update fails, the agent automatically restores the .old binary
backup. For manual rollbacks: