Capacity Planning¶
This page provides sizing guidance for AppControl deployments — from a 10-app pilot to a 1000-app enterprise install. Except for explicitly identified microbenchmarks, the numbers below are planning estimates, not validated production capacity. The supported evaluation topology uses one backend with Recreate updates; see the dated OpenShift audit.
For the topology these resources support, see High Availability. For storage growth math, see Backup & Restore.
Table of contents¶
- Benchmarks
- Headline numbers
- Backend sizing
- PostgreSQL sizing
- Agent host footprint
- Gateway sizing
- Agents per backend replica
- WebSocket fan-out and latency targets
- Worked example: 200 apps × 30 components × 1 check / min
- Worked example: 5000 agents, mostly idle
Benchmarks¶
All performance numbers in this page that mention (measured) come from a real
criterion benchmark suite that ships with the repository. The suite lives in
crates/benchmarks/ and covers three hot paths:
| Benchmark | What it measures | Why it matters |
|---|---|---|
fsm |
is_valid_transition and next_state_from_check (pure CPU) |
Runs once per health-check result — must be ≪ 1 µs |
dag_topo |
topological_levels for chains, diamonds and a 500-component wide DAG |
Runs once per POST /apps/:id/start, must stay sub-millisecond on real-world graphs |
permission_resolution |
effective_permission(pool, user, app) on a seeded SQLite |
Runs on every authenticated API request |
How to reproduce locally¶
# Full run (~10s per bench)
cargo bench -p appcontrol-benchmarks --bench fsm
cargo bench -p appcontrol-benchmarks --features sqlite-bench --bench dag_topo
cargo bench -p appcontrol-benchmarks --features sqlite-bench --bench permission_resolution
# Quick run (~2s per bench, fine for spot-checks)
cargo bench -p appcontrol-benchmarks --features sqlite-bench -- --quick
Run the backend DAG and SQLite permission benchmarks separately from the PostgreSQL workspace:
Cargo unifies dependency features within an invocation. Its dedicated CI job
checks both its fixture tests and all benchmark targets with sqlite-bench.
Criterion writes a full HTML report under target/criterion/; open
target/criterion/report/index.html in a browser to drill into the
per-benchmark distributions, flame plots and history.
CI artefacts¶
The Benchmarks workflow (.github/workflows/benchmarks.yaml) runs the suite
on workflow_dispatch, every Monday at 06:00 UTC, and on every push to main
that touches crates/**. Each run uploads two artefacts (90-day retention):
criterion-report— the full HTML tree fromtarget/criterion/bencher-results— the single-file bencher-format output pluslscpuof the runner so numbers can be normalised against the host
The most recent baseline used in this document comes from a GitHub-hosted ubuntu-latest x86_64 runner. Numbers on bare-metal Xeon / EPYC are typically 1.5–2× faster.
Baseline numbers (measured)¶
Recorded against an ubuntu-latest GitHub-hosted runner, criterion --quick
mode (mean of 3 best samples):
| Bench | Per-call wall clock | Throughput |
|---|---|---|
fsm::is_valid_transition (single call) |
1.83 ns | 545 Melem/s |
fsm::is_valid_transition (full 8×8 matrix) |
95.8 ns | 670 Melem/s per entry |
fsm::next_state_from_check (Running, exit 0 hot path) |
0.98 ns | 1.02 Gelem/s |
fsm::next_state_from_check (Running, exit 1 → Failed) |
1.63 ns | 613 Melem/s |
fsm::next_state_from_check (8×3 grid) |
27.4 ns | 876 Melem/s per entry |
dag::topological_levels (5-node linear chain) |
1.14 µs | 880 sorts/s |
dag::topological_levels (52-node diamond) |
19.85 µs | 50 400 sorts/s |
dag::topological_levels (500-node, 10×50 wide DAG) |
2.73 ms | 367 sorts/s |
dag::find_all_dependencies (500-node DAG, deepest node) |
445 µs | 2 248 lookups/s |
permissions::effective_permission (10 users / 10 teams / 10 apps / 100 grants) |
370 µs | 2 700 lookups/s |
permissions::effective_permission (1k users / 100 teams / 1k apps / 10k grants) |
345 µs | 2 900 lookups/s |
permissions::effective_permission (10k users / 1k teams / 10k apps / 100k grants) |
363 µs | 2 754 lookups/s |
Two observations worth calling out:
- The FSM is free:
next_state_from_checkon the hot path runs in under a nanosecond — i.e. one L1 cache hit. The FSM will never be the bottleneck; the database INSERT that follows always dominates. - Permission resolution is constant in data set size: the SQLite query
plan is two index-lookups, so the cost is dominated by the per-query
round-trip (~ 350 µs on SQLite, faster on PostgreSQL where pgbouncer keeps
connections warm). A 1000× bigger fixture does not move the number. This
is why we cache
effective_permissionper-request rather than caching results.
Headline numbers¶
For quick reference:
| Metric | Value | Source |
|---|---|---|
| Backend RAM per replica | 1–2 GiB | production-values.yaml requests/limits |
| Backend CPU per replica | 1–2 vCPU | production-values.yaml requests/limits |
DB_POOL_SIZE formula |
10 + apps × 0.5, capped at 50 |
sqlx pool sizing |
action_log row size |
≈ 500 bytes | typical JSONB + IDs |
state_transitions row size |
≈ 300 bytes | small JSONB |
check_events row size |
≈ 200 bytes | bounded fields, truncated stdout |
| Agent RSS | 20–40 MB | sysinfo sampling |
| Agent CPU | < 1 % steady state | scheduled checks |
| Agent buffer | < 100 MB | sled FIFO cap |
| Gateway connections per agent | 1 WS | by design |
| Agents per backend replica | 5 000–10 000 | empirical limit, single-shard |
Backend sizing¶
CPU per HTTP RPS¶
The backend serves REST + WebSocket and retains live agent routing and operation state in memory. Typical per-request CPU:
| Endpoint class | CPU per request | Latency P50 / P95 |
|---|---|---|
GET /apps (list, no joins) |
0.3 ms | 5 / 20 ms |
GET /apps/:id (with components, deps) |
2 ms | 20 / 80 ms |
POST /apps/:id/start (DAG + sequencer kickoff) |
8 ms | 80 / 200 ms |
GET /reports/availability (aggregations) |
50 ms | 200 / 800 ms |
These endpoint estimates require a workload benchmark. They do not establish a throughput guarantee or authorize horizontal backend scaling.
RAM per replica¶
The backend's RSS is dominated by:
- sqlx connection pool buffers (≈ 5 MB per connection ×
DB_POOL_SIZE) - WebSocket subscriber maps (
DashMap<user_id, Vec<Sender>>) — ≈ 2 KB per active subscriber - Tokio task stacks (negligible)
For DB_POOL_SIZE=20 and 500 active WebSocket subscribers: ≈ 100 MB + 1 MB + base. The Helm defaults set requests=1 GiB / limits=2 GiB which gives ample headroom.
DB_POOL_SIZE formula¶
The recommended formula is:
Where apps is the number of applications under management. The rationale:
- The base of 10 covers baseline traffic (health checks, dashboards).
- Each application adds about 0.5 concurrent connections at steady-state (one for the user looking at it, one occasional for the FSM update).
- The cap at 50 prevents exhausting PostgreSQL's
max_connectionswhen running many replicas.
Examples:
| Apps | Pool size | Why |
|---|---|---|
| 20 | 20 | 10 + 10 |
| 100 | 50 (cap) | 10 + 50, capped |
| 500 | 50 (cap) | always capped — add PgBouncer if needed |
For the supported single backend, reserve database connections for administration and migrations in addition to the application pool.
Replicas¶
Use one backend with Recreate updates. Distributed command routing and operation recovery are not qualified; increasing the replica count or enabling HA_MODE does not make them safe. Plan a maintenance interruption.
PostgreSQL sizing¶
Storage growth model¶
Where:
action_log_rows≈ 50 × users × active_apps_per_day (one row per user action; rough average)state_transitions_rows≈ check_events × 0.05 (only ~ 5 % of checks change exit code)check_events_rows≈ components × checks_per_minute × 60 × 24
Worked storage math¶
For an install with 1000 components, 1 health check / 30 s, 30 days retention on check_events:
check_events_per_day = 1000 × 2 × 60 × 24 = 2 880 000 rows/day
check_events_30_days = 86 400 000 rows
storage ≈ 86.4 M × 200 B = 17.3 GB
Add action_log and state_transitions:
action_log_per_day = 50 × 20 × 100 = 100 000 (~ 50 MB / day)
state_transitions_per_day = 2 880 000 × 0.05 = 144 000 (~ 43 MB / day)
Total daily growth: ≈ 18 GB / month for the example above. With 90-day retention on check_events: ≈ 54 GB.
Partition retention math¶
check_events is partitioned by month. With RETENTION_CHECK_EVENTS_DAYS=90:
- 3–4 monthly partitions are kept live at any time.
- The background retention task drops partitions older than the cutoff (daily check).
- Drop time is O(1) — PostgreSQL just unlinks the file.
To halve storage, halve the retention; to keep audit forever, set 0 and offload monthly partitions to cold storage (see Backup & Restore).
Index and WAL overhead¶
Add ≈ 30 % on top of raw row size for:
- B-tree indexes on
(component_id, created_at)for every event table. - WAL records during write-heavy periods.
- Visibility map and free-space map.
So the 17 GB row total above turns into ≈ 22 GB on disk.
Compute sizing¶
| Component count | Recommended PostgreSQL size |
|---|---|
| < 100 | 2 vCPU / 8 GB RAM / 100 GB SSD |
| 100–500 | 4 vCPU / 16 GB / 250 GB SSD |
| 500–2000 | 8 vCPU / 32 GB / 500 GB SSD |
| > 2000 | 16 vCPU / 64 GB / 1 TB SSD + PgBouncer |
For managed databases on cloud providers:
- AWS RDS:
db.r6g.largefor 100–500 components,db.r6g.xlargefor 500–2000. - GCP CloudSQL:
db-custom-2-7680todb-custom-8-32768. - Azure DB for PostgreSQL:
GP_Gen5_2toGP_Gen5_8.
Agent host footprint¶
The agent is a single Rust binary (~ 8 MB static). On a stable host with ~ 50 managed components:
| Resource | Typical | Peak |
|---|---|---|
| RSS | 25 MB | 50 MB (during a burst of checks) |
| CPU | < 1 % | 5–10 % during start_cmd execution |
| Disk (binary) | 8 MB | 8 MB |
Disk (buffer DB at /var/lib/appcontrol/buffer-{id}) |
< 1 MB normal | < 100 MB during prolonged disconnect |
| Network outbound | ~ 1–5 KB/s baseline | up to 100 KB/s during buffer replay |
| Network inbound | nil — agent is outbound-only | nil |
The agent has no inbound port; outbound TCP 4443 (or 443) to the gateway is sufficient.
Process detachment cost¶
When the agent runs a detached start_cmd (double-fork + setsid), each child consumes the resources of the started process. The agent itself does not retain them — waitpid reaps the intermediate child. So your sizing for start_cmd processes is independent of the agent footprint.
Resource limits applied to checks¶
The agent applies these limits before exec for sync check commands:
RLIMIT_CPU |
RLIMIT_AS |
RLIMIT_NOFILE |
RLIMIT_NPROC |
Default timeout |
|---|---|---|---|---|
| 30 s | 512 MB | 512 fd | 64 child processes | 120 s |
A runaway check command cannot exhaust the host. See Security Architecture §8.
Gateway sizing¶
Gateways are stateless WebSocket relays. Sizing is driven by:
- Number of concurrent agent connections — each holds one WebSocket.
- Messages per second — each WS message is a few hundred bytes.
Per-gateway capacity¶
| Resource | Per 1000 agents | Per 5000 agents | Per 10 000 agents |
|---|---|---|---|
| RAM | ~ 128 MB | ~ 512 MB | ~ 1 GB |
| CPU | < 0.5 vCPU | 1 vCPU | 2 vCPU |
| File descriptors | 1000+ | 5000+ | 10 000+ |
| TCP connections | 1001 (+ 1 to backend) | 5001 | 10 001 |
ulimit adjustment¶
For > 1024 agents per gateway, raise the open-file limit:
# /etc/security/limits.d/appcontrol.conf
appcontrol soft nofile 65536
appcontrol hard nofile 65536
# Or in the systemd unit
[Service]
LimitNOFILE=65536
For Kubernetes, set securityContext.sysctls or use a node-level adjustment.
Recommended replicas¶
For redundancy, deploy at least 2 gateways even for small fleets. For large fleets, distribute by zone:
| Total agents | Recommended gateway count |
|---|---|
| < 500 | 2 (N+1) |
| 500–5000 | 3–4 (distribute load) |
| 5000–20 000 | 5–10 (zone-scoped) |
Agents per backend replica¶
Rough rule of thumb: 5 000 – 10 000 agents per backend replica.
This is bounded by:
- WebSocket fan-out load. Every agent's heartbeat (every 60 s) and CheckResult (delta-only) terminates at a backend replica via the gateway → backend WS. With 10 000 agents, the backend handles ~ 170 inbound WS messages / s baseline.
- DB pool size. Each agent's state update is a brief DB write. At 10 000 agents and 30 s check interval, the worst case is ~ 330 writes / s — well within a 20-connection pool.
- CPU. Each WS frame requires deserialization (serde_json), FSM logic, and an SQL
INSERT. Empirical: 1 vCPU handles ~ 500 inbound WS frames / s.
The 10 000-agent estimate is not a qualified limit. Do not add backend replicas to address it; benchmark the single-backend deployment and qualify distributed command routing before considering another topology.
WebSocket fan-out and latency targets¶
Events per second¶
A typical AppControl event source breakdown for 1000 components:
| Source | Frequency | Events / s |
|---|---|---|
| Health-check deltas | 5 % of 1000 components × 30 s | ~ 1.7 |
| State transitions | derived from deltas | ~ 1.7 |
| Agent heartbeats (60 s) | 100 agents | ~ 1.7 |
| Operator actions | bursty | ~ 0.5 |
Total: ≈ 5–10 events/s sustained, with bursts to 50/s during operations. Multiply linearly with component count.
Latency budget¶
| Hop | Target P95 | Notes |
|---|---|---|
| Agent → Gateway (WS frame) | 5 ms | network-bound |
| Gateway → Backend (WS frame) | 5 ms | network-bound |
| Backend FSM decision | < 1 µs | measured — next_state_from_check is ≈ 1 ns/call; FSM cost is invisible against the rest of the chain |
| Backend → DB INSERT | 10 ms | state_transitions INSERT, sqlx round-trip dominates |
| Backend → WS Hub fan-out | 5 ms | DashMap iteration; permission check is < 400 µs (measured) |
| Backend → Frontend WS frame | 10 ms | network-bound |
| Total agent-event → UI render | 35 ms |
If your install consistently exceeds 100 ms agent-event → UI latency, monitor:
agent_message_latency_msPrometheus histogram (labels:message_type=heartbeat|check_result|discovery_report) — emitted by the backend on every agent message that carries a timestamp; recommended alert: P95 > 1000 ms over 5 min. The backend additionally emits atracing::warn!line ("High agent → backend message latency") above the hard-coded thresholdAGENT_MESSAGE_LATENCY_WARN_MS = 10_000ms (defined incrates/backend/src/websocket/mod.rs).ws_connections_activeper replica — imbalance is a hint of broken affinity- DB
INSERTtime — slowstate_transitionsinserts cascade
WebSocket fan-out scaling¶
Each backend replica maintains an in-memory DashMap of subscribers. The cross-replica notification path uses PostgreSQL LISTEN/NOTIFY. This scales linearly with the number of backend replicas — there is no global fan-out queue.
Worked example: 200 apps × 30 components × 1 check / min¶
A medium enterprise install. The goal is to derive concrete resource requirements.
Inputs¶
- 200 applications
- 30 components per application on average → 6000 total components
- 60 s check interval (1 check / min per component)
- 30 distinct agents per application → 6000 / 30 = 200 components per agent on average → 200 active agents
- Working hours: 50 operator actions / day; 50 dashboard users
- 90-day retention on
check_events
Computed values¶
Check events¶
checks/sec = 6000 components / 60 s = 100 checks/s
deltas/sec = 100 × 5 % = 5 deltas/s (95 % deduplicated)
check_events_per_day = 100 × 86400 = 8 640 000 rows/day
storage_30d ≈ 8.64 M × 30 × 200 B = 52 GB raw + 30 % index/WAL = 68 GB on disk
State transitions¶
Action log¶
DB pool¶
Recommended backend¶
| Resource | Value |
|---|---|
| Replicas | 3 |
| CPU per replica | 2 vCPU |
| RAM per replica | 2 GB |
| Total backend CPU | 6 vCPU |
| Total backend RAM | 6 GB |
Recommended PostgreSQL¶
| Resource | Value |
|---|---|
| CPU | 4 vCPU |
| RAM | 16 GB |
| Storage (90 d, no archive) | 200 GB SSD |
| Connections | max_connections = 200 |
Recommended gateway¶
| Resource | Value |
|---|---|
| Replicas | 2 |
| CPU per replica | 0.5 vCPU |
| RAM per replica | 256 MB |
Per-agent profile¶
- 200 components → ~ 35 MB RSS, < 1 % CPU
- 200 checks/min × delta ratio ≈ 10 deltas/min × 200 bytes ≈ 2 KB/min network egress (negligible)
Per-frontend session¶
- ~ 1 MB RAM in the browser
- ~ 5 KB/s WebSocket inbound from backend during dashboard view
Helm values¶
backend:
replicaCount: 1
resources:
requests: { cpu: "1", memory: 1Gi }
limits: { cpu: "2", memory: 2Gi }
env:
HA_MODE: "false"
DB_POOL_SIZE: "50"
RETENTION_CHECK_EVENTS_DAYS: "90"
gateway:
replicaCount: 2
resources:
requests: { cpu: "500m", memory: 256Mi }
limits: { cpu: "1", memory: 512Mi }
Cost rough order-of-magnitude (AWS pricing)¶
- Backend: 3 ×
t3.large≈ $200/mo - Gateway: 2 ×
t3.small≈ $30/mo - PostgreSQL:
db.r6g.largeMulti-AZ ≈ $300/mo - Storage (200 GB SSD + backup) ≈ $50/mo
- LB and data transfer: ≈ $40/mo
- Total ≈ $620/mo for a control plane managing 6000 components across 200 hosts.
Worked example: 5000 agents, mostly idle¶
A surveillance install — many agents, low check frequency, infrequent operations.
Inputs¶
- 5000 agents, 5 components each → 25 000 components
- Check interval: 5 minutes
- 99 % stable (deltas only 1 %)
Computed values¶
checks/sec = 25 000 / 300 = 83 checks/s
deltas/sec = 83 × 1 % = 0.8 deltas/s
check_events/day = 83 × 86400 = 7.2 M rows/day → 1.5 GB/day raw
storage_30d ≈ 45 GB raw + 13 GB index ≈ 60 GB
The DB grows slower than the previous example because the check interval is 5× longer.
Gateway sizing¶
5000 agents at 5 different sites → 1 gateway per site = 5 gateways minimum. Each gateway handles 1000 agents — well within 1 vCPU / 256 MB.
Backend sizing¶
WS throughput is low (< 1 frame/s baseline). One backend replica suffices, but always run ≥ 2 for HA. CPU is dominated by 50 dashboard users on the frontend.
backend:
replicaCount: 1
resources:
requests: { cpu: "500m", memory: 512Mi }
limits: { cpu: "1", memory: 1Gi }
gateway:
replicaCount: 5 # 1 per site
File descriptor planning¶
Each gateway holds 1000 agent WebSockets + 1 backend WebSocket = 1001 fds. The default ulimit -n 1024 is too tight; set LimitNOFILE=4096.
Reference¶
- BACKUP_RESTORE.md — how the storage volume is preserved and retained
- HIGH_AVAILABILITY.md — replicas and failure-domain distribution
- LIMITS.md — explicit per-entity limits
- PRODUCTION_DEPLOYMENT.md — full deployment topology