Skip to content

Capacity Planning

This page provides sizing guidance for AppControl deployments — from a 10-app pilot to a 1000-app enterprise install. Except for explicitly identified microbenchmarks, the numbers below are planning estimates, not validated production capacity. The supported evaluation topology uses one backend with Recreate updates; see the dated OpenShift audit.

For the topology these resources support, see High Availability. For storage growth math, see Backup & Restore.

Table of contents


Benchmarks

All performance numbers in this page that mention (measured) come from a real criterion benchmark suite that ships with the repository. The suite lives in crates/benchmarks/ and covers three hot paths:

Benchmark What it measures Why it matters
fsm is_valid_transition and next_state_from_check (pure CPU) Runs once per health-check result — must be ≪ 1 µs
dag_topo topological_levels for chains, diamonds and a 500-component wide DAG Runs once per POST /apps/:id/start, must stay sub-millisecond on real-world graphs
permission_resolution effective_permission(pool, user, app) on a seeded SQLite Runs on every authenticated API request

How to reproduce locally

# Full run (~10s per bench)
cargo bench -p appcontrol-benchmarks --bench fsm
cargo bench -p appcontrol-benchmarks --features sqlite-bench --bench dag_topo
cargo bench -p appcontrol-benchmarks --features sqlite-bench --bench permission_resolution

# Quick run (~2s per bench, fine for spot-checks)
cargo bench -p appcontrol-benchmarks --features sqlite-bench -- --quick

Run the backend DAG and SQLite permission benchmarks separately from the PostgreSQL workspace: Cargo unifies dependency features within an invocation. Its dedicated CI job checks both its fixture tests and all benchmark targets with sqlite-bench.

Criterion writes a full HTML report under target/criterion/; open target/criterion/report/index.html in a browser to drill into the per-benchmark distributions, flame plots and history.

CI artefacts

The Benchmarks workflow (.github/workflows/benchmarks.yaml) runs the suite on workflow_dispatch, every Monday at 06:00 UTC, and on every push to main that touches crates/**. Each run uploads two artefacts (90-day retention):

  • criterion-report — the full HTML tree from target/criterion/
  • bencher-results — the single-file bencher-format output plus lscpu of the runner so numbers can be normalised against the host

The most recent baseline used in this document comes from a GitHub-hosted ubuntu-latest x86_64 runner. Numbers on bare-metal Xeon / EPYC are typically 1.5–2× faster.

Baseline numbers (measured)

Recorded against an ubuntu-latest GitHub-hosted runner, criterion --quick mode (mean of 3 best samples):

Bench Per-call wall clock Throughput
fsm::is_valid_transition (single call) 1.83 ns 545 Melem/s
fsm::is_valid_transition (full 8×8 matrix) 95.8 ns 670 Melem/s per entry
fsm::next_state_from_check (Running, exit 0 hot path) 0.98 ns 1.02 Gelem/s
fsm::next_state_from_check (Running, exit 1 → Failed) 1.63 ns 613 Melem/s
fsm::next_state_from_check (8×3 grid) 27.4 ns 876 Melem/s per entry
dag::topological_levels (5-node linear chain) 1.14 µs 880 sorts/s
dag::topological_levels (52-node diamond) 19.85 µs 50 400 sorts/s
dag::topological_levels (500-node, 10×50 wide DAG) 2.73 ms 367 sorts/s
dag::find_all_dependencies (500-node DAG, deepest node) 445 µs 2 248 lookups/s
permissions::effective_permission (10 users / 10 teams / 10 apps / 100 grants) 370 µs 2 700 lookups/s
permissions::effective_permission (1k users / 100 teams / 1k apps / 10k grants) 345 µs 2 900 lookups/s
permissions::effective_permission (10k users / 1k teams / 10k apps / 100k grants) 363 µs 2 754 lookups/s

Two observations worth calling out:

  1. The FSM is free: next_state_from_check on the hot path runs in under a nanosecond — i.e. one L1 cache hit. The FSM will never be the bottleneck; the database INSERT that follows always dominates.
  2. Permission resolution is constant in data set size: the SQLite query plan is two index-lookups, so the cost is dominated by the per-query round-trip (~ 350 µs on SQLite, faster on PostgreSQL where pgbouncer keeps connections warm). A 1000× bigger fixture does not move the number. This is why we cache effective_permission per-request rather than caching results.

Headline numbers

For quick reference:

Metric Value Source
Backend RAM per replica 1–2 GiB production-values.yaml requests/limits
Backend CPU per replica 1–2 vCPU production-values.yaml requests/limits
DB_POOL_SIZE formula 10 + apps × 0.5, capped at 50 sqlx pool sizing
action_log row size ≈ 500 bytes typical JSONB + IDs
state_transitions row size ≈ 300 bytes small JSONB
check_events row size ≈ 200 bytes bounded fields, truncated stdout
Agent RSS 20–40 MB sysinfo sampling
Agent CPU < 1 % steady state scheduled checks
Agent buffer < 100 MB sled FIFO cap
Gateway connections per agent 1 WS by design
Agents per backend replica 5 000–10 000 empirical limit, single-shard

Backend sizing

CPU per HTTP RPS

The backend serves REST + WebSocket and retains live agent routing and operation state in memory. Typical per-request CPU:

Endpoint class CPU per request Latency P50 / P95
GET /apps (list, no joins) 0.3 ms 5 / 20 ms
GET /apps/:id (with components, deps) 2 ms 20 / 80 ms
POST /apps/:id/start (DAG + sequencer kickoff) 8 ms 80 / 200 ms
GET /reports/availability (aggregations) 50 ms 200 / 800 ms

These endpoint estimates require a workload benchmark. They do not establish a throughput guarantee or authorize horizontal backend scaling.

RAM per replica

The backend's RSS is dominated by:

  • sqlx connection pool buffers (≈ 5 MB per connection × DB_POOL_SIZE)
  • WebSocket subscriber maps (DashMap<user_id, Vec<Sender>>) — ≈ 2 KB per active subscriber
  • Tokio task stacks (negligible)

For DB_POOL_SIZE=20 and 500 active WebSocket subscribers: ≈ 100 MB + 1 MB + base. The Helm defaults set requests=1 GiB / limits=2 GiB which gives ample headroom.

DB_POOL_SIZE formula

The recommended formula is:

DB_POOL_SIZE = min(50, 10 + apps × 0.5)

Where apps is the number of applications under management. The rationale:

  • The base of 10 covers baseline traffic (health checks, dashboards).
  • Each application adds about 0.5 concurrent connections at steady-state (one for the user looking at it, one occasional for the FSM update).
  • The cap at 50 prevents exhausting PostgreSQL's max_connections when running many replicas.

Examples:

Apps Pool size Why
20 20 10 + 10
100 50 (cap) 10 + 50, capped
500 50 (cap) always capped — add PgBouncer if needed

For the supported single backend, reserve database connections for administration and migrations in addition to the application pool.

Replicas

Use one backend with Recreate updates. Distributed command routing and operation recovery are not qualified; increasing the replica count or enabling HA_MODE does not make them safe. Plan a maintenance interruption.


PostgreSQL sizing

Storage growth model

daily_bytes ≈
  action_log_rows × 500
  + state_transitions_rows × 300
  + check_events_rows × 200

Where:

  • action_log_rows ≈ 50 × users × active_apps_per_day (one row per user action; rough average)
  • state_transitions_rows ≈ check_events × 0.05 (only ~ 5 % of checks change exit code)
  • check_events_rows ≈ components × checks_per_minute × 60 × 24

Worked storage math

For an install with 1000 components, 1 health check / 30 s, 30 days retention on check_events:

check_events_per_day  = 1000 × 2 × 60 × 24 = 2 880 000 rows/day
check_events_30_days  = 86 400 000 rows
storage               ≈ 86.4 M × 200 B = 17.3 GB

Add action_log and state_transitions:

action_log_per_day        = 50 × 20 × 100 = 100 000  (~ 50 MB / day)
state_transitions_per_day = 2 880 000 × 0.05 = 144 000  (~ 43 MB / day)

Total daily growth: ≈ 18 GB / month for the example above. With 90-day retention on check_events: ≈ 54 GB.

Partition retention math

check_events is partitioned by month. With RETENTION_CHECK_EVENTS_DAYS=90:

  • 3–4 monthly partitions are kept live at any time.
  • The background retention task drops partitions older than the cutoff (daily check).
  • Drop time is O(1) — PostgreSQL just unlinks the file.

To halve storage, halve the retention; to keep audit forever, set 0 and offload monthly partitions to cold storage (see Backup & Restore).

Index and WAL overhead

Add ≈ 30 % on top of raw row size for:

  • B-tree indexes on (component_id, created_at) for every event table.
  • WAL records during write-heavy periods.
  • Visibility map and free-space map.

So the 17 GB row total above turns into ≈ 22 GB on disk.

Compute sizing

Component count Recommended PostgreSQL size
< 100 2 vCPU / 8 GB RAM / 100 GB SSD
100–500 4 vCPU / 16 GB / 250 GB SSD
500–2000 8 vCPU / 32 GB / 500 GB SSD
> 2000 16 vCPU / 64 GB / 1 TB SSD + PgBouncer

For managed databases on cloud providers:

  • AWS RDS: db.r6g.large for 100–500 components, db.r6g.xlarge for 500–2000.
  • GCP CloudSQL: db-custom-2-7680 to db-custom-8-32768.
  • Azure DB for PostgreSQL: GP_Gen5_2 to GP_Gen5_8.

Agent host footprint

The agent is a single Rust binary (~ 8 MB static). On a stable host with ~ 50 managed components:

Resource Typical Peak
RSS 25 MB 50 MB (during a burst of checks)
CPU < 1 % 5–10 % during start_cmd execution
Disk (binary) 8 MB 8 MB
Disk (buffer DB at /var/lib/appcontrol/buffer-{id}) < 1 MB normal < 100 MB during prolonged disconnect
Network outbound ~ 1–5 KB/s baseline up to 100 KB/s during buffer replay
Network inbound nil — agent is outbound-only nil

The agent has no inbound port; outbound TCP 4443 (or 443) to the gateway is sufficient.

Process detachment cost

When the agent runs a detached start_cmd (double-fork + setsid), each child consumes the resources of the started process. The agent itself does not retain them — waitpid reaps the intermediate child. So your sizing for start_cmd processes is independent of the agent footprint.

Resource limits applied to checks

The agent applies these limits before exec for sync check commands:

RLIMIT_CPU RLIMIT_AS RLIMIT_NOFILE RLIMIT_NPROC Default timeout
30 s 512 MB 512 fd 64 child processes 120 s

A runaway check command cannot exhaust the host. See Security Architecture §8.


Gateway sizing

Gateways are stateless WebSocket relays. Sizing is driven by:

  1. Number of concurrent agent connections — each holds one WebSocket.
  2. Messages per second — each WS message is a few hundred bytes.

Per-gateway capacity

Resource Per 1000 agents Per 5000 agents Per 10 000 agents
RAM ~ 128 MB ~ 512 MB ~ 1 GB
CPU < 0.5 vCPU 1 vCPU 2 vCPU
File descriptors 1000+ 5000+ 10 000+
TCP connections 1001 (+ 1 to backend) 5001 10 001

ulimit adjustment

For > 1024 agents per gateway, raise the open-file limit:

# /etc/security/limits.d/appcontrol.conf
appcontrol soft nofile 65536
appcontrol hard nofile 65536

# Or in the systemd unit
[Service]
LimitNOFILE=65536

For Kubernetes, set securityContext.sysctls or use a node-level adjustment.

For redundancy, deploy at least 2 gateways even for small fleets. For large fleets, distribute by zone:

Total agents Recommended gateway count
< 500 2 (N+1)
500–5000 3–4 (distribute load)
5000–20 000 5–10 (zone-scoped)

Agents per backend replica

Rough rule of thumb: 5 000 – 10 000 agents per backend replica.

This is bounded by:

  1. WebSocket fan-out load. Every agent's heartbeat (every 60 s) and CheckResult (delta-only) terminates at a backend replica via the gateway → backend WS. With 10 000 agents, the backend handles ~ 170 inbound WS messages / s baseline.
  2. DB pool size. Each agent's state update is a brief DB write. At 10 000 agents and 30 s check interval, the worst case is ~ 330 writes / s — well within a 20-connection pool.
  3. CPU. Each WS frame requires deserialization (serde_json), FSM logic, and an SQL INSERT. Empirical: 1 vCPU handles ~ 500 inbound WS frames / s.

The 10 000-agent estimate is not a qualified limit. Do not add backend replicas to address it; benchmark the single-backend deployment and qualify distributed command routing before considering another topology.


WebSocket fan-out and latency targets

Events per second

A typical AppControl event source breakdown for 1000 components:

Source Frequency Events / s
Health-check deltas 5 % of 1000 components × 30 s ~ 1.7
State transitions derived from deltas ~ 1.7
Agent heartbeats (60 s) 100 agents ~ 1.7
Operator actions bursty ~ 0.5

Total: ≈ 5–10 events/s sustained, with bursts to 50/s during operations. Multiply linearly with component count.

Latency budget

Hop Target P95 Notes
Agent → Gateway (WS frame) 5 ms network-bound
Gateway → Backend (WS frame) 5 ms network-bound
Backend FSM decision < 1 µs measured — next_state_from_check is ≈ 1 ns/call; FSM cost is invisible against the rest of the chain
Backend → DB INSERT 10 ms state_transitions INSERT, sqlx round-trip dominates
Backend → WS Hub fan-out 5 ms DashMap iteration; permission check is < 400 µs (measured)
Backend → Frontend WS frame 10 ms network-bound
Total agent-event → UI render 35 ms

If your install consistently exceeds 100 ms agent-event → UI latency, monitor:

  • agent_message_latency_ms Prometheus histogram (labels: message_type=heartbeat|check_result|discovery_report) — emitted by the backend on every agent message that carries a timestamp; recommended alert: P95 > 1000 ms over 5 min. The backend additionally emits a tracing::warn! line ("High agent → backend message latency") above the hard-coded threshold AGENT_MESSAGE_LATENCY_WARN_MS = 10_000 ms (defined in crates/backend/src/websocket/mod.rs).
  • ws_connections_active per replica — imbalance is a hint of broken affinity
  • DB INSERT time — slow state_transitions inserts cascade

WebSocket fan-out scaling

Each backend replica maintains an in-memory DashMap of subscribers. The cross-replica notification path uses PostgreSQL LISTEN/NOTIFY. This scales linearly with the number of backend replicas — there is no global fan-out queue.


Worked example: 200 apps × 30 components × 1 check / min

A medium enterprise install. The goal is to derive concrete resource requirements.

Inputs

  • 200 applications
  • 30 components per application on average → 6000 total components
  • 60 s check interval (1 check / min per component)
  • 30 distinct agents per application → 6000 / 30 = 200 components per agent on average → 200 active agents
  • Working hours: 50 operator actions / day; 50 dashboard users
  • 90-day retention on check_events

Computed values

Check events

checks/sec  = 6000 components / 60 s = 100 checks/s
deltas/sec  = 100 × 5 %         = 5 deltas/s  (95 % deduplicated)
check_events_per_day = 100 × 86400 = 8 640 000 rows/day
storage_30d ≈ 8.64 M × 30 × 200 B = 52 GB raw + 30 % index/WAL = 68 GB on disk

State transitions

state_transitions/day ≈ 8.64 M × 5 % = 432 000 rows/day
storage_30d ≈ 432 K × 30 × 300 B = ~ 4 GB

Action log

50 actions × 30 days = 1500 rows/month — negligible

DB pool

DB_POOL_SIZE = min(50, 10 + 200 × 0.5) = min(50, 110) = 50
Resource Value
Replicas 3
CPU per replica 2 vCPU
RAM per replica 2 GB
Total backend CPU 6 vCPU
Total backend RAM 6 GB
Resource Value
CPU 4 vCPU
RAM 16 GB
Storage (90 d, no archive) 200 GB SSD
Connections max_connections = 200
Resource Value
Replicas 2
CPU per replica 0.5 vCPU
RAM per replica 256 MB

Per-agent profile

  • 200 components → ~ 35 MB RSS, < 1 % CPU
  • 200 checks/min × delta ratio ≈ 10 deltas/min × 200 bytes ≈ 2 KB/min network egress (negligible)

Per-frontend session

  • ~ 1 MB RAM in the browser
  • ~ 5 KB/s WebSocket inbound from backend during dashboard view

Helm values

backend:
  replicaCount: 1
  resources:
    requests: { cpu: "1",   memory: 1Gi }
    limits:   { cpu: "2",   memory: 2Gi }
  env:
    HA_MODE: "false"
    DB_POOL_SIZE: "50"
    RETENTION_CHECK_EVENTS_DAYS: "90"

gateway:
  replicaCount: 2
  resources:
    requests: { cpu: "500m", memory: 256Mi }
    limits:   { cpu: "1",    memory: 512Mi }

Cost rough order-of-magnitude (AWS pricing)

  • Backend: 3 × t3.large ≈ $200/mo
  • Gateway: 2 × t3.small ≈ $30/mo
  • PostgreSQL: db.r6g.large Multi-AZ ≈ $300/mo
  • Storage (200 GB SSD + backup) ≈ $50/mo
  • LB and data transfer: ≈ $40/mo
  • Total ≈ $620/mo for a control plane managing 6000 components across 200 hosts.

Worked example: 5000 agents, mostly idle

A surveillance install — many agents, low check frequency, infrequent operations.

Inputs

  • 5000 agents, 5 components each → 25 000 components
  • Check interval: 5 minutes
  • 99 % stable (deltas only 1 %)

Computed values

checks/sec = 25 000 / 300 = 83 checks/s
deltas/sec = 83 × 1 % = 0.8 deltas/s
check_events/day = 83 × 86400 = 7.2 M rows/day → 1.5 GB/day raw
storage_30d ≈ 45 GB raw + 13 GB index ≈ 60 GB

The DB grows slower than the previous example because the check interval is 5× longer.

Gateway sizing

5000 agents at 5 different sites → 1 gateway per site = 5 gateways minimum. Each gateway handles 1000 agents — well within 1 vCPU / 256 MB.

Backend sizing

WS throughput is low (< 1 frame/s baseline). One backend replica suffices, but always run ≥ 2 for HA. CPU is dominated by 50 dashboard users on the frontend.

backend:
  replicaCount: 1
  resources:
    requests: { cpu: "500m", memory: 512Mi }
    limits:   { cpu: "1",    memory: 1Gi }
gateway:
  replicaCount: 5     # 1 per site

File descriptor planning

Each gateway holds 1000 agent WebSockets + 1 backend WebSocket = 1001 fds. The default ulimit -n 1024 is too tight; set LimitNOFILE=4096.


Reference