Metrics reference¶
Auto-generated by scanning
crates/**/src/**/*.rsformetrics::{counter,gauge,histogram}!invocations. Runscripts/docs/regen.pyto refresh.
The backend exposes Prometheus metrics at GET /metrics. The endpoint requires no authentication by default; restrict it at the load-balancer or reverse-proxy layer if you do not want it scraped from the public network.
# Local check
curl -s http://localhost:3000/metrics | head -40
# Prometheus scrape config snippet
# (see monitoring/prometheus/prometheus.yml for the full file)
scrape_configs:
- job_name: appcontrol-backend
metrics_path: /metrics
static_configs:
- targets: ['backend:3000']
Exposed metrics¶
| Name | Kind | Unit | Labels | Description |
|---|---|---|---|---|
agent_message_latency_ms |
histogram | milliseconds | message_type |
End-to-end latency between an agent emitting a message and the backend processing it (the message's at timestamp minus server-side receive time). Labelled by message_type. |
agents_connected |
gauge | agents | (none) | Number of unique agents currently routed through the WebSocket hub. Diverges from the total agent count when an agent is registered in the DB but disconnected. |
commands_executed_total |
counter | commands | command |
Count of custom commands dispatched to agents via POST /components/:id/command/:cmd. Labelled by command name. |
db_pool_connections |
gauge | connections | state |
Postgres / SQLite connection-pool occupancy. Label state is one of idle, active, total — alert on total ≥ DB_POOL_SIZE for sustained periods. |
db_type_info |
gauge | info | type |
Constant gauge (= 1) labelled with type=postgres or type=sqlite, exposing the compile-time database backend. |
http_request_duration_seconds |
histogram | seconds | method, path |
Histogram of HTTP request latency. Use the standard _bucket series for P50/P95/P99 quantiles; _sum / _count gives the mean. |
http_requests_total |
counter | requests | method |
Count of HTTP requests handled by the backend, labelled by HTTP method and response status code. Use this to compute request rate and the error ratio (rate by status=~"5.."). |
state_transitions_total |
counter | transitions | from |
Count of FSM transitions persisted to state_transitions. Labelled with from and to (uppercase state names). Heartbeat-timeout transitions show to="UNREACHABLE". |
ws_connections_active |
gauge | connections | (none) | Open WebSocket connections to the backend hub at sampling time (frontend + gateways). Sampled every 5 seconds by a background task. |
Call sites¶
If you need to trace a metric back to the code that emits it:
| Metric | Source |
|---|---|
agent_message_latency_ms |
crates/backend/src/websocket/mod.rs:999 |
agents_connected |
crates/backend/src/main.rs:457 |
commands_executed_total |
crates/backend/src/api/components.rs:1348 |
db_pool_connections |
crates/backend/src/db.rs:486, crates/backend/src/db.rs:487, crates/backend/src/db.rs:488 |
db_type_info |
crates/backend/src/db.rs:492 |
http_request_duration_seconds |
crates/backend/src/lib.rs:187 |
http_requests_total |
crates/backend/src/lib.rs:181 |
state_transitions_total |
crates/backend/src/core/fsm.rs:164, crates/backend/src/core/fsm.rs:281, crates/backend/src/core/heartbeat_monitor.rs:244 |
ws_connections_active |
crates/backend/src/main.rs:456 |
Recommended alerts¶
Three baseline alerts cover most real outages:
groups:
- name: appcontrol
rules:
- alert: BackendErrorRateHigh
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.02
for: 10m
annotations:
summary: ">2% 5xx on the backend for 10 minutes"
- alert: DbPoolSaturated
expr: db_pool_connections{state="total"} >= 18
for: 5m
annotations:
summary: "DB pool ≥90% of DB_POOL_SIZE for 5 minutes — increase pool or investigate slow queries"
- alert: AgentMessageLatencyHigh
expr: histogram_quantile(0.95, rate(agent_message_latency_ms_bucket[5m])) > 5000
for: 10m
annotations:
summary: "P95 agent → backend latency above 5s — check gateway, network, or backend CPU"
See also¶
- Check command metrics — the application-level metrics extracted from your
check_cmdstdout (queue depth, replication lag, etc.). These are stored incheck_events.metricsJSON and surfaced in the UI, separately from the Prometheus metrics documented on this page. - Capacity planning — sizing the backend based on metric load.