Skip to content

Metrics reference

Auto-generated by scanning crates/**/src/**/*.rs for metrics::{counter,gauge,histogram}! invocations. Run scripts/docs/regen.py to refresh.

The backend exposes Prometheus metrics at GET /metrics. The endpoint requires no authentication by default; restrict it at the load-balancer or reverse-proxy layer if you do not want it scraped from the public network.

# Local check
curl -s http://localhost:3000/metrics | head -40

# Prometheus scrape config snippet
# (see monitoring/prometheus/prometheus.yml for the full file)
scrape_configs:
  - job_name: appcontrol-backend
    metrics_path: /metrics
    static_configs:
      - targets: ['backend:3000']

Exposed metrics

Name Kind Unit Labels Description
agent_message_latency_ms histogram milliseconds message_type End-to-end latency between an agent emitting a message and the backend processing it (the message's at timestamp minus server-side receive time). Labelled by message_type.
agents_connected gauge agents (none) Number of unique agents currently routed through the WebSocket hub. Diverges from the total agent count when an agent is registered in the DB but disconnected.
commands_executed_total counter commands command Count of custom commands dispatched to agents via POST /components/:id/command/:cmd. Labelled by command name.
db_pool_connections gauge connections state Postgres / SQLite connection-pool occupancy. Label state is one of idle, active, total — alert on total ≥ DB_POOL_SIZE for sustained periods.
db_type_info gauge info type Constant gauge (= 1) labelled with type=postgres or type=sqlite, exposing the compile-time database backend.
http_request_duration_seconds histogram seconds method, path Histogram of HTTP request latency. Use the standard _bucket series for P50/P95/P99 quantiles; _sum / _count gives the mean.
http_requests_total counter requests method Count of HTTP requests handled by the backend, labelled by HTTP method and response status code. Use this to compute request rate and the error ratio (rate by status=~"5..").
state_transitions_total counter transitions from Count of FSM transitions persisted to state_transitions. Labelled with from and to (uppercase state names). Heartbeat-timeout transitions show to="UNREACHABLE".
ws_connections_active gauge connections (none) Open WebSocket connections to the backend hub at sampling time (frontend + gateways). Sampled every 5 seconds by a background task.

Call sites

If you need to trace a metric back to the code that emits it:

Metric Source
agent_message_latency_ms crates/backend/src/websocket/mod.rs:999
agents_connected crates/backend/src/main.rs:457
commands_executed_total crates/backend/src/api/components.rs:1348
db_pool_connections crates/backend/src/db.rs:486, crates/backend/src/db.rs:487, crates/backend/src/db.rs:488
db_type_info crates/backend/src/db.rs:492
http_request_duration_seconds crates/backend/src/lib.rs:187
http_requests_total crates/backend/src/lib.rs:181
state_transitions_total crates/backend/src/core/fsm.rs:164, crates/backend/src/core/fsm.rs:281, crates/backend/src/core/heartbeat_monitor.rs:244
ws_connections_active crates/backend/src/main.rs:456

Three baseline alerts cover most real outages:

groups:
  - name: appcontrol
    rules:
      - alert: BackendErrorRateHigh
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.02
        for: 10m
        annotations:
          summary: ">2% 5xx on the backend for 10 minutes"

      - alert: DbPoolSaturated
        expr: db_pool_connections{state="total"} >= 18
        for: 5m
        annotations:
          summary: "DB pool ≥90% of DB_POOL_SIZE for 5 minutes — increase pool or investigate slow queries"

      - alert: AgentMessageLatencyHigh
        expr: histogram_quantile(0.95, rate(agent_message_latency_ms_bucket[5m])) > 5000
        for: 10m
        annotations:
          summary: "P95 agent → backend latency above 5s — check gateway, network, or backend CPU"

See also

  • Check command metrics — the application-level metrics extracted from your check_cmd stdout (queue depth, replication lag, etc.). These are stored in check_events.metrics JSON and surfaced in the UI, separately from the Prometheus metrics documented on this page.
  • Capacity planning — sizing the backend based on metric load.