Skip to content

FSM reference

Auto-generated from crates/common/src/fsm.rs and crates/common/src/types.rs. Run scripts/docs/regen.py to refresh.

AppControl drives every component through a deterministic finite-state machine. The same logic runs in the backend (authoritative) and in the agent (for local optimisation). A transition is only persisted when is_valid_transition(from, to) returns true; invalid transitions are dropped and logged at warn level — they never produce a row in state_transitions.

States

State Meaning
UNKNOWN The component has been registered but no health check has reported back yet. The next check determines the state.
RUNNING Latest health check returned exit 0. The component is considered operational.
DEGRADED Reserved state for business-logic-defined partial health. Not currently entered by next_state_from_check; reachable only via explicit FSM driver code.
FAILED Latest health check returned a non-zero exit code while the component was RUNNING or DEGRADED, or the sequencer's start timeout elapsed before the process came up.
STOPPED Stop sequence completed successfully — the agent confirmed the process is gone (check returns non-zero). Also the initial state for components on a freshly enrolled agent.
STARTING Start command has been dispatched. The component stays in STARTING until the next successful health check (→ RUNNING) or the sequencer timeout (→ FAILED). Non-zero check exits during STARTING do not transition: the start command is detached and may take time.
STOPPING Stop command has been dispatched. The component stays in STOPPING until the agent confirms the process is gone (→ STOPPED). A zero exit during STOPPING keeps the component in STOPPING.
UNREACHABLE The agent hosting the component has missed heartbeat_timeout_seconds (default 180s, configurable per organisation). All in-flight states (RUNNING / DEGRADED / STARTING) collapse to UNREACHABLE. STOPPED and STOPPING are preserved because they are intentional.

Transition diagram

Edges are valid from → to pairs accepted by is_valid_transition. The Any → UNREACHABLE and UNREACHABLE → Any rules are expanded explicitly so the diagram is self-contained.

stateDiagram-v2
    Unknown : UNKNOWN
    Running : RUNNING
    Degraded : DEGRADED
    Failed : FAILED
    Stopped : STOPPED
    Starting : STARTING
    Stopping : STOPPING
    Unreachable : UNREACHABLE
    Unknown --> Unreachable
    Running --> Unreachable
    Degraded --> Unreachable
    Failed --> Unreachable
    Stopped --> Unreachable
    Starting --> Unreachable
    Stopping --> Unreachable
    Unreachable --> Unknown
    Unreachable --> Running
    Unreachable --> Degraded
    Unreachable --> Failed
    Unreachable --> Stopped
    Unreachable --> Starting
    Unreachable --> Stopping

Total accepted transitions: 14.

Transition matrix

✓ = valid; · = rejected by is_valid_transition; — = identity (not a transition).

from to UNKNOWN RUNNING DEGRADED FAILED STOPPED STARTING STOPPING UNREACHABLE
UNKNOWN — · · · · · · ✓
RUNNING · — · · · · · ✓
DEGRADED · · — · · · · ✓
FAILED · · · — · · · ✓
STOPPED · · · · — · · ✓
STARTING · · · · · — · ✓
STOPPING · · · · · · — ✓
UNREACHABLE ✓ ✓ ✓ ✓ ✓ ✓ ✓ —

Check-result decision table

next_state_from_check(current, exit_code) is the function the agent invokes after every health check. It returns the new state (if any) before is_valid_transition validates it — the validation pass is what catches any logic error in the decision table.

Current Exit code Result Notes
UNKNOWN 0 → RUNNING Unknown: first check determines state
UNKNOWN any non-zero → FAILED
STARTING 0 → RUNNING Starting: check OK → Running, check KO → stay in Starting (wait for timeout) The start command is async/detached, so the process may take time to start. Failed health checks during Starting are expect
STARTING any non-zero (no transition)
RUNNING 0 (no transition) Stay in Starting, let sequencer handle timeout Running: exit 0 = stay, exit != 0 = Failed
RUNNING any non-zero → FAILED
DEGRADED 0 → RUNNING Degraded: exit 0 = Running, exit != 0 = Failed (Degraded state is reserved for future business logic, not exit codes)
DEGRADED any non-zero → FAILED
STOPPING 0 (no transition) Stopping: check KO (process gone) → Stopped, check OK = stay in Stopping
STOPPING any non-zero → STOPPED
FAILED 0 → RUNNING Failed: exit 0 = Running (auto-recovery), exit != 0 = stay Failed
FAILED any non-zero (no transition)
UNREACHABLE 0 → RUNNING Unreachable: if agent comes back and check passes, restore to Running
UNREACHABLE any non-zero → FAILED
STOPPED 0 → RUNNING Stopped: exit 0 = Running (external start detected), exit != 0 = stay Stopped
STOPPED any non-zero (no transition)

Heartbeat timeout

The backend's heartbeat_monitor task runs every 30 seconds. It finds agents whose last_heartbeat_at is older than organizations.heartbeat_timeout_seconds (default 180s) and transitions every active component on that agent to UNREACHABLE (except STOPPED and STOPPING, which are preserved). The trigger is recorded as heartbeat_timeout in state_transitions.trigger, and the previous state is preserved in state_transitions.details->>'previous_state' so the original state can be restored when the agent reconnects.

Where to look next