FSM reference¶
Auto-generated from
crates/common/src/fsm.rsandcrates/common/src/types.rs. Runscripts/docs/regen.pyto refresh.
AppControl drives every component through a deterministic finite-state machine. The same logic runs in the backend (authoritative) and in the agent (for local optimisation). A transition is only persisted when is_valid_transition(from, to) returns true; invalid transitions are dropped and logged at warn level — they never produce a row in state_transitions.
States¶
| State | Meaning |
|---|---|
UNKNOWN |
The component has been registered but no health check has reported back yet. The next check determines the state. |
RUNNING |
Latest health check returned exit 0. The component is considered operational. |
DEGRADED |
Reserved state for business-logic-defined partial health. Not currently entered by next_state_from_check; reachable only via explicit FSM driver code. |
FAILED |
Latest health check returned a non-zero exit code while the component was RUNNING or DEGRADED, or the sequencer's start timeout elapsed before the process came up. |
STOPPED |
Stop sequence completed successfully — the agent confirmed the process is gone (check returns non-zero). Also the initial state for components on a freshly enrolled agent. |
STARTING |
Start command has been dispatched. The component stays in STARTING until the next successful health check (→ RUNNING) or the sequencer timeout (→ FAILED). Non-zero check exits during STARTING do not transition: the start command is detached and may take time. |
STOPPING |
Stop command has been dispatched. The component stays in STOPPING until the agent confirms the process is gone (→ STOPPED). A zero exit during STOPPING keeps the component in STOPPING. |
UNREACHABLE |
The agent hosting the component has missed heartbeat_timeout_seconds (default 180s, configurable per organisation). All in-flight states (RUNNING / DEGRADED / STARTING) collapse to UNREACHABLE. STOPPED and STOPPING are preserved because they are intentional. |
Transition diagram¶
Edges are valid from → to pairs accepted by is_valid_transition. The Any → UNREACHABLE and UNREACHABLE → Any rules are expanded explicitly so the diagram is self-contained.
stateDiagram-v2
Unknown : UNKNOWN
Running : RUNNING
Degraded : DEGRADED
Failed : FAILED
Stopped : STOPPED
Starting : STARTING
Stopping : STOPPING
Unreachable : UNREACHABLE
Unknown --> Unreachable
Running --> Unreachable
Degraded --> Unreachable
Failed --> Unreachable
Stopped --> Unreachable
Starting --> Unreachable
Stopping --> Unreachable
Unreachable --> Unknown
Unreachable --> Running
Unreachable --> Degraded
Unreachable --> Failed
Unreachable --> Stopped
Unreachable --> Starting
Unreachable --> Stopping
Total accepted transitions: 14.
Transition matrix¶
✓ = valid; · = rejected by is_valid_transition; — = identity (not a transition).
| from to | UNKNOWN |
RUNNING |
DEGRADED |
FAILED |
STOPPED |
STARTING |
STOPPING |
UNREACHABLE |
|---|---|---|---|---|---|---|---|---|
UNKNOWN |
— | · | · | · | · | · | · | ✓ |
RUNNING |
· | — | · | · | · | · | · | ✓ |
DEGRADED |
· | · | — | · | · | · | · | ✓ |
FAILED |
· | · | · | — | · | · | · | ✓ |
STOPPED |
· | · | · | · | — | · | · | ✓ |
STARTING |
· | · | · | · | · | — | · | ✓ |
STOPPING |
· | · | · | · | · | · | — | ✓ |
UNREACHABLE |
✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — |
Check-result decision table¶
next_state_from_check(current, exit_code) is the function the agent invokes after every health check. It returns the new state (if any) before is_valid_transition validates it — the validation pass is what catches any logic error in the decision table.
| Current | Exit code | Result | Notes |
|---|---|---|---|
UNKNOWN |
0 | → RUNNING |
Unknown: first check determines state |
UNKNOWN |
any non-zero | → FAILED |
|
STARTING |
0 | → RUNNING |
Starting: check OK → Running, check KO → stay in Starting (wait for timeout) The start command is async/detached, so the process may take time to start. Failed health checks during Starting are expect |
STARTING |
any non-zero | (no transition) | |
RUNNING |
0 | (no transition) | Stay in Starting, let sequencer handle timeout Running: exit 0 = stay, exit != 0 = Failed |
RUNNING |
any non-zero | → FAILED |
|
DEGRADED |
0 | → RUNNING |
Degraded: exit 0 = Running, exit != 0 = Failed (Degraded state is reserved for future business logic, not exit codes) |
DEGRADED |
any non-zero | → FAILED |
|
STOPPING |
0 | (no transition) | Stopping: check KO (process gone) → Stopped, check OK = stay in Stopping |
STOPPING |
any non-zero | → STOPPED |
|
FAILED |
0 | → RUNNING |
Failed: exit 0 = Running (auto-recovery), exit != 0 = stay Failed |
FAILED |
any non-zero | (no transition) | |
UNREACHABLE |
0 | → RUNNING |
Unreachable: if agent comes back and check passes, restore to Running |
UNREACHABLE |
any non-zero | → FAILED |
|
STOPPED |
0 | → RUNNING |
Stopped: exit 0 = Running (external start detected), exit != 0 = stay Stopped |
STOPPED |
any non-zero | (no transition) |
Heartbeat timeout¶
The backend's heartbeat_monitor task runs every 30 seconds. It finds agents whose last_heartbeat_at is older than organizations.heartbeat_timeout_seconds (default 180s) and transitions every active component on that agent to UNREACHABLE (except STOPPED and STOPPING, which are preserved). The trigger is recorded as heartbeat_timeout in state_transitions.trigger, and the previous state is preserved in state_transitions.details->>'previous_state' so the original state can be restored when the agent reconnects.
Where to look next¶
- Component state in the UI — colour coding
- Sequencer behaviour — how starts and stops drive transitions
- Runbooks — when a component is stuck in a state
- Troubleshooting — diagnose oscillation, stale UNREACHABLE