Operate Forge Projects

TL;DR

Treat project state as a set of related signals rather than a single running flag. A process can be running while its runtime is degraded. Use the escalation order when diagnosing unhealthy projects.

Inspect inventory

aurum list --server prod
aurum status --server prod
aurum status signal-watch --server prod

status distinguishes installation, desired lifecycle state, physical process state, runtime availability, and health. Each signal answers a different question.

Read logs

aurum logs signal-watch --server prod
aurum logs signal-watch --server prod --tail
aurum logs signal-watch --server prod --journal

--tail follows the project runtime log. --journal selects the managed service journal instead — useful when the project fails during startup before the runtime log is available.

Stop a version

aurum stop signal-watch --server prod --version 0.0.1 --wait

Use --timeout-ms when automation needs a bounded convergence wait. A successful command submission is not the same as a converged process state; verify with status.

Operate in Terminal

Terminal’s Projects view provides search, state filters, refresh, start, stop, and restart actions. Use the project Summary for product metrics and Inspect for live resource-level diagnosis.

Tip

Terminal and the CLI show the same state signals — they use the same Forge API. If a state change made in Terminal is not visible in the CLI, it has not converged yet. Poll until the signals agree.

Escalation order

When a project is unhealthy, follow this order. Each step eliminates a class of failure before moving to the next:

1. Check Forge /health and /doctor
   → Proves the API process is alive and can reach aurumd.
   → If /health fails: API process is down. Check the Forge service.
   → If /doctor returns 502: aurumd is unreachable. Check the daemon.

2. Inspect the project's desired and process states
   → Proves aurumd can manage the project's OS process.
   → If desired != actual: a lifecycle command may not have converged.
   → If process is dead: check managed service logs (--journal).

3. Confirm the runtime endpoint and capabilities
   → Proves the Engine is accepting connections.
   → If runtime is unreachable: check project runtime logs (--tail).
   → If a capability is false: check if this is the correct contract.

4. Read project logs, then managed journal logs
   → Runtime logs show application behavior.
   → Journal logs show process startup and OS-level failures.

5. Check resource freshness and ClickHouse history separately
   → Live data may be flowing while history is behind.
   → History may be available while live data is stopped.

Note

Most incidents are resolved at step 1 or 2. If you reach step 4, the problem is likely in the strategy code or the venue connection, not in the Forge infrastructure. The distinction matters because the debugging path is different.

See Troubleshooting for symptom-based checks.