Provisioning Debugger

Systematically diagnose why an agent provisioning failed or is stuck.

Diagnostic steps

Run these checks in order, stopping at the first failure found:

1. Check agent status in database

# Get the agent record — look for status != 'active'
docker compose exec -T postgres psql -U postgres -d livia -c \
  "SELECT id, \"serverName\", status, ip, domain, \"provisionedAt\", \"updatedAt\" FROM \"Agent\" ORDER BY \"updatedAt\" DESC LIMIT 5;"

2. Check provisioner logs

Prod: Railway dashboard → provisioner service → logs. Local: the provisioner runs via pnpm dev (no compose service) — read its terminal output.

Look for:

• hcloud API errors (rate limit, invalid token, quota exceeded)
• Ansible playbook failures (task name + error message)
• Callback errors to NestJS API

3. Check API logs for provisioning events

docker compose logs api --tail=100 --no-log-prefix | grep -i "provis"

4. Check Hetzner server state

# Requires HCLOUD_TOKEN in environment
hcloud server list -o columns=name,status,ipv4,created

5. SSH into the agent (if server exists)

# Get IP from step 1 or 4, then:
ssh -i ~/.ssh/livia_deployer -o ConnectTimeout=5 -o StrictHostKeyChecking=no root@<IP> \
  "systemctl status openclaw caddy && docker ps"

6. Check OpenClaw service on agent

ssh -i ~/.ssh/livia_deployer root@<IP> "journalctl -u openclaw --no-pager -n 50"

7. Check Caddy (reverse proxy) on agent

ssh -i ~/.ssh/livia_deployer root@<IP> "journalctl -u caddy --no-pager -n 30"

Common failure patterns

Symptom	Likely cause	Fix
Status stuck on provisioning	Ansible playbook timeout	Check provisioner logs for task failure, re-run playbook
Status error	hcloud or Ansible failure	Check provisioner logs, fix issue, retry via API
Server exists but pending	Callback to API failed	Check API logs, manually update status
SSH timeout	Firewall or server not booted	Check hcloud console, verify security group
OpenClaw not running	Config error in openclaw.json	Check journalctl -u openclaw, fix config, restart
Caddy 502	OpenClaw not listening	Verify OpenClaw port, check Caddyfile upstream

After diagnosis

Report:

1. Root cause: What failed and why
2. Current state: What's running, what's not
3. Fix: Concrete steps to resolve
4. Prevention: What could prevent this in the future