Skip to content
Salitech Docs

Runbook: Server Down (production-01)

Runbook: Server Down (production-01)

Section titled “Runbook: Server Down (production-01)”

Trigger: Host production-01 (178.105.204.32 / salesnow-prod) fails network ping, SSH connectivity, or edge monitoring checks.

Step 1: Verify Outage Scope

Section titled “Step 1: Verify Outage Scope”
  1. Test ping to 178.105.204.32.
  2. Check Cloudflare edge status for automation.salitech.de and analytics.salitech.de.
  3. Check Hetzner Cloud Console status / maintenance notices.

Step 2: Attempt Management Console Access

Section titled “Step 2: Attempt Management Console Access”
  1. Log into Hetzner Cloud Console.
  2. Locate server salesnow-prod (production-01).
  3. Open the VNC / Web Console.
  4. Check kernel dmesg or OOM-killer logs:
    Terminal window
    dmesg -T | tail -n 50

Step 3: Service & Docker Recovery

Section titled “Step 3: Service & Docker Recovery”

Once host terminal access is restored:

  1. Verify Docker Daemon:

    Terminal window
    systemctl status docker
    sudo systemctl restart docker

  2. Verify Reverse Proxy (Caddy):

    Terminal window
    systemctl status caddy
    sudo systemctl restart caddy

  3. Check Container Health:

    Terminal window
    cd /opt/n8n && docker compose ps
    cd /opt/umami && docker compose ps

  4. Bring up failed stacks:

    Terminal window
    cd /opt/n8n && docker compose up -d
    cd /opt/umami && docker compose up -d

Step 4: Verification

Section titled “Step 4: Verification”
  • Verify https://automation.salitech.de returns HTTP 200.
  • Verify https://analytics.salitech.de returns HTTP 200.