---
name: hetzner-cloud-cli
description: "Provision and manage Hetzner Cloud VPS servers via the hcloud CLI."
version: 1.0.0
author: Alex + Hermes
license: MIT
metadata:
  hermes:
    tags: [hetzner, vps, hcloud, provisioning, devops]
---

# Hetzner Cloud CLI

Alex has a Hetzner Cloud project with `hcloud` CLI installed and a working token. Use it instead of asking him to provision things through the web console.

## Setup state (as of Crawl4AI deploy)

- **Binary:** `~/.local/bin/hcloud` (v1.55.0) — not on default PATH, use absolute path
- **Token:** `HCLOUD_TOKEN` in `~/.hermes/.env`. Load with `export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)`
- **SSH key in project:** `avalon-current-vps-astral` (use this for `--ssh-key`)
- **Default region:** `hil` (Hillsboro, OR) — all existing servers are there

## Common commands

```bash
export HOME=/home/avalon  # subagent envs sometimes lack this
export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)
HC=~/.local/bin/hcloud

$HC server list
$HC server-type list           # see available sizes & arch
$HC location list              # ID/name/network zone
$HC ssh-key list
$HC server create --name <n> --type cpx11 --image ubuntu-24.04 --location hil --ssh-key avalon-current-vps-astral --user-data-from-file /tmp/cloudinit.yaml
$HC server delete <name>
$HC server poweroff <name>
$HC server reboot <name>
$HC server describe <name>     # full info incl IPv4/IPv6
```

## Pricing reference (monthly cap, hourly billed)

| Type | Specs | Price/mo |
|---|---|---|
| cpx11 | 2 vCPU x86, 2GB, 40GB | ~€4.35 |
| cpx21 | 3 vCPU x86, 4GB, 80GB | ~€7.55 |
| cax11 | 2 vCPU ARM, 4GB, 40GB | ~€4.51 |
| cax21 | 4 vCPU ARM, 8GB, 80GB | ~€8 |

## Gotchas

- **ARM (cax*) not available in `hil`** — only EU locations host ARM. Use `cpx*` for Hillsboro deploys.
- **Names must be unique** across the project — `hermes-crawl4ai` etc.
- **Hetzner only bills hourly/monthly** — destroying mid-month only charges hours used. No annual prepay exists.
- **cloud-init** is the cheap way to provision. Write a yaml file with packages/write_files/runcmd then pass `--user-data-from-file`. SSH in to debug with `cat /var/log/cloud-init-output.log`.
- After server create, wait ~2-5 min for cloud-init before testing services.

## Expanding disk on an existing server — volumes (no downtime path)

When a VPS hits >95% root disk full, the cheapest fix is a Hetzner Volume (block storage), not a server resize. Volumes are ~$0.048/GB/mo, live-attached, resizable up, and survive server destroy.

### Provision + attach in one command

```bash
# --server and --location are MUTUALLY EXCLUSIVE on volume create.
# When attaching to an existing server, OMIT --location — it inherits the server's location.
$HC volume create --name data-100 --size 100 --format ext4 --server ubuntu-8gb-hil-1 --automount
# Output: Volume 105790860 created
```

`--automount` writes a `/etc/fstab` entry via `/dev/disk/by-id/scsi-0HC_Volume_<id>` with `nofail,defaults` so a missing volume won't brick boot. Default mount path is `/mnt/HC_Volume_<id>` — ugly. Rename it:

```bash
sudo mkdir -p /data
sudo umount /mnt/HC_Volume_<id>
sudo sed -i "s|/mnt/HC_Volume_<id>|/data|" /etc/fstab
sudo mount -a
sudo chown avalon:avalon /data
df -h /data    # confirm ~98G usable on a 100GB volume
```

### Move heavy dirs with symlink-back pattern (apps see no change)

```bash
move_dir() {
  local src="$1"; local name="$(basename "$src")"
  [ -L "$src" ] && { echo "skip: $src is symlink"; return; }
  mv "$src" "/data/$name"
  ln -s "/data/$name" "$src"
}
# Safe candidates: static asset vaults, wikis, build artifacts, cache dirs.
# Risky candidates: running app dirs (PM2 cwd) — possible but stop the app first.
move_dir /home/avalon/hermes-media-vault
move_dir /home/avalon/hyperframes
```

### Migrating `/var/lib/docker` to a volume (BIG win, ~3 min downtime)

Docker data-root is usually the largest single dir on a VPS running containers. To shift it:

```bash
# 1. Stop the engine and its socket
sudo systemctl stop docker docker.socket containerd

# 2. rsync with --aHAX to preserve hardlinks/ACLs/xattrs (CRITICAL for docker)
sudo rsync -aHAX --info=progress2 /var/lib/docker/ /data/docker/

# 3. Verify sizes match (rough — docker compacts on next start)
sudo du -sh /var/lib/docker /data/docker

# 4. Move old aside, write daemon.json
sudo mv /var/lib/docker /var/lib/docker.OLD
sudo mkdir -p /etc/docker
echo '{"data-root": "/data/docker"}' | sudo tee /etc/docker/daemon.json

# 5. Start docker, verify
sudo systemctl start docker
sudo docker info | grep "Docker Root Dir"   # → /data/docker
sudo docker ps                              # auto-restart=always containers come back
```

### Restart=no containers won't auto-start after daemon restart

Containers created without `--restart unless-stopped` (or `always`) stay Exited after `systemctl restart docker`. Manually `docker start <name>` each one. To make them resilient for next time:

```bash
sudo docker update --restart unless-stopped <container_name>
```

### Pitfalls (volumes + data-root migration)

- **`--server` and `--location` are mutually exclusive on `volume create`** — error is `hcloud: found mutually exclusive fields [Server, Location] in [hcloud.VolumeCreateOpts]`. Drop `--location` when attaching to an existing server.
- **Hetzner-generated `/mnt/HC_Volume_<id>` paths are ugly** — rename via `sed -i` on `/etc/fstab` + remount before any app sees the path. Don't symlink `/mnt/HC_Volume_<id>` → `/data`, it complicates `/etc/fstab` parsing.
- **rsync MUST use `-aHAX`** for Docker — overlay2 relies on hardlinks across image layers. Plain `cp -r` or `rsync -a` will silently inflate disk usage 3-5x and may corrupt layer references.
- **DON'T `rm -rf /var/lib/docker.OLD` until you've verified all containers restart cleanly** — keep the old dir around as a fallback for one cycle.
- **`nofail` in fstab is non-negotiable** — if the volume detaches or is destroyed, the server still boots without it. Without `nofail`, you get an emergency-mode boot screen and need rescue mode to fix.
- **Volumes can be resized up but NOT down** — start smaller and grow than oversize. Resize via `$HC volume resize <name> --size <new_gb>` then `sudo resize2fs /dev/disk/by-id/scsi-0HC_Volume_<id>`.

### Server-resize path (alternative — adds CPU/RAM but requires reboot)

When you need disk + CPU + RAM together, resize the server. Cost-effective tiers from `ccx13` (8GB / 80GB / $20):

| Type | vCPU | RAM | Disk | $/mo (hil) |
|---|---|---|---|---|
| cpx32 (shared) | 4 | 8GB | 160GB | ~$18 — cheaper but loses dedicated CPU |
| ccx23 (dedicated) | 4 | 16GB | 160GB | ~$40 — clean 2x upgrade |
| ccx33 (dedicated) | 8 | 32GB | 240GB | ~$80 |

```bash
$HC server poweroff ubuntu-8gb-hil-1
$HC server change-type ubuntu-8gb-hil-1 ccx23 --upgrade-disk
$HC server poweron ubuntu-8gb-hil-1
```

`--upgrade-disk` is irreversible — Hetzner won't let you shrink disk on downgrade later, you'd have to rebuild. Volumes don't have this problem.

## Existing servers (do not destroy)

- `ubuntu-8gb-hil-1` 5.78.190.92 — main poofc.com VPS (Hermes Agent host)
- `ubuntu-2gb-hil-1` 5.78.199.81
- `astral-node-1` 5.78.199.26 — Astral
- `hermes-crawl4ai` 5.78.214.66 — Crawl4AI service on :11235 (CPX11 €4.35/mo)

## Server access

Use `ssh root@<ip>` from the main VPS — the `avalon-current-vps-astral` SSH key is wired up for that login.