hetzner-cloud-cli

Hetzner Cloud CLI

Alex has a Hetzner Cloud project with hcloud CLI installed and a working token. Use it instead of asking him to provision things through the web console.

Setup state (as of Crawl4AI deploy)

• Binary: ~/.local/bin/hcloud (v1.55.0) — not on default PATH, use absolute path
• Token: HCLOUD_TOKEN in ~/.hermes/.env. Load with export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)
• SSH key in project: avalon-current-vps-astral (use this for --ssh-key)
• Default region: hil (Hillsboro, OR) — all existing servers are there

Common commands

export HOME=/home/avalon  # subagent envs sometimes lack this
export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)
HC=~/.local/bin/hcloud

$HC server list
$HC server-type list           # see available sizes & arch
$HC location list              # ID/name/network zone
$HC ssh-key list
$HC server create --name <n> --type cpx11 --image ubuntu-24.04 --location hil --ssh-key avalon-current-vps-astral --user-data-from-file /tmp/cloudinit.yaml
$HC server delete <name>
$HC server poweroff <name>
$HC server reboot <name>
$HC server describe <name>     # full info incl IPv4/IPv6

Pricing reference (monthly cap, hourly billed)

Gotchas

• ARM (cax*) not available in hil — only EU locations host ARM. Use cpx* for Hillsboro deploys.
• Names must be unique across the project — hermes-crawl4ai etc.
• Hetzner only bills hourly/monthly — destroying mid-month only charges hours used. No annual prepay exists.
• cloud-init is the cheap way to provision. Write a yaml file with packages/write_files/runcmd then pass --user-data-from-file. SSH in to debug with cat /var/log/cloud-init-output.log.
• After server create, wait ~2-5 min for cloud-init before testing services.

Expanding disk on an existing server — volumes (no downtime path)

When a VPS hits >95% root disk full, the cheapest fix is a Hetzner Volume (block storage), not a server resize. Volumes are ~$0.048/GB/mo, live-attached, resizable up, and survive server destroy.

Provision + attach in one command

# --server and --location are MUTUALLY EXCLUSIVE on volume create.
# When attaching to an existing server, OMIT --location — it inherits the server's location.
$HC volume create --name data-100 --size 100 --format ext4 --server ubuntu-8gb-hil-1 --automount
# Output: Volume 105790860 created

--automount writes a /etc/fstab entry via /dev/disk/by-id/scsi-0HC_Volume_<id> with nofail,defaults so a missing volume won't brick boot. Default mount path is /mnt/HC_Volume_<id> — ugly. Rename it:

sudo mkdir -p /data
sudo umount /mnt/HC_Volume_<id>
sudo sed -i "s|/mnt/HC_Volume_<id>|/data|" /etc/fstab
sudo mount -a
sudo chown avalon:avalon /data
df -h /data    # confirm ~98G usable on a 100GB volume

Move heavy dirs with symlink-back pattern (apps see no change)

move_dir() {
  local src="$1"; local name="$(basename "$src")"
  [ -L "$src" ] && { echo "skip: $src is symlink"; return; }
  mv "$src" "/data/$name"
  ln -s "/data/$name" "$src"
}
# Safe candidates: static asset vaults, wikis, build artifacts, cache dirs.
# Risky candidates: running app dirs (PM2 cwd) — possible but stop the app first.
move_dir /home/avalon/hermes-media-vault
move_dir /home/avalon/hyperframes

Migrating /var/lib/docker to a volume (BIG win, ~3 min downtime)

Docker data-root is usually the largest single dir on a VPS running containers. To shift it:

# 1. Stop the engine and its socket
sudo systemctl stop docker docker.socket containerd

# 2. rsync with --aHAX to preserve hardlinks/ACLs/xattrs (CRITICAL for docker)
sudo rsync -aHAX --info=progress2 /var/lib/docker/ /data/docker/

# 3. Verify sizes match (rough — docker compacts on next start)
sudo du -sh /var/lib/docker /data/docker

# 4. Move old aside, write daemon.json
sudo mv /var/lib/docker /var/lib/docker.OLD
sudo mkdir -p /etc/docker
echo '{"data-root": "/data/docker"}' | sudo tee /etc/docker/daemon.json

# 5. Start docker, verify
sudo systemctl start docker
sudo docker info | grep "Docker Root Dir"   # → /data/docker
sudo docker ps                              # auto-restart=always containers come back

Restart=no containers won't auto-start after daemon restart

Containers created without --restart unless-stopped (or always) stay Exited after systemctl restart docker. Manually docker start <name> each one. To make them resilient for next time:

sudo docker update --restart unless-stopped <container_name>

Pitfalls (volumes + data-root migration)

• --server and --location are mutually exclusive on volume create — error is hcloud: found mutually exclusive fields [Server, Location] in [hcloud.VolumeCreateOpts]. Drop --location when attaching to an existing server.
• Hetzner-generated /mnt/HC_Volume_<id> paths are ugly — rename via sed -i on /etc/fstab + remount before any app sees the path. Don't symlink /mnt/HC_Volume_<id> → /data, it complicates /etc/fstab parsing.
• rsync MUST use -aHAX for Docker — overlay2 relies on hardlinks across image layers. Plain cp -r or rsync -a will silently inflate disk usage 3-5x and may corrupt layer references.
• DON'T rm -rf /var/lib/docker.OLD until you've verified all containers restart cleanly — keep the old dir around as a fallback for one cycle.
• nofail in fstab is non-negotiable — if the volume detaches or is destroyed, the server still boots without it. Without nofail, you get an emergency-mode boot screen and need rescue mode to fix.
• Volumes can be resized up but NOT down — start smaller and grow than oversize. Resize via $HC volume resize <name> --size <new_gb> then sudo resize2fs /dev/disk/by-id/scsi-0HC_Volume_<id>.

Server-resize path (alternative — adds CPU/RAM but requires reboot)

When you need disk + CPU + RAM together, resize the server. Cost-effective tiers from ccx13 (8GB / 80GB / $20):

$HC server poweroff ubuntu-8gb-hil-1
$HC server change-type ubuntu-8gb-hil-1 ccx23 --upgrade-disk
$HC server poweron ubuntu-8gb-hil-1

--upgrade-disk is irreversible — Hetzner won't let you shrink disk on downgrade later, you'd have to rebuild. Volumes don't have this problem.

Existing servers (do not destroy)

• ubuntu-8gb-hil-1 5.78.190.92 — main poofc.com VPS (Hermes Agent host)
• ubuntu-2gb-hil-1 5.78.199.81
• astral-node-1 5.78.199.26 — Astral
• hermes-crawl4ai 5.78.214.66 — Crawl4AI service on :11235 (CPX11 €4.35/mo)

Server access

Use ssh root@<ip> from the main VPS — the avalon-current-vps-astral SSH key is wired up for that login.