hetzner-cloud-cli

Hetzner Cloud CLI

Alex has a Hetzner Cloud project with hcloud CLI installed and a working token. Use it instead of asking him to provision things through the web console.

Setup state (as of Crawl4AI deploy)

• Binary: ~/.local/bin/hcloud (v1.55.0) — not on default PATH, use absolute path
• Token: HCLOUD_TOKEN in ~/.hermes/.env. Load with export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)
• SSH key in project: avalon-current-vps-astral (use this for --ssh-key)
• Default region: hil (Hillsboro, OR) — all existing servers are there

Common commands

export HOME=/home/avalon  # subagent envs sometimes lack this
export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)
HC=~/.local/bin/hcloud

$HC server list
$HC server-type list           # see available sizes & arch
$HC location list              # ID/name/network zone
$HC ssh-key list
$HC server create --name <n> --type cpx11 --image ubuntu-24.04 --location hil --ssh-key avalon-current-vps-astral --user-data-from-file /tmp/cloudinit.yaml
$HC server delete <name>
$HC server poweroff <name>
$HC server reboot <name>
$HC server describe <name>     # full info incl IPv4/IPv6

Pricing reference (monthly cap, hourly billed)

Always verify live with $HC server-type describe <type> -o json: Hetzner US locations (hil, ash) are much more expensive than EU locations for shared CPU cpx* types. Dedicated CPU ccx* types may have little/no region spread, so do not assume moving every server saves money.

See references/hetzner-us-to-eu-migration-notes.md for the Hetzner-first overpayment workflow: compare exact type pricing, remember there is no in-place datacenter swap, recreate in fsn1/nbg1/hel1, migrate data/config, update DNS/firewalls, smoke test, then delete the US source.

See references/freemix-cost-storage-migration.md for the Freemix-derived cost/storage accounting pattern: verify server types, attached volumes, inside-guest mounts, volume pricing, temporary overlap cost, and whether the current EU topology is migration-safe versus final cost-optimized.

See references/freemix-eu-volume-cutover-runbook.md for the follow-on pattern when a large app was staged onto an oversized EU root disk due to volume quota, and you need to prove live storage parity, delete/free the old volume quota if approved, create a new EU volume, and mount it at the live storage path with minimal downtime.

Gotchas

• No in-place datacenter swap for servers — hcloud server change-type changes size only, and hcloud server rebuild keeps the same location. To leave expensive hil/ash pricing, create a replacement server in the target EU location (fsn1, nbg1, or hel1), migrate data/config, update DNS/IP allowlists/firewalls, smoke test, then delete the old server.
• Volumes are location-bound — a hil volume cannot simply attach to an EU server. Plan rsync/snapshot/object-storage migration for large volumes before promising savings, especially 900GB+ data volumes.
• Project volume-size limits can block safe side-by-side migration — if creating an equal EU replacement volume fails with volumes size limit exceeded, do not delete the old volume just to free quota. Use references/large-volume-migration-staging.md: create a temporary EU server with enough root disk for the actually-used data, rsync/verify there while the old volume remains rollback, then delete/recreate final storage only after app + data + local agents pass smokes.
• Freemix-style cost/storage migrations need live-data verification before deleting rollback volumes — see references/freemix-cost-storage-migration.md for the accounting pattern, quota-increase path, old-volume live-vs-recovery breakdown, and rsync dry-run/checksum verification commands before freeing quota by deleting old storage.
• Primary IPs are location/datacenter-bound and usually auto-delete — unless Floating IPs were planned, cross-location migration means new IPs and all DNS/service/firewall allowlists must be updated.
• ARM (cax*) not available in hil — only EU locations host ARM. Use cpx* for Hillsboro deploys.
• Names must be unique across the project — hermes-crawl4ai etc.
• Hetzner only bills hourly/monthly — destroying mid-month only charges hours used. No annual prepay exists.
• There is no in-place datacenter/location swap for existing servers. server change-type changes size only, server rebuild stays in the same location, and server create --location <fsn1|nbg1|hel1|...> is the path for a different datacenter. Treat US→EU cost reductions as replacement-server migrations: create the new server in the cheap location, copy data/config, update DNS/firewalls, smoke test, then delete the old one. For nginx/DNS cutover details, see vps-app-deployment → references/hetzner-eu-migration-dns-cutover.md.
• Primary IPs and Volumes are location/datacenter-bound. Cross-location migrations get new server IPs unless a suitable floating/primary-IP strategy already exists; attached volumes must be recreated/copied in the target location. Do not assume data-100 or a large app volume can simply attach to an EU server.
• cloud-init is the cheap way to provision. Write a yaml file with packages/write_files/runcmd then pass --user-data-from-file. SSH in to debug with cat /var/log/cloud-init-output.log.
• After server create, wait ~2-5 min for cloud-init before testing services.

Expanding disk on an existing server — volumes (no downtime path)

When a VPS hits >95% root disk full, the cheapest fix is a Hetzner Volume (block storage), not a server resize. Volumes are ~$0.048/GB/mo, live-attached, resizable up, and survive server destroy.

Provision + attach in one command

# --server and --location are MUTUALLY EXCLUSIVE on volume create.
# When attaching to an existing server, OMIT --location — it inherits the server's location.
$HC volume create --name data-100 --size 100 --format ext4 --server ubuntu-8gb-hil-1 --automount
# Output: Volume 105790860 created

--automount writes a /etc/fstab entry via /dev/disk/by-id/scsi-0HC_Volume_<id> with nofail,defaults so a missing volume won't brick boot. Default mount path is /mnt/HC_Volume_<id> — ugly. Rename it:

sudo mkdir -p /data
sudo umount /mnt/HC_Volume_<id>
sudo sed -i "s|/mnt/HC_Volume_<id>|/data|" /etc/fstab
sudo mount -a
sudo chown avalon:avalon /data
df -h /data    # confirm ~98G usable on a 100GB volume

Move heavy dirs with symlink-back pattern (apps see no change)

move_dir() {
  local src="$1"; local name="$(basename "$src")"
  [ -L "$src" ] && { echo "skip: $src is symlink"; return; }
  mv "$src" "/data/$name"
  ln -s "/data/$name" "$src"
}
# Safe candidates: static asset vaults, wikis, build artifacts, cache dirs.
# Risky candidates: running app dirs (PM2 cwd) — possible but stop the app first.
move_dir /home/avalon/hermes-media-vault
move_dir /home/avalon/hyperframes

Migrating /var/lib/docker to a volume (BIG win, ~3 min downtime)

Docker data-root is usually the largest single dir on a VPS running containers. To shift it:

# 1. Stop the engine and its socket
sudo systemctl stop docker docker.socket containerd

# 2. rsync with --aHAX to preserve hardlinks/ACLs/xattrs (CRITICAL for docker)
sudo rsync -aHAX --info=progress2 /var/lib/docker/ /data/docker/

# 3. Verify sizes match (rough — docker compacts on next start)
sudo du -sh /var/lib/docker /data/docker

# 4. Move old aside, write daemon.json
sudo mv /var/lib/docker /var/lib/docker.OLD
sudo mkdir -p /etc/docker
echo '{"data-root": "/data/docker"}' | sudo tee /etc/docker/daemon.json

# 5. Start docker, verify
sudo systemctl start docker
sudo docker info | grep "Docker Root Dir"   # → /data/docker
sudo docker ps                              # auto-restart=always containers come back

Restart=no containers won't auto-start after daemon restart

Containers created without --restart unless-stopped (or always) stay Exited after systemctl restart docker. Manually docker start <name> each one. To make them resilient for next time:

sudo docker update --restart unless-stopped <container_name>

Pitfalls (volumes + data-root migration)

• --server and --location are mutually exclusive on volume create — error is hcloud: found mutually exclusive fields [Server, Location] in [hcloud.VolumeCreateOpts]. Drop --location when attaching to an existing server.
• Hetzner-generated /mnt/HC_Volume_<id> paths are ugly — rename via sed -i on /etc/fstab + remount before any app sees the path. Don't symlink /mnt/HC_Volume_<id> → /data, it complicates /etc/fstab parsing.
• rsync MUST use -aHAX for Docker — overlay2 relies on hardlinks across image layers. Plain cp -r or rsync -a will silently inflate disk usage 3-5x and may corrupt layer references.
• DON'T rm -rf /var/lib/docker.OLD until you've verified all containers restart cleanly — keep the old dir around as a fallback for one cycle.
• nofail in fstab is non-negotiable — if the volume detaches or is destroyed, the server still boots without it. Without nofail, you get an emergency-mode boot screen and need rescue mode to fix.
• Volumes can be resized up but NOT down — start smaller and grow than oversize. Resize via $HC volume resize <name> --size <new_gb> then sudo resize2fs /dev/disk/by-id/scsi-0HC_Volume_<id>.

Server-resize path (alternative — adds CPU/RAM but requires reboot)

When you need disk + CPU + RAM together, resize the server. Cost-effective tiers from ccx13 (8GB / 80GB / $20):

$HC server poweroff ubuntu-8gb-hil-1
$HC server change-type ubuntu-8gb-hil-1 ccx23 --upgrade-disk
$HC server poweron ubuntu-8gb-hil-1

--upgrade-disk is irreversible — Hetzner won't let you shrink disk on downgrade later, you'd have to rebuild. Volumes don't have this problem.

Existing servers (do not destroy)

• ubuntu-8gb-hil-1 5.78.190.92 — main *.apps.poofc.com / Hermes Agent host; Hetzner type ccx13 dedicated CPU; volume data-100 mounted at /data.
• ubuntu-2gb-hil-1 5.78.199.81 — small nginx host; current default TLS cert is sacred-geometry-teacher.apps.theferalfemale.art; direct SSH from main VPS failed during 2026-06 inventory.
• astral-node-eu-1 167.233.226.57 — Astral Hermes / Spawn worker-node in Hetzner fsn1, type cx43; Docker containers named astral-tenant-* / spawn-tenant-*; no nginx/PM2. Replaced deleted US astral-node-1 (5.78.199.26, cpx41) on 2026-07-01.
• freemix-eu-final 91.99.95.203 — final Freemix production host in Hetzner fsn1, type cx43 (8 vCPU / 16GB / 160GB root); Docker compose app + Mongo + Redis; dedicated hermes-freemix user with primary and remote-peer gateways; 500GB freemix-storage-eu volume mounted persistently at /opt/freemix/app/storage. It replaced and deleted both old US freemix-revival-1 (5.78.84.191, cpx31) and oversized temporary EU freemix-migration-stage-eu (cpx52) on 2026-07-10.

Server access

Use SSH from the main VPS when possible. astral-node-eu-1 (167.233.226.57) and freemix-eu-final (91.99.95.203) accept root@<ip> using the main VPS key. ubuntu-8gb-hil-1 should be inspected locally as avalon (root SSH denied from itself), and ubuntu-2gb-hil-1 may require recovering/updating authorized SSH keys before shell-level inspection. The old hermes-crawl4ai Hetzner server was deleted on 2026-07-01 after migrating Crawl4AI to Hostinger 187.127.70.43.