---
name: hetzner-cloud-cli
description: "Provision and manage Hetzner Cloud VPS servers via the hcloud CLI."
version: 1.0.0
author: Alex + Hermes
license: MIT
metadata:
  hermes:
    tags: [hetzner, vps, hcloud, provisioning, devops]
---

# Hetzner Cloud CLI

Alex has a Hetzner Cloud project with `hcloud` CLI installed and a working token. Use it instead of asking him to provision things through the web console.

## Setup state (as of Crawl4AI deploy)

- **Binary:** `~/.local/bin/hcloud` (v1.55.0) — not on default PATH, use absolute path
- **Token:** `HCLOUD_TOKEN` in `~/.hermes/.env`. Load with `export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)`
- **SSH key in project:** `avalon-current-vps-astral` (use this for `--ssh-key`)
- **Default region:** `hil` (Hillsboro, OR) — all existing servers are there

## Common commands

```bash
export HOME=/home/avalon  # subagent envs sometimes lack this
export HCLOUD_TOKEN=$(grep HCLOUD_TOKEN ~/.hermes/.env | cut -d= -f2)
HC=~/.local/bin/hcloud

$HC server list
$HC server-type list           # see available sizes & arch
$HC location list              # ID/name/network zone
$HC ssh-key list
$HC server create --name <n> --type cpx11 --image ubuntu-24.04 --location hil --ssh-key avalon-current-vps-astral --user-data-from-file /tmp/cloudinit.yaml
$HC server delete <name>
$HC server poweroff <name>
$HC server reboot <name>
$HC server describe <name>     # full info incl IPv4/IPv6
```

## Pricing reference (monthly cap, hourly billed)

Always verify live with `$HC server-type describe <type> -o json`: Hetzner US locations (`hil`, `ash`) are much more expensive than EU locations for shared CPU `cpx*` types. Dedicated CPU `ccx*` types may have little/no region spread, so do not assume moving every server saves money.

See `references/hetzner-us-to-eu-migration-notes.md` for the Hetzner-first overpayment workflow: compare exact type pricing, remember there is no in-place datacenter swap, recreate in `fsn1`/`nbg1`/`hel1`, migrate data/config, update DNS/firewalls, smoke test, then delete the US source.

See `references/freemix-cost-storage-migration.md` for the Freemix-derived cost/storage accounting pattern: verify server types, attached volumes, inside-guest mounts, volume pricing, temporary overlap cost, and whether the current EU topology is migration-safe versus final cost-optimized.

See `references/freemix-eu-volume-cutover-runbook.md` for the follow-on pattern when a large app was staged onto an oversized EU root disk due to volume quota, and you need to prove live storage parity, delete/free the old volume quota if approved, create a new EU volume, and mount it at the live storage path with minimal downtime.

| Type | Specs | `hil` / US-west price | EU price examples |
|---|---|---:|---:|
| cpx11 | 2 vCPU x86, 2GB, 40GB | ~€20.49/mo | ~€5.99/mo |
| cpx21 | 3 vCPU x86, 4GB, 80GB | ~€37.49/mo | ~€10.99/mo |
| cpx31 | 4 vCPU x86, 8GB, 160GB | ~€73.49/mo | ~€20.49/mo |
| cax11 | 2 vCPU ARM, 4GB, 40GB | not in `hil` | ~€6.99/mo |
| cax21 | 4 vCPU ARM, 8GB, 80GB | not in `hil` | ~€12.49/mo |
| cx33 | 4 vCPU x86, 8GB, 80GB | not in `hil` | ~€8.99/mo |

## Gotchas

- **No in-place datacenter swap for servers** — `hcloud server change-type` changes size only, and `hcloud server rebuild` keeps the same location. To leave expensive `hil`/`ash` pricing, create a replacement server in the target EU location (`fsn1`, `nbg1`, or `hel1`), migrate data/config, update DNS/IP allowlists/firewalls, smoke test, then delete the old server.
- **Volumes are location-bound** — a `hil` volume cannot simply attach to an EU server. Plan rsync/snapshot/object-storage migration for large volumes before promising savings, especially 900GB+ data volumes.
- **Project volume-size limits can block safe side-by-side migration** — if creating an equal EU replacement volume fails with `volumes size limit exceeded`, do not delete the old volume just to free quota. Use `references/large-volume-migration-staging.md`: create a temporary EU server with enough root disk for the actually-used data, rsync/verify there while the old volume remains rollback, then delete/recreate final storage only after app + data + local agents pass smokes.
- **Freemix-style cost/storage migrations need live-data verification before deleting rollback volumes** — see `references/freemix-cost-storage-migration.md` for the accounting pattern, quota-increase path, old-volume live-vs-recovery breakdown, and rsync dry-run/checksum verification commands before freeing quota by deleting old storage.
- **Primary IPs are location/datacenter-bound and usually auto-delete** — unless Floating IPs were planned, cross-location migration means new IPs and all DNS/service/firewall allowlists must be updated.
- **ARM (cax*) not available in `hil`** — only EU locations host ARM. Use `cpx*` for Hillsboro deploys.
- **Names must be unique** across the project — `hermes-crawl4ai` etc.
- **Hetzner only bills hourly/monthly** — destroying mid-month only charges hours used. No annual prepay exists.
- **There is no in-place datacenter/location swap for existing servers.** `server change-type` changes size only, `server rebuild` stays in the same location, and `server create --location <fsn1|nbg1|hel1|...>` is the path for a different datacenter. Treat US→EU cost reductions as replacement-server migrations: create the new server in the cheap location, copy data/config, update DNS/firewalls, smoke test, then delete the old one. For nginx/DNS cutover details, see `vps-app-deployment` → `references/hetzner-eu-migration-dns-cutover.md`.
- **Primary IPs and Volumes are location/datacenter-bound.** Cross-location migrations get new server IPs unless a suitable floating/primary-IP strategy already exists; attached volumes must be recreated/copied in the target location. Do not assume `data-100` or a large app volume can simply attach to an EU server.
- **cloud-init** is the cheap way to provision. Write a yaml file with packages/write_files/runcmd then pass `--user-data-from-file`. SSH in to debug with `cat /var/log/cloud-init-output.log`.
- After server create, wait ~2-5 min for cloud-init before testing services.

## Expanding disk on an existing server — volumes (no downtime path)

When a VPS hits >95% root disk full, the cheapest fix is a Hetzner Volume (block storage), not a server resize. Volumes are ~$0.048/GB/mo, live-attached, resizable up, and survive server destroy.

### Provision + attach in one command

```bash
# --server and --location are MUTUALLY EXCLUSIVE on volume create.
# When attaching to an existing server, OMIT --location — it inherits the server's location.
$HC volume create --name data-100 --size 100 --format ext4 --server ubuntu-8gb-hil-1 --automount
# Output: Volume 105790860 created
```

`--automount` writes a `/etc/fstab` entry via `/dev/disk/by-id/scsi-0HC_Volume_<id>` with `nofail,defaults` so a missing volume won't brick boot. Default mount path is `/mnt/HC_Volume_<id>` — ugly. Rename it:

```bash
sudo mkdir -p /data
sudo umount /mnt/HC_Volume_<id>
sudo sed -i "s|/mnt/HC_Volume_<id>|/data|" /etc/fstab
sudo mount -a
sudo chown avalon:avalon /data
df -h /data    # confirm ~98G usable on a 100GB volume
```

### Move heavy dirs with symlink-back pattern (apps see no change)

```bash
move_dir() {
  local src="$1"; local name="$(basename "$src")"
  [ -L "$src" ] && { echo "skip: $src is symlink"; return; }
  mv "$src" "/data/$name"
  ln -s "/data/$name" "$src"
}
# Safe candidates: static asset vaults, wikis, build artifacts, cache dirs.
# Risky candidates: running app dirs (PM2 cwd) — possible but stop the app first.
move_dir /home/avalon/hermes-media-vault
move_dir /home/avalon/hyperframes
```

### Migrating `/var/lib/docker` to a volume (BIG win, ~3 min downtime)

Docker data-root is usually the largest single dir on a VPS running containers. To shift it:

```bash
# 1. Stop the engine and its socket
sudo systemctl stop docker docker.socket containerd

# 2. rsync with --aHAX to preserve hardlinks/ACLs/xattrs (CRITICAL for docker)
sudo rsync -aHAX --info=progress2 /var/lib/docker/ /data/docker/

# 3. Verify sizes match (rough — docker compacts on next start)
sudo du -sh /var/lib/docker /data/docker

# 4. Move old aside, write daemon.json
sudo mv /var/lib/docker /var/lib/docker.OLD
sudo mkdir -p /etc/docker
echo '{"data-root": "/data/docker"}' | sudo tee /etc/docker/daemon.json

# 5. Start docker, verify
sudo systemctl start docker
sudo docker info | grep "Docker Root Dir"   # → /data/docker
sudo docker ps                              # auto-restart=always containers come back
```

### Restart=no containers won't auto-start after daemon restart

Containers created without `--restart unless-stopped` (or `always`) stay Exited after `systemctl restart docker`. Manually `docker start <name>` each one. To make them resilient for next time:

```bash
sudo docker update --restart unless-stopped <container_name>
```

### Pitfalls (volumes + data-root migration)

- **`--server` and `--location` are mutually exclusive on `volume create`** — error is `hcloud: found mutually exclusive fields [Server, Location] in [hcloud.VolumeCreateOpts]`. Drop `--location` when attaching to an existing server.
- **Hetzner-generated `/mnt/HC_Volume_<id>` paths are ugly** — rename via `sed -i` on `/etc/fstab` + remount before any app sees the path. Don't symlink `/mnt/HC_Volume_<id>` → `/data`, it complicates `/etc/fstab` parsing.
- **rsync MUST use `-aHAX`** for Docker — overlay2 relies on hardlinks across image layers. Plain `cp -r` or `rsync -a` will silently inflate disk usage 3-5x and may corrupt layer references.
- **DON'T `rm -rf /var/lib/docker.OLD` until you've verified all containers restart cleanly** — keep the old dir around as a fallback for one cycle.
- **`nofail` in fstab is non-negotiable** — if the volume detaches or is destroyed, the server still boots without it. Without `nofail`, you get an emergency-mode boot screen and need rescue mode to fix.
- **Volumes can be resized up but NOT down** — start smaller and grow than oversize. Resize via `$HC volume resize <name> --size <new_gb>` then `sudo resize2fs /dev/disk/by-id/scsi-0HC_Volume_<id>`.

### Server-resize path (alternative — adds CPU/RAM but requires reboot)

When you need disk + CPU + RAM together, resize the server. Cost-effective tiers from `ccx13` (8GB / 80GB / $20):

| Type | vCPU | RAM | Disk | $/mo (hil) |
|---|---|---|---|---|
| cpx32 (shared) | 4 | 8GB | 160GB | ~$18 — cheaper but loses dedicated CPU |
| ccx23 (dedicated) | 4 | 16GB | 160GB | ~$40 — clean 2x upgrade |
| ccx33 (dedicated) | 8 | 32GB | 240GB | ~$80 |

```bash
$HC server poweroff ubuntu-8gb-hil-1
$HC server change-type ubuntu-8gb-hil-1 ccx23 --upgrade-disk
$HC server poweron ubuntu-8gb-hil-1
```

`--upgrade-disk` is irreversible — Hetzner won't let you shrink disk on downgrade later, you'd have to rebuild. Volumes don't have this problem.

## Existing servers (do not destroy)

- `ubuntu-8gb-hil-1` 5.78.190.92 — main `*.apps.poofc.com` / Hermes Agent host; Hetzner type `ccx13` dedicated CPU; volume `data-100` mounted at `/data`.
- `ubuntu-2gb-hil-1` 5.78.199.81 — small nginx host; current default TLS cert is `sacred-geometry-teacher.apps.theferalfemale.art`; direct SSH from main VPS failed during 2026-06 inventory.
- `astral-node-eu-1` 167.233.226.57 — Astral Hermes / Spawn worker-node in Hetzner `fsn1`, type `cx43`; Docker containers named `astral-tenant-*` / `spawn-tenant-*`; no nginx/PM2. Replaced deleted US `astral-node-1` (`5.78.199.26`, `cpx41`) on 2026-07-01.
- `freemix-eu-final` 91.99.95.203 — final Freemix production host in Hetzner `fsn1`, type `cx43` (8 vCPU / 16GB / 160GB root); Docker compose app + Mongo + Redis; dedicated `hermes-freemix` user with primary and remote-peer gateways; 500GB `freemix-storage-eu` volume mounted persistently at `/opt/freemix/app/storage`. It replaced and deleted both old US `freemix-revival-1` (`5.78.84.191`, `cpx31`) and oversized temporary EU `freemix-migration-stage-eu` (`cpx52`) on 2026-07-10.

## Server access

Use SSH from the main VPS when possible. `astral-node-eu-1` (`167.233.226.57`) and `freemix-eu-final` (`91.99.95.203`) accept `root@<ip>` using the main VPS key. `ubuntu-8gb-hil-1` should be inspected locally as `avalon` (root SSH denied from itself), and `ubuntu-2gb-hil-1` may require recovering/updating authorized SSH keys before shell-level inspection. The old `hermes-crawl4ai` Hetzner server was deleted on 2026-07-01 after migrating Crawl4AI to Hostinger `187.127.70.43`.