---
name: hetzner-s3-storage
description: "Set up and use Hetzner Object Storage (S3-compatible) for any app. Covers bucket creation, Node.js (AWS SDK v3), Python (boto3), env config, public URLs, and PWA caching. Alex's VPS has credentials ready."
tags: [s3, hetzner, object-storage, aws-sdk, boto3, uploads, files, media]
triggers:
  - S3
  - object storage
  - hetzner storage
  - file upload to cloud
  - media storage
  - bucket
  - persistent file hosting
---

# Hetzner S3 Object Storage

S3-compatible object storage from Hetzner. Used for persistent file hosting (images, video, audio, documents) across apps on Alex's VPS.

## Credentials (Alex's VPS)

Stored per-app in `.env` files. The stable shared values are:

```
HETZNER_S3_ENDPOINT=https://hel1.your-objectstorage.com
HETZNER_S3_ACCESS_KEY=SSZMXUFZ61GAXI19I5I9
HETZNER_S3_SECRET_KEY=(read from a known-good app env, preferably /home/avalon/apps/video-story/.env)
HETZNER_S3_REGION=hel1
```

Region `hel1` = Helsinki. All buckets use this region unless specified otherwise.

### Canonical known-good reference on Alex's VPS

As of April 2026, `video-story` is the canonical working Hetzner S3 integration on the VPS:
- App path: `/home/avalon/apps/video-story`
- Bucket: `video-story`
- Upload test with that app's `.env` succeeds
- Public bucket root check returns `403 AccessDenied` (this means the bucket exists but listing is private)

Use `video-story` as the source of truth when wiring S3 into a new app.
Do NOT assume another app's `.env` is correct just because it has `HETZNER_S3_*` keys.

## 1. Bucket Management

Hetzner doesn't have a CLI — use the Hetzner Cloud Console (https://console.hetzner.cloud) or create buckets via SDK:

### Create bucket (Node.js)
```js
import {
  S3Client,
  CreateBucketCommand,
  HeadBucketCommand,
  PutBucketPolicyCommand,
} from '@aws-sdk/client-s3'

const s3 = new S3Client({
  endpoint: process.env.HETZNER_S3_ENDPOINT,
  region: process.env.HETZNER_S3_REGION || 'hel1',
  credentials: {
    accessKeyId: process.env.HETZNER_S3_ACCESS_KEY,
    secretAccessKey: process.env.HETZNER_S3_SECRET_KEY,
  },
  forcePathStyle: true,  // REQUIRED for Hetzner
})

async function ensurePublicBucket(bucket) {
  try {
    await s3.send(new HeadBucketCommand({ Bucket: bucket }))
  } catch {
    await s3.send(new CreateBucketCommand({ Bucket: bucket }))
  }

  // Needed when URLs should be browser-fetchable/public. Bucket root may still be 403.
  await s3.send(new PutBucketPolicyCommand({
    Bucket: bucket,
    Policy: JSON.stringify({
      Version: '2012-10-17',
      Statement: [{
        Sid: 'PublicReadObjects',
        Effect: 'Allow',
        Principal: '*',
        Action: ['s3:GetObject'],
        Resource: [`arn:aws:s3:::${bucket}/*`],
      }],
    }),
  }))
}

await ensurePublicBucket('my-app-name')
```

### Create bucket (Python)
```python
import boto3

s3 = boto3.client(
    's3',
    endpoint_url=os.environ['HETZNER_S3_ENDPOINT'],
    aws_access_key_id=os.environ['HETZNER_S3_ACCESS_KEY'],
    aws_secret_access_key=os.environ['HETZNER_S3_SECRET_KEY'],
    region_name=os.environ.get('HETZNER_S3_REGION', 'hel1'),
)

s3.create_bucket(Bucket='my-app-name')
```

### Bucket naming convention
Use the app name for app-wide storage: `video-story`, `mocha-results`, `jungle-studio`, etc.

For the Viewer app, use a page-scoped bucket so each generated web page can carry its own independent media library: `viewer-{slug}` (example: `viewer-36-decan-image-atlas`). This matches Alex's desired model of dynamic pages with separate designs/assets and makes cleanup/export/provenance easier.

For provisioned agent/tenant systems (Hermes Spawn, Astral-like tenant factories), prefer a tenant-scoped bucket and quota contract:

- Bucket convention: `<platform>-<tenant-id>` after strict lowercase slug cleaning; for Hermes Spawn use `spawn-[tenant-id]`.
- Cap bucket names to 63 chars and avoid user-controlled punctuation.
- Store tenant metadata in the tenant home (for Spawn: `.spawn/s3.json`) with bucket, endpoint, region, quotaBytes, and timestamps.
- Inject tenant env: `HETZNER_S3_ENDPOINT`, `HETZNER_S3_REGION`, `HETZNER_S3_ACCESS_KEY`, `HETZNER_S3_SECRET_KEY`, `HETZNER_S3_BUCKET`, `HETZNER_S3_QUOTA_BYTES`.
- Give tenant agents a generated local skill/runtime note that says they must check usage before writes and stop/notify the user if a planned action would exceed quota.
- Provision an executable guard helper (Python/boto3 is fine) that supports `usage`, `can-add BYTES`, and guarded `upload LOCAL_PATH KEY` so the agent can enforce the cap without reimplementing quota math every time.
- Add a control-plane usage API that lists objects with continuation tokens and returns `usedBytes`, `objectCount`, `quotaBytes`, and `percentUsed`; surface the same info in the admin dashboard.

## 2. Node.js Integration (AWS SDK v3)

### Install
```bash
npm install @aws-sdk/client-s3
```

### storage.js module (copy-paste ready)
```js
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import fs from 'fs'
import path from 'path'

const ENDPOINT = process.env.HETZNER_S3_ENDPOINT
const BUCKET = process.env.HETZNER_S3_BUCKET || 'my-app'
const REGION = process.env.HETZNER_S3_REGION || 'hel1'

let s3 = null

function getClient() {
  if (!s3) {
    if (!ENDPOINT || !process.env.HETZNER_S3_ACCESS_KEY) {
      throw new Error('Hetzner S3 not configured — set HETZNER_S3_* env vars')
    }
    s3 = new S3Client({
      endpoint: ENDPOINT,
      region: REGION,
      credentials: {
        accessKeyId: process.env.HETZNER_S3_ACCESS_KEY,
        secretAccessKey: process.env.HETZNER_S3_SECRET_KEY,
      },
      forcePathStyle: true,
    })
  }
  return s3
}

/**
 * Upload a buffer or local file path to S3. Returns the public URL.
 */
export async function upload(input, key, contentType) {
  const client = getClient()
  let body
  if (Buffer.isBuffer(input)) {
    body = input
  } else if (typeof input === 'string') {
    body = fs.readFileSync(input)
  } else {
    throw new Error('upload() input must be a Buffer or file path string')
  }

  await client.send(new PutObjectCommand({
    Bucket: BUCKET,
    Key: key,
    Body: body,
    ContentType: contentType,
  }))

  return `${ENDPOINT}/${BUCKET}/${key}`
}

/**
 * Download a remote URL and re-upload to S3. Returns public S3 URL.
 * Use for persisting ephemeral URLs (e.g. Replicate output URLs that expire).
 */
export async function downloadAndUpload(remoteUrl, key) {
  const res = await fetch(remoteUrl)
  if (!res.ok) throw new Error(`Download failed: ${res.status} ${remoteUrl}`)
  const buffer = Buffer.from(await res.arrayBuffer())
  const contentType = res.headers.get('content-type') || guessMime(key)
  return upload(buffer, key, contentType)
}

function guessMime(key) {
  const ext = path.extname(key).toLowerCase()
  const map = {
    '.webp': 'image/webp', '.png': 'image/png',
    '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg',
    '.mp4': 'video/mp4', '.mp3': 'audio/mpeg',
    '.ogg': 'audio/ogg', '.pdf': 'application/pdf',
    '.json': 'application/json',
  }
  return map[ext] || 'application/octet-stream'
}

export function isConfigured() {
  return !!(ENDPOINT && process.env.HETZNER_S3_ACCESS_KEY)
}
```

### Usage in routes
```js
import { upload, downloadAndUpload, isConfigured } from './storage.js'

// Upload from multipart form (multer in-memory)
app.post('/api/upload', multerMemory.single('file'), async (req, res) => {
  const key = `uploads/${Date.now()}_${req.file.originalname}`
  const url = await upload(req.file.buffer, key, req.file.mimetype)
  res.json({ url })
})

// Persist an ephemeral URL (e.g. AI-generated image)
const s3Url = await downloadAndUpload(replicateOutputUrl, `projects/${pid}/images/shot_001.webp`)
```

## 3. Python Integration (boto3)

### Install
```bash
pip install boto3
```

### storage module (copy-paste ready)
```python
import os, base64, mimetypes
import boto3

S3_ENDPOINT = os.environ.get('HETZNER_S3_ENDPOINT', '')
S3_BUCKET = os.environ.get('HETZNER_S3_BUCKET', 'my-app')
S3_ACCESS_KEY = os.environ.get('HETZNER_S3_ACCESS_KEY', '')
S3_SECRET_KEY = os.environ.get('HETZNER_S3_SECRET_KEY', '')
S3_REGION = os.environ.get('HETZNER_S3_REGION', 'hel1')

def get_client():
    return boto3.client(
        's3',
        endpoint_url=S3_ENDPOINT,
        aws_access_key_id=S3_ACCESS_KEY,
        aws_secret_access_key=S3_SECRET_KEY,
        region_name=S3_REGION,
    )

def upload_file(local_path, key, content_type=None):
    """Upload a local file to S3. Returns public URL."""
    if not S3_ENDPOINT or not S3_ACCESS_KEY:
        raise ValueError('S3 not configured — set HETZNER_S3_* env vars')

    if not content_type:
        content_type, _ = mimetypes.guess_type(local_path)
        content_type = content_type or 'application/octet-stream'

    s3 = get_client()
    s3.upload_file(
        local_path, S3_BUCKET, key,
        ExtraArgs={'ACL': 'public-read', 'ContentType': content_type}
    )
    return f"{S3_ENDPOINT}/{S3_BUCKET}/{key}"

def upload_bytes(data, key, content_type='application/octet-stream'):
    """Upload bytes/buffer to S3. Returns public URL."""
    s3 = get_client()
    s3.put_object(
        Bucket=S3_BUCKET, Key=key, Body=data,
        ACL='public-read', ContentType=content_type
    )
    return f"{S3_ENDPOINT}/{S3_BUCKET}/{key}"

def is_configured():
    return bool(S3_ENDPOINT and S3_ACCESS_KEY)
```

### With base64 fallback (for GPU workers)
```python
def upload_or_base64(local_path, key, content_type='video/mp4'):
    """Upload to S3 if configured, otherwise return base64 data URI."""
    if not is_configured():
        print("No S3 configured, returning base64")
        with open(local_path, 'rb') as f:
            data = base64.b64encode(f.read()).decode('utf-8')
        return f"data:{content_type};base64,{data}"
    return upload_file(local_path, key, content_type)
```

## 4. Environment Variables (.env)

Add to the app's `.env` file:
```
# Hetzner Object Storage (S3-compatible)
HETZNER_S3_ENDPOINT=https://hel1.your-objectstorage.com
HETZNER_S3_ACCESS_KEY=SSZMXUFZ61GAXI19I5I9
HETZNER_S3_SECRET_KEY=<get from existing app .env>
HETZNER_S3_BUCKET=<app-name>
HETZNER_S3_REGION=hel1
```

If using PM2 ecosystem, add to the app's `env` block:
```js
env: {
  HETZNER_S3_ENDPOINT: 'https://hel1.your-objectstorage.com',
  HETZNER_S3_ACCESS_KEY: 'SSZMXUFZ61GAXI19I5I9',
  HETZNER_S3_SECRET_KEY: process.env.HETZNER_S3_SECRET_KEY,
  HETZNER_S3_BUCKET: 'app-name',
  HETZNER_S3_REGION: 'hel1',
}
```

## 5. Public URL Pattern

All uploaded objects are publicly accessible at:
```
https://hel1.your-objectstorage.com/<bucket>/<key>
```

Example: `https://hel1.your-objectstorage.com/video-story/projects/abc123/images/char_001.webp`

### Key naming conventions
- Images: `projects/{id}/images/{name}.webp`
- Videos: `projects/{id}/videos/{name}.mp4`
- Audio: `projects/{id}/audio/{name}.mp3`
- Generic uploads: `uploads/{timestamp}_{filename}`

## 6. PWA Service Worker Caching

For Vite PWA apps, cache S3 images with CacheFirst strategy:
```js
// vite.config.js
workbox: {
  runtimeCaching: [{
    urlPattern: /^https:\/\/hel1\.your-objectstorage\.com\/.*/,
    handler: 'CacheFirst',
    options: {
      cacheName: 'images-cache',
      expiration: { maxEntries: 300, maxAgeSeconds: 7 * 24 * 60 * 60 }
    }
  }]
}
```

## Pitfalls

1. **forcePathStyle: true** — REQUIRED for Hetzner. Without it, the SDK tries virtual-hosted-style URLs that don't work.

2. **Good pattern for multi-user apps** — store only provider API keys (like each user's fal.ai key) in your app DB, but persist generated media itself to Hetzner S3 immediately after generation. A solid reusable shape is:
   - app DB row for `users`
   - app DB row for `assets` with `user_id`, `model`, `prompt`, `result_kind`, `storage_key`, `storage_url`, optional params JSON
   - on generation: call provider with user's API key -> download returned ephemeral media URL -> upload buffer to Hetzner S3 -> save asset row -> return S3 URL to frontend
   This avoids ephemeral provider URLs expiring and cleanly separates per-user credentials from shared media infrastructure.

3. **Per-user key + S3 hybrid works well on Alex's VPS** — for apps like fal-studio, keep S3 credentials in server `.env`, but let each signed-in user store their own third-party API key in the app database/account settings. Do NOT put end-user provider keys in shared app `.env`.

2. **Public object URLs require BOTH object/bucket permissions and a real fetch test** — A successful `PutObject` can still return a URL that gives `403`. For browser-public URLs on newly created buckets, set a bucket policy allowing `s3:GetObject` on `arn:aws:s3:::BUCKET/*`; optionally include `ACL: 'public-read'` in Node `PutObjectCommand` as well. Verify a concrete uploaded object with `curl -s -o /dev/null -w '%{http_code}' "$URL"` and expect `200`. The bucket root returning `403 AccessDenied` is normal and only means listing is private.

3. **ACL: 'public-read'** — Required in Python/boto3 `ExtraArgs` if you want public URLs. In Node.js AWS SDK v3, include `ACL: 'public-read'` when available, but do not rely on ACL alone; make sure the bucket policy allows public `GetObject`.

4. **Bucket must exist first** — Unlike some providers, Hetzner won't auto-create buckets. Create via Console or SDK before uploading.

4. **Ephemeral URL persistence** — AI services (Replicate, fal.ai) return temporary URLs that expire in hours. Always `downloadAndUpload()` to S3 immediately after generation.

5. **Do a real write test, not just a code diff** — before declaring S3 wired up, run an actual `PutObject` using the new app's `.env`. Also probe the public bucket URL:
   - `GET https://ENDPOINT/BUCKET` → `403 AccessDenied` usually means the bucket exists but listing is private
   - `GET https://ENDPOINT/BUCKET` → `404 NoSuchBucket` means the bucket name is wrong or the bucket was never created
   On Alex's VPS this caught the exact issue where `video-story` worked but `fal-studio` pointed at a nonexistent `fal-studio` bucket.

6. **Different apps may have different full secrets even with the same access key** — compare secret lengths and do not assume parity. In the April 2026/May 2026 fal-studio incident, `video-story/.env` had a real 40-char secret and successful uploads/S3 HEAD checks, while `fal-studio/.env` had a different 35-char secret and S3 requests failed with `SignatureDoesNotMatch` or generic 403. When in doubt, inspect `/home/avalon/apps/video-story/.env` and test with that app's actual runtime env. If a script loads known-good credentials as fallback, do not use `os.environ.setdefault` when the parent process may already have stale/bad `HETZNER_S3_*` values; intentionally override from the known-good env for the verification subprocess.

7. **PM2 can preserve stale S3 env across restarts** — app code that parses `.env` with `if (!process.env[key]) process.env[key] = value` may silently keep an old bucket/secret inherited from PM2 or the shell. For per-app storage, prefer project-local `.env` as source of truth and intentionally overwrite matching `HETZNER_S3_*` keys at boot, then `pm2 restart APP --update-env` and run a real upload/HEAD smoke test. This caught Hermes Social accidentally inheriting an old S3 bucket value before switching to bucket `hermes-social`.

7. **Video Story pattern = best-practice fallback for generated media** — `video-story` uploads provider outputs to S3 with `downloadAndUpload(remoteUrl, key)` and, if upload fails, logs the failure and continues using the provider's remote URL instead of crashing the generation flow. For multi-user apps like fal-studio, an even safer variant is local-file fallback with a DB asset row so user galleries still work when S3 is unavailable.

8. **Background cron/swarm jobs need their own S3 runtime, not just interactive env** — for long-running Hermes jobs, copy known-good Hetzner values into a private project `.env` (mode `0600`), create a project-local `.venv`, and add a `scripts/bootstrap_runtime.sh` that sources the env and installs missing `boto3`/`awscli` before each run. Then verify with real `HeadBucket` + `PutObject` + `HeadObject` + `DeleteObject` operations before clearing any `blocked` state. This prevented the Human Design wiki swarm from repeatedly blocking on absent S3 tooling/credentials under cron.

9. **fal-studio concurrency + local persistence pattern** — for mobile-first multi-user AI apps, do NOT block the entire UI on a single global `generating` state. Allow multiple generation requests to run concurrently from the frontend, track them in a lightweight job list (`queued`/`running`/`completed`/`failed`), and auto-save each finished asset immediately. This lets users start one job, switch models, and launch another while earlier jobs are still running.

9. **Preserve reference uploads across compatible model switches** — when users switch between similar image/video models, keep only overlapping upload buckets (`generalReferences`, `firstFrame`, `lastFrame`, `storyboardReference`) and trim to the new model's `maxFiles`. This avoids wiping a user's reference images when they are just comparing adjacent models in the same family.

10. **Graceful fallback pattern for app media storage** — when S3 upload fails due to missing bucket or bad credentials, store the asset locally under a per-user path such as `data/local-assets/users/<userId>/<kind>/...`, expose it via an authenticated/static route (or a dedicated media route), and return a warning like `Saved locally because S3 is unavailable`. This prevents the storage layer from blocking core generation flows.

9. **CORS** — If the frontend needs to fetch S3 objects directly (not through your API proxy), you need to configure CORS on the bucket via the Hetzner Console.

10. **Large files** — For files >5GB, use multipart upload. Both AWS SDK v3 and boto3 handle this automatically with their high-level upload methods.

11. **Env var naming** — Use `HETZNER_S3_*` prefix (not generic `S3_*`) to avoid conflicts with AWS or other S3-compatible services. Exception: RunPod workers use generic `S3_*` since they only talk to one storage backend.