S3-compatible object storage from Hetzner. Used for persistent file hosting (images, video, audio, documents) across apps on Alex's VPS.
Stored per-app in .env files. The stable shared values are:
HETZNER_S3_ENDPOINT=https://hel1.your-objectstorage.com
HETZNER_S3_ACCESS_KEY=SSZMXUFZ61GAXI19I5I9
HETZNER_S3_SECRET_KEY=(read from a known-good app env, preferably /home/avalon/apps/video-story/.env)
HETZNER_S3_REGION=hel1
Region hel1 = Helsinki. All buckets use this region unless specified otherwise.
As of April 2026, video-story is the canonical working Hetzner S3 integration on the VPS:
- App path: /home/avalon/apps/video-story
- Bucket: video-story
- Upload test with that app's .env succeeds
- Public bucket root check returns 403 AccessDenied (this means the bucket exists but listing is private)
Use video-story as the source of truth when wiring S3 into a new app.
Do NOT assume another app's .env is correct just because it has HETZNER_S3_* keys.
Hetzner doesn't have a CLI — use the Hetzner Cloud Console (https://console.hetzner.cloud) or create buckets via SDK:
import {
S3Client,
CreateBucketCommand,
HeadBucketCommand,
PutBucketPolicyCommand,
} from '@aws-sdk/client-s3'
const s3 = new S3Client({
endpoint: process.env.HETZNER_S3_ENDPOINT,
region: process.env.HETZNER_S3_REGION || 'hel1',
credentials: {
accessKeyId: process.env.HETZNER_S3_ACCESS_KEY,
secretAccessKey: process.env.HETZNER_S3_SECRET_KEY,
},
forcePathStyle: true, // REQUIRED for Hetzner
})
async function ensurePublicBucket(bucket) {
try {
await s3.send(new HeadBucketCommand({ Bucket: bucket }))
} catch {
await s3.send(new CreateBucketCommand({ Bucket: bucket }))
}
// Needed when URLs should be browser-fetchable/public. Bucket root may still be 403.
await s3.send(new PutBucketPolicyCommand({
Bucket: bucket,
Policy: JSON.stringify({
Version: '2012-10-17',
Statement: [{
Sid: 'PublicReadObjects',
Effect: 'Allow',
Principal: '*',
Action: ['s3:GetObject'],
Resource: [`arn:aws:s3:::${bucket}/*`],
}],
}),
}))
}
await ensurePublicBucket('my-app-name')
import boto3
s3 = boto3.client(
's3',
endpoint_url=os.environ['HETZNER_S3_ENDPOINT'],
aws_access_key_id=os.environ['HETZNER_S3_ACCESS_KEY'],
aws_secret_access_key=os.environ['HETZNER_S3_SECRET_KEY'],
region_name=os.environ.get('HETZNER_S3_REGION', 'hel1'),
)
s3.create_bucket(Bucket='my-app-name')
Use the app name for app-wide storage: video-story, mocha-results, jungle-studio, etc.
For the Viewer app, use a page-scoped bucket so each generated web page can carry its own independent media library: viewer-{slug} (example: viewer-36-decan-image-atlas). This matches Alex's desired model of dynamic pages with separate designs/assets and makes cleanup/export/provenance easier.
For provisioned agent/tenant systems (Hermes Spawn, Astral-like tenant factories), prefer a tenant-scoped bucket and quota contract:
<platform>-<tenant-id> after strict lowercase slug cleaning; for Hermes Spawn use spawn-[tenant-id]..spawn/s3.json) with bucket, endpoint, region, quotaBytes, and timestamps.HETZNER_S3_ENDPOINT, HETZNER_S3_REGION, HETZNER_S3_ACCESS_KEY, HETZNER_S3_SECRET_KEY, HETZNER_S3_BUCKET, HETZNER_S3_QUOTA_BYTES.usage, can-add BYTES, and guarded upload LOCAL_PATH KEY so the agent can enforce the cap without reimplementing quota math every time.usedBytes, objectCount, quotaBytes, and percentUsed; surface the same info in the admin dashboard.npm install @aws-sdk/client-s3
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import fs from 'fs'
import path from 'path'
const ENDPOINT = process.env.HETZNER_S3_ENDPOINT
const BUCKET = process.env.HETZNER_S3_BUCKET || 'my-app'
const REGION = process.env.HETZNER_S3_REGION || 'hel1'
let s3 = null
function getClient() {
if (!s3) {
if (!ENDPOINT || !process.env.HETZNER_S3_ACCESS_KEY) {
throw new Error('Hetzner S3 not configured — set HETZNER_S3_* env vars')
}
s3 = new S3Client({
endpoint: ENDPOINT,
region: REGION,
credentials: {
accessKeyId: process.env.HETZNER_S3_ACCESS_KEY,
secretAccessKey: process.env.HETZNER_S3_SECRET_KEY,
},
forcePathStyle: true,
})
}
return s3
}
/**
* Upload a buffer or local file path to S3. Returns the public URL.
*/
export async function upload(input, key, contentType) {
const client = getClient()
let body
if (Buffer.isBuffer(input)) {
body = input
} else if (typeof input === 'string') {
body = fs.readFileSync(input)
} else {
throw new Error('upload() input must be a Buffer or file path string')
}
await client.send(new PutObjectCommand({
Bucket: BUCKET,
Key: key,
Body: body,
ContentType: contentType,
}))
return `${ENDPOINT}/${BUCKET}/${key}`
}
/**
* Download a remote URL and re-upload to S3. Returns public S3 URL.
* Use for persisting ephemeral URLs (e.g. Replicate output URLs that expire).
*/
export async function downloadAndUpload(remoteUrl, key) {
const res = await fetch(remoteUrl)
if (!res.ok) throw new Error(`Download failed: ${res.status} ${remoteUrl}`)
const buffer = Buffer.from(await res.arrayBuffer())
const contentType = res.headers.get('content-type') || guessMime(key)
return upload(buffer, key, contentType)
}
function guessMime(key) {
const ext = path.extname(key).toLowerCase()
const map = {
'.webp': 'image/webp', '.png': 'image/png',
'.jpg': 'image/jpeg', '.jpeg': 'image/jpeg',
'.mp4': 'video/mp4', '.mp3': 'audio/mpeg',
'.ogg': 'audio/ogg', '.pdf': 'application/pdf',
'.json': 'application/json',
}
return map[ext] || 'application/octet-stream'
}
export function isConfigured() {
return !!(ENDPOINT && process.env.HETZNER_S3_ACCESS_KEY)
}
import { upload, downloadAndUpload, isConfigured } from './storage.js'
// Upload from multipart form (multer in-memory)
app.post('/api/upload', multerMemory.single('file'), async (req, res) => {
const key = `uploads/${Date.now()}_${req.file.originalname}`
const url = await upload(req.file.buffer, key, req.file.mimetype)
res.json({ url })
})
// Persist an ephemeral URL (e.g. AI-generated image)
const s3Url = await downloadAndUpload(replicateOutputUrl, `projects/${pid}/images/shot_001.webp`)
pip install boto3
import os, base64, mimetypes
import boto3
S3_ENDPOINT = os.environ.get('HETZNER_S3_ENDPOINT', '')
S3_BUCKET = os.environ.get('HETZNER_S3_BUCKET', 'my-app')
S3_ACCESS_KEY = os.environ.get('HETZNER_S3_ACCESS_KEY', '')
S3_SECRET_KEY = os.environ.get('HETZNER_S3_SECRET_KEY', '')
S3_REGION = os.environ.get('HETZNER_S3_REGION', 'hel1')
def get_client():
return boto3.client(
's3',
endpoint_url=S3_ENDPOINT,
aws_access_key_id=S3_ACCESS_KEY,
aws_secret_access_key=S3_SECRET_KEY,
region_name=S3_REGION,
)
def upload_file(local_path, key, content_type=None):
"""Upload a local file to S3. Returns public URL."""
if not S3_ENDPOINT or not S3_ACCESS_KEY:
raise ValueError('S3 not configured — set HETZNER_S3_* env vars')
if not content_type:
content_type, _ = mimetypes.guess_type(local_path)
content_type = content_type or 'application/octet-stream'
s3 = get_client()
s3.upload_file(
local_path, S3_BUCKET, key,
ExtraArgs={'ACL': 'public-read', 'ContentType': content_type}
)
return f"{S3_ENDPOINT}/{S3_BUCKET}/{key}"
def upload_bytes(data, key, content_type='application/octet-stream'):
"""Upload bytes/buffer to S3. Returns public URL."""
s3 = get_client()
s3.put_object(
Bucket=S3_BUCKET, Key=key, Body=data,
ACL='public-read', ContentType=content_type
)
return f"{S3_ENDPOINT}/{S3_BUCKET}/{key}"
def is_configured():
return bool(S3_ENDPOINT and S3_ACCESS_KEY)
def upload_or_base64(local_path, key, content_type='video/mp4'):
"""Upload to S3 if configured, otherwise return base64 data URI."""
if not is_configured():
print("No S3 configured, returning base64")
with open(local_path, 'rb') as f:
data = base64.b64encode(f.read()).decode('utf-8')
return f"data:{content_type};base64,{data}"
return upload_file(local_path, key, content_type)
Add to the app's .env file:
# Hetzner Object Storage (S3-compatible)
HETZNER_S3_ENDPOINT=https://hel1.your-objectstorage.com
HETZNER_S3_ACCESS_KEY=SSZMXUFZ61GAXI19I5I9
HETZNER_S3_SECRET_KEY=<get from existing app .env>
HETZNER_S3_BUCKET=<app-name>
HETZNER_S3_REGION=hel1
If using PM2 ecosystem, add to the app's env block:
env: {
HETZNER_S3_ENDPOINT: 'https://hel1.your-objectstorage.com',
HETZNER_S3_ACCESS_KEY: 'SSZMXUFZ61GAXI19I5I9',
HETZNER_S3_SECRET_KEY: process.env.HETZNER_S3_SECRET_KEY,
HETZNER_S3_BUCKET: 'app-name',
HETZNER_S3_REGION: 'hel1',
}
All uploaded objects are publicly accessible at:
https://hel1.your-objectstorage.com/<bucket>/<key>
Example: https://hel1.your-objectstorage.com/video-story/projects/abc123/images/char_001.webp
projects/{id}/images/{name}.webpprojects/{id}/videos/{name}.mp4projects/{id}/audio/{name}.mp3uploads/{timestamp}_{filename}For Vite PWA apps, cache S3 images with CacheFirst strategy:
// vite.config.js
workbox: {
runtimeCaching: [{
urlPattern: /^https:\/\/hel1\.your-objectstorage\.com\/.*/,
handler: 'CacheFirst',
options: {
cacheName: 'images-cache',
expiration: { maxEntries: 300, maxAgeSeconds: 7 * 24 * 60 * 60 }
}
}]
}
forcePathStyle: true — REQUIRED for Hetzner. Without it, the SDK tries virtual-hosted-style URLs that don't work.
Good pattern for multi-user apps — store only provider API keys (like each user's fal.ai key) in your app DB, but persist generated media itself to Hetzner S3 immediately after generation. A solid reusable shape is:
- app DB row for users
- app DB row for assets with user_id, model, prompt, result_kind, storage_key, storage_url, optional params JSON
- on generation: call provider with user's API key -> download returned ephemeral media URL -> upload buffer to Hetzner S3 -> save asset row -> return S3 URL to frontend
This avoids ephemeral provider URLs expiring and cleanly separates per-user credentials from shared media infrastructure.
Per-user key + S3 hybrid works well on Alex's VPS — for apps like fal-studio, keep S3 credentials in server .env, but let each signed-in user store their own third-party API key in the app database/account settings. Do NOT put end-user provider keys in shared app .env.
Public object URLs require BOTH object/bucket permissions and a real fetch test — A successful PutObject can still return a URL that gives 403. For browser-public URLs on newly created buckets, set a bucket policy allowing s3:GetObject on arn:aws:s3:::BUCKET/*; optionally include ACL: 'public-read' in Node PutObjectCommand as well. Verify a concrete uploaded object with curl -s -o /dev/null -w '%{http_code}' "$URL" and expect 200. The bucket root returning 403 AccessDenied is normal and only means listing is private.
ACL: 'public-read' — Required in Python/boto3 ExtraArgs if you want public URLs. In Node.js AWS SDK v3, include ACL: 'public-read' when available, but do not rely on ACL alone; make sure the bucket policy allows public GetObject.
Bucket must exist first — Unlike some providers, Hetzner won't auto-create buckets. Create via Console or SDK before uploading.
Ephemeral URL persistence — AI services (Replicate, fal.ai) return temporary URLs that expire in hours. Always downloadAndUpload() to S3 immediately after generation.
Do a real write test, not just a code diff — before declaring S3 wired up, run an actual PutObject using the new app's .env. Also probe the public bucket URL:
- GET https://ENDPOINT/BUCKET → 403 AccessDenied usually means the bucket exists but listing is private
- GET https://ENDPOINT/BUCKET → 404 NoSuchBucket means the bucket name is wrong or the bucket was never created
On Alex's VPS this caught the exact issue where video-story worked but fal-studio pointed at a nonexistent fal-studio bucket.
Different apps may have different full secrets even with the same access key — compare secret lengths and do not assume parity. In the April 2026/May 2026 fal-studio incident, video-story/.env had a real 40-char secret and successful uploads/S3 HEAD checks, while fal-studio/.env had a different 35-char secret and S3 requests failed with SignatureDoesNotMatch or generic 403. When in doubt, inspect /home/avalon/apps/video-story/.env and test with that app's actual runtime env. If a script loads known-good credentials as fallback, do not use os.environ.setdefault when the parent process may already have stale/bad HETZNER_S3_* values; intentionally override from the known-good env for the verification subprocess.
PM2 can preserve stale S3 env across restarts — app code that parses .env with if (!process.env[key]) process.env[key] = value may silently keep an old bucket/secret inherited from PM2 or the shell. For per-app storage, prefer project-local .env as source of truth and intentionally overwrite matching HETZNER_S3_* keys at boot, then pm2 restart APP --update-env and run a real upload/HEAD smoke test. This caught Hermes Social accidentally inheriting an old S3 bucket value before switching to bucket hermes-social.
Video Story pattern = best-practice fallback for generated media — video-story uploads provider outputs to S3 with downloadAndUpload(remoteUrl, key) and, if upload fails, logs the failure and continues using the provider's remote URL instead of crashing the generation flow. For multi-user apps like fal-studio, an even safer variant is local-file fallback with a DB asset row so user galleries still work when S3 is unavailable.
Background cron/swarm jobs need their own S3 runtime, not just interactive env — for long-running Hermes jobs, copy known-good Hetzner values into a private project .env (mode 0600), create a project-local .venv, and add a scripts/bootstrap_runtime.sh that sources the env and installs missing boto3/awscli before each run. Then verify with real HeadBucket + PutObject + HeadObject + DeleteObject operations before clearing any blocked state. This prevented the Human Design wiki swarm from repeatedly blocking on absent S3 tooling/credentials under cron.
fal-studio concurrency + local persistence pattern — for mobile-first multi-user AI apps, do NOT block the entire UI on a single global generating state. Allow multiple generation requests to run concurrently from the frontend, track them in a lightweight job list (queued/running/completed/failed), and auto-save each finished asset immediately. This lets users start one job, switch models, and launch another while earlier jobs are still running.
Preserve reference uploads across compatible model switches — when users switch between similar image/video models, keep only overlapping upload buckets (generalReferences, firstFrame, lastFrame, storyboardReference) and trim to the new model's maxFiles. This avoids wiping a user's reference images when they are just comparing adjacent models in the same family.
Graceful fallback pattern for app media storage — when S3 upload fails due to missing bucket or bad credentials, store the asset locally under a per-user path such as data/local-assets/users/<userId>/<kind>/..., expose it via an authenticated/static route (or a dedicated media route), and return a warning like Saved locally because S3 is unavailable. This prevents the storage layer from blocking core generation flows.
CORS — If the frontend needs to fetch S3 objects directly (not through your API proxy), you need to configure CORS on the bucket via the Hetzner Console.
Large files — For files >5GB, use multipart upload. Both AWS SDK v3 and boto3 handle this automatically with their high-level upload methods.
Env var naming — Use HETZNER_S3_* prefix (not generic S3_*) to avoid conflicts with AWS or other S3-compatible services. Exception: RunPod workers use generic S3_* since they only talk to one storage backend.