← all skills

vps-app-deployment

/home/avalon/.hermes/skills/devops/vps-app-deployment/SKILL.md · raw

VPS App Deployment — *.apps.poofc.com

Full-stack deployment workflow for Alex's self-hosted VPS infrastructure.

When to Use

Infrastructure

Item Value
Main VPS IP 5.78.190.92
User avalon
Sudo/root access May require interactive/root authorization; do not assume passwordless sudo is available from Hermes. If sudo -n true reports a password is required, avoid making /etc/nginx changes unless root access is available, and report the exact blocked edit.
Domain *.apps.poofc.com (wildcard DNS)
Apps directory /home/avalon/apps/
PM2 ecosystem /home/avalon/apps/ecosystem.config.js
Nginx configs /etc/nginx/sites-available/
Nginx auth admin / SecurePass123!
GitHub author firemountain (firemountain@gmail.com)
GitHub token ~/.openclaw/credentials/github

For provisioning additional Hetzner Cloud VPS nodes when hcloud is absent, use the direct API workflow in references/hetzner-cloud-api-provisioning.md. Distinguish Hetzner Cloud API tokens from Hetzner Object Storage/S3 credentials.

When the root disk fills up (>90%), the cheapest non-disruptive fix is a Hetzner Cloud Volume mounted at /data plus symlink-migrating heavy dirs (and Docker data-root) onto it. Full recipe, pricing, and pitfalls (including the restart=no container gotcha post-migration) in references/hetzner-volume-disk-expansion.md.

For Railway-hosted projects in Alex's or partner orgs (Exxir's Projects → Exxir App, Alex's personal projects), the Railway CLI is installed at ~/.npm-global/bin/railway. The auth env var is RAILWAY_API_TOKEN, NOT RAILWAY_TOKEN — every official doc snippet uses RAILWAY_TOKEN but that only works for project-scoped tokens; account/team tokens (the "no workspace" option in the Railway UI, which reaches all workspaces) require RAILWAY_API_TOKEN. The token is saved to ~/.hermes/.env. Useful one-liners:

# List all projects across workspaces
RAILWAY_API_TOKEN=... railway list

# Discover services + deploy state + domains via GraphQL (works without `railway link`):
curl -s -X POST https://backboard.railway.com/graphql/v2 \
  -H "Authorization: Bearer $RAILWAY_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"query":"query { project(id: \"<PROJECT_ID>\") { services { edges { node { name serviceInstances { edges { node { source { repo image } domains { serviceDomains { domain } customDomains { domain } } latestDeployment { status createdAt staticUrl } } } } } } } } }"}'

Note that the GraphQL schema does not expose source directly on Service — it's on serviceInstances.edges.node.source. Using Service { source } errors with Cannot query field "source" on type "Service".

Tech Stack

Port Convention

App N gets: - API port: 3000 + N - Client port: 4000 + N

Current allocations: - hello-1: 3001/4001 - hello-2: 3002/4002 - dashboard: 4003 - todo-app: 4004 - meta-leads: 3003 - dolphin-rider: 4005 (static game, no API)

Repo Structure (Unified)

hermes-<app-name>/          # GitHub: private repo
├── client/                 # Next.js frontend
│   ├── pages/
│   ├── components/
│   │   └── LogPanel.tsx    # MANDATORY slide-out log panel
│   └── package.json
├── server/                 # Express backend
│   ├── src/
│   │   ├── utils/logger.js # Pino logger
│   │   ├── api/logs.js     # WebSocket log streaming
│   │   └── index.js
│   ├── database.db
│   └── package.json
├── README.md
└── .gitignore

Workflow for New Apps

1. Plan

2. Create GitHub Repo

GITHUB_TOKEN=$(cat ~/.openclaw/credentials/github)
curl -H "Authorization: token $GITHUB_TOKEN" \
  -d '{"name":"hermes-APP_NAME","private":true}' \
  https://api.github.com/user/repos

3. Build Server

4. Build Client

5. Deploy

# Add to PM2 ecosystem or start directly
pm2 start server/src/index.js --name "APP-server" -- --port 300N
pm2 start client/.next/standalone/server.js --name "APP-client" -- --port 400N
pm2 save

# Create nginx config
sudo cp /etc/nginx/sites-available/TEMPLATE /etc/nginx/sites-available/APP.apps.poofc.com
# Edit with correct ports and server_name
sudo ln -s /etc/nginx/sites-available/APP.apps.poofc.com /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# SSL
sudo certbot --nginx -d APP.apps.poofc.com

6. Commit & Push (MANDATORY)

git add .
git commit -m "descriptive message"
git push origin main

7. Test & Report

Nginx Config Template

server {
    listen 80;
    listen [::]:80;
    server_name APP.apps.poofc.com;

    # Basic auth (optional)
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location /api {
        proxy_pass http://localhost:300N;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }

    location /socket.io {
        proxy_pass http://localhost:300N;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }

    location / {
        proxy_pass http://localhost:400N;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

CORS Pattern

origin: (origin, callback) => {
  if (!origin) return callback(null, true);
  if (origin.endsWith('.poofc.com') || origin === 'http://localhost:3000') {
    return callback(null, true);
  }
  callback(new Error('Not allowed by CORS'));
}

Standard CRUD API

GET    /api/items       # List all
GET    /api/items/:id   # Get one
POST   /api/items       # Create {title, content}
PUT    /api/items/:id   # Update {title, content}
DELETE /api/items/:id   # Delete

Mandatory: Real-Time Log Streaming

Every app MUST have: 1. Pino logger — JSON format, log all CRUD operations 2. Socket.IO — stream logs to authenticated clients 3. LogPanel component — slide-out drawer, newest logs on top, filters, pause/resume 4. Nginx auth — only authenticated users access logs 5. Security — never log tokens/passwords, sanitize PII

Critical Rules

  1. ALWAYS commit and push after every change
  2. ALWAYS deploy after code changes (not optional)
  3. ALWAYS test deployments before declaring done
  4. ALWAYS provide URL after deployment
  5. NEVER use Vercel — self-host everything
  6. NEVER hardcode credentials — read from ~/.openclaw/credentials/
  7. ALWAYS trim whitespace from user-input credentials
  8. ALWAYS copy .next/static after Next.js standalone build
  9. ALWAYS verify OTHER apps after nginx changes — run sudo nginx -t first, and after reload, spot-check 2-3 existing apps with curl -s -o /dev/null -w "%{http_code}" https://APP.apps.poofc.com -u admin:SecurePass123! to confirm nothing broke
  10. ALWAYS nginx -t before systemctl reload nginx — NEVER reload if the test fails. A failed reload leaves nginx down for ALL apps
  11. ALWAYS add/update the app launcher at launcher.apps.poofc.com for every new deployed app — the launcher is Alex's canonical index for live VPS apps. After deploying a new app, update /home/avalon/apps/app-launcher/index.html with a visible card/link (or regenerate from live inventory), verify locally with curl --resolve launcher.apps.poofc.com:443:127.0.0.1 https://launcher.apps.poofc.com/, then commit and push firemountain/app-launcher. Do not let new apps exist only as standalone subdomains.
  12. ALWAYS create a real GPT Image branded app icon for new apps — use the configured GPT Image workflow (optionally with a design/image subagent to write/refine the prompt) to create a 1024×1024 source icon. Do not substitute quick Pillow/programmatic placeholder tiles unless Alex explicitly asks for a temporary placeholder or GPT Image is unavailable and you clearly label it temporary. The icon should be a single iOS-style rounded-square composition — no icon-inside-icon, no extra stacked card layer, no labels/text, no status dots. Save it in the app repo (for example public/app-icon.png plus generated 512/192/180 variants), use the same icon in the PWA manifest/apple-touch-icon, and surface it on launcher.apps.poofc.com. For launcher entries, also add/update the launcher's local icon asset (/home/avalon/apps/app-launcher/icons/<slug>.png) and app metadata (apps.json or the equivalent inventory source) so the launcher remains a phone-style grid of app icons.

Active Apps

Root app launcher / inventory page

App Subdomain API Port Client Port
hello-1 apps.poofc.com 3001 4001
hello-2 — 3002 4002
dashboard dashboard.apps.poofc.com — 4003
todo-app — — 4004
meta-leads leads.apps.poofc.com 3003 3000
bingo bingo.apps.poofc.com — —
hd-prism hd-prism.apps.poofc.com — —
portfolio-demo portfolio-demo.apps.poofc.com — —
video-censor video-censor.apps.poofc.com — —
dolphin-rider dolphin-rider.apps.poofc.com — 4005
mocha-pod mocha-pod.apps.poofc.com — 4006
jungle-studio jungle-studio.apps.poofc.com — 4010 (single port, Vite+Express)
street-fighter street-fighter.apps.poofc.com — 4012 (static game)
street-fighter-2 street-fighter-2.apps.poofc.com — 4013 (static game)
jungle-mobile jungle-mobile.apps.poofc.com — 4011 (Vite+Express PWA, shares jungle-studio DB)
todos todos.apps.poofc.com — 4014 (Vite+Express+SQLite, single port)
video-story video-story.apps.poofc.com — 4015 (Vite+Express+SQLite, AI video pipeline)
video-story video-story.apps.poofc.com — 4015 (Vite+Express+SQLite, AI film pipeline)
hd-prism hd-prism.apps.poofc.com 3005 4016 (Next.js client + Express/SQLite API, separate processes)

Viewer App — Dynamic Per-Response HTML Pages

Use the existing viewer app when Alex wants Hermes responses turned into rich web pages/visualizations with page-specific HTML/CSS/JS rather than a new standalone app.

Workflow for a new Viewer page: 1. Create/select a stable slug (kebab-case, descriptive, durable). 2. Create the page record with isolated markup/style/script/data for that page; do NOT force global styling if the page asks for its own visual language. 3. Create/ensure the page bucket viewer-{slug} and upload media assets there. 4. Index asset metadata in SQLite with page_slug, source/provenance, content type, storage key, and public URL. 5. Export a JSON snapshot under exported-pages/{slug}.json so the page is repo-tracked/recoverable. 6. Build/restart PM2, verify the public /p/:slug URL returns 200, verify representative S3 asset URLs return 200, then add the new page as a direct card in the Viewer pages section of /home/avalon/apps/app-launcher/index.html so it is discoverable from launcher.apps.poofc.com. 7. Commit and push both the Viewer repo changes and the launcher repo changes.

For source-atlas pages (manuscript/book/image catalogs), distinguish clearly between seed/reference imagery and direct source scans/crops. Store provenance fields such as source title, edition/manuscript, folio/page, crop notes, and extraction method so future passes can replace or add historical layers without losing the seed layer.

See references/viewer-app-pages.md for the first Viewer page pattern and verification checklist.

Vite/React Apps (Non-Next.js)

For apps built with Vite + React (not Next.js), the deployment is different.

For legacy split frontend/backend apps that need to be refactored into Alex's VPS stack — especially Vite + Firebase client apps with a separate Express/Firebase backend repo, committed node_modules, gitignored config.js, Firebase service-account JSONs, and hardcoded AI models/providers — use references/legacy-firebase-vite-app-vps-refactor.md. The target shape is usually one production Vite+Express app on a single PM2 port, with env-driven frontend config, an OpenAI-compatible model adapter, app-level auth, and no nginx Basic Auth on the PWA shell. If the refactor must live outside the original app repo, also use references/separate-vps-refactor-repo.md to create/cleanly switch to an APP-vps repo and deployed path. For a lightweight “2nd version” refactor where the old app’s assets should be preserved but the code becomes a fresh React/Vite/PWA app, see references/tarot2-react-pwa-refactor.md for the Tarot II example.

For older Vite/Vercel-style repos with a root Vite frontend plus api/server.js Express module, use references/legacy-vite-vercel-api-to-pm2.md: make the Express API serve built dist/ on one port, listen under PM2 in production, configure OpenAI SDK baseURL/headers when using OpenRouter as an OpenAI-compatible provider, verify a real model-backed endpoint, and check spoken domains against DNS before creating certs.

For exploratory project cockpits that read a markdown wiki plus a Hermes Kanban board, use the pattern in references/wiki-kanban-cockpit-pwa.md: keep wiki/Kanban as sources of truth, expose read-only Express APIs, build a mobile-first PWA, verify source-grounding/path traversal, and use bounded cron continuation only when Alex explicitly asks for autonomous work while away.

For reusable in-app Hermes chat/voice/file components, use references/hermes-app-bridge-vps-pattern.md: browser UI -> app-owned /api/hermes bridge -> Hermes /v1/runs, with server-side credentials, app context injection, SSE event proxying, voice/file affordances, Vite local-package dependency pitfalls, and the duplicate-React blank-screen fix (resolve.dedupe: ['react','react-dom']).

Before the first commit in any Vite/React app, create .gitignore with node_modules/, dist/, .env, logs, and OS junk. Never let node_modules or built dist enter the initial commit; if they do, immediately git rm -r --cached node_modules dist, commit the cleanup, and push.

For apps built with Vite + React (not Next.js), the deployment is different:

Key Differences

Extracting Vite Plugin API to Express Server

If the app has its API as a Vite plugin: 1. Read the plugin file — copy ALL route handlers into a standalone server.ts 2. Use Express with cors, compression, express.static(distDir) 3. Add SPA catch-all: app.get('/{*splat}', ...) (NOT '*' — Express 5 requires named wildcards) 4. Connect to DB using env vars (DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME)

Express 5 Gotcha

Express 5 (installed by default with newer npm) uses path-to-regexp v8+ which does NOT support bare * catch-all routes. Use '/{*splat}' instead:

// WRONG — crashes with "Missing parameter name at index 1"
app.get('*', (req, res) => res.sendFile('index.html'))

// CORRECT — Express 5 named wildcard
app.get('/{*splat}', (req, res) => res.sendFile('index.html'))

Docker Database

If the app needs a Docker database (e.g., Postgres): - Check if the default port is already in use: ss -tlnp | grep 5432 - If so, remap to a different host port (e.g., 5433:5432) - Use sudo docker-compose (not docker compose — may not work on all installs) - DB schema in docker-entrypoint-initdb.d/ runs automatically on first container start - Seed separately after container is healthy

Deployment Steps (Vite App)

# 1. Install & build frontend
npm install --legacy-peer-deps  # may need legacy flag
npx vite build                  # outputs to dist/

# 2. Start Docker DB (if needed)
sudo docker-compose -f docker-compose.prod.yml up -d

# 3. Seed database
DATABASE_URL="postgresql://user:pass@localhost:PORT/dbname" npx tsx db/seed.ts

# 4. Start with PM2 (single process — serves both API + static)
DB_PORT=5433 PORT=4010 pm2 start npx --name app-name -- tsx server.ts
pm2 save

# 5. Nginx — single upstream (no separate API/client ports)
# proxy_pass http://127.0.0.1:4010;  (everything goes to one port)

Nginx Config (Vite Single-Port App)

server {
    listen 443 ssl;
    server_name APP.apps.poofc.com;
    # ... SSL config ...

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.htpasswd;

    # API routes MUST skip basic auth — browser fetch() won't forward
    # Basic Auth credentials reliably, causing silent 401s on all API calls
    location /api/ {
        auth_basic off;
        proxy_pass http://127.0.0.1:PORT;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass http://127.0.0.1:PORT;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}

Port Allocations (Vite apps — single port)

App Subdomain Port
jungle-studio jungle-studio.apps.poofc.com 4010
jungle-mobile jungle-mobile.apps.poofc.com 4011
todos todos.apps.poofc.com 4014
video-story video-story.apps.poofc.com 4015
thursday-web thursday.apps.poofc.com 4019
transit-list-demo transit-list-demo.apps.poofc.com 4021
obsidian-quartz obsidian.apps.poofc.com 4022
language-wheels language-wheels.apps.poofc.com 4023
hermes-workspace hermes workspace service 4024
viewer viewer.apps.poofc.com 4025
astral-hermes-web astral.apps.poofc.com 4026
hermes-ads hermes-ads.apps.poofc.com planned 4027
magi magi.apps.poofc.com 4028 (Vite+Express+Postgres/local-S3; launched as firemountain/magi-vps)
hermes-social hermes-social.apps.poofc.com 4029
hermes-creative hermes-creative.apps.poofc.com 4030
hermes-spawn spawn.apps.poofc.com (live; local 127.0.0.1:4031) 4031
hermes-trader hermes-trader.apps.poofc.com (Vite+Express+SQLite, astrology/crypto paper trading cockpit) 4032
tarot tarot.apps.poofc.com (legacy alexTarot, Vite/Vercel-style Express static/API server) 4033
tarot2 tarot2.apps.poofc.com (React/Vite/Express PWA refactor of alexTarot; see references/tarot2-react-pwa-refactor.md) 4034

Hermes Trader quick reference: - Path: /home/avalon/apps/hermes-trader - Repo: firemountain/hermes-trader - PM2: hermes-trader - Port: 4032 - URL: https://hermes-trader.apps.poofc.com - Current state: Vite+React/Express + SQLite dashboard for paper-only astrology-informed crypto swap research. It has app-level developer auth, seeded strategy definitions, experiment creation, blind schedule sealing with deterministic SHA-256 hashes, paper-run reveal/alpha accounting, confidence/live-usability metrics, executable research-task queue, /leaderboard result ranking with sober one-sentence evidence summaries and data-centric grades, /monitor queue/runtime visibility, research-cycle auditing, and a cron snapshot viewer. - Auth env: live password hash is HERMES_TRADER_PASSWORD_HASH in .env; raw local mirror is /home/avalon/.hermes/secrets/hermes-trader-password.txt (0600). - Project-specific skills live inside the repo under skills/hermes-trader-*; keep them project-local rather than adding narrow global skills. Current bundle includes blind backtesting, astrology strategy lab, and ongoing research. - Hermes cron awareness uses ~/.hermes/scripts/hermes-trader-cron-snapshot.py via no-agent cron job Hermes Trader cron awareness snapshot; cronjob script paths must be relative to ~/.hermes/scripts/. - Ongoing research loop: agent cron Hermes Trader autonomous research worker (every 10m, local), no-agent script cron Hermes Trader deterministic cycle recorder (every 10m, runs hermes-trader-ongoing-research.sh -> seeds executable full-history/finer-granularity research tasks and invokes scripts/execute-research-tasks.mjs), no-agent Hermes Trader cron awareness snapshot (every 10m), and origin reporter Hermes Trader daily research digest. For BTC full-history research, default to public/free Bitstamp BTC/USD OHLC (MARKET_DATA_SOURCE=bitstamp, verified from 2011-08-18T13:00:00Z) rather than Coinbase-only history; keep Coinbase as a comparison source. - See references/hermes-trader-strategy-lab.md for the API/schema/cron/TDD verification pattern and the self-improving research-cycle architecture.

Magi debugging note: if birth submit hides the form and then shows a blank page, check for a client exception after #inputContainer is hidden and verify the chart provider. One known root cause was config.js exporting planets as an array while dataFetcher.js called .split(" "); another was astrodataserver.com TLS expiry. Normalize planet values, add submit rollback UI, and use/fallback to transit-list-demo for chart JSON/SVG. If Mercury/Venus/Mars are missing, do not rewrite the original SVG astrology-font entities; preserve the original SVG and append Unicode overlays. Prefer original body-glyph x/y coordinates when the <g class="mercury ..." data-longitude="..."> body group exists, and only calculate from JSON longitude when the body marker is absent. Ensure detection does not count aspect groups as planet markers. If interpretation sections duplicate, verify the SSE endpoint emits one title/end pair per selected planet, then check frontend lifecycle: stale EventSource, pending delayed summary timeout, stale async fetchSystemMessage/setup race, double-tap submit, every terminal stream path closing its EventSource, and HandyCollapse reinitialization. If the chart appears but the reading area is blank, inspect /api/getSummary2 SSE output and LLM provider status; OpenRouter needs OpenAI-compatible chat-completions format plus HTTP-Referer/X-Title headers. If aspect lists/SPG previews/slide-out aspect details disappear in the refactor, check the provider aspect flag translation first (jsonincludeaspects -> aspects=true for transit-list-demo) and restore the same-origin chart-data + client drawer pattern in references/magi-vps-aspect-previews.md. If registration succeeds but the modal stays open, check for overwritten global window.closeModal and treat chart-save failure as non-fatal after account creation. See references/magi-vps-chart-submit-debugging.md and references/magi-vps-aspect-previews.md.

Hermes Workspace Knowledge Tab / LLM Wiki Integration

For hermes-workspace knowledge-browser work, treat upstream outsourc-e/hermes-workspace as the base project and Alex's fork firemountain/hermes-workspace as the deploy fork. Before editing, fetch/rebase latest upstream, stash local work if dirty, then reapply changes so Workspace does not drift from the project:

cd /home/avalon/apps/hermes-workspace
git fetch --all --prune
git stash push -u -m "pre-knowledge-work-$(date +%Y%m%d-%H%M%S)"  # if dirty
git rebase origin/main

Knowledge-base discovery should be Hermes-native and profile-safe where possible: - Prefer ~/.hermes/knowledge-config.json, ~/.hermes/knowledge/, and ~/.hermes/knowledge-cache/. - Keep ~/.claude/knowledge only as a legacy fallback, not the primary path. - For Alex's current HD wiki, config points to /home/avalon/wiki-human-design; the ingestion swarm state is under /home/avalon/hd-wiki-swarm. - If the Knowledge tab is empty, first verify the config file and the server module rather than assuming the wiki is missing.

Useful verification pattern for Workspace knowledge changes:

pnpm build
node --import tsx /tmp/check-workspace-knowledge.mjs   # or an equivalent script importing src/server modules
pm2 restart hermes-workspace --update-env
pm2 save
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4024/
curl -k --resolve hermes.apps.poofc.com:443:127.0.0.1 -o /dev/null -w '%{http_code}\n' https://hermes.apps.poofc.com/

For Hermes Workspace as an installable PWA, public root should ultimately be 200 without nginx Basic Auth; if it is still 401, standalone iOS/Android launches can fail before app-level auth loads. Local 127.0.0.1:4024 should return 200. App/API routes are app-auth protected, so direct unauthenticated curls to /api/* may return 401; verify protected behavior with a valid hermes-workspace-auth cookie from ~/.hermes/workspace-sessions.json or by importing server modules with node --import tsx when no session cookie is available. See software-development/pwa-configuration reference references/hermes-workspace-pwa.md for the current PWA/auth/nginx workflow.

Lint caveat: full-repo pnpm lint in hermes-workspace can OOM or fail on large pre-existing upstream lint debt. Use NODE_OPTIONS='--max-old-space-size=4096' and run targeted ESLint on changed files. Do not block deploy on unrelated repo-wide lint failures when pnpm build and targeted changed-file lint pass.

See references/hermes-workspace-knowledge-integration.md for the session-specific multi-base Knowledge tab workflow, verification script pattern, and current HD wiki baseline. | astral-hermes-web | astral.apps.poofc.com | 4026 | | hermes-ads | hermes-ads.apps.poofc.com | planned 4027 |

Hermes Ads quick reference: - Path: /home/avalon/apps/hermes-ads - Repo: firemountain/hermes-ads (private) - PM2: hermes-ads - Port: 4027 - URL: https://hermes-ads.apps.poofc.com - Current state: Vite+React/Express + SQLite dashboard MVP. It serves a mobile-first Meta ads cockpit with Command, Account, Reports, Drafts, Setup, and Audit tabs; read-only Meta sync/reporting works for Astro Mage; local drafts/approvals/audit events exist; Meta writes are still intentionally disabled until ads_management and explicit approved paused writes. - Local DB: /home/avalon/apps/hermes-ads/data/hermes-ads.sqlite (uncommitted; .gitignore ignores data/ and SQLite WAL/SHM files). - For architecture/build guidance, load the paid-ads-agent-platforms skill, especially references/hermes-ads-dashboard-mvp.md.

Astral Hermes quick reference: - Path: /home/avalon/apps/astral-hermes-platform - Repo: firemountain/astral-hermes-platform - Public URL: https://astral.apps.poofc.com - Public root /: strict minimal mock chat/knowledge-console landing page. Use the original soft light-gray Astral palette, not the later parchment/reference-image treatment. No big titles/subtitles/filter chips/decorative orbits. Mobile-first: hidden side panel, almost blank chat, one animated prompt, bottom input. Empty input shows a working mic recorder with a proper SVG microphone icon; typed input swaps to send; text/audio messages post into chat and trigger randomized mock replies. Voice note playback should be custom and Telegram-like in layout only — large circular play/pause, waveform bars, duration/status, small ••• control — but never Telegram green/blue; use Astral's own warm ink/espresso, muted bronze/gold, and soft gray palette. Avoid native browser audio controls, volume/mute buttons, or bulky audio chrome. Chat composer UX: it should behave like a normal multiline message box — Return/Enter inserts a newline, only the visible send button/form submit sends, multiline bubbles render with white-space: pre-wrap, and the textarea auto-grows to a small max height before internal scrolling. For exact code checkpoints and a Playwright --no-sandbox verification recipe, see references/astral-composer-multiline-verification.md. For chat interaction polish, prefer very subtle Telegram-quality micro-animations: 140–220ms control feedback, 300–420ms message/overlay transitions, tiny rise/scale/blur fades, typing dots during delayed mock replies, input focus lift, tactile press compression, waveform bar entrance/progress transitions, eased sidebar slide + scrim fade, all respecting prefers-reduced-motion. The actual invite-code provisioning wizard lives at /onboarding; keep smoke/provision tests pointed at /onboarding. - Admin URL: https://astral.apps.poofc.com/admin (PWA-safe public app shell; admin API access is protected by app-level bearer token stored in the client, not nginx Basic Auth. /api/admin/* must require the token and must never return raw tenant secrets.) - PM2: astral-hermes-web - Port: 4026 - Worker node: astral-node-1 (5.78.199.26), runtime root /srv/astral - Purpose: invite-code guarded onboarding/control-plane MVP that provisions Dockerized tenant Hermes homes/containers on the worker node. - Admin console exposes tenant status, local skill inventory/inspector, planned skill profiles, and worker bundle listing. Admin APIs must return only secret variable names such as OPENAI_API_KEY/TELEGRAM_BOT_TOKEN, never values from tenant .env. - Admin tenant counts should distinguish real tenants from benchmark/test directories: benchmark dirs like bench-* are real filesystem directories but should not inflate headline customer/real tenant metrics. - Astral UI should be mobile-first, compact, minimal light-gray, and Typeform-inspired for onboarding. On mobile, keep admin skill/profile cards tight, avoid oversized padding, and ensure <a> elements styled as buttons use flex centering (display:inline-flex; align-items:center; justify-content:center) so text is visually centered. - Astral public chat landing preference: preserve the original pre-reference-image palette from around commit 0046acf (#eeeeeb/soft light-gray gradient, translucent white panels, muted gray/gold accents). Keep the chat landing strictly minimal: no big titles, subtitles, unnecessary section headers, chat-message blocks, decorative orbits, or filter chips. Mobile-first means the knowledge/sidebar panel is hidden completely behind a hamburger/scrim, the page is almost blank, one small animated prompt appears near the top of the chat, and the input sits directly at the bottom of the screen rather than nested inside a card. Desktop can use a simple ChatGPT-like left sidebar plus blank main chat and bottom-centered input. - Current skill rollout state: bin/astral.mjs supports bundle create/publish/install/status; production tenants should have astral-core@0.2.0 plus astral-hd@0.2.0. astral-core includes astral-chart-api so tenants call transit-list-demo for astrology chart calculation rather than falling back to vague/non-tool responses. astral-hd includes hd-prism-tenant-api, which calls shared HD Prism POST https://hdprism.apps.poofc.com/api/tenant/bodygraph instead of tenant-local HD dependencies. Scoped tenant containers must set HERMES_BUNDLED_SKILLS=/data/hermes/.astral/no-bundled-skills or Hermes will auto-sync the full bundled skill catalog back into $HERMES_HOME/skills on startup. For openai-codex tenants, voice transcription needs a control-plane fallback key and chart/HD skills need a real service-call path (usually terminal enabled); see references/astral-hermes-platform.md “Voice transcription and tenant API-call capability”. - PWA/control-plane admin pattern: keep installable app-shell routes public at nginx and protect /api/admin/* with app-level bearer/session auth; see references/pwa-control-plane-admin.md. - Onboarding provisioning UX should expose durable progress on the submit/review step, not just a momentary disabled button. For Astral, verify /api/provision directly with a disposable tenant before assuming backend failure; clean up remote tenant/container and entitlement ledger rows afterward. Return telegramBotUsername/telegramLink so success can show an Open Telegram bot chat button. See references/astral-hermes-platform.md “Onboarding provisioning UX and debugging”. - BYO OpenAI API key provisioning must generate Hermes config as provider: custom, base_url: https://api.openai.com/v1, api_mode: chat_completions, api_key: ${OPENAI_API_KEY}. Do not use provider: openai (Hermes may reject it) or openai-codex (OAuth/subscription flow, not raw API key). For subscription-first onboarding, default to openai-codex / ChatGPT subscription, skip raw API-key collection, then complete device authorization from the Astral control plane and write the OAuth credential into tenant auth.json; see references/astral-onboarding-subscription-provider.md and references/hermes-spawn-control-plane.md. Verify inside the tenant container with hermes chat -q 'Reply with exactly: ok' -Q after restarting the container. - Session-specific details, endpoints, bundle rollout, Stripe hosted-beta billing, customer-onboarding Checkout flow, tenant knowledge-base/transit scaffolding, and the current web-chat-to-tenant-Hermes bridge/readiness gates: references/astral-hermes-platform.md. For the live MVP bridge, /chat is protected by either a logged-in customer session that owns the requested tenant or the operator ASTRAL_CHAT_TOKEN; /api/chat/* uses the Astral backend to SSH/docker-exec into tenant containers with safe base64 prompt transport and Hermes session-id parsing; root / remains a mock demo. - Astral customer auth pattern: expose /account as the customer login/registration dashboard, keep registration invite-code gated, hash passwords server-side (current implementation uses PBKDF2), store sessions in signed HttpOnly cookies (ASTRAL_SESSION_SECRET must exist in .env), and never return passwordHash from account APIs/admin entitlements. Onboarding should avoid double sign-up: call the optional /api/auth/session endpoint, reuse an existing logged-in account, and skip the account/password step when a session exists; only collect owner email + account password when there is no session. Provisioning should set the account password after provisioning, set the session cookie, and link success to /account and /chat?tenant=.... Customer chat/status endpoints must authorize tenant ownership from the entitlement ledger; keep operator token support as an override only for admin/testing. Regression tests should cover safe account serialization, bad login rejection, owned-tenant access, and cross-tenant denial. For the current onboarding/subscription-provider shape and verification checklist, see references/astral-onboarding-subscription-provider.md. - Astral live web chat UX: chat is the primary surface, especially on mobile. Do not stack account/tenant/token/status controls above the chat on small screens; put those secondary controls in a proper left slide-out drawer behind a hamburger + scrim + close button, leaving the chat and bottom composer visible immediately. Desktop may keep a left sidebar, but make it wide enough for cards/emails and not a crushed strip. If the user is signed in, do not keep public signup/onboarding CTAs in the chat chrome. Put logged-in identity, tenant connect controls, tenant online/status, and logout inside a collapsed Settings group in the side panel; keep the primary chat action (New thread) in the top-right chat header. If using <details> for Settings, visually verify that closed Settings contents have zero visible layout/rect area (add an explicit .settings-section:not([open]) .settings-stack { display: none; } guard if custom CSS causes hidden children to reserve space). If the chat shell probes auth on load, use an optional session endpoint that returns { account:null, tenants:[] } when logged out so public /chat smoke tests do not emit expected-but-noisy 401 console errors; keep strict /api/auth/me protected for security regression coverage.

Simple Static Apps (No Next.js)

For games, demos, or simple frontends that don't need SSR/API:

app-name/
├── server.js          # Express serving static files from project root
├── index.html         # Entry point (may use ES modules)
├── src/               # JS source files
├── public/            # Assets (images, sounds)
├── package.json
├── README.md
└── .gitignore

Skip the full client/server split, LogPanel, and Socket.IO — use a single Express server. Reserve the full stack for CRUD apps.

Static App Server Template

When serving static sites that use ES modules (<script type="module">), the catch-all route MUST NOT serve index.html for .js/.css/asset requests — browsers enforce strict MIME type checking for module scripts and will reject text/html responses.

const express = require('express');
const path = require('path');
const app = express();
const PORT = process.env.PORT || 4012;

// Serve static files FIRST with correct MIME types
app.use(express.static(path.join(__dirname), {
  setHeaders: (res, filePath) => {
    if (filePath.endsWith('.js') || filePath.endsWith('.mjs')) {
      res.setHeader('Content-Type', 'application/javascript');
    }
  }
}));

// Catch-all: only serve index.html for non-file requests
app.get('/{*splat}', (req, res) => {
  if (path.extname(req.path)) {
    return res.status(404).send('Not found');
  }
  res.sendFile(path.join(__dirname, 'index.html'));
});

app.listen(PORT, '::', () => console.log(`Running on port ${PORT}`));

Deploying a cloned GitHub static app

# 1. Clone to /home/avalon/apps/app-name
git clone https://github.com/user/repo /home/avalon/apps/app-name
cd /home/avalon/apps/app-name

# 2. Add server.js (template above), npm init -y, npm install express

# 3. Start with PM2
pm2 start server.js --name app-name
pm2 save

Pitfalls

app.get('/{*splat}', (req, res, next) => {
  if (path.extname(req.path)) {
    return res.status(404).send('Not found');
  }
  res.sendFile(path.join(__dirname, 'index.html'));
});

import numpy as np
from PIL import Image
logo = Image.open('logo.png').convert('RGBA')
arr = np.array(logo)
alpha = arr[:,:,3].copy()
# Recolor to desired brand color
arr[:,:,0], arr[:,:,1], arr[:,:,2] = RED, GREEN, BLUE
arr[:,:,3] = alpha
logo_clean = Image.fromarray(arr, 'RGBA')

<meta name="theme-color" content="#2B2926" media="(prefers-color-scheme: dark)">
<meta name="theme-color" content="#2B2926">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">

And in CSS ensure body { background-color: var(--color-background); padding-top: env(safe-area-inset-top); }. This makes the iOS status bar area blend seamlessly into the app background instead of showing a contrasting color. - manifest.json needs no-cache headers too: Just like sw.js, serve manifest.json with Cache-Control: no-cache, no-store, must-revalidate from Express so icon, theme, and name changes propagate immediately to installed PWAs without requiring a reinstall. - Containerized tenants: env vars in tenant .env files do NOT auto-propagate into child processes spawned by the runtime: When a multi-tenant control plane provisions docker containers (e.g. one container per tenant Hermes home), the container's Config.Env is fixed at docker run time. Writing FOO=bar into the mounted tenant .env file does not put FOO into os.environ for Python/Node tools the agent runs inside the container. Symptom: skill or tool code reads os.environ.get('SOMETHING'), finds nothing, silently falls back to a default path (often ~/.hermes/... inside the container's writable layer instead of the mounted volume), claims success, but nothing persists to the host. Fix: bake required env vars into the docker run -e FOO=bar command at container creation. The .env file is for documentation and for the agent to read explicitly via skills — not for ambient inheritance. After fixing the docker run line, you must RECREATE the existing containers (docker rm -f then docker run); a plain docker restart keeps the original Config.Env. Verify with docker inspect <container> --format '{{.Config.Env}}' and docker exec <container> env | grep FOO. This bit me on Astral when ASTRAL_KB_ROOT was set in tenant .env files but the astral-tenant-kb skill couldn't see it — agent confidently said "saved to KB" while writing to a path that vanished on container restart. - Verifying multi-tenant isolation when concerned about cross-tenant leakage: When the user worries about whether one tenant can see another's data, give grounded evidence not assurances. Five quick checks: (1) ls /srv/<root>/tenants/<id>/hermes-home/knowledge per tenant — confirm each has its own dir with 0700 perms; (2) docker inspect <container> --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}' per container — confirm each only mounts its own tenant dir; (3) docker inspect <container> --format 'CapDrop={{.HostConfig.CapDrop}} Privileged={{.HostConfig.Privileged}}' — confirm --cap-drop=ALL and no privileged mode; (4) probe test: write a sentinel file in tenant A's volume, then docker exec <container-B> ls /data/hermes/<path> to confirm B can't see it; (5) review app-level middleware that gates tenant access (requireTenantOwner or equivalent) — confirm the user-to-tenant binding is enforced server-side, not just client-side. Capture the actual command outputs in your response — the user wants to see the evidence, not be told "yes it's secure".

Redeploy Workflow (GitHub as Truth)

For updating existing apps when the remote repo is the source of truth.

1. Check Current State

cd /home/avalon/apps/APP_NAME
git status --short
git log --oneline -1
git fetch origin
git status -sb
git log --oneline HEAD..origin/main

2. Preserve or Discard Local Changes

For Alex's existing apps, do not blindly discard uncommitted local work when asked to "pull and deploy latest". First inspect it. If there are tracked/untracked changes and the request is only to deploy remote latest, preserve them in a named stash, then fast-forward pull:

git stash push -u -m "pre-deploy-local-changes-$(date +%Y%m%d-%H%M%S)"
git pull --ff-only

Report the stash name in the final response so the work can be recovered.

Only discard local changes when the user explicitly asks to throw them away or when you have confirmed they are disposable:

git checkout -- .          # discard tracked changes
git clean -fd              # remove untracked files

If git clean -fd hangs, remove untracked files manually:

rm -f UNTRACKED_FILE1 UNTRACKED_FILE2

3. Pull Latest

If the working tree is clean, pull directly:

git pull --ff-only

4. Build (App-Specific)

App Build Command Notes
jungle-studio npm run build tsc -b \|\| true && vite build — TS errors don't block
jungle-mobile npm run deploy npx vite build — skips TS checking entirely
video-story npm run build Standard Vite build
todos npm run build Standard Vite build

Always verify dist/ was actually updated: stat dist/assets/index-*.js | grep Modify

5. Restart PM2

pm2 restart APP_NAME

Verify runtime paths if started manually:

pm2 show APP_NAME | grep -E "cwd|script|args"

6. Health Check

# Local port
curl -s -o /dev/null -w "%{http_code}" http://localhost:PORT/

# Public URL (jungle-studio returns 401 = expected basic auth)
curl -s -o /dev/null -w "%{http_code}" https://APP.apps.poofc.com/

Jungle Apps Quick Reference

App Path Port Public URL Auth
jungle-studio /home/avalon/apps/jungle-studio-dashboard 4010 jungle-studio.apps.poofc.com Basic auth
jungle-mobile /home/avalon/apps/jungle-studio-mobile 4011 jungle-mobile.apps.poofc.com None

Thursday Quick Reference

Item Value
Path /home/avalon/apps/thursday
Repo firemountain/thursday
URL https://thursday.apps.poofc.com
PM2 thursday-api, thursday-web
API port 3019 (API_PORT=3019)
Web port 4019
DB Postgres via DATABASE_URL in .env, host port 55432
Build npm run build (tsc -b && vite build)
DB commands npm run db:generate, npm run db:push, npm run db:seed

Thursday is a Vite React web app plus compiled TypeScript Fastify API. PM2 runs API from dist/api/src/server.js, so API source changes require npm run build before pm2 restart thursday-api. The web build outputs to dist/web and is served by thursday-web. After schema changes, run npm run db:generate before lint/build so Prisma Client exposes new models; run npm run db:push before exercising new endpoints. App-level admin auth belongs in Thursday itself; do not add nginx basic auth to guest/PWA flows. Required admin env vars include THURSDAY_ADMIN_PASSWORD, THURSDAY_ADMIN_SESSION_SECRET, and THURSDAY_ADMIN_PIN=3693; fail closed if missing, do not provide default credentials or hardcoded PINs. The PIN-protected table QR manager endpoint is /api/admin/paradiso/tables?pin=...; after PIN/admin changes verify the correct PIN returns 200 and an old/wrong PIN returns 401, e.g. public curls against https://thursday.apps.poofc.com/api/admin/paradiso/tables?pin=3693 and ?pin=3696. Promotion/admin features should verify with local curls against port 3019, then public curls against https://thursday.apps.poofc.com/ and /admin. Thursday API health is /api/health (not /health); use curl http://localhost:3019/api/health and curl https://thursday.apps.poofc.com/api/health for deploy verification. For live Toast changes, also smoke-test a configured bill endpoint such as https://thursday.apps.poofc.com/api/t/paradiso/table-19/bill.

Thursday production env gotcha

Do NOT copy local-machine .env ports directly onto the VPS. The VPS production runtime uses API port 3019 and web port 4019 behind nginx, with Postgres on host port 55432. For the built public web app, prefer VITE_API_URL="" (same-origin /api through nginx) instead of http://localhost:4000; otherwise browsers on the public site may try to call their own localhost or the wrong port. Toast live data requires local, uncommitted .env values for TOAST_API_BASE, TOAST_CREDENTIAL_SET, TOAST_CLIENT_ID, TOAST_CLIENT_SECRET, optional ERA credentials, TOAST_GUID_PARADISO, TOAST_LIVE_TABLE_19_ENABLED, and TOAST_LIVE_TABLES. Never commit real Toast credentials.

Thursday live Toast verification

After changing Thursday .env, restart API with env update: pm2 restart thursday-api --update-env && pm2 save. Verify in order:

curl -s http://localhost:3019/api/dev/toast/auth
curl -s "http://localhost:3019/api/dev/toast/orders?businessDate=$(TZ=America/Chicago date +%Y%m%d)&pageSize=5"
curl -s "https://thursday.apps.poofc.com/api/admin/paradiso/tables?pin=3693"
curl -s https://thursday.apps.poofc.com/api/t/paradiso/table-19/bill

/api/dev/toast/auth should show configured: true. If a table is listed in TOAST_LIVE_TABLES but /api/t/paradiso/table-N/bill returns Bill not found, try POST /api/dev/toast/sync/table-N; a 404 No open live Toast check found for that table means env/auth is working but Toast currently has no open check for that table.

HD Prism — Compiled TypeScript API

HD Prism has a separate Express/SQLite API (apps/api/) that runs from compiled JavaScript in dist/, NOT from the TypeScript source directly.

Architecture

CRITICAL: API must be recompiled after TypeScript changes

The API runs from apps/api/dist/routes/*.js. Editing src/routes/charts.ts does NOTHING until you recompile:

cd /home/avalon/apps/hd-prism/apps/api && npx tsc

Then restart: pm2 restart hd-prism-api

Symptoms of stale dist/: New fields added to API routes don't appear in responses. Frontend filters/features that depend on new API fields silently fail because the data isn't there.

Extensible Filter Architecture

The sidebar uses a FILTER_CATEGORIES registry pattern for chart filtering: - Each category has: key (field name), label, mode ("scalar" or "array"), optional staticOptions/dynamicOptions - chartPassesFilters() and FilterDrawer work generically over the registry - To add a new filter (e.g., gates, channels, authority): just add an entry to FILTER_CATEGORIES in Sidebar.tsx - Current active filters: type, profile. Future: authority, definition, gates, channels (all already returned by API, commented out in registry)

API returns full chart metadata for filtering

GET /api/charts now returns per-chart: type, profile, authority, definition, gates (number[]), channels (string[]) — all parsed from the chart_json blob without DB migration.

Build + Deploy

cd /home/avalon/apps/hd-prism/apps/api && npx tsc       # rebuild API
cd /home/avalon/apps/hd-prism/apps/hdprism && npx next build  # rebuild client
pm2 restart hd-prism-api hd-prism-client