---
name: hermes-spawn-control-plane
description: Manage Alex's Hermes Spawn app/control plane to create and administer containerized Hermes tenants via the existing web/API workflow.
version: 1.0.0
tags: [hermes, spawn, tenants, docker, control-plane, vps]
metadata:
  hermes:
    tags: [hermes, spawn, tenants, docker, control-plane, vps]
---

# Hermes Spawn Control Plane

Use this when Alex asks to create/manage a Hermes Spawn tenant, Spawn user, tenant container, Telegram setup, provider setup, or the Spawn web app. **Do not use `hermes profile create` for these requests.** Hermes Spawn is a separate app/control plane that provisions Dockerized Hermes homes.

## Source of truth

- App: `/home/avalon/apps/hermes-spawn`
- Public URL: `https://spawn.apps.poofc.com`
- PM2 process: `hermes-spawn`
- Local API: `http://127.0.0.1:4031`
- Worker host: `avalon@167.233.226.57`
- Tenant root: `/home/avalon/hermes-spawn-tenants`
- Container naming: `spawn-tenant-<tenant-id>`
- Tenant home: `/home/avalon/hermes-spawn-tenants/tenants/<tenant-id>/hermes-home`
- Data DB: `/home/avalon/apps/hermes-spawn/data/spawn-db.json`

## Important distinction

If a voice message/transcript is ambiguous about product taxonomy, stop before creating manifests, migration plans, diagrams, or Viewer artifacts. Transcribe/listen to the original audio, then echo the hierarchy back as:

```text
platform product
→ product customer account
→ product instance
→ product-derived tenant stack
→ tenant-local portfolio/workspace
→ companies/businesses/projects
```

For Caduceus specifically, the tenant stack contains the complete Portfolio OS/database/media/Hermes brain/workspace/hosting entitlement. Avalon Caduceus central owns only account identity/login, billing/entitlements, tenant directory/routing, stack versions and provisioning orders. Proper nouns and phrases such as “Caduceus user” or “tenant” are too consequential to infer from a rough platform transcript.

If Alex says “Hermes Spawn”, use the Spawn app/API. Do **not** create raw local Hermes profiles under `~/.hermes/profiles` unless he explicitly asks for a normal Hermes profile.

Raw profiles are not Spawn tenants. Current general Spawn tenants are Docker containers managed by `/home/avalon/apps/hermes-spawn/server.mjs`; product-derived tenants may be larger vertical stacks. In particular, a Caduceus tenant is not merely its Hermes container.

Hermes Spawn also has two product-family roles: it remains a general-purpose customer product, and its control-plane/runtime machinery is the intended invisible shared platform behind specialized products such as Astral Hermes and Caduceus. Keep this split explicit:

- `@hermes/agent-engine` owns pure runtime profile/config/container helpers;
- the private Spawn Platform API owns tenant-stack provisioning and lifecycle;
- each specialized product owns its own customer accounts, billing, onboarding, and branded health/recovery UX;
- for Caduceus, central product state is limited to account/billing/entitlement/directory/routing/order metadata while business domain state lives in the full customer tenant stack;
- each tenant has one fixed primary specialization.

Do not infer that shared runtime infrastructure means shared customer identity, billing, memory, credentials, or knowledge. Product services authenticate privately with opaque product owner/instance references. Provider modes are authorized per product. Cross-product access is off by default and requires an explicit scoped connector. When centralizing previously separate control planes, start with side-effect-free product manifests, service auth, request validation, and an idempotent operation ledger; migrate live provisioners only after characterization and rollback tests. The class-level architecture and acceptance gates live in `applied-agent-platforms` → `references/shared-spawn-platform-specialized-products.md`.

For Phase 1 internal tenant operation contracts, keep the contract module pure and side-effect-free: product callers send opaque owner/instance refs and a product id, while the platform derives the fixed profile from the manifest, validates provider mode/id against manifest allowlists, accepts only symbolic vault credential refs for BYO, strips credential refs from public operation projections, rejects raw secret-shaped input recursively, and keeps future containers such as channels/metadata fail-closed until schematized. See `references/internal-tenant-operation-contracts.md` for the TDD checklist and pitfalls.

For Phase 1 product-service authentication, keep auth in a pure platform module with no server/DB/deploy side effects; require exact product service ids (`spawn`, `astral-hermes`, `caduceus`), reject customer/portfolio names such as Wild Earth, hash configured tokens, compare fixed-length digests with `timingSafeEqual`, fail closed when config is incomplete or malformed, and expose only a frozen `{ productId, keyFingerprint }` identity. See `references/product-service-authentication.md` for the test checklist, header pitfalls, and env example pattern.

For Phase 1 durable operation ledgers, keep idempotency/storage in a pure platform module with no server/DB/deploy side effects; scope replay by `productId + sha256(idempotencyKey)`, persist only public operation projections plus internal digests/request hashes, atomic-write a standalone schema-versioned ledger, treat corrupt/version-drifted ledgers as unavailable without overwriting, and make retries replay-safe even after post-rename directory fsync uncertainty. Validate persisted records against current manifest semantics, reject operation-ID collisions before writing, and remember that an instance-local queue is not a multi-instance/process lock. See `references/durable-idempotent-operation-ledger.md` for the TDD checklist, atomic write recipe, privacy boundary, semantic-integrity checks, and concurrency pitfalls.

For Phase 1 authenticated internal v1 APIs, expose only side-effect-free acceptance/read surfaces around the pure primitives: authenticate before parsing, mount before broad global body parsers, use a router-local small JSON limit, require raw-header-checked idempotency keys, return JSON-only sanitized envelopes, re-project public operations, and prove tests did not mutate `spawn-db.json` or create the default live ledger. See `references/authenticated-side-effect-free-internal-v1-api.md` for route shape, Express header-materialization pitfall, TDD checklist, and verification gates.

For final security approval after provider-authority hardening, prove the platform pins exact product/profile/provider authority rather than trusting upstream/injected registry policy directly. Run an external temp probe that mutates every product's provider set (extra `evil-provider`, missing, duplicate, malformed), checks both HTTP router and direct operation-store boundaries fail closed without ledger writes, verifies reordered exact sets still normalize/succeed, and confirms gzip operation bodies are rejected with `inflate: false`. See `references/provider-authority-final-approval.md`.

For Phase 1 docs and safe operator verification, document the product taxonomy, route contracts, idempotency/ledger semantics, explicit no-runtime-execution warning, and sanitized error taxonomy. The operator verifier should be a gated external HTTP client with no endpoint/token defaults, `startNow:false`, concise safe output, and tests against an ephemeral router/temp ledger proving no `spawn-db.json` or default ledger mutation. See `references/platform-docs-and-operator-verifier.md`.

For final Phase 1 security/code-quality approval across the complete Spawn Platform authority chain, keep the pass audit-only and verify both repos together (`hermes-agent-engine` + `hermes-spawn`). Run a temporary `/tmp` probe covering contract secret-key sanitization, auth raw/normalized header smuggling, registry/product/provider drift through HTTP and direct store boundaries, gzip rejection, cross-instance same-key replay, UUID collision cleanup, duplicate/escaped JSON keys, symlink rejection, operator-verifier preflight, and live-state side-effect proof. See `references/final-phase1-security-approval.md`.

For final Phase 1 integration / operational safety approval, distinguish **code integration + side-effect safety** from **final operational approval**. Even if temp-ledger probes and tests pass, hold final approval when operational docs/config/runtime drift exists (for example live `.env`/`/api/health` worker host differs from README or `.env.example`). Use temp build output, ephemeral verifier targets, before/after live-state hashes, PM2 unchanged evidence, and no default-ledger mutation. See `references/final-phase1-operational-approval.md`.

For final Phase 1 specification approval across `@hermes/agent-engine` and Spawn Platform, run a read-only acceptance audit: canonical suites, syntax/diff gates, temp-output Spawn build, verifier refusal smoke, live-state hashes, static side-effect scan, and an external temp probe that tests provider-authority drift through both HTTP router and direct operation-store boundaries. See `references/phase1-final-spec-approval.md` for the complete approval matrix and reporting format.

For pure open-profile provisioning planning, add the side-effect-free planner and tests before any route/executor wiring: no `server.mjs`, filesystem, SSH, S3, Docker, DB, ledger, PM2, docs, or deploy changes; return a deterministic frozen plan with exact provider authority, artifact payloads, preflight classification, compensation metadata, and redacted public summary. See `references/pure-open-profile-provisioning-planner.md` for the TDD checklist and safety boundary. For final Phase 2 P2.1 planner specification approval after error-path hardening, run an external `/tmp` adversarial probe in addition to the canonical suites, including secret-shaped property-name leaks through cycles/accessors/nonenumerables/symbols/depth/proxy paths and before/after `spawn-db.json`/default-ledger hashes; see `references/phase2-p21-planner-final-approval.md`.

For pure Caduceus stack-v1 contracts, keep the module manifest/projection-only: derive the complete tenant-local Caduceus stack from fixed server authority and accepted public operation, expose only central directory/routing/status/operator-policy projections, reject caller resource/route/version/profile/credential authority, and prove no production wiring or live/default-state mutation. See `references/pure-caduceus-stack-v1-contracts.md` for exact TDD shape, central-vs-tenant-local boundaries, adversarial probes, and verification gates.

For branded one-click Caduceus signup and seamless tenant entry, keep central signup thin and tenant-local state isolated: email-only magic links store only token digests, customer-visible provisioning goes through Caduceus product-service operations with server-generated opaque refs/idempotency keys, central-to-tenant entry uses a short-lived one-time handoff to a host-only tenant session, and operator inspection stays sanitized/status-only. Do not reuse legacy general Spawn `/api/provision` or assume `/internal/v1/operations` executes unless a Caduceus executor/reconciler is actually wired. When asked for the smallest production-grade email-only signup backend, include the central schema/routes/modules, exact Spawn operation body, disabled-by-default executor wiring, and public ingress publication test gates. See `references/caduceus-one-click-signup-security.md` and `references/caduceus-passwordless-signup-tenant-create.md`.

For the first generic open-profile executor checkpoint, keep it dependency-injected and fake-adapter only: temp `OperationStore`, injected planner, fake effect adapters, `accepted -> planned -> running -> succeeded/failed` transitions, no live SSH/Docker/S3/PM2/DB mutation, and compensation only for resources evidenced as created by this operation. Optional router scheduling must be inert when no executor is injected and must not reschedule idempotent replays. See `references/open-profile-executor-checkpoint.md` for the TDD checklist, adapter surface, evidence-based compensation rules, and durable evidence boundary.

For durable forward execution claims and outcomes in the operation store, use strict TDD around `claimExecutionStep(...)` and `recordExecutionResult(...)`: claims must bind to immutable fingerprints, execute in effect order, use store-owned attempts/timestamps, persist bounded public-safe evidence, replay byte-identically after acknowledgement loss, reject stale/conflicting claims without mutation, and keep live/default state untouched. See `references/durable-forward-execution-claims.md`.

For the strictly local synthetic disposable lifecycle proof, use a temp-sandbox harness with durable `apply/reconcile/compensate/reconcileCompensation` adapters, loopback-only health runtimes, operation-owned resource markers, forced failure at every adapter boundary, and exact zero-residual cleanup. This proves create/start/status/stop/delete and durable adapter reconciliation without touching Docker, S3, PM2, worker hosts, `spawn-db.json`, default ledgers, or production router wiring. See `references/strictly-local-disposable-tenant-lifecycle.md`.

For the first unwired production remote Caduceus stack adapter protocol, keep the adapter injectable and fake-runner tested only: derive exact remote names/paths/labels/resource keys from the canonical `caduceus-stack-v1` manifest plus server-owned config, require exact `contract + tenantId + operationId` ownership for mutating lifecycle verbs, model backup/restore catalogs and upgrade/rollback pre-snapshots without payload leakage, make delete exact-name/no-glob with protected legacy names forbidden, and prove no production wiring or live/default-state mutation. See `references/production-remote-caduceus-stack-adapter.md`.

For independent high-risk review of the Caduceus durable-store/router/lock/adapter boundary, reconcile untracked tests before trusting status claims, run the focused suites that cover newly added modules, treat missing imported modules as blockers, and specifically probe durable catalog authority across adapter restart, lock scope against every mutable authority (global for a shared file-backed ledger; tenant/resource scoped only for transactionally partitioned state), host/PID-aware stale lock recovery, and router scheduling durability/outbox semantics. See `references/caduceus-durable-store-router-lock-adapter-boundary-review.md`.

For production execution locks, PID liveness is authoritative only on the hostname that acquired the lock. Include `hostname` and `lockId` in busy/manual-recovery diagnostics, treat foreign-host PID state as unknown/present, and keep foreign-host stale recovery fail-closed. See `references/production-execution-lock-host-aware.md` for the exact implementation and regression pattern.

Before designing any Spawn migration, classify the nouns explicitly: **platform product → product customer account → product instance → product-derived tenant stack → tenant-local portfolio/workspace → companies/businesses/projects**. For Caduceus, consult `/home/avalon/apps/caduceus/docs/idea-research/adr-004-per-tenant-caduceus-stack.md`; never infer a brain-only tenant or central shared Portfolio OS.

When migrating an existing general tenant into a specialized product, do not mutate its profile/home in place. Provision through the specialized product's ordinary customer/product-instance path and derive the complete target stack required by that product. Keep channels and production deployment authority singular, move only classified knowledge/business data and separately authorized credential references, run side-by-side isolation and lifecycle acceptance, then perform an audited cutover with a proven rollback. For Caduceus, Wild Earth receives an independent full Portfolio OS/data/Hermes/hosting stack and never becomes rows in Alex's database. See `references/specialized-product-canary-migration.md` for the reusable workflow.

For portable specialized tenant appliances, do not approve Dockerfile/Compose shape tests alone. Preserve dirty development sources through immutable secret-free source bundles, bind health to validated tenant/operation/contract identity, require exact non-legacy container names, use SQLite online backup plus checksummed tenant manifests, exclude raw secrets/auth/WAL, and restore offline through fully validated staging and atomic swaps with rollback. See `references/portable-specialized-tenant-appliance.md` for the executable TDD and canary gates.

For Caduceus managed Hermes bootstrap fixes, review the complete host-worker durable-I/O, backup acceptance, and runtime pickup path adversarially: final env paths, resource/topology metadata, backup catalogs/payloads/manifests, and all temp paths must use fd-based no-follow atomic I/O; archive members must be canonical, link-free, secret-free, and independently size/SHA-verified from actual tar bytes; already-created Docker containers will not pick up new `--env-file` or Hermes `.env`/`config.yaml` values unless owned containers are recreated/restarted; and idempotent repair should fix stale bad `config.yaml`, not only create missing files. Treat asynchronous independent review completion as an approval/claim gate rather than inferring approval from focused tests. See `references/caduceus-managed-hermes-bootstrap-review.md` for adversarial probes, production-backup proof, provider/API-key consistency checks, and reviewer timing rules. For fresh tenants whose UI loads but chat fails, follow `references/caduceus-fresh-tenant-chat-readiness.md`; it covers exact deployed-transport inspection, managed-provider acceptance, route-port-preserving repair, copy-only application-bridge proof, and duplicate-free recovery of a persisted failed turn.

For upgrading an existing Spawn-managed Caduceus tenant release, first derive the real topology from Docker labels/mounts and Spawn's server-owned image authority; do not confuse the portable image recipe with standalone deployment authority. Build an immutable exact-commit image with retained logs, run disposable image-level Normal/Pro/Admin/upgrade acceptance, then deploy only through the Spawn upgrade path with a durable pre-snapshot. Activate handoff contract v2 only after upgraded health, and promote the product account only after v2 activation. See `references/caduceus-managed-release-upgrade.md` for the build, acceptance-harness, permission, key, rollback, and sequencing checklist.

For a running tenant, never infer that idempotent `provision(...)` reconciles image drift: prove the app and brain immutable image IDs change in a disposable tenant before touching production. Do not delete live containers to discover reconcile semantics. Use `references/caduceus-running-stack-image-reconciliation.md` for preflight, volume UID/GID and transport-dependency compatibility, copied-home API canaries, partial-rollout recovery, random-port/tunnel repair, and required release evidence.

For destructive test-environment resets such as “keep only Alex,” use `references/caduceus-tenant-retirement-and-reset.md`. It covers positive keep sets, a concurrency cut line for user-created tenants, exact-ownership purges, per-connection SQLite foreign-key enforcement, product-scoped ledger cleanup, canonical keeper routing, and zero-orphan verification.

For Caduceus Social/OAuth releases, add the stricter gate in `references/caduceus-social-release-safety.md`: integrate latest main before exact-commit review, disarm overdue imported schedules, resolve credentials from the exact surface connection, validate grants/status/expiry before provider writes, checkpoint multi-item Story mutations, derive only the public handoff build key, run disposable Normal/Pro/Admin handoff acceptance, and upgrade via adapter snapshot/rollback rather than direct Docker replacement.

When synthetic full-stack lifecycle and restore rehearsal are complete but real provisioning is not yet wired, do not collapse “foundation complete” into “production ready.” Promote through portable appliance → inter-process lock → product-scoped router scheduling → exact second durable executor policy → unwired reviewed adapters → distinct private blank canary → zero-residual lifecycle proof. Inventory and protect any legacy runtime whose name resembles the target before choosing the canary identity, and keep feature development on the migration-source checkout isolated with a repo-local session-seed prompt. See `references/specialized-stack-production-promotion.md` for the complete production-promotion gate and exact-policy operation-store pattern.

## Basic checks

```bash
cd /home/avalon/apps/hermes-spawn
curl -fsS http://127.0.0.1:4031/api/health
pm2 list | grep hermes-spawn
```

List remote tenant containers:

```bash
ssh -o BatchMode=yes -o ConnectTimeout=10 avalon@167.233.226.57 \
  "docker ps -a --filter name=spawn-tenant --format '{{.Names}}|{{.Status}}|{{.Image}}'"
```

The current Hermes runtime may not have direct local Docker socket access; use the Spawn API or SSH worker path rather than local `docker`.

### Caduceus upgrade ingress pitfall

The Caduceus host worker publishes app port `4043` to a random worker loopback port. A successful `upgrade` removes and recreates the owned app container, so the worker port can change even though identity-bound container health passes. If public ingress uses a PM2-managed SSH forward, immediately compare `docker inspect ... .NetworkSettings.Ports` with that PM2 process's `-L local:remote` argument, retarget only the affected tenant forward, `pm2 save`, then verify local forwarded health and the public tenant URL. Treat a post-upgrade `502` with a healthy container as likely stale port forwarding, not application failure.

### Future-tenant image selector authority

Do not infer that future tenants will use a release because the desired image tags appear in a repository `.env`. First inspect how `server.mjs` loads configuration and read the **effective supervisor environment** of the running `hermes-spawn` process. PM2 can retain older injected values indefinitely, and a Node service that never imports dotenv will ignore a correct-looking `.env` entirely.

For selector changes:

1. Compare file values, PM2's effective `SPAWN_CADUCEUS_*_IMAGE` values, and the source lines that construct the production runtime.
2. Export the intended selectors into the operator shell, run `pm2 restart hermes-spawn --update-env`, then `pm2 save`.
3. Re-read PM2's effective environment and require exact values.
4. Verify both immutable tags/IDs exist on the worker and contain their claimed capabilities.
5. Prove the next disposable or real new tenant's app/brain `.Config.Image` values. Existing tenants do not inherit selector changes automatically.

A healthy Spawn endpoint plus a correct `.env` is not selector proof; the created container image IDs are the final evidence.

## Create a tenant through the API

The web UI calls `POST /api/provision`. Use an existing authenticated user session from `spawn-db.json`, or register/login via the API if needed. Avoid printing bearer tokens, passwords, API keys, or bot tokens.

Example pattern:

```bash
cd /home/avalon/apps/hermes-spawn
python - <<'PY'
import json, subprocess
from pathlib import Path
DB=Path('data/spawn-db.json')
db=json.loads(DB.read_text())
token=next(s['token'] for s in db['sessions'] if s['userId']=='user_firemountain')
payload={
  'tenantId':'wild-earth-hermes',
  'name':'Wild Earth Hermes',
  'ownerEmail':'firemountain@gmail.com',
  'provider':'openai-codex',
  'model':'gpt-5.5',
  'telegramAllowedUsers':'1932991162',
  'telegramToken':'',
  'telegramBotUsername':'',
  'selectedSkills':[],
  'curtailedSkills':[],
  'startNow': True
}
res=subprocess.run([
  'curl','-fsS','-X','POST','http://127.0.0.1:4031/api/provision',
  '-H',f'Authorization: Bearer {token}',
  '-H','Content-Type: application/json',
  '--data-binary',json.dumps(payload)
],capture_output=True,text=True,timeout=180)
print(res.returncode)
data=json.loads(res.stdout)
for k in list(data):
    if 'token' in k.lower(): data[k]='[redacted]'
print(json.dumps(data, indent=2))
PY
```

Verify:

```bash
ssh avalon@167.233.226.57 \
  "docker ps -a --filter name=^/spawn-tenant-wild-earth-hermes$ --format '{{.Names}}|{{.Status}}|{{.Image}}'"
```

## Telegram setup

Before provisioning or updating a Telegram-connected Spawn tenant, ask Alex for the necessary Telegram details unless he already provided them:
- Bot token from `@BotFather`
- Bot username, e.g. `WildEarthHermesBot`
- Allowed Telegram user IDs and/or chat IDs; default Alex user ID is `1932991162` if he says only he should access it

Spawn captures Telegram through `POST /api/telegram/update`:

Required:
- `tenantId`
- `telegramAllowedUsers` as numeric IDs, comma-separated
- optional `telegramToken`
- optional `telegramBotUsername`

If no token is available yet, provision with `telegramAllowedUsers` and blank token, then update later when Alex provides the BotFather token.

Official Telegram Bot API cannot create bots or fetch BotFather tokens. The recommended path is guided BotFather creation by the user, then paste token into Spawn. MTProto/Telethon user-account automation is possible but sensitive and should not be the default.

## Provider setup

Provider choices include:
- `openai-api` with API key
- `openai-codex` for ChatGPT/Codex OAuth
- `openrouter`
- `anthropic`

For `openai-codex`, provisioning creates the tenant configured for Codex but `setupStep` may remain `provider` until OAuth credentials are installed. The app has:
- `POST /api/provider/device-auth` — returns the OpenAI/Codex device auth URL and code for the owner to approve.
- `POST /api/provider/complete` — after approval, polls/exchanges and installs the Codex credential, then advances setup to chat.

Default path: use those endpoints / the Spawn UI so the owner explicitly hooks up the provider.

If Alex explicitly asks to reuse this Hermes' current Codex credential for a new or existing Spawn tenant, it is acceptable to copy `/home/avalon/.hermes/auth.json` into the tenant home as `auth.json`, set restrictive permissions, restart the tenant container, and smoke-test with `HERMES_HOME=/data/hermes hermes chat ...`. Do not print credential contents. Do not do this silently; it shares the same OAuth account across tenants.

Before replacement, preserve the tenant's old credential as a timestamped backup. After the real smoke test, do **not** require the two `auth.json` files to remain byte-identical: a successful Codex request can rotate the tenant token and rewrite its file. Verify the same durable OAuth identity (`active_provider`, `auth_mode`, and non-secret `account_id`) plus a successful model request instead.

### Caduceus brain image/provider repair pitfalls

- The live Caduceus host worker's managed bootstrap can drift from the operation manifest: inspect the tenant's actual `hermes-home/config.yaml`, `.env` key names, `auth.json` provider identity, and container image before believing central metadata. A current failure mode is a hardcoded OpenRouter bootstrap (`openai/gpt-5.5`, `max_tokens: 8192`) even when an internal test tenant is intended to use shared Codex OAuth.
- Share only the requested provider material. For a Codex-only test tenant, derive a least-privilege `auth.json` containing `active_provider`, only `providers.openai-codex`, and only `credential_pool.openai-codex`; do not copy unrelated Photon, XAI, Anthropic, or OpenRouter credentials.
- Old `astral-hermes-runner:dev` images may lack `python-telegram-bot`; adding it to the system Python is insufficient because that image's `hermes` wrapper executes its private venv. Install the dependency into the interpreter that the wrapper actually uses and prove `import telegram` there.
- Read-only brain containers need writable runtime state under the mounted Hermes home. Set container-start environment such as `HOME=/data/hermes`, `XDG_STATE_HOME=/data/hermes/.local/state`, `XDG_CACHE_HOME=/data/hermes/.cache`, and `XDG_CONFIG_HOME=/data/hermes/.config`; `.env` may load too late for import-time XDG path decisions.
- When recreating an old container on a newer runner image, do not blindly preserve the old image's `PATH`; it can hide the new image's Hermes venv. Also use the new image's actual Python path for health checks. A gateway can be healthy even when a stale health probe points at a nonexistent old venv.
- Some release entrypoints incorrectly require a server-owned read-only bundled-skills mount to be writable. Require it to be readable when it already exists; require only `HERMES_HOME` to be writable.
- Verify every repaired tenant with: owned container running on the intended image, private API health, actual `hermes chat -Q` model canary returning expected text, configured/active provider identity, absence of the superseded provider key from brain `.env`, and Telegram polling connected where enabled. Keep rollback containers/credential backups only through the real canary, then remove them and secret-bearing temp files.
- Do not make Alex's personal OAuth the universal production default merely to fix test tenants. The canonical provisioner needs an explicit test-only managed-Codex policy or a customer-owned provider flow; otherwise a global bootstrap change would silently share one personal account across customer tenants.

## Tenant skill propagation

When Alex asks to push skills from the operator Hermes into a Spawn tenant:

1. Inventory both source and tenant skill sets first.
2. Prefer the smallest class-level umbrella that covers the request; retain complementary tenant-specific skills rather than replacing them blindly.
3. Copy the **entire skill directory**, including `references/`, `templates/`, `scripts/`, and assets—not only `SKILL.md`.
4. Verify deterministic source/destination file counts and content digests.
5. Restart only the target tenant container.
6. Confirm discovery with `HERMES_HOME=/data/hermes hermes skills list` inside the container.
7. Run an agent-level smoke test that makes the tenant select/load the skill for its intended trigger. Filesystem presence alone is not sufficient proof.

See `references/tenant-credential-and-skill-sync.md` for exact verification patterns and the OAuth token-rotation nuance.

## Tenant app deployment broker

When a Spawn tenant needs to deploy websites/APIs on the worker host, do **not** give it unrestricted SSH, sudo, the Docker socket, `/etc/nginx`, or shared-host PM2 access. Use a token-authenticated deploy broker with a fixed command surface:

- tenant builds an artifact and manifest
- Spawn API binds a high-entropy deploy token to one tenant and stores only its hash
- root-owned worker executor validates checksum, archive safety, runtime, port range, and exact domain allowlist
- app runs under a dedicated tenant service account, never shared `avalon`
- broker renders nginx, runs `nginx -t`, reloads, performs DNS-aware Certbot, and records layer-specific status
- submission returns `202` quickly and the tenant polls status; do not hold an HTTP request open through Certbot/public checks

Public DNS may converge differently across recursive resolvers, authoritative nameservers, and Let's Encrypt secondary validators. Query every authoritative nameserver directly before certificate issuance; if they disagree, do not blame nginx/Certbot or claim HTTPS is ready. Add requested hostnames to nginx promptly so traffic that already reaches the VPS cannot fall through to the default welcome site, but issue the primary certificate only after all names in that request resolve consistently. Prefer apex + `www` + the requested public app hostname over letting stale optional subdomains block the main site. Support partial certificate issuance for propagated hostnames, report `live_partial`/pending names, and use a bounded silent DNS/TLS watchdog when registrar propagation is the only remaining blocker.

Certbot/TLS recovery rules:

- A prior partial issuance can leave a certificate lineage whose file path looks like the apex while its SAN list covers only one subdomain. Inspect the certificate actually served, not only `certbot certificates` or the lineage name.
- Noninteractive expansion of a partial multi-domain certificate requires `--expand`; use it with `--allow-subset-of-names` when the broker intentionally grows an existing lineage.
- Read the live PM2 `PORT`, broker deployment metadata, nginx `proxy_pass`, and artifact SHA during diagnosis. Do not assume the port or package from an earlier deployment is still current.
- Pause recovery watchdogs during manual broker repair. A watchdog must point to the current signed artifact/request, verify its SHA before submission, check authoritative plus public recursive DNS, and be removed after full HTTPS success so it cannot roll back a newer app.
- Keep app activation and TLS status layer-specific. If Certbot fails after nginx regeneration, preserve or restore the last known-good TLS configuration where possible rather than taking working HTTPS offline.

See `references/scoped-tenant-deploy-broker.md` for the full security boundary, request validation, isolated PM2 account pattern, asynchronous API contract, `EACCES` cwd pitfall, DNS/TLS behavior, and end-to-end verification. See `references/dns-tls-partial-certificate-recovery.md` for the partial-lineage `--expand` failure, stale recovery artifact prevention, and a complete recovery/verification sequence.

## User/account notes

Existing users live in `data/spawn-db.json`. Common owner user:
- `user_firemountain` / `firemountain@gmail.com`

Do not print password hashes, bearer tokens, or secrets. If creating a user, prefer the app's `/api/auth/register` or edit DB only when the API cannot satisfy the task.

## Verification checklist

After provisioning, verify:
1. API response has `ok: true` and expected `tenantId`.
2. `GET /api/instances` shows the tenant for the owner.
3. SSH worker shows `spawn-tenant-<tenant-id>` exists/running.
4. Tenant root exists under Spawn root.
5. If Telegram token was supplied, the instance shows `telegramTokenSaved: true` and link.
6. If provider is Codex, complete provider auth before claiming live model access.

## Cleanup mistaken raw profile

If a Spawn request was accidentally handled as a local profile:

```bash
hermes profile delete <profile-name> --yes
```

Then provision through the Spawn API instead.