← all skills

hermes-spawn-control-plane

/home/avalon/.hermes/skills/devops/hermes-spawn-control-plane/SKILL.md · raw

Hermes Spawn Control Plane

Use this when Alex asks to create/manage a Hermes Spawn tenant, Spawn user, tenant container, Telegram setup, provider setup, or the Spawn web app. Do not use hermes profile create for these requests. Hermes Spawn is a separate app/control plane that provisions Dockerized Hermes homes.

Source of truth

Important distinction

If a voice message/transcript is ambiguous about product taxonomy, stop before creating manifests, migration plans, diagrams, or Viewer artifacts. Transcribe/listen to the original audio, then echo the hierarchy back as:

platform product
→ product customer account
→ product instance
→ product-derived tenant stack
→ tenant-local portfolio/workspace
→ companies/businesses/projects

For Caduceus specifically, the tenant stack contains the complete Portfolio OS/database/media/Hermes brain/workspace/hosting entitlement. Avalon Caduceus central owns only account identity/login, billing/entitlements, tenant directory/routing, stack versions and provisioning orders. Proper nouns and phrases such as “Caduceus user” or “tenant” are too consequential to infer from a rough platform transcript.

If Alex says “Hermes Spawn”, use the Spawn app/API. Do not create raw local Hermes profiles under ~/.hermes/profiles unless he explicitly asks for a normal Hermes profile.

Raw profiles are not Spawn tenants. Current general Spawn tenants are Docker containers managed by /home/avalon/apps/hermes-spawn/server.mjs; product-derived tenants may be larger vertical stacks. In particular, a Caduceus tenant is not merely its Hermes container.

Hermes Spawn also has two product-family roles: it remains a general-purpose customer product, and its control-plane/runtime machinery is the intended invisible shared platform behind specialized products such as Astral Hermes and Caduceus. Keep this split explicit:

Do not infer that shared runtime infrastructure means shared customer identity, billing, memory, credentials, or knowledge. Product services authenticate privately with opaque product owner/instance references. Provider modes are authorized per product. Cross-product access is off by default and requires an explicit scoped connector. When centralizing previously separate control planes, start with side-effect-free product manifests, service auth, request validation, and an idempotent operation ledger; migrate live provisioners only after characterization and rollback tests. The class-level architecture and acceptance gates live in applied-agent-platforms → references/shared-spawn-platform-specialized-products.md.

For Phase 1 internal tenant operation contracts, keep the contract module pure and side-effect-free: product callers send opaque owner/instance refs and a product id, while the platform derives the fixed profile from the manifest, validates provider mode/id against manifest allowlists, accepts only symbolic vault credential refs for BYO, strips credential refs from public operation projections, rejects raw secret-shaped input recursively, and keeps future containers such as channels/metadata fail-closed until schematized. See references/internal-tenant-operation-contracts.md for the TDD checklist and pitfalls.

For Phase 1 product-service authentication, keep auth in a pure platform module with no server/DB/deploy side effects; require exact product service ids (spawn, astral-hermes, caduceus), reject customer/portfolio names such as Wild Earth, hash configured tokens, compare fixed-length digests with timingSafeEqual, fail closed when config is incomplete or malformed, and expose only a frozen { productId, keyFingerprint } identity. See references/product-service-authentication.md for the test checklist, header pitfalls, and env example pattern.

For Phase 1 durable operation ledgers, keep idempotency/storage in a pure platform module with no server/DB/deploy side effects; scope replay by productId + sha256(idempotencyKey), persist only public operation projections plus internal digests/request hashes, atomic-write a standalone schema-versioned ledger, treat corrupt/version-drifted ledgers as unavailable without overwriting, and make retries replay-safe even after post-rename directory fsync uncertainty. Validate persisted records against current manifest semantics, reject operation-ID collisions before writing, and remember that an instance-local queue is not a multi-instance/process lock. See references/durable-idempotent-operation-ledger.md for the TDD checklist, atomic write recipe, privacy boundary, semantic-integrity checks, and concurrency pitfalls.

For Phase 1 authenticated internal v1 APIs, expose only side-effect-free acceptance/read surfaces around the pure primitives: authenticate before parsing, mount before broad global body parsers, use a router-local small JSON limit, require raw-header-checked idempotency keys, return JSON-only sanitized envelopes, re-project public operations, and prove tests did not mutate spawn-db.json or create the default live ledger. See references/authenticated-side-effect-free-internal-v1-api.md for route shape, Express header-materialization pitfall, TDD checklist, and verification gates.

For final security approval after provider-authority hardening, prove the platform pins exact product/profile/provider authority rather than trusting upstream/injected registry policy directly. Run an external temp probe that mutates every product's provider set (extra evil-provider, missing, duplicate, malformed), checks both HTTP router and direct operation-store boundaries fail closed without ledger writes, verifies reordered exact sets still normalize/succeed, and confirms gzip operation bodies are rejected with inflate: false. See references/provider-authority-final-approval.md.

For Phase 1 docs and safe operator verification, document the product taxonomy, route contracts, idempotency/ledger semantics, explicit no-runtime-execution warning, and sanitized error taxonomy. The operator verifier should be a gated external HTTP client with no endpoint/token defaults, startNow:false, concise safe output, and tests against an ephemeral router/temp ledger proving no spawn-db.json or default ledger mutation. See references/platform-docs-and-operator-verifier.md.

For final Phase 1 security/code-quality approval across the complete Spawn Platform authority chain, keep the pass audit-only and verify both repos together (hermes-agent-engine + hermes-spawn). Run a temporary /tmp probe covering contract secret-key sanitization, auth raw/normalized header smuggling, registry/product/provider drift through HTTP and direct store boundaries, gzip rejection, cross-instance same-key replay, UUID collision cleanup, duplicate/escaped JSON keys, symlink rejection, operator-verifier preflight, and live-state side-effect proof. See references/final-phase1-security-approval.md.

For final Phase 1 integration / operational safety approval, distinguish code integration + side-effect safety from final operational approval. Even if temp-ledger probes and tests pass, hold final approval when operational docs/config/runtime drift exists (for example live .env//api/health worker host differs from README or .env.example). Use temp build output, ephemeral verifier targets, before/after live-state hashes, PM2 unchanged evidence, and no default-ledger mutation. See references/final-phase1-operational-approval.md.

For final Phase 1 specification approval across @hermes/agent-engine and Spawn Platform, run a read-only acceptance audit: canonical suites, syntax/diff gates, temp-output Spawn build, verifier refusal smoke, live-state hashes, static side-effect scan, and an external temp probe that tests provider-authority drift through both HTTP router and direct operation-store boundaries. See references/phase1-final-spec-approval.md for the complete approval matrix and reporting format.

For pure open-profile provisioning planning, add the side-effect-free planner and tests before any route/executor wiring: no server.mjs, filesystem, SSH, S3, Docker, DB, ledger, PM2, docs, or deploy changes; return a deterministic frozen plan with exact provider authority, artifact payloads, preflight classification, compensation metadata, and redacted public summary. See references/pure-open-profile-provisioning-planner.md for the TDD checklist and safety boundary. For final Phase 2 P2.1 planner specification approval after error-path hardening, run an external /tmp adversarial probe in addition to the canonical suites, including secret-shaped property-name leaks through cycles/accessors/nonenumerables/symbols/depth/proxy paths and before/after spawn-db.json/default-ledger hashes; see references/phase2-p21-planner-final-approval.md.

For pure Caduceus stack-v1 contracts, keep the module manifest/projection-only: derive the complete tenant-local Caduceus stack from fixed server authority and accepted public operation, expose only central directory/routing/status/operator-policy projections, reject caller resource/route/version/profile/credential authority, and prove no production wiring or live/default-state mutation. See references/pure-caduceus-stack-v1-contracts.md for exact TDD shape, central-vs-tenant-local boundaries, adversarial probes, and verification gates.

For branded one-click Caduceus signup and seamless tenant entry, keep central signup thin and tenant-local state isolated: email-only magic links store only token digests, customer-visible provisioning goes through Caduceus product-service operations with server-generated opaque refs/idempotency keys, central-to-tenant entry uses a short-lived one-time handoff to a host-only tenant session, and operator inspection stays sanitized/status-only. Do not reuse legacy general Spawn /api/provision or assume /internal/v1/operations executes unless a Caduceus executor/reconciler is actually wired. When asked for the smallest production-grade email-only signup backend, include the central schema/routes/modules, exact Spawn operation body, disabled-by-default executor wiring, and public ingress publication test gates. See references/caduceus-one-click-signup-security.md and references/caduceus-passwordless-signup-tenant-create.md.

For the first generic open-profile executor checkpoint, keep it dependency-injected and fake-adapter only: temp OperationStore, injected planner, fake effect adapters, accepted -> planned -> running -> succeeded/failed transitions, no live SSH/Docker/S3/PM2/DB mutation, and compensation only for resources evidenced as created by this operation. Optional router scheduling must be inert when no executor is injected and must not reschedule idempotent replays. See references/open-profile-executor-checkpoint.md for the TDD checklist, adapter surface, evidence-based compensation rules, and durable evidence boundary.

For durable forward execution claims and outcomes in the operation store, use strict TDD around claimExecutionStep(...) and recordExecutionResult(...): claims must bind to immutable fingerprints, execute in effect order, use store-owned attempts/timestamps, persist bounded public-safe evidence, replay byte-identically after acknowledgement loss, reject stale/conflicting claims without mutation, and keep live/default state untouched. See references/durable-forward-execution-claims.md.

For the strictly local synthetic disposable lifecycle proof, use a temp-sandbox harness with durable apply/reconcile/compensate/reconcileCompensation adapters, loopback-only health runtimes, operation-owned resource markers, forced failure at every adapter boundary, and exact zero-residual cleanup. This proves create/start/status/stop/delete and durable adapter reconciliation without touching Docker, S3, PM2, worker hosts, spawn-db.json, default ledgers, or production router wiring. See references/strictly-local-disposable-tenant-lifecycle.md.

For the first unwired production remote Caduceus stack adapter protocol, keep the adapter injectable and fake-runner tested only: derive exact remote names/paths/labels/resource keys from the canonical caduceus-stack-v1 manifest plus server-owned config, require exact contract + tenantId + operationId ownership for mutating lifecycle verbs, model backup/restore catalogs and upgrade/rollback pre-snapshots without payload leakage, make delete exact-name/no-glob with protected legacy names forbidden, and prove no production wiring or live/default-state mutation. See references/production-remote-caduceus-stack-adapter.md.

For independent high-risk review of the Caduceus durable-store/router/lock/adapter boundary, reconcile untracked tests before trusting status claims, run the focused suites that cover newly added modules, treat missing imported modules as blockers, and specifically probe durable catalog authority across adapter restart, lock scope against every mutable authority (global for a shared file-backed ledger; tenant/resource scoped only for transactionally partitioned state), host/PID-aware stale lock recovery, and router scheduling durability/outbox semantics. See references/caduceus-durable-store-router-lock-adapter-boundary-review.md.

For production execution locks, PID liveness is authoritative only on the hostname that acquired the lock. Include hostname and lockId in busy/manual-recovery diagnostics, treat foreign-host PID state as unknown/present, and keep foreign-host stale recovery fail-closed. See references/production-execution-lock-host-aware.md for the exact implementation and regression pattern.

Before designing any Spawn migration, classify the nouns explicitly: platform product → product customer account → product instance → product-derived tenant stack → tenant-local portfolio/workspace → companies/businesses/projects. For Caduceus, consult /home/avalon/apps/caduceus/docs/idea-research/adr-004-per-tenant-caduceus-stack.md; never infer a brain-only tenant or central shared Portfolio OS.

When migrating an existing general tenant into a specialized product, do not mutate its profile/home in place. Provision through the specialized product's ordinary customer/product-instance path and derive the complete target stack required by that product. Keep channels and production deployment authority singular, move only classified knowledge/business data and separately authorized credential references, run side-by-side isolation and lifecycle acceptance, then perform an audited cutover with a proven rollback. For Caduceus, Wild Earth receives an independent full Portfolio OS/data/Hermes/hosting stack and never becomes rows in Alex's database. See references/specialized-product-canary-migration.md for the reusable workflow.

For portable specialized tenant appliances, do not approve Dockerfile/Compose shape tests alone. Preserve dirty development sources through immutable secret-free source bundles, bind health to validated tenant/operation/contract identity, require exact non-legacy container names, use SQLite online backup plus checksummed tenant manifests, exclude raw secrets/auth/WAL, and restore offline through fully validated staging and atomic swaps with rollback. See references/portable-specialized-tenant-appliance.md for the executable TDD and canary gates.

For Caduceus managed Hermes bootstrap fixes, review the complete host-worker durable-I/O, backup acceptance, and runtime pickup path adversarially: final env paths, resource/topology metadata, backup catalogs/payloads/manifests, and all temp paths must use fd-based no-follow atomic I/O; archive members must be canonical, link-free, secret-free, and independently size/SHA-verified from actual tar bytes; already-created Docker containers will not pick up new --env-file or Hermes .env/config.yaml values unless owned containers are recreated/restarted; and idempotent repair should fix stale bad config.yaml, not only create missing files. Treat asynchronous independent review completion as an approval/claim gate rather than inferring approval from focused tests. See references/caduceus-managed-hermes-bootstrap-review.md for adversarial probes, production-backup proof, provider/API-key consistency checks, and reviewer timing rules. For fresh tenants whose UI loads but chat fails, follow references/caduceus-fresh-tenant-chat-readiness.md; it covers exact deployed-transport inspection, managed-provider acceptance, route-port-preserving repair, copy-only application-bridge proof, and duplicate-free recovery of a persisted failed turn.

For upgrading an existing Spawn-managed Caduceus tenant release, first derive the real topology from Docker labels/mounts and Spawn's server-owned image authority; do not confuse the portable image recipe with standalone deployment authority. Build an immutable exact-commit image with retained logs, run disposable image-level Normal/Pro/Admin/upgrade acceptance, then deploy only through the Spawn upgrade path with a durable pre-snapshot. Activate handoff contract v2 only after upgraded health, and promote the product account only after v2 activation. See references/caduceus-managed-release-upgrade.md for the build, acceptance-harness, permission, key, rollback, and sequencing checklist.

For a running tenant, never infer that idempotent provision(...) reconciles image drift: prove the app and brain immutable image IDs change in a disposable tenant before touching production. Do not delete live containers to discover reconcile semantics. Use references/caduceus-running-stack-image-reconciliation.md for preflight, volume UID/GID and transport-dependency compatibility, copied-home API canaries, partial-rollout recovery, random-port/tunnel repair, and required release evidence.

For destructive test-environment resets such as “keep only Alex,” use references/caduceus-tenant-retirement-and-reset.md. It covers positive keep sets, a concurrency cut line for user-created tenants, exact-ownership purges, per-connection SQLite foreign-key enforcement, product-scoped ledger cleanup, canonical keeper routing, and zero-orphan verification.

For Caduceus Social/OAuth releases, add the stricter gate in references/caduceus-social-release-safety.md: integrate latest main before exact-commit review, disarm overdue imported schedules, resolve credentials from the exact surface connection, validate grants/status/expiry before provider writes, checkpoint multi-item Story mutations, derive only the public handoff build key, run disposable Normal/Pro/Admin handoff acceptance, and upgrade via adapter snapshot/rollback rather than direct Docker replacement.

When synthetic full-stack lifecycle and restore rehearsal are complete but real provisioning is not yet wired, do not collapse “foundation complete” into “production ready.” Promote through portable appliance → inter-process lock → product-scoped router scheduling → exact second durable executor policy → unwired reviewed adapters → distinct private blank canary → zero-residual lifecycle proof. Inventory and protect any legacy runtime whose name resembles the target before choosing the canary identity, and keep feature development on the migration-source checkout isolated with a repo-local session-seed prompt. See references/specialized-stack-production-promotion.md for the complete production-promotion gate and exact-policy operation-store pattern.

Basic checks

cd /home/avalon/apps/hermes-spawn
curl -fsS http://127.0.0.1:4031/api/health
pm2 list | grep hermes-spawn

List remote tenant containers:

ssh -o BatchMode=yes -o ConnectTimeout=10 avalon@167.233.226.57 \
  "docker ps -a --filter name=spawn-tenant --format '{{.Names}}|{{.Status}}|{{.Image}}'"

The current Hermes runtime may not have direct local Docker socket access; use the Spawn API or SSH worker path rather than local docker.

Caduceus upgrade ingress pitfall

The Caduceus host worker publishes app port 4043 to a random worker loopback port. A successful upgrade removes and recreates the owned app container, so the worker port can change even though identity-bound container health passes. If public ingress uses a PM2-managed SSH forward, immediately compare docker inspect ... .NetworkSettings.Ports with that PM2 process's -L local:remote argument, retarget only the affected tenant forward, pm2 save, then verify local forwarded health and the public tenant URL. Treat a post-upgrade 502 with a healthy container as likely stale port forwarding, not application failure.

Future-tenant image selector authority

Do not infer that future tenants will use a release because the desired image tags appear in a repository .env. First inspect how server.mjs loads configuration and read the effective supervisor environment of the running hermes-spawn process. PM2 can retain older injected values indefinitely, and a Node service that never imports dotenv will ignore a correct-looking .env entirely.

For selector changes:

  1. Compare file values, PM2's effective SPAWN_CADUCEUS_*_IMAGE values, and the source lines that construct the production runtime.
  2. Export the intended selectors into the operator shell, run pm2 restart hermes-spawn --update-env, then pm2 save.
  3. Re-read PM2's effective environment and require exact values.
  4. Verify both immutable tags/IDs exist on the worker and contain their claimed capabilities.
  5. Prove the next disposable or real new tenant's app/brain .Config.Image values. Existing tenants do not inherit selector changes automatically.

A healthy Spawn endpoint plus a correct .env is not selector proof; the created container image IDs are the final evidence.

Create a tenant through the API

The web UI calls POST /api/provision. Use an existing authenticated user session from spawn-db.json, or register/login via the API if needed. Avoid printing bearer tokens, passwords, API keys, or bot tokens.

Example pattern:

cd /home/avalon/apps/hermes-spawn
python - <<'PY'
import json, subprocess
from pathlib import Path
DB=Path('data/spawn-db.json')
db=json.loads(DB.read_text())
token=next(s['token'] for s in db['sessions'] if s['userId']=='user_firemountain')
payload={
  'tenantId':'wild-earth-hermes',
  'name':'Wild Earth Hermes',
  'ownerEmail':'firemountain@gmail.com',
  'provider':'openai-codex',
  'model':'gpt-5.5',
  'telegramAllowedUsers':'1932991162',
  'telegramToken':'',
  'telegramBotUsername':'',
  'selectedSkills':[],
  'curtailedSkills':[],
  'startNow': True
}
res=subprocess.run([
  'curl','-fsS','-X','POST','http://127.0.0.1:4031/api/provision',
  '-H',f'Authorization: Bearer {token}',
  '-H','Content-Type: application/json',
  '--data-binary',json.dumps(payload)
],capture_output=True,text=True,timeout=180)
print(res.returncode)
data=json.loads(res.stdout)
for k in list(data):
    if 'token' in k.lower(): data[k]='[redacted]'
print(json.dumps(data, indent=2))
PY

Verify:

ssh avalon@167.233.226.57 \
  "docker ps -a --filter name=^/spawn-tenant-wild-earth-hermes$ --format '{{.Names}}|{{.Status}}|{{.Image}}'"

Telegram setup

Before provisioning or updating a Telegram-connected Spawn tenant, ask Alex for the necessary Telegram details unless he already provided them: - Bot token from @BotFather - Bot username, e.g. WildEarthHermesBot - Allowed Telegram user IDs and/or chat IDs; default Alex user ID is 1932991162 if he says only he should access it

Spawn captures Telegram through POST /api/telegram/update:

Required: - tenantId - telegramAllowedUsers as numeric IDs, comma-separated - optional telegramToken - optional telegramBotUsername

If no token is available yet, provision with telegramAllowedUsers and blank token, then update later when Alex provides the BotFather token.

Official Telegram Bot API cannot create bots or fetch BotFather tokens. The recommended path is guided BotFather creation by the user, then paste token into Spawn. MTProto/Telethon user-account automation is possible but sensitive and should not be the default.

Provider setup

Provider choices include: - openai-api with API key - openai-codex for ChatGPT/Codex OAuth - openrouter - anthropic

For openai-codex, provisioning creates the tenant configured for Codex but setupStep may remain provider until OAuth credentials are installed. The app has: - POST /api/provider/device-auth — returns the OpenAI/Codex device auth URL and code for the owner to approve. - POST /api/provider/complete — after approval, polls/exchanges and installs the Codex credential, then advances setup to chat.

Default path: use those endpoints / the Spawn UI so the owner explicitly hooks up the provider.

If Alex explicitly asks to reuse this Hermes' current Codex credential for a new or existing Spawn tenant, it is acceptable to copy /home/avalon/.hermes/auth.json into the tenant home as auth.json, set restrictive permissions, restart the tenant container, and smoke-test with HERMES_HOME=/data/hermes hermes chat .... Do not print credential contents. Do not do this silently; it shares the same OAuth account across tenants.

Before replacement, preserve the tenant's old credential as a timestamped backup. After the real smoke test, do not require the two auth.json files to remain byte-identical: a successful Codex request can rotate the tenant token and rewrite its file. Verify the same durable OAuth identity (active_provider, auth_mode, and non-secret account_id) plus a successful model request instead.

Caduceus brain image/provider repair pitfalls

Tenant skill propagation

When Alex asks to push skills from the operator Hermes into a Spawn tenant:

  1. Inventory both source and tenant skill sets first.
  2. Prefer the smallest class-level umbrella that covers the request; retain complementary tenant-specific skills rather than replacing them blindly.
  3. Copy the entire skill directory, including references/, templates/, scripts/, and assets—not only SKILL.md.
  4. Verify deterministic source/destination file counts and content digests.
  5. Restart only the target tenant container.
  6. Confirm discovery with HERMES_HOME=/data/hermes hermes skills list inside the container.
  7. Run an agent-level smoke test that makes the tenant select/load the skill for its intended trigger. Filesystem presence alone is not sufficient proof.

See references/tenant-credential-and-skill-sync.md for exact verification patterns and the OAuth token-rotation nuance.

Tenant app deployment broker

When a Spawn tenant needs to deploy websites/APIs on the worker host, do not give it unrestricted SSH, sudo, the Docker socket, /etc/nginx, or shared-host PM2 access. Use a token-authenticated deploy broker with a fixed command surface:

Public DNS may converge differently across recursive resolvers, authoritative nameservers, and Let's Encrypt secondary validators. Query every authoritative nameserver directly before certificate issuance; if they disagree, do not blame nginx/Certbot or claim HTTPS is ready. Add requested hostnames to nginx promptly so traffic that already reaches the VPS cannot fall through to the default welcome site, but issue the primary certificate only after all names in that request resolve consistently. Prefer apex + www + the requested public app hostname over letting stale optional subdomains block the main site. Support partial certificate issuance for propagated hostnames, report live_partial/pending names, and use a bounded silent DNS/TLS watchdog when registrar propagation is the only remaining blocker.

Certbot/TLS recovery rules:

See references/scoped-tenant-deploy-broker.md for the full security boundary, request validation, isolated PM2 account pattern, asynchronous API contract, EACCES cwd pitfall, DNS/TLS behavior, and end-to-end verification. See references/dns-tls-partial-certificate-recovery.md for the partial-lineage --expand failure, stale recovery artifact prevention, and a complete recovery/verification sequence.

User/account notes

Existing users live in data/spawn-db.json. Common owner user: - user_firemountain / firemountain@gmail.com

Do not print password hashes, bearer tokens, or secrets. If creating a user, prefer the app's /api/auth/register or edit DB only when the API cannot satisfy the task.

Verification checklist

After provisioning, verify: 1. API response has ok: true and expected tenantId. 2. GET /api/instances shows the tenant for the owner. 3. SSH worker shows spawn-tenant-<tenant-id> exists/running. 4. Tenant root exists under Spawn root. 5. If Telegram token was supplied, the instance shows telegramTokenSaved: true and link. 6. If provider is Codex, complete provider auth before claiming live model access.

Cleanup mistaken raw profile

If a Spawn request was accidentally handled as a local profile:

hermes profile delete <profile-name> --yes

Then provision through the Spawn API instead.