caduceus-tenant-operations

/home/avalon/.hermes/skills/devops/caduceus-tenant-operations/SKILL.md · raw

Caduceus Tenant Operations

Use this when a Caduceus tenant is healthy but inaccessible, has the wrong account authority, opens with an unexpectedly empty portfolio, needs legacy business state restored, or requires a verified recovery/cutover.

This skill owns tenant-level recovery and state continuity. Use hermes-spawn-control-plane alongside it for controller/executor/container lifecycle work and vps-app-deployment for shared-host deployment conventions.

Operational model

Keep these authorities separate:

  1. Telegram allowlisting authenticates a messaging user to a tenant brain.
  2. Caduceus central identity authenticates the product account.
  3. A signed central handoff creates the browser session inside one tenant stack.
  4. Business/domain state and media live in the tenant stack, not in central identity.

Never report “the admin login works” merely because the bot replies, and never report “the portfolio is restored” merely because cards render.

Admin access recovery

First inspect the live account mechanism. If the account is passkey-only, do not invent a conventional password in conversation. If the verified admin explicitly requests a password fallback, implement a stable account-scoped route rather than repeating short-lived static recovery files.

Required properties:

A connected Telegram bot is not a browser login. A 60–90 second static HTML bridge that later returns Cannot GET is not durable recovery.

Empty portfolio recovery

Before seeding anything, compare the live tenant database with prior canonical Caduceus databases and backups. A healthy Admin/Pro bootstrap may intentionally contain a portfolio profile and zero projects. If a prior canonical database contains the exact expected businesses while the tenant has none, classify it as a missing migration and restore real state rather than creating lookalike placeholders.

Use an SQLite online source backup, a durable target rollback backup, an explicit business-table allowlist, a dry run against a disposable target copy, one transactional apply, integrity_check, and foreign_key_check. Preserve target-owned identity, handoff, profile, entitlement, and schema state. Do not migrate encrypted credential tokens, user passwords/sessions, agent keys, pending jobs, or stale schema rows. Imported connection metadata must say needs_connection unless credentials were separately re-authorized.

Media is part of state continuity: inventory every populated path column, safely resolve symlinked vault paths, transfer only referenced files, verify size and SHA-256, rewrite absolute paths to the tenant-visible vault root, and perform an authenticated media-fetch digest canary.

Full procedure and acceptance matrix: references/admin-access-and-legacy-portfolio-recovery.md. Reusable allowlisted SQLite importer: scripts/import-business-state.py.

Unexpected operational notifications

When an operator receives a money, approval, sentinel, or review notification they did not expect, trace the exact message before changing live state. Search distinctive payload text in current source, all worktrees, and Git history; then map the central account to the exact tenant stack and inspect that tenant's database and container logs. Keep proposal, approval, and execution evidence separate.

A common leakage path is an integration test that correctly uses a temporary database but inherits a real Telegram token or home channel. Repeated runs then emit identical fixture messages with fresh random agent IDs while leaving no proposal in the current live database. Literal fixture matches plus the test assertions establish this provenance; an absent live row alone does not.

For the full provenance workflow and durable test-suppression pattern, see references/unexpected-telegram-notification-provenance.md.

Completion standard

Do not finish until all relevant layers are proven:

Retain the durable rollback backup until the user has used the restored tenant successfully.