← all skills

hermes-agent-runtime-debugging

/home/avalon/.hermes/skills/devops/hermes-agent-runtime-debugging/SKILL.md · raw

Hermes Agent Runtime Debugging

Use this skill when Alex asks why Hermes itself is behaving oddly at runtime: repeated compacting/compression, provider errors, Telegram gateway failures, stale code after a patch, context/token anomalies, session resume issues, or unexplained agent loop behavior.

Do not use this as a replacement for the protected bundled hermes-agent skill. Load hermes-agent first for official commands/config reference, then use this skill for local runtime forensics and patch patterns discovered in Alex's install.

Core workflow

  1. Identify the active or relevant session - Use session IDs from user context, gateway logs, or ~/.hermes/state.db. - If the user references “the other/current session,” search recent session records and match by title/content/source/user rather than guessing.

  2. Correlate logs and DB state - Primary logs:

  3. Separate symptoms from root cause - Repeated “compacting” can be a failed compression loop, not necessarily runaway user content. - Provider/API error text in logs may point at an auxiliary subsystem (compression, vision, title generation), not the main user-facing model response. - If the same warning repeats with a pause interval, inspect the subsystem retry/backoff path.

  4. Patch narrowly and verify locally - Prefer a minimal response-shape guard, parser fix, or adapter normalization over broad disabling of compression/provider features. - Verify with syntax checks and a tiny shape test when possible. - Do not restart the Telegram gateway automatically if another active session may be interrupted; explain that the on-disk fix needs a gateway restart to take effect.

  5. Report operational next step clearly - Distinguish:

Compression loop diagnosis pattern

When a session repeatedly compacts or feels like it is compacting too often:

  1. Search errors.log for the session ID and terms like: - context summary - conversation_compression - Failed to generate context summary - No auxiliary LLM provider for compression - Auxiliary compression
  2. Check config: - compression.enabled - compression.threshold - compression.abort_on_summary_failure - auxiliary.compression.provider/model/timeout
  3. If summary generation fails, identify whether compression is repeatedly triggered but unable to produce a usable summary.
  4. If the error is provider-shape related, inspect the auxiliary provider adapter first.

Known local pitfall: Codex auxiliary output=None

In Alex's install, Codex auxiliary Responses calls can return a final object with output present but set to None. Python's getattr(final, "output", []) does not fall back to [] when the attribute exists and is None; iterating it raises:

'NoneType' object is not iterable

For auxiliary response extraction, use the null-coalescing form:

for item in (getattr(final, "output", None) or []):
    ...

This pattern is especially relevant in agent/auxiliary_client.py for compression/vision/title-generation paths that adapt Codex Responses to chat-completions-like objects.

See references/codex-auxiliary-compression-null-output.md for the session-specific reproduction and patch note.

Known deployment pitfall: local Codex fix vs tenant runner image

When openai-codex works in the local Hermes checkout but fails inside an Astral/Hermes tenant Docker runner with:

TypeError: 'NoneType' object is not iterable
Provider: openai-codex
Endpoint: https://chatgpt.com/backend-api/codex

suspect a stale runner image or missing main Codex stream hardening before changing tenant provider config. A minimal smoke test inside the same container should be:

hermes chat -q "Reply with exactly: ASTRAL_OK" -Q --provider openai-codex -m gpt-5.5 --source debug

The durable fix pattern is to collect response.output_item.done and text deltas during streaming, catch the OpenAI SDK output:null TypeError around both stream iteration and get_final_response(), synthesize a minimal final response from collected items/deltas, and treat output is None the same as an empty output list during backfill. Then rebuild/restart tenant runner containers; a local Hermes pass does not prove deployed tenants have the patch.

See references/codex-openai-output-null-tenant-runner.md for the tenant-runner reproduction recipe and deployment checklist.

Photon / iMessage outbound debugging

When Alex asks Hermes to send an iMessage/SMS-style message through Photon Spectrum:

  1. Use the gateway send path when possible - Target format for direct phone recipients is photon:+1XXXXXXXXXX (E.164). - If calling from outside the live gateway process, a standalone send may fail with Photon standalone send requires a running sidecar with PHOTON_SIDECAR_TOKEN; in that case inspect the running gateway/sidecar process rather than concluding Photon is down.

  2. Verify the live sidecar before debugging content - Gateway status should show photon connected and a Node sidecar process like plugins/platforms/photon/sidecar/index.mjs. - Recent ~/.hermes/logs/gateway.log lines from hermes_plugins.photon_platform.adapter contain the useful sidecar stack/error even when the HTTP caller only receives internal sidecar error.

  3. Interpret Target not allowed for this project correctly - This means Photon/Spectrum rejected the recipient for project allowlisting/pairing reasons, not that the message text or Hermes send tool is malformed. - Do not retry repeatedly or try alternate opaque spaceId shapes (+E164, any;-;+E164) as a “fix”; the durable next step is to add/approve that target in Photon/Spectrum project settings or the configured allowlist, then retry once. - User-facing reply should be concise: say the exact recipient is not currently allowed by the Photon project and quote the attempted message text if helpful.

  4. Keep privacy in transcripts - Do not expose full phone numbers in the final answer unless the user explicitly needs it; identify the issue by role/recipient or masked number.

Telegram in read-only tenant runners

When Telegram is newly configured in a Docker tenant, do not stop at token validation or gateway is running. A healthy gateway may have only API Server connected while Telegram failed independently.

Debug in this order:

  1. Validate the bot identity and webhook mode without printing the token.
  2. Inspect the exact interpreter used by the container's hermes wrapper; install the runner-pinned messaging dependency into that venv, not system Python.
  3. If the Telegram scoped lock targets an immutable home, provide HOME and XDG_{STATE,CACHE,CONFIG}_HOME paths under the writable tenant mount at container start.
  4. Replace only the affected brain/container transactionally, preserving exact labels, network, mounts, read-only/tmpfs settings, user, command, and private API transport.
  5. Require the post-restart gateway log to say Telegram connected and prove the paired app can still reach the brain API.

See references/telegram-readonly-tenant-runner.md for the secret-safe preflight, interpreter/venv trap, read-only XDG repair, rollback pattern, and acceptance evidence.

Profile-isolated API multiplexer appliances

When a tenant appliance runs one private Hermes API multiplexer for several isolated profile homes, distinguish bootstrap readiness from operational readiness. Profile files must be materialized before the sidecar can start, but public health must remain 503 until every existing profile has an error-free active activation. Treat materialized: true, container Up, or hermes --help as intermediate evidence only.

Before debugging profile routing, verify that the pinned image includes the HTTP adapter required by the Hermes API server. A gateway warning that no API adapter is available and that execution will continue for cron means the multiplexer never started. For custom OpenAI-compatible providers, verify the profile config includes validated base_url plus an allowlisted profile-local ${KEY_ENV} reference—not just provider and model names.

Hermes discovers Python plugins once at gateway startup from the multiplexer root. Profile-local plugin copies are necessary for profile-owned config and assets but are not sufficient to register toolsets. Materialize the same versioned plugin bundle under the private root and enable it in the root config before starting Hermes; keep project bindings, Gate capabilities, and provider secrets profile-local.

Do not use GET /v1/skills as proof that plugin-registered skills are absent. Hermes ctx.register_skill(...) creates explicit namespaced plugin skills, while this endpoint enumerates the flat filesystem skill index and may return data: []. Verify the actual plugin registry/qualified skill load, or provide a deterministic plugin-capability endpoint. Likewise, a successful Caduceus toolset listing is not enough if generic terminal/file/browser toolsets remain runnable: acceptance must prove forbidden capabilities fail from a real profile run and that port 8642 is private and API-key protected.

Use references/profile-isolated-multiplexer-appliance-debugging.md for exact build checks, Normal/Pro disposable topology, root-versus-profile plugin installation, endpoint semantics, two-stage health, real acceptance assertions, ambient-tool denial probes, shadow-activation debug order, legacy Compose recreation, and secret-safe cleanup.

Remote provider-auth incident triage

When Alex reports that a separate/project-specific Hermes on another VPS has a provider authentication problem:

  1. Resolve the exact runtime before diagnosing: distinguish the project name, public app/domain, VPS IP/hostname, OS user, Hermes profile/home, and whether Hermes runs on the host or in Docker. Do not silently equate a business/app label with an older server label merely because they may share infrastructure.
  2. Inspect the original live source first: if shell access exists, inspect that remote Hermes config/auth/logs directly. Session history is secondary context, not proof of the current remote state.
  3. Separate host availability from provider health: HTTP/TLS/nginx/OAuth probes can prove the host and front door are alive, but they cannot prove the Hermes gateway or model provider works.
  4. Avoid destructive auth recovery as diagnosis: do not reset credentials, rebuild the VPS, or restart unrelated services merely to gain visibility. Recover scoped SSH/container access or obtain the exact user-facing error first.
  5. Once connected, verify in this order: runtime location → gateway/service state → active model/provider → hermes auth list <provider> → recent gateway/error logs → minimal direct hermes chat probe with the same provider/model → narrow credential-store repair → affected gateway restart → repeat probe and platform smoke test.
  6. For Codex OAuth, inspect both state and shape: a credential can exist but be exhausted/rate-limited, or be written only to one auth-store shape. Check before forcing reauthentication.
  7. Report evidence boundaries clearly: say what is confirmed externally, what requires remote access, and what remains only a hypothesis.

See references/remote-provider-auth-triage.md for a concise probe and verification checklist.

Known gateway memory pitfall: unreaped LSP clients across worktrees

When a long-lived gateway grows to several GB and systemd reports an OOM kill with many Node children, first compare host uptime/boot history with the user-service journal. Long host uptime plus Failed with result 'oom-kill' means the gateway restarted; do not misreport that as a VPS reboot. Then inspect the gateway cgroup before blaming Telegram or the model provider. Hermes LSP is process-wide and keys clients by (server_id, workspace_root), so editing files across many git worktrees—or multiple package-level roots inside one monorepo—can spawn one TypeScript language-server tree per root.

Diagnostic pattern:

  1. Check systemctl --user show hermes-gateway.service -p MemoryCurrent -p MemoryPeak -p MemorySwapCurrent -p TasksCurrent -p MemoryHigh -p MemoryMax -p OOMScoreAdjust and inspect memory.stat/memory.pressure for the cgroup.
  2. Enumerate only processes whose /proc/<pid>/cgroup contains hermes-gateway.service; group typescript-language-server, tsserver.js, and typingsInstaller.js by /proc/<pid>/cwd. One TypeScript LSP tree can use roughly 0.5–0.6 GB RSS on a large monorepo.
  3. Take two quick snapshots. Rapid increases in workspace roots, tasks, and cgroup memory are stronger leak evidence than a single large total. Avoid edits in additional roots while auditing a near-OOM gateway.
  4. Inspect agent/lsp/manager.py. A declared DEFAULT_IDLE_TIMEOUT and _last_used map do not prove reaping works: verify _idle_timeout is actually read by an eviction/reaper path and that the config factory parses and passes an idle-timeout setting.
  5. If _idle_timeout is only assigned and never consumed, LSP clients persist until whole-process shutdown. Multiple roots then accumulate Node trees and can push the gateway cgroup into OOM.
  6. Separate incidents where the gateway reached several GB from host-wide OOM events where its unit peak remained small. A positive OOMScoreAdjust explains why the gateway was selected as victim, not what consumed host memory.
  7. Temporary mitigation: disable lsp.enabled for the gateway and restart during a safe window. Config-on-disk alone does not reclaim current child trees. Durable repair: implement tested idle eviction plus a client-count/LRU bound and atomically remove client/timestamp state before shutdown.
  8. Add systemd MemoryHigh/MemoryMax only as containment after fixing/reducing LSP growth; limits protect the VPS but do not fix the leak and can still restart the gateway.

Correlate this with host headroom: nearly-full swap can be stale rather than active thrashing, so attribute VmSwap by PID and inspect vmstat swap-in/out plus /proc/pressure/memory. Do not run swapoff unless available RAM can hold all swapped pages safely.

See references/lsp-worktree-oom-forensics.md for the detailed cgroup attribution, code-proof, incident-separation, and mitigation workflow. See references/gateway-lsp-oom-containment.md for the controlled external-systemd restart pattern, post-restart acceptance checks, cgroup cache decomposition, and false-positive-safe LSP process accounting.

Restart discipline

After patching Hermes runtime code used by the gateway:

Pitfall: one-shot cron restart script can self-loop

If the gateway is restarting every ~1-2 minutes and journalctl --user -u hermes-gateway.service shows repeated Stopping hermes-gateway.service shortly after startup, check for a no-agent cron job whose script calls systemctl --user restart hermes-gateway.service.

Root cause pattern:

CGroup: hermes-gateway.service
  ├─... python -m hermes_cli.main gateway run
  ├─... bash ~/.hermes/scripts/restart_gateway_*.sh
  └─... systemctl --user restart hermes-gateway.service

A cron job running inside the gateway asks systemd to restart the gateway, killing the gateway process before the cron scheduler can record the one-shot as completed. On the next gateway startup, the due one-shot still has last_run_at: null / enabled: true, so it fires again and creates a restart loop.

Recovery:

  1. List cron jobs: hermes cron list --all or cronjob(action='list').
  2. Remove or pause the stuck one-shot restart job.
  3. Let the in-flight restart finish, then verify hermes gateway status is active and ps no longer shows the restart script/systemctl child.
  4. For future gateway reloads, prefer a direct hermes gateway restart or an external systemd timer/service that is not supervised by the gateway's own cron runner.

User-facing style for runtime incidents

Alex prefers root-cause evidence and compact summaries. Use this shape: