---
name: hermes-agent-runtime-debugging
description: Debug Hermes Agent runtime failures in live gateway/CLI sessions — context compression loops, provider adapter response-shape bugs, stale gateway code, session DB/log correlation, and safe verification without interrupting active work.
version: 1.3.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [hermes, debugging, gateway, compression, provider-adapters, session-db, logs]
    related_skills: [hermes-agent, systematic-debugging]
---

# Hermes Agent Runtime Debugging

Use this skill when Alex asks why Hermes itself is behaving oddly at runtime: repeated compacting/compression, provider errors, Telegram gateway failures, stale code after a patch, context/token anomalies, session resume issues, or unexplained agent loop behavior.

Do **not** use this as a replacement for the protected bundled `hermes-agent` skill. Load `hermes-agent` first for official commands/config reference, then use this skill for local runtime forensics and patch patterns discovered in Alex's install.

## Core workflow

1. **Identify the active or relevant session**
   - Use session IDs from user context, gateway logs, or `~/.hermes/state.db`.
   - If the user references “the other/current session,” search recent session records and match by title/content/source/user rather than guessing.

2. **Correlate logs and DB state**
   - Primary logs:
     - `~/.hermes/logs/gateway.log`
     - `~/.hermes/logs/errors.log`
   - Primary DB:
     - `~/.hermes/state.db`
   - Look for the exact session ID in logs before attributing cause.
   - For token/context issues, inspect `sessions` plus recent `messages` rows, not just chat transcript summaries.

3. **Separate symptoms from root cause**
   - Repeated “compacting” can be a *failed compression loop*, not necessarily runaway user content.
   - Provider/API error text in logs may point at an auxiliary subsystem (compression, vision, title generation), not the main user-facing model response.
   - If the same warning repeats with a pause interval, inspect the subsystem retry/backoff path.

4. **Patch narrowly and verify locally**
   - Prefer a minimal response-shape guard, parser fix, or adapter normalization over broad disabling of compression/provider features.
   - Verify with syntax checks and a tiny shape test when possible.
   - Do not restart the Telegram gateway automatically if another active session may be interrupted; explain that the on-disk fix needs a gateway restart to take effect.

5. **Report operational next step clearly**
   - Distinguish:
     - “fixed on disk”
     - “loaded by current gateway process”
     - “requires restart/reset/new session”
   - Avoid raw traceback dumps in the user-facing summary unless Alex explicitly asks for logs.

## Compression loop diagnosis pattern

When a session repeatedly compacts or feels like it is compacting too often:

1. Search `errors.log` for the session ID and terms like:
   - `context summary`
   - `conversation_compression`
   - `Failed to generate context summary`
   - `No auxiliary LLM provider for compression`
   - `Auxiliary compression`
2. Check config:
   - `compression.enabled`
   - `compression.threshold`
   - `compression.abort_on_summary_failure`
   - `auxiliary.compression.provider/model/timeout`
3. If summary generation fails, identify whether compression is repeatedly triggered but unable to produce a usable summary.
4. If the error is provider-shape related, inspect the auxiliary provider adapter first.

## Known local pitfall: Codex auxiliary `output=None`

In Alex's install, Codex auxiliary Responses calls can return a final object with `output` present but set to `None`. Python's `getattr(final, "output", [])` does **not** fall back to `[]` when the attribute exists and is `None`; iterating it raises:

```text
'NoneType' object is not iterable
```

For auxiliary response extraction, use the null-coalescing form:

```python
for item in (getattr(final, "output", None) or []):
    ...
```

This pattern is especially relevant in `agent/auxiliary_client.py` for compression/vision/title-generation paths that adapt Codex Responses to chat-completions-like objects.

See `references/codex-auxiliary-compression-null-output.md` for the session-specific reproduction and patch note.

## Known deployment pitfall: local Codex fix vs tenant runner image

When `openai-codex` works in the local Hermes checkout but fails inside an Astral/Hermes tenant Docker runner with:

```text
TypeError: 'NoneType' object is not iterable
Provider: openai-codex
Endpoint: https://chatgpt.com/backend-api/codex
```

suspect a stale runner image or missing main Codex stream hardening before changing tenant provider config. A minimal smoke test inside the same container should be:

```bash
hermes chat -q "Reply with exactly: ASTRAL_OK" -Q --provider openai-codex -m gpt-5.5 --source debug
```

The durable fix pattern is to collect `response.output_item.done` and text deltas during streaming, catch the OpenAI SDK `output:null` `TypeError` around both stream iteration and `get_final_response()`, synthesize a minimal final response from collected items/deltas, and treat `output is None` the same as an empty output list during backfill. Then rebuild/restart tenant runner containers; a local Hermes pass does not prove deployed tenants have the patch.

See `references/codex-openai-output-null-tenant-runner.md` for the tenant-runner reproduction recipe and deployment checklist.

## Photon / iMessage outbound debugging

When Alex asks Hermes to send an iMessage/SMS-style message through Photon Spectrum:

1. **Use the gateway send path when possible**
   - Target format for direct phone recipients is `photon:+1XXXXXXXXXX` (E.164).
   - If calling from outside the live gateway process, a standalone send may fail with `Photon standalone send requires a running sidecar with PHOTON_SIDECAR_TOKEN`; in that case inspect the running gateway/sidecar process rather than concluding Photon is down.

2. **Verify the live sidecar before debugging content**
   - Gateway status should show `photon connected` and a Node sidecar process like `plugins/platforms/photon/sidecar/index.mjs`.
   - Recent `~/.hermes/logs/gateway.log` lines from `hermes_plugins.photon_platform.adapter` contain the useful sidecar stack/error even when the HTTP caller only receives `internal sidecar error`.

3. **Interpret `Target not allowed for this project` correctly**
   - This means Photon/Spectrum rejected the recipient for project allowlisting/pairing reasons, not that the message text or Hermes send tool is malformed.
   - Do not retry repeatedly or try alternate opaque `spaceId` shapes (`+E164`, `any;-;+E164`) as a “fix”; the durable next step is to add/approve that target in Photon/Spectrum project settings or the configured allowlist, then retry once.
   - User-facing reply should be concise: say the exact recipient is not currently allowed by the Photon project and quote the attempted message text if helpful.

4. **Keep privacy in transcripts**
   - Do not expose full phone numbers in the final answer unless the user explicitly needs it; identify the issue by role/recipient or masked number.

## Telegram in read-only tenant runners

When Telegram is newly configured in a Docker tenant, do not stop at token validation or `gateway is running`. A healthy gateway may have only API Server connected while Telegram failed independently.

Debug in this order:

1. Validate the bot identity and webhook mode without printing the token.
2. Inspect the exact interpreter used by the container's `hermes` wrapper; install the runner-pinned messaging dependency into that venv, not system Python.
3. If the Telegram scoped lock targets an immutable home, provide `HOME` and `XDG_{STATE,CACHE,CONFIG}_HOME` paths under the writable tenant mount at container start.
4. Replace only the affected brain/container transactionally, preserving exact labels, network, mounts, read-only/tmpfs settings, user, command, and private API transport.
5. Require the post-restart gateway log to say Telegram connected **and** prove the paired app can still reach the brain API.

See `references/telegram-readonly-tenant-runner.md` for the secret-safe preflight, interpreter/venv trap, read-only XDG repair, rollback pattern, and acceptance evidence.

## Profile-isolated API multiplexer appliances

When a tenant appliance runs one private Hermes API multiplexer for several isolated profile homes, distinguish **bootstrap readiness** from **operational readiness**. Profile files must be materialized before the sidecar can start, but public health must remain `503` until every existing profile has an error-free `active` activation. Treat `materialized: true`, container `Up`, or `hermes --help` as intermediate evidence only.

Before debugging profile routing, verify that the pinned image includes the HTTP adapter required by the Hermes API server. A gateway warning that no API adapter is available and that execution will continue for cron means the multiplexer never started. For custom OpenAI-compatible providers, verify the profile config includes validated `base_url` plus an allowlisted profile-local `${KEY_ENV}` reference—not just provider and model names.

Hermes discovers Python plugins once at gateway startup from the **multiplexer root**. Profile-local plugin copies are necessary for profile-owned config and assets but are not sufficient to register toolsets. Materialize the same versioned plugin bundle under the private root and enable it in the root config before starting Hermes; keep project bindings, Gate capabilities, and provider secrets profile-local.

Do not use `GET /v1/skills` as proof that plugin-registered skills are absent. Hermes `ctx.register_skill(...)` creates explicit namespaced plugin skills, while this endpoint enumerates the flat filesystem skill index and may return `data: []`. Verify the actual plugin registry/qualified skill load, or provide a deterministic plugin-capability endpoint. Likewise, a successful Caduceus toolset listing is not enough if generic terminal/file/browser toolsets remain runnable: acceptance must prove forbidden capabilities fail from a real profile run and that port `8642` is private and API-key protected.

Use `references/profile-isolated-multiplexer-appliance-debugging.md` for exact build checks, Normal/Pro disposable topology, root-versus-profile plugin installation, endpoint semantics, two-stage health, real acceptance assertions, ambient-tool denial probes, shadow-activation debug order, legacy Compose recreation, and secret-safe cleanup.

## Remote provider-auth incident triage

When Alex reports that a separate/project-specific Hermes on another VPS has a provider authentication problem:

1. **Resolve the exact runtime before diagnosing**: distinguish the project name, public app/domain, VPS IP/hostname, OS user, Hermes profile/home, and whether Hermes runs on the host or in Docker. Do not silently equate a business/app label with an older server label merely because they may share infrastructure.
2. **Inspect the original live source first**: if shell access exists, inspect that remote Hermes config/auth/logs directly. Session history is secondary context, not proof of the current remote state.
3. **Separate host availability from provider health**: HTTP/TLS/nginx/OAuth probes can prove the host and front door are alive, but they cannot prove the Hermes gateway or model provider works.
4. **Avoid destructive auth recovery as diagnosis**: do not reset credentials, rebuild the VPS, or restart unrelated services merely to gain visibility. Recover scoped SSH/container access or obtain the exact user-facing error first.
5. **Once connected, verify in this order**: runtime location → gateway/service state → active model/provider → `hermes auth list <provider>` → recent gateway/error logs → minimal direct `hermes chat` probe with the same provider/model → narrow credential-store repair → affected gateway restart → repeat probe and platform smoke test.
6. **For Codex OAuth, inspect both state and shape**: a credential can exist but be exhausted/rate-limited, or be written only to one auth-store shape. Check before forcing reauthentication.
7. **Report evidence boundaries clearly**: say what is confirmed externally, what requires remote access, and what remains only a hypothesis.

See `references/remote-provider-auth-triage.md` for a concise probe and verification checklist.

## Known gateway memory pitfall: unreaped LSP clients across worktrees

When a long-lived gateway grows to several GB and systemd reports an OOM kill with many Node children, first compare host uptime/boot history with the user-service journal. Long host uptime plus `Failed with result 'oom-kill'` means the gateway restarted; do not misreport that as a VPS reboot. Then inspect the gateway cgroup before blaming Telegram or the model provider. Hermes LSP is process-wide and keys clients by `(server_id, workspace_root)`, so editing files across many git worktrees—or multiple package-level roots inside one monorepo—can spawn one TypeScript language-server tree per root.

Diagnostic pattern:

1. Check `systemctl --user show hermes-gateway.service -p MemoryCurrent -p MemoryPeak -p MemorySwapCurrent -p TasksCurrent -p MemoryHigh -p MemoryMax -p OOMScoreAdjust` and inspect `memory.stat`/`memory.pressure` for the cgroup.
2. Enumerate only processes whose `/proc/<pid>/cgroup` contains `hermes-gateway.service`; group `typescript-language-server`, `tsserver.js`, and `typingsInstaller.js` by `/proc/<pid>/cwd`. One TypeScript LSP tree can use roughly 0.5–0.6 GB RSS on a large monorepo.
3. Take two quick snapshots. Rapid increases in workspace roots, tasks, and cgroup memory are stronger leak evidence than a single large total. Avoid edits in additional roots while auditing a near-OOM gateway.
4. Inspect `agent/lsp/manager.py`. A declared `DEFAULT_IDLE_TIMEOUT` and `_last_used` map do not prove reaping works: verify `_idle_timeout` is actually read by an eviction/reaper path and that the config factory parses and passes an idle-timeout setting.
5. If `_idle_timeout` is only assigned and never consumed, LSP clients persist until whole-process shutdown. Multiple roots then accumulate Node trees and can push the gateway cgroup into OOM.
6. Separate incidents where the gateway reached several GB from host-wide OOM events where its unit peak remained small. A positive `OOMScoreAdjust` explains why the gateway was selected as victim, not what consumed host memory.
7. Temporary mitigation: disable `lsp.enabled` for the gateway and restart during a safe window. Config-on-disk alone does not reclaim current child trees. Durable repair: implement tested idle eviction plus a client-count/LRU bound and atomically remove client/timestamp state before shutdown.
8. Add systemd `MemoryHigh`/`MemoryMax` only as containment after fixing/reducing LSP growth; limits protect the VPS but do not fix the leak and can still restart the gateway.

Correlate this with host headroom: nearly-full swap can be stale rather than active thrashing, so attribute `VmSwap` by PID and inspect `vmstat` swap-in/out plus `/proc/pressure/memory`. Do not run `swapoff` unless available RAM can hold all swapped pages safely.

See `references/lsp-worktree-oom-forensics.md` for the detailed cgroup attribution, code-proof, incident-separation, and mitigation workflow. See `references/gateway-lsp-oom-containment.md` for the controlled external-systemd restart pattern, post-restart acceptance checks, cgroup cache decomposition, and false-positive-safe LSP process accounting.

## Restart discipline

After patching Hermes runtime code used by the gateway:

- A successful `py_compile` means the file is syntactically valid, not that the running gateway has loaded it.
- A live Telegram gateway likely needs `hermes gateway restart` before using patched code.
- If Alex is actively running another long session, do **not** restart immediately unless asked; warn that restart can interrupt active agents.

### Pitfall: one-shot cron restart script can self-loop

If the gateway is restarting every ~1-2 minutes and `journalctl --user -u hermes-gateway.service` shows repeated `Stopping hermes-gateway.service` shortly after startup, check for a no-agent cron job whose script calls `systemctl --user restart hermes-gateway.service`.

Root cause pattern:

```text
CGroup: hermes-gateway.service
  ├─... python -m hermes_cli.main gateway run
  ├─... bash ~/.hermes/scripts/restart_gateway_*.sh
  └─... systemctl --user restart hermes-gateway.service
```

A cron job running inside the gateway asks systemd to restart the gateway, killing the gateway process before the cron scheduler can record the one-shot as completed. On the next gateway startup, the due one-shot still has `last_run_at: null` / `enabled: true`, so it fires again and creates a restart loop.

Recovery:

1. List cron jobs: `hermes cron list --all` or `cronjob(action='list')`.
2. Remove or pause the stuck one-shot restart job.
3. Let the in-flight restart finish, then verify `hermes gateway status` is active and `ps` no longer shows the restart script/systemctl child.
4. For future gateway reloads, prefer a direct `hermes gateway restart` or an external systemd timer/service that is not supervised by the gateway's own cron runner.

## User-facing style for runtime incidents

Alex prefers root-cause evidence and compact summaries. Use this shape:

- **Session ID / component:** exact identifier if known.
- **Observed log line:** one short quote, not full traceback.
- **Root cause:** plain English.
- **Fix applied:** file/function/pattern.
- **Status:** verified on disk vs needs restart.
- **Next step:** one clear action.