agent-capability-surface-audits

Agent Capability Surface Audits

Use this skill for source audits that ask whether capabilities are available to agents, humans, both, or neither—especially plugin-to-API integrations, typed tool layers, multi-worktree migrations, and claims of portability into an existing agent runtime.

Core rule

A capability is not agent-native merely because business logic, a human REST route, or a UI exists. Prove the complete chain:

native registration → typed handler → transport request → authenticated agent route → canonical command → durable state/effect

Audit human and internal surfaces separately.

Classification

Classify every requested capability as one or more of:

• native tool — registered typed agent tool with a verified reachable backend;
• raw HTTP — authenticated agent route exists but no typed tool wraps it;
• UI — human/admin route and mounted UI exist;
• internal — command/table/job exists without an exposed agent or human surface;
• planned — documentation or projection exists without an operational chain;
• missing — no meaningful implementation evidence;
• declared-but-broken — tool/manifest claims exist, but route, binding, or runtime wiring fails.

Workflow

1. Establish snapshot truth

1. Enumerate the main checkout and every relevant worktree.
2. Record branch/HEAD, upstream/default-branch HEAD, divergence, and git status --short before testing. Refresh remote refs when the audit permits it; a stale local checkout can falsely prove that a plugin is absent or branch-only.
3. Include untracked source/tests in the audit.
4. Distinguish current upstream/default branch, local checkout, branch-only work, and dirty-worktree additions in the final report. Never let an old local main override newer upstream source without saying so.

2. Cluster duplicated worktrees

Large migrations often copy the same plugin and Gate across many worktrees. Hash plugin source and Gate source, count registered tools, manifest tools, and routes, then group identical snapshots. Deep-read one representative per unique cluster plus every dirty variant.

Do not imply that copied snapshots are independent implementations.

3. Build the evidence map

For every capability, inspect independently:

1. plugin manifest and registration;
2. tool schema and handler;
3. emitted method/path/query/body/headers;
4. authenticated agent route and scope checks;
5. canonical command or provider adapter;
6. human API route and mounted UI;
7. internal tables/jobs/projections;
8. runtime/profile provisioning and health.

Record exact paths and line ranges while reading.

4. Compare registration with declaration

Programmatically derive all registered/schema tool names and compare them with the plugin manifest. Report duplicates and both set differences. Beware tests that convert names to a set or validate only one toolset subset.

5. Verify tool-route parity behaviorally

Normalize native tools and server routes to (tool, method, path template). For mismatches, start an in-memory server and call both the plugin-emitted path and any suspected legacy path. Mocked client tests establish formatting only, not server reachability.

6. Probe default routing and authorization separately

A capability key may authorize a project while the URL still requires a real project ID/slug. Exercise the plugin's default project/profile reference against the real resolver. Never assume symbolic values such as current are supported without server code and a passing probe.

7. Trace provenance end to end

Verify the exact client representation is consumed by the Gate and appears in durable audit/activity rows:

• GET requests cannot depend on JSON bodies;
• header names and body fields must match server expectations;
• strict command validation must consume or strip metadata;
• direct core audit tests do not prove plugin→Gate provenance.

8. Separate profile records from runtime provisioning

A profile database row is not proof that an arbitrary existing agent runtime can use the capability. Require plugin installation/discovery, enabled config, Gate URL, secret delivery, exact project binding, identity binding, runtime launch/health, and a real native tool call through the runtime loader.

Hard-coded hosts, local monorepo imports, fixed domains, and appliance-specific scripts are coupling assumptions to report explicitly.

9. Audit installation into an existing or multiplex Hermes

A disposable subprocess with HERMES_HOME set proves only single-process discovery. It does not prove safety in a gateway serving multiple profiles.

Inspect whether the plugin uses Hermes's current profile-aware APIs for active home/config/secret resolution rather than reading process-global HERMES_HOME, profile names, or credentials directly. Prove two profiles with different endpoints and keys cannot resolve each other's connection. Also determine:

• whether plugin toolsets become globally/platform-visible automatically;
• whether large mutating toolsets are exposed to unrelated sessions after enablement;
• whether installation or enablement requires a gateway restart;
• whether the current conversation can continue after restart;
• whether target selection is profile-local, session-local, or process-global;
• whether one Hermes can represent multiple tenant connections without tool-name collisions or credential leakage.

For a general-purpose Hermes with broad host authority, treat global exposure of vertical mutating tools as a blast-radius finding. Prefer explicit activation, a read-only discovery surface, opaque connection aliases, and session-local target selection.

10. Check action capability and offline boundaries

Project/module scope is not necessarily action authorization. Verify a server-enforced matrix across tenant/connection × business/project × module × action class such as read, propose, execute-under-contract, and administer. Role names and project bindings alone are insufficient.

Distinguish remote use of a tenant from relocating its brain. Confirm the vertical product remains manually operable without an external Hermes and that deterministic schedules, reconciliation, audit, credentials, and canonical state stay tenant-local. Compare deployment flags with scheduler startup code: implemented-but-disabled workers are not offline autonomy.

11. Run focused verification

Prefer focused plugin tests, command parity tests, Gate integration tests, and small adversarial probes. Tests must use temporary state and keep audit-only repositories unchanged. Re-run git status --short afterward.

Reporting format

1. Outcome — whether the claimed native/portable surface is real.
2. Capability matrix — native, raw HTTP, UI/internal, verdict.
3. Coupling and drift findings — ordered by impact, with exact paths.
4. Verification — pass/fail counts and behavioral probe output.
5. Cleanliness — files changed by the audit versus pre-existing dirt.

Use precise labels such as “branch-only,” “declared-but-broken,” and “internal-only.” Avoid collapsing a rich human Studio into an agent capability merely because both ultimately touch the same table.

Pitfalls

• Treating docs, route names, or green registration tests as end-to-end proof.
• Auditing only the main checkout when capability work is branch-only.
• Reading every copied worktree manually instead of clustering snapshots.
• Assuming manifest declarations match runtime registration.
• Assuming mocked HTTP handlers prove real Gate route parity.
• Calling a UI/admin route an agent route.
• Calling a profile registry row “provisioned.”
• Losing provenance because client and server use different headers/body fields.
• Failing to report dirty worktree state in a read-only audit.

Reference

See references/caduceus-hermes-plugin-agent-gate-capability-audit.md for a concrete Caduceus path map, probe recipes, and portability checks.