---
name: hermes-knowledge-foundry
description: Use when operating, extending, testing, deploying, or troubleshooting Hermes Knowledge Foundry; ingesting reviewed documents; importing compiled wikis; publishing evidence projections to OpenNotebook or Quartz; or validating Hermes Brain graph behavior.
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [knowledge-foundry, ingestion, provenance, graph, opennotebook, quartz]
    related_skills: [coding-quality-workflow, vps-app-deployment, llm-wiki]
---

# Hermes Knowledge Foundry Operations

## Overview

Hermes Knowledge Foundry is the source-faithful ingestion and knowledge-production layer at `/home/avalon/apps/hermes-knowledge-foundry`. Runtime data lives at `/data/hermes-knowledge-foundry`; PM2 process `hermes-knowledge-foundry` binds to `127.0.0.1:4044`; nginx exposes the authenticated site at `https://knowledge-foundry.apps.poofc.com`.

Foundry owns classification, bounded extraction, provenance, QA, source packages, retrieval evidence, and typed graph production. OpenNotebook and Quartz consume validated projections; neither is canonical truth.

## When to Use

- Ingest a reviewed PDF, EPUB, document, presentation, email, webpage, spreadsheet, audio, or video.
- Add or repair extraction routing, QA, graph, retrieval, or evidence contracts.
- Import Human Design, astrology, or another compiled Markdown wiki.
- Publish a validated projection to OpenNotebook or Quartz.
- Diagnose the operator cockpit, Hermes Brain 3D, PM2 service, nginx route, or build.
- Run pre-deployment or post-deployment verification.

Do not bypass quarantine/review boundaries for arbitrary downloaded files. Do not crawl a wiki's `raw` tree when compiled `concepts`, `sources`, `queries`, or `summaries` exist.

## Invariants

1. **Canonical source package:** immutable original, manifest, extraction JSONL, graph JSON, and benchmark JSON under `/data/hermes-knowledge-foundry/tenants/<tenant>/sources/<source_id>/`.
2. **Evidence first:** every surfaced claim or graph relationship must resolve to an evidence element and provenance handle where the relationship semantics require evidence.
3. **Extraction is not knowledge:** never promote first tokens, frequent words, filenames, hashes, or raw evidence IDs into concepts. Extraction may publish a source/evidence package; concepts require a trusted synthesis or compiled-wiki stage.
4. **Curated graph wins:** `/data/hermes-knowledge-foundry/tenants/<tenant>/knowledge-graph.json` is the human-facing semantic graph and takes precedence over per-source extraction graphs. Keep demos and smoke fixtures in separate tenants.
5. **Fail closed:** malformed routes, unreadable text, unsafe local paths, extraction bounds, bad provenance, and QA failures become `needs_review`, `blocked`, or `rejected`; never silently publish.
6. **Safe identifiers:** tenant and source IDs are 1–64 characters from `[A-Za-z0-9_-]` and cannot become filesystem paths.
7. **Bounded work:** one extraction at a time in the deployed process; upload, PDF-page, archive-member, decompression, evidence-count, and graph-query limits remain enforced.
8. **No raw paths or secrets:** API responses and projections use `foundry://` evidence handles, source IDs, validation hashes, pages, timestamps, rows, slides, or relative wiki references.
9. **No Dense Summary:** OpenNotebook source requests send `transformations: []`.

## Safe Ingestion Workflow

1. Confirm the file is already security-reviewed and located under one of `FOUNDRY_ALLOWED_LOCAL_ROOTS`. Current deployment roots are listed in `.env.example` and `ecosystem.config.cjs`.
2. Intake the source:
   ```bash
   curl -fsS -X POST http://127.0.0.1:4044/api/sources/intake \
     -H 'Content-Type: application/json' \
     -d '{"tenant_id":"default","local_path":"/data/open-notebook-corpus/reviewed/example.pdf"}'
   ```
3. Capture `source_id`, then process synchronously:
   ```bash
   curl -fsS -X POST "http://127.0.0.1:4044/api/sources/<source_id>/process?tenant_id=default"
   ```
4. Accept completion only when the result says `state: published` and `qa.passed: true`. For `needs_review`, inspect routing and golden-sample reason; do not force publication.
5. Verify the package contains `manifest.json`, `originals/`, `extraction.jsonl`, `graph.json`, and `benchmark.json` for a published source.
6. Verify provenance keys match the medium: page for PDF, container/DOM for EPUB, row/columns for CSV, slide for PPTX, paragraph for DOCX, line/byte for text, or metadata-only for untranscribed media.

Completion criterion: API, ledger state, package files, QA result, and medium-specific provenance all agree.

## Development and Quality Gates

From the repository root:

```bash
uv sync --extra dev
uv run --extra dev pytest -q
uv run --extra dev ruff check foundry tests
uvx bandit -q -r foundry
git diff --check

npm --prefix web test
npm --prefix web run build
npm --prefix web audit --omit=dev --audit-level=high
```

A Starlette `httpx2` migration warning may appear from `TestClient`; it is upstream and does not currently fail tests. Do not suppress new warnings without proving their origin.

For extractor changes, use TDD and include provenance assertions plus bounds/failure behavior. For graph changes, test missing endpoints, tenant mismatch, mandatory evidence references, semantic descriptions, and curated-graph precedence. For frontend changes, test accessible plain-language navigation, named concepts and connections, source attribution, absence of raw IDs/jargon, simple filters, and both 3D and card views.

Completion criterion: Python tests, frontend tests, Ruff, Bandit, build, npm audit, and diff check all pass.

## Existing Wiki Imports

Use `foundry.integrations.wiki_import.ExistingWikiImporter`. It scans compiled sections only: `concepts`, `entities`, `sources`, `queries`, and `summaries`. It must not traverse `raw`.

When auditing or comparing older book-to-wiki work against Foundry semantic compilers, inspect the original wiki production artifacts before judging equivalence: working scripts, golden-sample reports, raw package manifests, section maps, per-chapter extraction JSON/Markdown, indexes, compiled concept/query/entity pages, wiki logs, git commits, and prior session evidence. A Foundry `semantic-graph.json` can be better evidenced than a wiki page, but it is not a replacement for chapter markdown, compiled notes, rendered-page assets, or human-readable synthesis stages unless those projections were actually produced.

Importer pitfall: `ExistingWikiImporter` currently derives descriptions from the first non-heading body paragraph. For sectioned compiled pages whose first body text is metadata such as `Degree range`, the human-facing graph may miss richer sections like `## Headline` or `## Imagery`. If graph descriptions look thin, inspect the compiled Markdown and consider a section-aware importer rather than assuming the source wiki lacks semantic content.

When importing the Hermes `astro` profile or astrology stores, first classify content into four separate graph/import boundaries: (1) shared compiled source wiki, (2) profile skill methods/workflows, (3) private tenant KB people/charts/readings/transits, and (4) operational/tool docs. Do not let skills, private tenant facts, or ops docs masquerade as source-backed astrology concepts. See `references/astro-profile-knowledge-import-boundaries.md` for node kinds, relationship types, and import boundaries.

For the active astrology graph, import both the compiled source wiki and the read-only knowledge-bearing layer from the Hermes `astro` profile:

```bash
cd /home/avalon/apps/hermes-knowledge-foundry
.venv/bin/python scripts/import_astrology_profile_graph.py
```

This preserves source-wiki books/concepts/people; adds distinctly typed profile methods, tools, workflows, systems, and reference notes; and imports the bounded `Planets in Transit` heading index as source-backed `transit_section` nodes with exact chapter-line/PDF-page provenance. `raw/books/**` remains evidence, not free-floating concepts. Private tenant people/charts/readings/transits, generic non-astrology profile skills, node-module docs, sessions, memories, credentials, and operational state remain excluded. `Planets in Transit` remains a `book`; each indexed transit section is `PART_OF` the book and related to the `Transits` concept. The import is bounded, read-only against the other profile, and writes only the default tenant's Foundry projection.

Use `scripts/import_wiki_graph.py` only for intentional wiki-only imports or diagnostics; running it against `astro-sources-wiki` will replace the active combined graph with the smaller source-wiki-only projection.

The wiki importer creates named books, concepts, people and questions; descriptions and source references from frontmatter/body; `RELATED_TO` edges from wikilinks; and `CITES` edges from `src:` citations. Keep file-count and per-file byte limits. The generated tenant-level `knowledge-graph.json` takes precedence over raw per-source graphs.

For programmatic inspection:

```python
from foundry.integrations.wiki_import import ExistingWikiImporter

graph = ExistingWikiImporter().import_tree(
    "/home/avalon/wiki-human-design",
    tenant_id="default",
)
```

For `/home/avalon/wiki-human-design` specifically, see `references/human-design-wiki-import-audit.md` for the current read-only audit probes, observed counts, raw/provenance boundaries, privacy scan patterns, and safe import recommendation. In short: treat it as one Human Design library corpus/source collection with many internal source/book/source_identity nodes; import only compiled sections (`concepts`, `entities`, `sources`, `queries`, `summaries`); keep `raw/` and `_meta/source-registry.json` as provenance/evidence, not primary graph input; prefer a separate tenant such as `human-design` unless the user explicitly wants to replace the active default graph.

Treat any numeric import figures as historical smoke baselines, not permanent invariants. Verify semantic labels, descriptions, source references, and named connections—not just counts.

## Human-Readable Ingestion Inspection

A knowledge cloud is a navigation surface, not proof that ingestion captured or organized a source well. Every ingested source should have an **Inspect ingestion** view that lets Alex assess the complete result without interpreting storage files or internal IDs.

Keep five things distinct in the interface:

1. complete retained source evidence;
2. the complete source-specific semantic inventory;
3. all cited relationships produced by compilation;
4. short graph/card descriptions;
5. complete compiled Markdown notes when they exist.

The inspector should default to plain language and show **Overview**, **Knowledge**, and **Extracted text** views. Overview explains each pipeline stage, totals, semantic type counts, recognized coverage, and honest limitations. Knowledge lists every compiled item and relationship with names, pages/timestamps, and citation counts—not only the nodes currently rendered in the cloud. Extracted text provides bounded pagination through every evidence element and visibly confirms when the final element has been reached.

Never answer “is that the whole file?” ambiguously. State explicitly whether the user is seeing a bounded description, full semantic item, complete Markdown note, or retained source evidence. Replace filename/ID-only technical disclosures with actions such as **Read full note** and **Inspect ingestion**. Keep absolute paths, hashes, source IDs, evidence IDs, and raw JSON out of the primary view.

### Visible-artifact acceptance rule

When Alex asks for compiled Markdown, the acceptance artifact is the **actual rendered note**, not node/edge counts, file counts, paths, character totals, architecture summaries, or claims that generation succeeded.

1. Immediately read and show a representative complete note or substantial verbatim Markdown excerpt.
2. Label its lifecycle truth precisely: staged filesystem artifact, active compiled corpus, and deployed UI-visible note are three different states.
3. Do not call the feature complete until the deployed interaction is exercised end to end: click the knowledge-map card/node or ingestion concept card → drawer opens → full-note action resolves → styled Markdown headings/body appear.
4. Make the whole knowledge card keyboard- and pointer-clickable when it visually presents as an action; a dead card with a hidden or tiny action does not satisfy this contract.
5. Browser/API tests must assert real note content and headings, not only that a drawer exists or an endpoint returns 200.
6. Report implementation details only after showing the requested artifact and verified click path. If the live site is still on the old bundle/corpus, say so directly rather than treating staged files as delivered.

For the API/UI contract, safe Markdown-note resolution, pagination, security boundaries, and verification checklist, see `references/human-readable-ingestion-inspector.md`. For compiled-note and library-note security reviews, also use `references/compiled-note-symlink-security.md`: reject symlinks before resolving paths and verify the resolved target stays under both the configured root and the intended compiled-section root.

## UI lifecycle, activity progress, and Checks audits

When auditing or redesigning the Sources, Activity, or Checks tabs, keep source lifecycle, extraction jobs, and quality artifacts separate:

- Do not accept an unconditional **Extract text** button as correct. Published/compiled sources should default to **Inspect ingestion** or **View in knowledge map**; re-extraction should be a deliberate **Re-run extraction…** action with confirmation and a backend force/guard path.
- Treat ledger lifecycle events as event history, not full progress. Real activity progress needs job-level operation/status/stage/percent/current/total/timing fields and a pollable jobs list or active-jobs endpoint.
- Activity should be job-derived rather than source-list-derived. A hardcoded duration such as `recorded` or a source `updated_at` timestamp is not evidence of progress support.
- Top-level Checks cards are only useful if they preserve source association, scores, issues, sample counts, artifact timestamps, and distinguish extraction QA from semantic compilation checks. Otherwise prefer folding quality status into **Inspect ingestion** and removing/renaming the detached Checks tab.

For a read-only audit workflow, exact files/routes to inspect, current failure patterns, and a minimal redesign rubric, see `references/ui-lifecycle-progress-checks-audit.md`.

## Human-Facing Knowledge Graph

Treat Hermes Brain as a knowledge product, not an operator/debug graph. Default to books, concepts, people, descriptions, source attribution, and named connections. Keep raw IDs, hashes, evidence shards, tenant controls, graph levels, confidence sliders, and pathfinding out of the primary interface. Put technical provenance behind an optional disclosure.

Use plain product language. Replace “backpressure” with the actual consequence (“one source at a time so the server stays responsive”) and do not invent labels such as “trust fan-out.” Use the established Hermes cream/navy/gold visual language rather than dark/cyber control-room styling. The 3D map needs visible labels and a readable card/list alternative.

See `references/semantic-graph-and-plain-language-ui.md` for the failure mode, corrected architecture, UX rules, and regression checks.

## Quartz Publication

Use `WikiGraphBuilder` to compile evidence-aware Markdown pages and `QuartzExporter` to write them beneath:

```text
/home/avalon/apps/obsidian-quartz/content/hermes-knowledge-foundry
```

Build Quartz with:

```bash
cd /home/avalon/apps/obsidian-quartz
npx quartz build
```

Quartz builds can exceed five minutes under VPS pressure. Run bounded long builds with Hermes background process tracking and `notify_on_complete=true`; never leave `public/` half-cleaned after a foreground timeout. Completion requires `Done processing ... files` and a successful exit.

## OpenNotebook Publication

Use `OpenNotebookExporter` to build a validated text payload and `OpenNotebookClient` to execute the request. Runtime credentials must come from environment/configuration and are excluded from request representations.

Required source contract:

```json
{
  "type": "text",
  "notebooks": ["notebook:..."],
  "transformations": [],
  "embed": true,
  "delete_source": false,
  "async_processing": true
}
```

Endpoints are `/api/notebooks`, `/api/sources/json`, `/api/sources/{id}/status`, `/api/notes`, and `/api/search`.

OpenNotebook command acceptance is not completion. Poll source status and confirm populated `full_text`; if a command remains `new`, diagnose the OpenNotebook worker/Temporal queue rather than resubmitting repeatedly. Synchronous source processing may block on the same worker issue.

## Deployment

```bash
cd /home/avalon/apps/hermes-knowledge-foundry
npm --prefix web run build
pm2 reload ecosystem.config.cjs --only hermes-knowledge-foundry --update-env
```

Wait for health readiness, then:

```bash
.venv/bin/python scripts/smoke.py
pm2 save
curl -sS -o /dev/null -w '%{http_code}\n' https://knowledge-foundry.apps.poofc.com/
```

Expected unauthenticated public status is `401`; HTTP redirects to HTTPS. Verify PM2 is online, restart count is stable after reload, local API smoke passes, TLS verifies, and browser console has no JavaScript errors.

For nginx changes: install the site config, run `nginx -t`, then reload. Never expose port 4044 publicly or bind Uvicorn to all interfaces.

## Common Pitfalls

1. **Adapter exists but pipeline does not call it.** Verify `ProcessingService` invokes `extract_source`; add an end-to-end route-specific test.
2. **Wrong OpenNotebook paths.** The live API uses `/api/...`, and JSON source creation is `/api/sources/json`.
3. **Unbounded uploads.** Preserve `FOUNDRY_MAX_UPLOAD_BYTES`, streamed copy, cleanup of `.part` files, and HTTP 413 behavior.
4. **Tenant path traversal.** Validate identifiers at both API and storage-service boundaries.
5. **Unsafe archive XML.** Keep `defusedxml` and cumulative uncompressed archive limits.
6. **Quartz timeout after cleaning `public`.** Continue/re-run the build to successful completion before reporting Quartz healthy.
7. **Semantic poisoning:** extraction tokens are not concepts. A malformed text layer can have perfect provenance and still be gibberish. Require readability QA plus trusted synthesis; publish source-only state until synthesis exists.
8. **Demo leakage:** never seed the default tenant or mix smoke fixtures into the user’s knowledge graph. Use dedicated demo/smoke tenants and archive cleanup reversibly.
9. **Debug graph presented as knowledge:** the primary view must show named books/concepts/people, descriptions, source attribution, and human relationships. Hide raw IDs and operations jargon; use visible labels, simple filters, progressive expansion, and a card/list fallback.
10. **Bad 3D framing:** disconnected nodes can make `zoomToFit` shrink the useful cluster into a speck. Import explicit `related:` frontmatter and source-derived edges, assert there are no isolated public nodes, fit the camera once after the graph settles, and do not refit when a detail drawer opens.
11. **Treating an asynchronous security review as non-blocking.** If a review was requested as a pre-deployment gate, do not commit/deploy merely because local tests pass while the reviewer is still running. Wait for its completed verdict, resolve every blocker with a regression test, rerun the full gates, and only then deploy. If a reviewer times out without a summary, perform and document a fresh manual security pass instead of assuming success.

## Semantic compilation after extraction

A source reaching `published` means its evidence extraction and QA passed. It does **not** mean its knowledge has been compiled. A compiled source should have both `semantic-graph.json` and `semantic-status.json` in its source package. The status report must state page/evidence coverage, compiler identity, typed node/edge counts, and known limitations.

When integrating Foundry with Caduceus-style agent/business systems, keep Foundry as the canonical source/evidence/semantic layer and let the product database store pointers, review state, dossier projections, and UI status. Use the Foundry HTTP API for intake/process/inspect; do not write source packages directly from the product app. Treat social/profile/email/vendor-MCP outputs as evidence first; promote them to semantic knowledge only after a trusted compiler emits `semantic-graph.json` plus `semantic-status.json`. See `references/caduceus-knowledge-integration.md` for the recommended bridge flow, dossier citation shape, and platform/image-source limits.

For the reviewed *The Inner Sky* source, run:

```bash
cd /home/avalon/apps/hermes-knowledge-foundry
.venv/bin/python scripts/compile_inner_sky.py
.venv/bin/python scripts/import_astrology_profile_graph.py
```

The structured-book adapter creates book parts, chapters, concepts, methods, principles, workflow steps, interpretive components, people, and worked examples. Every semantic edge must carry extraction evidence IDs, every node must expose page-based `source_refs`, and `GraphCompiler` must validate endpoints and evidence before output is written. Shared books/concepts/people are merged by normalized kind + label; source descriptions, tags, references, and evidence are combined instead of creating duplicates.

Agent-facing `/api/graph/search` searches labels, types, descriptions, source references, and tags. Verify representative method and concept queries after each semantic import.

Do not imply that an arbitrary new book or video has become knowledge merely because text/transcript extraction passed. Unsupported sources remain visibly `not_compiled` until a trusted structure/domain adapter or validated model-assisted compiler produces a cited semantic graph.

## PDF ingestion integrity pitfalls

- A bounded PDF probe must inspect both the header and a bounded tail. Searching only the first 512 KiB for `%%EOF` falsely labels valid PDFs larger than the probe window as malformed.
- Never publish a book because the first N chunks passed QA. PDF extraction must carry `page` and `page_count`, cover every page, and raise `ExtractionError` when page/element bounds are exceeded. Partial extraction fails closed into review.
- Do not split books into one element per text line under a tiny global cap. Chunk normalized text page-by-page within the character bound so provenance remains meaningful and complete books fit bounded limits.
- Security QA must distinguish ordinary prose containing words such as “secret” from credential-shaped values. Check local paths in provenance/metadata; check source text for precise API-key/password/token assignment or Bearer-token patterns rather than broad word matching.
- After repairing a quarantined book, verify the persisted extraction itself: first page, last page, unique `page_count`, element count, total characters, QA score/issues, and final lifecycle state.

## Verification Checklist

- [ ] Repository clean and pushed to `firemountain/hermes-knowledge-foundry`
- [ ] Python and frontend gates pass
- [ ] Bandit has no findings and npm production audit reports no vulnerabilities
- [ ] Real reviewed PDF and EPUB smokes retain provenance and pass QA
- [ ] Malformed source smoke routes to `needs_review`; unreadable extraction cannot become concept nodes
- [ ] Default tenant has no demo/smoke records or synthetic evidence nodes
- [ ] Curated graph contains meaningful books, concepts and people with descriptions, source references and named connections
- [ ] Primary UI contains no raw IDs, hashes, “backpressure,” “fan-out,” or unexplained graph controls
- [ ] Desktop and mobile browser visual QA confirm Hermes cream/navy/gold styling, visible graph labels, readable card view and zero console errors
- [ ] Wiki importer excludes raw trees and stays bounded
- [ ] Quartz projection builds successfully
- [ ] OpenNotebook payload has no transformations and no raw paths
- [ ] PM2 is online and saved
- [ ] Local smoke passes
- [ ] Public endpoint requires auth and TLS verifies
- [ ] Browser console has zero errors