Hermes Knowledge Foundry is the source-faithful ingestion and knowledge-production layer at /home/avalon/apps/hermes-knowledge-foundry. Runtime data lives at /data/hermes-knowledge-foundry; PM2 process hermes-knowledge-foundry binds to 127.0.0.1:4044; nginx exposes the authenticated site at https://knowledge-foundry.apps.poofc.com.
Foundry owns classification, bounded extraction, provenance, QA, source packages, retrieval evidence, and typed graph production. OpenNotebook and Quartz consume validated projections; neither is canonical truth.
Do not bypass quarantine/review boundaries for arbitrary downloaded files. Do not crawl a wiki's raw tree when compiled concepts, sources, queries, or summaries exist.
/data/hermes-knowledge-foundry/tenants/<tenant>/sources/<source_id>/./data/hermes-knowledge-foundry/tenants/<tenant>/knowledge-graph.json is the human-facing semantic graph and takes precedence over per-source extraction graphs. Keep demos and smoke fixtures in separate tenants.needs_review, blocked, or rejected; never silently publish.[A-Za-z0-9_-] and cannot become filesystem paths.foundry:// evidence handles, source IDs, validation hashes, pages, timestamps, rows, slides, or relative wiki references.transformations: [].FOUNDRY_ALLOWED_LOCAL_ROOTS. Current deployment roots are listed in .env.example and ecosystem.config.cjs.bash
curl -fsS -X POST http://127.0.0.1:4044/api/sources/intake \
-H 'Content-Type: application/json' \
-d '{"tenant_id":"default","local_path":"/data/open-notebook-corpus/reviewed/example.pdf"}'source_id, then process synchronously:
bash
curl -fsS -X POST "http://127.0.0.1:4044/api/sources/<source_id>/process?tenant_id=default"state: published and qa.passed: true. For needs_review, inspect routing and golden-sample reason; do not force publication.manifest.json, originals/, extraction.jsonl, graph.json, and benchmark.json for a published source.Completion criterion: API, ledger state, package files, QA result, and medium-specific provenance all agree.
From the repository root:
uv sync --extra dev
uv run --extra dev pytest -q
uv run --extra dev ruff check foundry tests
uvx bandit -q -r foundry
git diff --check
npm --prefix web test
npm --prefix web run build
npm --prefix web audit --omit=dev --audit-level=high
A Starlette httpx2 migration warning may appear from TestClient; it is upstream and does not currently fail tests. Do not suppress new warnings without proving their origin.
For extractor changes, use TDD and include provenance assertions plus bounds/failure behavior. For graph changes, test missing endpoints, tenant mismatch, mandatory evidence references, semantic descriptions, and curated-graph precedence. For frontend changes, test accessible plain-language navigation, named concepts and connections, source attribution, absence of raw IDs/jargon, simple filters, and both 3D and card views.
Completion criterion: Python tests, frontend tests, Ruff, Bandit, build, npm audit, and diff check all pass.
Use foundry.integrations.wiki_import.ExistingWikiImporter. It scans compiled sections only: concepts, entities, sources, queries, and summaries. It must not traverse raw.
When auditing or comparing older book-to-wiki work against Foundry semantic compilers, inspect the original wiki production artifacts before judging equivalence: working scripts, golden-sample reports, raw package manifests, section maps, per-chapter extraction JSON/Markdown, indexes, compiled concept/query/entity pages, wiki logs, git commits, and prior session evidence. A Foundry semantic-graph.json can be better evidenced than a wiki page, but it is not a replacement for chapter markdown, compiled notes, rendered-page assets, or human-readable synthesis stages unless those projections were actually produced.
Importer pitfall: ExistingWikiImporter currently derives descriptions from the first non-heading body paragraph. For sectioned compiled pages whose first body text is metadata such as Degree range, the human-facing graph may miss richer sections like ## Headline or ## Imagery. If graph descriptions look thin, inspect the compiled Markdown and consider a section-aware importer rather than assuming the source wiki lacks semantic content.
When importing the Hermes astro profile or astrology stores, first classify content into four separate graph/import boundaries: (1) shared compiled source wiki, (2) profile skill methods/workflows, (3) private tenant KB people/charts/readings/transits, and (4) operational/tool docs. Do not let skills, private tenant facts, or ops docs masquerade as source-backed astrology concepts. See references/astro-profile-knowledge-import-boundaries.md for node kinds, relationship types, and import boundaries.
For the active astrology graph, import both the compiled source wiki and the read-only knowledge-bearing layer from the Hermes astro profile:
cd /home/avalon/apps/hermes-knowledge-foundry
.venv/bin/python scripts/import_astrology_profile_graph.py
This preserves source-wiki books/concepts/people; adds distinctly typed profile methods, tools, workflows, systems, and reference notes; and imports the bounded Planets in Transit heading index as source-backed transit_section nodes with exact chapter-line/PDF-page provenance. raw/books/** remains evidence, not free-floating concepts. Private tenant people/charts/readings/transits, generic non-astrology profile skills, node-module docs, sessions, memories, credentials, and operational state remain excluded. Planets in Transit remains a book; each indexed transit section is PART_OF the book and related to the Transits concept. The import is bounded, read-only against the other profile, and writes only the default tenant's Foundry projection.
Use scripts/import_wiki_graph.py only for intentional wiki-only imports or diagnostics; running it against astro-sources-wiki will replace the active combined graph with the smaller source-wiki-only projection.
The wiki importer creates named books, concepts, people and questions; descriptions and source references from frontmatter/body; RELATED_TO edges from wikilinks; and CITES edges from src: citations. Keep file-count and per-file byte limits. The generated tenant-level knowledge-graph.json takes precedence over raw per-source graphs.
For programmatic inspection:
from foundry.integrations.wiki_import import ExistingWikiImporter
graph = ExistingWikiImporter().import_tree(
"/home/avalon/wiki-human-design",
tenant_id="default",
)
For /home/avalon/wiki-human-design specifically, see references/human-design-wiki-import-audit.md for the current read-only audit probes, observed counts, raw/provenance boundaries, privacy scan patterns, and safe import recommendation. In short: treat it as one Human Design library corpus/source collection with many internal source/book/source_identity nodes; import only compiled sections (concepts, entities, sources, queries, summaries); keep raw/ and _meta/source-registry.json as provenance/evidence, not primary graph input; prefer a separate tenant such as human-design unless the user explicitly wants to replace the active default graph.
Treat any numeric import figures as historical smoke baselines, not permanent invariants. Verify semantic labels, descriptions, source references, and named connections—not just counts.
A knowledge cloud is a navigation surface, not proof that ingestion captured or organized a source well. Every ingested source should have an Inspect ingestion view that lets Alex assess the complete result without interpreting storage files or internal IDs.
Keep five things distinct in the interface:
The inspector should default to plain language and show Overview, Knowledge, and Extracted text views. Overview explains each pipeline stage, totals, semantic type counts, recognized coverage, and honest limitations. Knowledge lists every compiled item and relationship with names, pages/timestamps, and citation counts—not only the nodes currently rendered in the cloud. Extracted text provides bounded pagination through every evidence element and visibly confirms when the final element has been reached.
Never answer “is that the whole file?” ambiguously. State explicitly whether the user is seeing a bounded description, full semantic item, complete Markdown note, or retained source evidence. Replace filename/ID-only technical disclosures with actions such as Read full note and Inspect ingestion. Keep absolute paths, hashes, source IDs, evidence IDs, and raw JSON out of the primary view.
When Alex asks for compiled Markdown, the acceptance artifact is the actual rendered note, not node/edge counts, file counts, paths, character totals, architecture summaries, or claims that generation succeeded.
For the API/UI contract, safe Markdown-note resolution, pagination, security boundaries, and verification checklist, see references/human-readable-ingestion-inspector.md. For compiled-note and library-note security reviews, also use references/compiled-note-symlink-security.md: reject symlinks before resolving paths and verify the resolved target stays under both the configured root and the intended compiled-section root.
When auditing or redesigning the Sources, Activity, or Checks tabs, keep source lifecycle, extraction jobs, and quality artifacts separate:
recorded or a source updated_at timestamp is not evidence of progress support.For a read-only audit workflow, exact files/routes to inspect, current failure patterns, and a minimal redesign rubric, see references/ui-lifecycle-progress-checks-audit.md.
Treat Hermes Brain as a knowledge product, not an operator/debug graph. Default to books, concepts, people, descriptions, source attribution, and named connections. Keep raw IDs, hashes, evidence shards, tenant controls, graph levels, confidence sliders, and pathfinding out of the primary interface. Put technical provenance behind an optional disclosure.
Use plain product language. Replace “backpressure” with the actual consequence (“one source at a time so the server stays responsive”) and do not invent labels such as “trust fan-out.” Use the established Hermes cream/navy/gold visual language rather than dark/cyber control-room styling. The 3D map needs visible labels and a readable card/list alternative.
See references/semantic-graph-and-plain-language-ui.md for the failure mode, corrected architecture, UX rules, and regression checks.
Use WikiGraphBuilder to compile evidence-aware Markdown pages and QuartzExporter to write them beneath:
/home/avalon/apps/obsidian-quartz/content/hermes-knowledge-foundry
Build Quartz with:
cd /home/avalon/apps/obsidian-quartz
npx quartz build
Quartz builds can exceed five minutes under VPS pressure. Run bounded long builds with Hermes background process tracking and notify_on_complete=true; never leave public/ half-cleaned after a foreground timeout. Completion requires Done processing ... files and a successful exit.
Use OpenNotebookExporter to build a validated text payload and OpenNotebookClient to execute the request. Runtime credentials must come from environment/configuration and are excluded from request representations.
Required source contract:
{
"type": "text",
"notebooks": ["notebook:..."],
"transformations": [],
"embed": true,
"delete_source": false,
"async_processing": true
}
Endpoints are /api/notebooks, /api/sources/json, /api/sources/{id}/status, /api/notes, and /api/search.
OpenNotebook command acceptance is not completion. Poll source status and confirm populated full_text; if a command remains new, diagnose the OpenNotebook worker/Temporal queue rather than resubmitting repeatedly. Synchronous source processing may block on the same worker issue.
cd /home/avalon/apps/hermes-knowledge-foundry
npm --prefix web run build
pm2 reload ecosystem.config.cjs --only hermes-knowledge-foundry --update-env
Wait for health readiness, then:
.venv/bin/python scripts/smoke.py
pm2 save
curl -sS -o /dev/null -w '%{http_code}\n' https://knowledge-foundry.apps.poofc.com/
Expected unauthenticated public status is 401; HTTP redirects to HTTPS. Verify PM2 is online, restart count is stable after reload, local API smoke passes, TLS verifies, and browser console has no JavaScript errors.
For nginx changes: install the site config, run nginx -t, then reload. Never expose port 4044 publicly or bind Uvicorn to all interfaces.
ProcessingService invokes extract_source; add an end-to-end route-specific test./api/..., and JSON source creation is /api/sources/json.FOUNDRY_MAX_UPLOAD_BYTES, streamed copy, cleanup of .part files, and HTTP 413 behavior.defusedxml and cumulative uncompressed archive limits.public. Continue/re-run the build to successful completion before reporting Quartz healthy.zoomToFit shrink the useful cluster into a speck. Import explicit related: frontmatter and source-derived edges, assert there are no isolated public nodes, fit the camera once after the graph settles, and do not refit when a detail drawer opens.A source reaching published means its evidence extraction and QA passed. It does not mean its knowledge has been compiled. A compiled source should have both semantic-graph.json and semantic-status.json in its source package. The status report must state page/evidence coverage, compiler identity, typed node/edge counts, and known limitations.
When integrating Foundry with Caduceus-style agent/business systems, keep Foundry as the canonical source/evidence/semantic layer and let the product database store pointers, review state, dossier projections, and UI status. Use the Foundry HTTP API for intake/process/inspect; do not write source packages directly from the product app. Treat social/profile/email/vendor-MCP outputs as evidence first; promote them to semantic knowledge only after a trusted compiler emits semantic-graph.json plus semantic-status.json. See references/caduceus-knowledge-integration.md for the recommended bridge flow, dossier citation shape, and platform/image-source limits.
For the reviewed The Inner Sky source, run:
cd /home/avalon/apps/hermes-knowledge-foundry
.venv/bin/python scripts/compile_inner_sky.py
.venv/bin/python scripts/import_astrology_profile_graph.py
The structured-book adapter creates book parts, chapters, concepts, methods, principles, workflow steps, interpretive components, people, and worked examples. Every semantic edge must carry extraction evidence IDs, every node must expose page-based source_refs, and GraphCompiler must validate endpoints and evidence before output is written. Shared books/concepts/people are merged by normalized kind + label; source descriptions, tags, references, and evidence are combined instead of creating duplicates.
Agent-facing /api/graph/search searches labels, types, descriptions, source references, and tags. Verify representative method and concept queries after each semantic import.
Do not imply that an arbitrary new book or video has become knowledge merely because text/transcript extraction passed. Unsupported sources remain visibly not_compiled until a trusted structure/domain adapter or validated model-assisted compiler produces a cited semantic graph.
%%EOF falsely labels valid PDFs larger than the probe window as malformed.page and page_count, cover every page, and raise ExtractionError when page/element bounds are exceeded. Partial extraction fails closed into review.page_count, element count, total characters, QA score/issues, and final lifecycle state.firemountain/hermes-knowledge-foundryneeds_review; unreadable extraction cannot become concept nodes