---
name: multimodal-knowledge-ingestion-platforms
description: Design, build, verify, and operate source-faithful multimodal ingestion systems that turn heterogeneous files into provenance-backed evidence, retrieval indexes, wikis, and typed knowledge graphs.
version: 1.0.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [ingestion, provenance, multimodal, knowledge-graph, rag, wiki, qa, security]
    category: software-development
    related_skills: [llm-wiki, ocr-and-documents, vps-app-deployment]
---

# Multimodal Knowledge Ingestion Platforms

Build ingestion systems for books, PDFs, scans, EPUBs, Office files, email, webpages, tables, audio, video, and folders without collapsing provenance into generic chunks.

## Activate when

- A user asks for a universal ingestion pipeline, evidence store, document intelligence system, knowledge operating system, or ingestion director.
- The output must feed several consumers: RAG, agents, notebooks, wikis, Quartz/Obsidian, search, or a knowledge graph.
- Extraction quality, immutable originals, citations, quarantine, resumability, or tenant isolation matters.
- Existing source-heavy wikis must be imported without crawling their entire raw media trees.

This skill complements `llm-wiki`: the wiki is a durable compiled knowledge consumer; this skill governs the source-package, extraction, QA, graph, and publisher machinery upstream.

## Non-negotiable architecture

```text
intake
→ quarantine and immutable package
→ probe/classify
→ route-specific golden sample
→ bounded extraction
→ evidence QA
→ canonical evidence package
→ retrieval/wiki/graph projections
→ consumer-specific publishers
```

The ingestion director owns source identity, hashes, manifests, routing, evidence, QA, lifecycle, and graph provenance. Notebook/RAG products and visualization sites are consumers, not the canonical evidence layer.

## Canonical source package

Use tenant-scoped, hash-addressed packages containing:

- immutable original;
- content SHA-256, byte size, and safe filename metadata;
- probe signals and selected route;
- append-only run/lifecycle events;
- extraction JSONL with stable evidence IDs;
- QA result and publication decision;
- retrieval/index artifacts;
- typed graph projection;
- public-safe manifest using storage URIs rather than host paths.

Deduplicate per tenant by content hash. Restrict local-path intake to explicit allowlisted roots. Browser uploads are copied into managed package storage.

## Route by medium

Do not pretend a generic text-line reader is universal extraction.

- Text/Markdown: line and byte provenance.
- HTML: suppress script/style/head chrome; preserve structural/DOM provenance.
- Email: headers, multipart body, attachment identities, thread structure.
- CSV/spreadsheets: row, sheet, and column provenance.
- DOCX/PPTX/EPUB: bounded archive/XML/HTML parsing; preserve paragraph/slide/container identity.
- PDF: real parser with page/block provenance; OCR/vision route for scans or artifacted pages.
- Audio/video: timestamped transcript/scene/slide routes; metadata-only output is a safe fallback, not a transcription claim.
- Unknown binary: metadata only, never raw payload surfacing.

Adapter code is not integrated until a pipeline-level test proves the chosen route invokes it and structural provenance survives into the package.

## Security and resource bounds

Treat every source as hostile until validated.

- Bound source bytes, emitted elements, characters per element, archive members, each decompressed member, total decompressed bytes, PDF pages, subprocess runtime, and model/API batch size.
- Use `defusedxml` or an equivalent hardened parser for untrusted XML.
- Reject symlinks and archive traversal.
- Keep extraction workers isolated where practical; never execute embedded macros/scripts.
- Sanitize API presentation models separately from internal manifests.
- Never return absolute host paths, credentials, raw stderr, or secret-bearing request objects.
- Batch and resume large corpora; persist checkpoints and terminal states.

## Golden sample and QA gates

Run a small representative extraction before full work. Fail closed:

- malformed/uncertain route → `needs_review`;
- extraction limit reached → `needs_review` with a stable public error code;
- missing provenance, path leaks, or secret-like values → publication blocked;
- only validated evidence may be indexed, graphed, or published.

Structural lint is necessary but insufficient. For valuable books/media, spot-check extracted evidence against original pages, frames, or timestamps.

## Typed graph

Compile graphs only after canonical evidence exists:

```text
canonical evidence
→ typed concepts/entities/claims
→ evidence-backed relationships
→ graph QA
→ multi-resolution projection
→ 2D and 3D viewers
```

Useful relations include `PART_OF`, `DERIVED_FROM`, `AUTHORED_BY`, `MENTIONS`, `DEFINES`, `SUPPORTS`, `CONTRADICTS`, `CITES`, `SAME_AS`, `RELATED_TO`, `DEPICTS`, `VALIDATED_BY`, and `PUBLISHED_AS`.

Edges that assert support or contradiction must carry exact evidence refs, confidence, source identity, extractor/model provenance, review status, and visibility scope.

Avoid graph hairballs. Default to global/collection/source levels and progressively expand neighborhoods. Provide filters, confidence thresholds, pathfinding, evidence drawers, and an accessible list fallback. Lazy-load heavy WebGL bundles.

## Consumer publishers

Each consumer gets a projection, not ownership of canonical truth.

- Notebook/RAG: validated Markdown/text with source IDs and validation hashes; avoid lossy default transformations unless explicitly requested.
- Wiki: frontmatter, compact source citations, wikilinks, index/log updates.
- Quartz/Obsidian: dedicated content subtree, backlinks/search/2D graph.
- Agents: narrow retrieval/tool contracts that return evidence IDs and citations.
- 3D graph: typed, multi-resolution projection rather than every raw chunk.

Validate every client against the live API/OpenAPI. Do not guess endpoint prefixes or send undocumented fields. Distinguish request accepted, queued, running, completed, and failed; an assigned job/source ID is not proof of completed processing.

## Existing-wiki import

For established wikis, orient to schema/index/source registry and import compiled sections such as `concepts/`, `sources/`, `queries/`, and summaries. Do not recursively crawl raw package/media trees.

- Enforce file-count and per-file byte limits.
- Build stable IDs from tenant plus logical relative identity, not absolute paths.
- Convert `[[wikilinks]]` into typed relationships.
- Convert compact `src:<id>` markers into source-identity nodes and `CITES` edges.
- Keep full artifact paths in registries/manifests, outside graph/API presentation.

## Required verification

Before calling a platform operational:

1. Unit, contract, static, and security checks.
2. Exact production frontend build and dependency audit.
3. Exact production ASGI/server target, not only test-client construction.
4. One reviewed real PDF and one reviewed real EPUB through intake → extraction → QA → graph → publication.
5. One malformed/unsupported fixture proving fail-closed review.
6. A bounded import of an existing compiled wiki with node/edge counts.
7. A publisher request against the live consumer API followed by terminal-state/result verification.
8. Final process reload, readiness loop, localhost API smoke, and public/auth boundary checks.

Report queued/accepted, completed, needs-review, and blocked states separately.

## Pitfalls

- Do not put all ingestion intelligence inside OpenNotebook or another RAG UI.
- Do not treat embeddings/chunks as canonical knowledge.
- Do not publish metadata-only extraction as if the document text was extracted.
- Do not crawl giant `raw/` trees when compiled pages and source registries exist.
- Do not run Dense Summary or comparable whole-book transformations by default.
- Do not claim a queued consumer job completed without polling and checking the resulting source/index.
- Do not expose every node/edge at once in 3D.
- Do not stop after writing adapters; wire them into the real pipeline and prove it with route-specific tests.

## Reference

See `references/source-faithful-foundry-contracts.md` for concrete publisher contracts, smoke-test expectations, and implementation lessons from a production multimodal Foundry build.