multimodal-knowledge-ingestion-platforms

Multimodal Knowledge Ingestion Platforms

Build ingestion systems for books, PDFs, scans, EPUBs, Office files, email, webpages, tables, audio, video, and folders without collapsing provenance into generic chunks.

Activate when

• A user asks for a universal ingestion pipeline, evidence store, document intelligence system, knowledge operating system, or ingestion director.
• The output must feed several consumers: RAG, agents, notebooks, wikis, Quartz/Obsidian, search, or a knowledge graph.
• Extraction quality, immutable originals, citations, quarantine, resumability, or tenant isolation matters.
• Existing source-heavy wikis must be imported without crawling their entire raw media trees.

This skill complements llm-wiki: the wiki is a durable compiled knowledge consumer; this skill governs the source-package, extraction, QA, graph, and publisher machinery upstream.

Non-negotiable architecture

intake
→ quarantine and immutable package
→ probe/classify
→ route-specific golden sample
→ bounded extraction
→ evidence QA
→ canonical evidence package
→ retrieval/wiki/graph projections
→ consumer-specific publishers

The ingestion director owns source identity, hashes, manifests, routing, evidence, QA, lifecycle, and graph provenance. Notebook/RAG products and visualization sites are consumers, not the canonical evidence layer.

Canonical source package

Use tenant-scoped, hash-addressed packages containing:

• immutable original;
• content SHA-256, byte size, and safe filename metadata;
• probe signals and selected route;
• append-only run/lifecycle events;
• extraction JSONL with stable evidence IDs;
• QA result and publication decision;
• retrieval/index artifacts;
• typed graph projection;
• public-safe manifest using storage URIs rather than host paths.

Deduplicate per tenant by content hash. Restrict local-path intake to explicit allowlisted roots. Browser uploads are copied into managed package storage.

Route by medium

Do not pretend a generic text-line reader is universal extraction.

• Text/Markdown: line and byte provenance.
• HTML: suppress script/style/head chrome; preserve structural/DOM provenance.
• Email: headers, multipart body, attachment identities, thread structure.
• CSV/spreadsheets: row, sheet, and column provenance.
• DOCX/PPTX/EPUB: bounded archive/XML/HTML parsing; preserve paragraph/slide/container identity.
• PDF: real parser with page/block provenance; OCR/vision route for scans or artifacted pages.
• Audio/video: timestamped transcript/scene/slide routes; metadata-only output is a safe fallback, not a transcription claim.
• Unknown binary: metadata only, never raw payload surfacing.

Adapter code is not integrated until a pipeline-level test proves the chosen route invokes it and structural provenance survives into the package.

Security and resource bounds

Treat every source as hostile until validated.

• Bound source bytes, emitted elements, characters per element, archive members, each decompressed member, total decompressed bytes, PDF pages, subprocess runtime, and model/API batch size.
• Use defusedxml or an equivalent hardened parser for untrusted XML.
• Reject symlinks and archive traversal.
• Keep extraction workers isolated where practical; never execute embedded macros/scripts.
• Sanitize API presentation models separately from internal manifests.
• Never return absolute host paths, credentials, raw stderr, or secret-bearing request objects.
• Batch and resume large corpora; persist checkpoints and terminal states.

Golden sample and QA gates

Run a small representative extraction before full work. Fail closed:

• malformed/uncertain route → needs_review;
• extraction limit reached → needs_review with a stable public error code;
• missing provenance, path leaks, or secret-like values → publication blocked;
• only validated evidence may be indexed, graphed, or published.

Structural lint is necessary but insufficient. For valuable books/media, spot-check extracted evidence against original pages, frames, or timestamps.

Typed graph

Compile graphs only after canonical evidence exists:

canonical evidence
→ typed concepts/entities/claims
→ evidence-backed relationships
→ graph QA
→ multi-resolution projection
→ 2D and 3D viewers

Useful relations include PART_OF, DERIVED_FROM, AUTHORED_BY, MENTIONS, DEFINES, SUPPORTS, CONTRADICTS, CITES, SAME_AS, RELATED_TO, DEPICTS, VALIDATED_BY, and PUBLISHED_AS.

Edges that assert support or contradiction must carry exact evidence refs, confidence, source identity, extractor/model provenance, review status, and visibility scope.

Avoid graph hairballs. Default to global/collection/source levels and progressively expand neighborhoods. Provide filters, confidence thresholds, pathfinding, evidence drawers, and an accessible list fallback. Lazy-load heavy WebGL bundles.

Consumer publishers

Each consumer gets a projection, not ownership of canonical truth.

• Notebook/RAG: validated Markdown/text with source IDs and validation hashes; avoid lossy default transformations unless explicitly requested.
• Wiki: frontmatter, compact source citations, wikilinks, index/log updates.
• Quartz/Obsidian: dedicated content subtree, backlinks/search/2D graph.
• Agents: narrow retrieval/tool contracts that return evidence IDs and citations.
• 3D graph: typed, multi-resolution projection rather than every raw chunk.

Validate every client against the live API/OpenAPI. Do not guess endpoint prefixes or send undocumented fields. Distinguish request accepted, queued, running, completed, and failed; an assigned job/source ID is not proof of completed processing.

Existing-wiki import

For established wikis, orient to schema/index/source registry and import compiled sections such as concepts/, sources/, queries/, and summaries. Do not recursively crawl raw package/media trees.

• Enforce file-count and per-file byte limits.
• Build stable IDs from tenant plus logical relative identity, not absolute paths.
• Convert [[wikilinks]] into typed relationships.
• Convert compact src:<id> markers into source-identity nodes and CITES edges.
• Keep full artifact paths in registries/manifests, outside graph/API presentation.

Required verification

Before calling a platform operational:

1. Unit, contract, static, and security checks.
2. Exact production frontend build and dependency audit.
3. Exact production ASGI/server target, not only test-client construction.
4. One reviewed real PDF and one reviewed real EPUB through intake → extraction → QA → graph → publication.
5. One malformed/unsupported fixture proving fail-closed review.
6. A bounded import of an existing compiled wiki with node/edge counts.
7. A publisher request against the live consumer API followed by terminal-state/result verification.
8. Final process reload, readiness loop, localhost API smoke, and public/auth boundary checks.

Report queued/accepted, completed, needs-review, and blocked states separately.

Pitfalls

• Do not put all ingestion intelligence inside OpenNotebook or another RAG UI.
• Do not treat embeddings/chunks as canonical knowledge.
• Do not publish metadata-only extraction as if the document text was extracted.
• Do not crawl giant raw/ trees when compiled pages and source registries exist.
• Do not run Dense Summary or comparable whole-book transformations by default.
• Do not claim a queued consumer job completed without polling and checking the resulting source/index.
• Do not expose every node/edge at once in 3D.
• Do not stop after writing adapters; wire them into the real pipeline and prove it with route-specific tests.

Reference

See references/source-faithful-foundry-contracts.md for concrete publisher contracts, smoke-test expectations, and implementation lessons from a production multimodal Foundry build.