← all skills

hermes-web-providers

/home/avalon/.hermes/skills/devops/hermes-web-providers/SKILL.md · raw

Hermes Web Providers — Configuration & Extension

The web_search and web_extract tools in Hermes are backed by pluggable provider modules under ~/.hermes/hermes-agent/plugins/web/<provider>/. This skill covers the architecture, the config surface, and how to add a new provider when none of the built-ins fit (Jina Reader, Crawl4AI, Playwright-direct, etc.).

Architecture at a glance

Config surface (~/.hermes/config.yaml)

web:
  backend: firecrawl        # shared fallback when per-capability keys empty
  search_backend: ''        # overrides backend for web_search
  extract_backend: ''       # overrides backend for web_extract

Resolution order: 1. web.search_backend / web.extract_backend (explicit per-capability) 2. web.backend (shared fallback) 3. Auto-detect from which API key/URL is present in .env

There is NO native tiered fallback chain. Hermes selects ONE backend per capability per call. If that backend fails, the tool fails. Crucially: do not promise the user a "Jina → Crawl4AI → Firecrawl chain" via config alone — that requires a custom plugin (see "Building a tiered provider" below).

Common per-capability splits

# Free search, paid extract for hard pages
web:
  search_backend: ddgs       # or searxng (self-hosted)
  extract_backend: firecrawl

# Free everything (search-only — extract will fail)
web:
  search_backend: searxng
  extract_backend: ''        # ERROR: searxng can't extract

# Cheapest mixed
web:
  search_backend: ddgs
  extract_backend: tavily    # or exa

hermes setup and hermes tools → "Web Search & Extract" both have wizards for this. CLI: hermes config set web.search_backend ddgs.

Auxiliary web_extract model (different concern, easy to conflate)

web_extract runs an auxiliary LLM call to summarize long pages before returning. By default this uses the main chat model — on Opus/Sonnet that's expensive. Configure separately:

auxiliary:
  web_extract:
    provider: openrouter
    model: google/gemini-3-flash-preview
    timeout: 360

This is orthogonal to the web.backend (which controls the fetcher, not the summarizer). Pointing this at a cheap model is often a bigger cost win than swapping fetchers.

Building a tiered/fallback provider plugin

When the user wants "try X, fall back to Y, fall back to Z" (the typical economical stack: Jina Reader → Crawl4AI → Firecrawl → Playwright-direct), you write a single plugin that wraps the chain.

Layout:

~/.hermes/hermes-agent/plugins/web/tiered/
├── __init__.py
└── provider.py           # implements the provider contract + chain logic

The provider contract is defined in agent/web_search_provider.py (read it before coding). Mirror the shape of an existing provider — plugins/web/firecrawl/provider.py is the richest reference. Key surface: - search(query, limit) -> list[dict] - extract(urls, mode) -> list[dict] - Capability flags (search vs extract vs crawl)

Inside the chain implementation: 1. Try the cheapest first (Jina: GET https://r.jina.ai/<url>, no key needed for low volume). 2. On HTTP non-2xx, content-too-small, or known JS-render failure markers, fall through. 3. Crawl4AI tier (only if self-hosted on the VPS) — point at the local service. 4. Firecrawl tier — uses the existing FIRECRAWL_API_KEY, preserves Alex's free tier as last-resort. 5. Final tier: the built-in browser_tool is the nuclear option for JS-heavy holdouts.

Register the new provider name in agent/web_search_registry.py (or whichever the current registration entry point is — read web_tools.py imports first; the layout has migrated before per PR #25182's comment in that file).

Adding Jina Reader as a standalone provider (no chain)

Lighter-weight than a tiered chain. Plain requests.get(f"https://r.jina.ai/{url}") returns clean markdown for most pages. Wrap it as a single provider, register it, set web.extract_backend: jina. Pair with searxng or ddgs for free search. ~80% of Firecrawl's value for $0.

Pitfalls

When the user asks "should I keep paying Firecrawl?"

The honest answer, captured here so future sessions don't have to re-derive it:

The strongest budget stack: search_backend: ddgs (or self-hosted SearXNG), custom jina provider as extract_backend with Firecrawl free tier as the in-plugin fallback inside that provider, and browser_* for the rare holdouts.

Pricing-format preference

When quoting infra costs (VPS, SaaS tiers) to Alex, lead with monthly figures, not annualized. He explicitly prefers it that way and will correct you if you do "$X/yr". Annual is fine as a secondary comparison line ("…or ~$X/yr for reference") but never the headline number. Hetzner specifically bills hourly/monthly with no annual prepay — mention that to defuse "am I committing for a year?" worries.

References