---
name: hermes-web-providers
description: "Configure and extend Hermes' web_search / web_extract backend providers. Use when changing which provider Hermes uses for web tools, splitting search vs extract across providers, adding a new provider (Jina Reader, Crawl4AI, Playwright-direct), or designing a tiered/fallback chain that doesn't exist natively."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [Hermes-Infra, Web-Tools, Plugins, Firecrawl, Jina, Crawl4AI, SearXNG, Tavily, Exa]
    related_skills: [terminal-docs-research-fallback, vps-app-deployment]
---

# Hermes Web Providers — Configuration & Extension

The `web_search` and `web_extract` tools in Hermes are backed by **pluggable provider modules** under `~/.hermes/hermes-agent/plugins/web/<provider>/`. This skill covers the architecture, the config surface, and how to add a new provider when none of the built-ins fit (Jina Reader, Crawl4AI, Playwright-direct, etc.).

## Architecture at a glance

- **Tools:** `tools/web_tools.py` exposes `web_search` and `web_extract` to the agent. It does NOT contain provider logic — it just dispatches.
- **Providers:** each is a self-contained package under `plugins/web/<name>/` (e.g. `plugins/web/firecrawl/provider.py`). Registered via the web-provider registry at `agent/web_search_registry.py` / `agent/web_search_provider.py`.
- **Built-in providers shipped with Hermes:**
  - `firecrawl` — paid SaaS, free tier ~500 pages/mo. Best DX. Recursive crawl supported.
  - `searxng` — free, self-hosted or public instance. **Search-only**, no extract.
  - `brave-free` — Brave Search free tier. Search-only.
  - `ddgs` — DuckDuckGo via `ddgs` Python lib. No API key. Search-only.
  - `tavily` — paid SaaS. Search + extract. Crawl supported.
  - `exa` — paid SaaS. Search + extract.
  - `parallel` — paid SaaS. Search + extract.
- **NOT built-in** (as of this writing): Jina Reader, Crawl4AI, Playwright-direct. Adding any of these requires a plugin.

## Config surface (`~/.hermes/config.yaml`)

```yaml
web:
  backend: firecrawl        # shared fallback when per-capability keys empty
  search_backend: ''        # overrides backend for web_search
  extract_backend: ''       # overrides backend for web_extract
```

**Resolution order:**
1. `web.search_backend` / `web.extract_backend` (explicit per-capability)
2. `web.backend` (shared fallback)
3. Auto-detect from which API key/URL is present in `.env`

**There is NO native tiered fallback chain.** Hermes selects ONE backend per capability per call. If that backend fails, the tool fails. Crucially: do not promise the user a "Jina → Crawl4AI → Firecrawl chain" via config alone — that requires a custom plugin (see "Building a tiered provider" below).

## Common per-capability splits

```yaml
# Free search, paid extract for hard pages
web:
  search_backend: ddgs       # or searxng (self-hosted)
  extract_backend: firecrawl

# Free everything (search-only — extract will fail)
web:
  search_backend: searxng
  extract_backend: ''        # ERROR: searxng can't extract

# Cheapest mixed
web:
  search_backend: ddgs
  extract_backend: tavily    # or exa
```

`hermes setup` and `hermes tools` → "Web Search & Extract" both have wizards for this. CLI: `hermes config set web.search_backend ddgs`.

## Auxiliary web_extract model (different concern, easy to conflate)

`web_extract` runs an **auxiliary LLM call** to summarize long pages before returning. By default this uses the main chat model — on Opus/Sonnet that's expensive. Configure separately:

```yaml
auxiliary:
  web_extract:
    provider: openrouter
    model: google/gemini-3-flash-preview
    timeout: 360
```

This is orthogonal to the `web.backend` (which controls the fetcher, not the summarizer). Pointing this at a cheap model is often a bigger cost win than swapping fetchers.

## Building a tiered/fallback provider plugin

When the user wants "try X, fall back to Y, fall back to Z" (the typical economical stack: Jina Reader → Crawl4AI → Firecrawl → Playwright-direct), you write a single plugin that wraps the chain.

Layout:
```
~/.hermes/hermes-agent/plugins/web/tiered/
├── __init__.py
└── provider.py           # implements the provider contract + chain logic
```

The provider contract is defined in `agent/web_search_provider.py` (read it before coding). Mirror the shape of an existing provider — `plugins/web/firecrawl/provider.py` is the richest reference. Key surface:
- `search(query, limit) -> list[dict]`
- `extract(urls, mode) -> list[dict]`
- Capability flags (search vs extract vs crawl)

Inside the chain implementation:
1. Try the cheapest first (Jina: `GET https://r.jina.ai/<url>`, no key needed for low volume).
2. On HTTP non-2xx, content-too-small, or known JS-render failure markers, fall through.
3. Crawl4AI tier (only if self-hosted on the VPS) — point at the local service.
4. Firecrawl tier — uses the existing `FIRECRAWL_API_KEY`, preserves Alex's free tier as last-resort.
5. Final tier: the built-in `browser_tool` is the nuclear option for JS-heavy holdouts.

Register the new provider name in `agent/web_search_registry.py` (or whichever the current registration entry point is — read `web_tools.py` imports first; the layout has migrated before per PR #25182's comment in that file).

## Adding Jina Reader as a standalone provider (no chain)

Lighter-weight than a tiered chain. Plain `requests.get(f"https://r.jina.ai/{url}")` returns clean markdown for most pages. Wrap it as a single provider, register it, set `web.extract_backend: jina`. Pair with `searxng` or `ddgs` for free search. ~80% of Firecrawl's value for $0.

## Pitfalls

- **`web.backend: firecrawl` is the default after install** — don't assume the user picked it deliberately. Confirm before changing.
- **The `FIRECRAWL_API_KEY` env var also gates auto-detection.** If you remove the key but leave `web.backend: firecrawl`, you get hard failures, not auto-fallback to another configured provider.
- **SearXNG is search-only.** Setting it as `web.backend` (not just `search_backend`) breaks `web_extract` with a "search-only backend" error. Always pair with an extract provider.
- **Plugin paths have migrated.** PR #25182 moved firecrawl from `tools/web_tools.py` into `plugins/web/firecrawl/provider.py`. Before writing new code, `read_file` an existing provider to confirm current shape — don't trust older docs/blog posts.
- **`web_extract` ≠ web fetcher alone.** It also runs an auxiliary LLM summary; check `auxiliary.web_extract` before blaming the fetcher for slow/expensive extracts.
- **Don't promise tiered fallback via config.** Hermes' `web.backend` is single-select. Tiered chains require code.

## When the user asks "should I keep paying Firecrawl?"

The honest answer, captured here so future sessions don't have to re-derive it:

- **Free tier (Firecrawl ~500 pages/mo) + Jina Reader fallback** covers most personal/agent workloads at $0 marginal cost.
- **Crawl4AI self-hosted** on the VPS is the next cheapest leap — but it's another service to maintain. Worth it only at real volume.
- **Browser tool (Playwright)** is already in Hermes and is the final fallback for JS-heavy pages no markdown fetcher can handle. No need to add Playwright as a separate web provider — it already exists as `browser_*` tools.

The strongest budget stack: `search_backend: ddgs` (or self-hosted SearXNG), custom `jina` provider as `extract_backend` with Firecrawl free tier as the in-plugin fallback inside that provider, and `browser_*` for the rare holdouts.

## Pricing-format preference

When quoting infra costs (VPS, SaaS tiers) to Alex, **lead with monthly figures**, not
annualized. He explicitly prefers it that way and will correct you if you do
"$X/yr". Annual is fine as a secondary comparison line ("…or ~$X/yr for reference")
but never the headline number. Hetzner specifically bills hourly/monthly with no
annual prepay — mention that to defuse "am I committing for a year?" worries.

## References

- `references/competitor-research.md` — distilled research dump from a Nov 2026 conversation comparing Firecrawl with Crawl4AI, Jina Reader, Apify, Browserbase, Bright Data, Crawlee, Playwright. Use when the user is re-evaluating the web-fetcher stack.
- `references/crawl4ai-self-hosting-cost.md` — RAM/CPU footprint, VPS sizing sweet spots, co-locate-vs-dedicated decision rule, current monthly prices across Hetzner / Fly / DO / Vultr / Contabo / Oracle Free, and a deployment outline. Load this when sizing the Crawl4AI tier or deciding whether to co-locate.