← all skills

tiered-web-extract

/home/avalon/.hermes/skills/devops/tiered-web-extract/SKILL.md · raw

Tiered Web Extract (Jina → Crawl4AI → Firecrawl)

Alex's cost-first web extract/search backend. Replaces direct Firecrawl usage. Saves the Firecrawl free tier for hardest-to-render pages only.

Architecture

Extract chain (per URL, falls through on miss): 1. r.jina.ai/<url> — free, no key, static pages, ~85% of cases 2. Self-hosted Crawl4AI on http://5.78.214.66:11235/md — JS-heavy pages 3. Firecrawl SDK — last-resort paid fallback (uses existing FIRECRAWL_API_KEY)

Search chain: 1. ddgs (DuckDuckGo) — free 2. Firecrawl search — fallback

Files

Verify it's working

# Live Crawl4AI health
curl -s --max-time 5 http://5.78.214.66:11235/health
# Expect: {"status":"ok","timestamp":...,"version":"0.8.6"}

# Live crawl test
curl -s -X POST http://5.78.214.66:11235/md \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

# Plugin loads & registers
cd /home/avalon/.hermes/hermes-agent
/home/avalon/.hermes/hermes-agent/venv/bin/python -c "
from hermes_cli.plugins import PluginManager
pm = PluginManager(); pm.discover_and_load()
from agent.web_search_registry import get_active_search_provider, get_active_extract_provider
print('search:', get_active_search_provider().name)   # tiered
print('extract:', get_active_extract_provider().name) # tiered
"

Pitfalls discovered during setup

Pitfall: gateway/MCP process predates the plugin

Symptom (the exact shape this bug takes): the agent's web_search MCP tool keeps returning Firecrawl search failed: Payment Required: Insufficient credits even though web.backend: tiered is in ~/.hermes/config.yaml and the plugin files exist on disk. Direct Python tests inside hermes-agent/venv resolve tiered correctly (get_active_search_provider().name == 'tiered') — the discrepancy is the giveaway.

Cause: the long-running gateway process (the one serving Telegram / MCP for the active session) was started BEFORE the tiered plugin was installed. Plugins are loaded once at gateway startup. Editing config.yaml and dropping new plugin files into ~/.hermes/plugins/ does NOT hot-reload them into the running process.

Diagnosis recipe:

# What's the gateway process and when did it start?
ps -ef | grep "hermes_cli.main gateway run" | grep -v grep
ps -o lstart= -p <pid>

# Compare to plugin install / config edit time
stat -c '%y %n' ~/.hermes/plugins/web/tiered/*.py ~/.hermes/config.yaml

If the gateway lstart is earlier than the plugin/config mtime → stale gateway, restart needed.

Fix: restart the gateway. Whatever launches it (pm2 / systemd / hand-launched) — kill the PID and let it respawn, or pm2 restart hermes-gateway. Brief Telegram drop (a few seconds), then the new chain is wired in.

Generalize: this same pattern bites any time config.yaml / .env / a plugin directory is edited while the gateway is running. Whenever Alex says "but we already configured X, why is it still doing the old thing", check gateway start time vs config/plugin mtime BEFORE re-debugging the config.

Maintaining Crawl4AI

SSH in via the main VPS (key already authorized):

ssh root@5.78.214.66
cd /root/crawl4ai
docker compose logs -f --tail 100
docker compose pull && docker compose up -d   # update
docker stats crawl4ai                          # mem/cpu

When to extend

Cost