AgentCrawl

What AgentCrawl does

AgentCrawl turns web pages and local documents into Markdown, text, links, metadata, JSON-LD and crawl results, so an agent does not need raw HTML pasted into chat or every URL routed through a hosted scraper. The default MCP tools are scrape_url, scrape_many, map_site, crawl_site and extract_structured, plus search_web when a search engine is configured, and setting AGENTCRAWL_MCP_PROFILE=full adds operator tools such as job history, retries and cache.

The MCP server fetches only URLs by default and refuses local file paths, since an agent can be steered by the pages it reads. Pages that need JavaScript use an optional browser extra built on Playwright, and a stealth extra retries refused pages with Patchright. A page that still demands a CAPTCHA is reported as error_type client_challenge rather than returned as content. An agent skill added with npx skills add JorG18/agentcrawl teaches Claude Code, Codex, Cursor, OpenClaw and other SKILL.md clients when and how to use it.

Key features

• Markdown, text, links, metadata and JSON-LD output
• Four interfaces: CLI, Python library, Docker API and MCP server
• MCP tools for scraping, batch scraping, site mapping, crawling and structured extraction
• Optional browser extra for JavaScript-rendered pages
• Explicit failure types such as client_challenge instead of fake content
• Local file paths refused over MCP unless enabled

When to use it

• Letting a Hermes agent read documentation sites as clean Markdown
• Mapping and crawling a site to gather research material
• Running a scraping API in Docker instead of sending URLs to a hosted service
• Reading a local docs folder when file access is explicitly enabled

Who it is for: Developers who want their agents to read the web through a self-hosted tool they control.

How it fits with Hermes Agent

AgentCrawl is a general MCP server, CLI and skill, and the repository is tagged hermes-agent. Hermes Agent can use it as an MCP tool source.

How to install AgentCrawl

These commands are copied from the project's README. Check the repository for the latest steps before you run them.

pip install agentcrawl-ai
python -m pip install "agentcrawl-ai[mcp]"
agentcrawl doctor
agentcrawl mcp

Note: The README describes the project as early and intentionally modest.

FAQ