Overview
jevcrawl is a web crawler for AI agents, available as both an MCP server and a command-line tool. Instead of passing whole pages to an agent, it uses a small decision model to choose links and score passages, helping the agent focus on material relevant to its question.
Extracted passages remain verbatim and include their source URL, heading path, and character offsets. The decision model scores page text rather than generating replacement text, and page content is never treated as an instruction.
Key Features
- Crawls toward pages likely to contain the answer rather than collecting every available page.
- Processes four pages at a time with one decision request per page.
- Returns focused passages with provenance information for later verification.
- Supports plain HTML and can optionally render pages detected as JavaScript shells.
- Works with MCP-compatible clients as well as a standalone CLI.
The published benchmark used the same 30-page budget, answering model, and judge for each crawler. On those tasks, jevcrawl found the answer page in 54% of cases, compared with 33% when Claude selected links, and the project reports roughly twice the speed of an LLM-driven crawl.
Setup and Use
jevcrawl requires a Jev API key and the uv Python package tool. MCP users configure the server through their client, while command-line users can run it with uvx and verify the setup with the included doctor command.
JavaScript rendering is optional. Installing the JavaScript extra and Chromium enables browser rendering only for pages identified as JavaScript shells, with no more than five such pages opened in one call; other pages continue to use plain HTML fetching.
Intended Uses
The tool is designed for agents that need to answer a specific question by navigating a website. It can be used from Claude Code, Claude Desktop, Cursor, other MCP clients, or directly from the command line, making the same focused crawling approach available across agent and terminal workflows.
