CrawlEngine MCP: Resilient web extraction with token-bounded markdown for Claude Desktop & Cursor (Now on TensorBlock & PyPI)

[johannschiffer]

Problem: Raw Web Scraping Breaks LLM Context Windows

When autonomous coding agents in Claude Desktop, Cursor, or Windsurf inspect web documentation or remote APIs, standard browser tools fail in three predictable ways:

1. Raw HTML Bloat: A single 2 MB landing page dumps 40,000 tokens of nested <div> and <script> tags directly into the context window, degrading attention and causing compaction.
2. Anti-Bot Defenses: Naive curl or headless requests get trapped in Cloudflare/DataDome challenges, stalling tool execution.
3. Complex Local Toolchains: Requiring local Chromium browser binaries or heavyweight node dependencies causes runtime failures on containerized or minimal developer seats.

Solution: CrawlEngine MCP (Lite Edition)

Following our recent inclusion in TensorBlock/awesome-mcp-servers (PR #4450) and our official distribution on PyPI, we designed CrawlEngine MCP specifically to solve LLM web ingestion.

Key Capabilities:

• Token-Bounded Clean Markdown: Converts pages into semantic Markdown with structural stripping of headers, footers, tracking scripts, and styling noise.
• Resilient Fallback Headers: Automated header rotation and realistic browser user-agent emulation to bypass basic scrape protection.
• Zero Local Footprint: Distributed as a standalone Python MCP package that runs instantly with uvx without installing persistent daemons.

Setup (2 Lines)

Add to claude_desktop_config.json or .cursor/mcp.json:

{
  "mcpServers": {
    "crawlengine-mcp": {
      "command": "uvx",
      "args": ["crawlengine-mcp"]
    }
  }
}

Links & Resources

• GitHub Repository: johannschiffer/crawlengine-mcp
• PyPI Package: pypi.org/project/crawlengine-mcp
• TensorBlock Directory: TensorBlock/awesome-mcp-servers
• Visual Config Builder: Interactive MCP Config & Ruleset Builder

Feedback and pull requests from the community are warmly welcome.

[wilsonccccc]

How is the token-bounded output enforced in CrawlEngine MCP? Which tokenizer or model is the limit based on, and when extracted content exceeds the budget, does the response include explicit truncation metadata or a continuation mechanism so the agent knows the result is incomplete?