WebClaw: Open-Source MCP Server for Web Extraction with Claude
WebClaw is an MCP server built in Rust that adds web extraction capabilities to Claude Desktop and Claude Code. It addresses the problem where Claude's built-in web_fetch gets blocked on most real websites, returning 403 Forbidden errors, Cloudflare challenges, or empty responses.
Technical Solution
The server uses TLS fingerprinting at the HTTP layer so websites see a real Chrome browser fingerprint instead of a bot. In testing against 10 popular sites, Claude's built-in web_fetch failed on all 10, while WebClaw successfully extracted content from 9 out of 10 sites.
Features
scrape: Extract clean content from any URLcrawl: Recursive site crawlingextract: Structured data extraction using JSON schema or natural language promptssummarize: Page summariesbrand: Extract colors, fonts, logos from any sitediff: Track content changesmap,batch,search,researchtools
Claude Code Development
The extraction pipeline was implemented with Claude Code, including:
- Scoring algorithm based on text density, semantic tags, and link ratio penalties
- Noise filter that strips navigation, ads, and cookie banners without false positives on Tailwind classes
- Multiple rounds of refinement for edge cases
Setup and Usage
Setup requires one command:
npx create-webclawThe tool detects Claude Desktop and Claude Code automatically and writes the configuration. No API key is needed for 8 of the 10 tools, and everything runs locally.
Performance Benefits
The output is optimized for Claude's context window. A typical news article goes from 4,820 tokens (raw HTML) to 1,590 tokens in WebClaw's LLM format - a 67% reduction while maintaining the same content.
WebClaw is free and open source under the MIT license, available at https://github.com/0xMassi/webclaw.
📖 Read the full source: r/ClaudeAI
👀 See Also
A 7-File Governance Layer to Prevent LLM Session Drift
A developer created a governance layer with seven files to prevent Claude from silently undoing architectural decisions across sessions. The system includes active_context.md, contracts.md, and decisions.md files with a strict execution loop.
RelayPlane Open Source Proxy Shows 73% Cost Reduction with Claude Model Routing
RelayPlane, an open source npm-native proxy for the Anthropic API, demonstrated 73.4% cost savings in benchmarks by routing requests to appropriate Claude models based on complexity. The tool reduced cost per 10 requests from $0.0323 to $0.0086 while improving p50 latency from 1.55s to 0.78s.
The Human Creativity Benchmark: Separating Convergence from Divergence in AI Creative Evaluation
Contra Labs introduces the Human Creativity Benchmark (HCB), a framework that distinguishes objectively verifiable criteria (e.g., prompt adherence) from subjective taste (e.g., visual appeal) in evaluating generative AI for creative work. The benchmark reveals that no current model is reliably both correct and steerable, addressing mode collapse and the need for differentiated output.
Canary: AI QA Agent for Automated Testing Based on Code Changes
Canary is an AI QA agent that reads codebases, analyzes pull request diffs, and generates end-to-end tests for affected user workflows. It connects to preview environments, runs tests, and comments results directly on PRs with recordings.