AgentCrawl Update Adds Critical Crawler Features and Enhancements
The latest update to AgentCrawl enhances its functionality as a TypeScript scraper/crawler, introducing several important features for developers using AI agents. This release focuses on production-readiness by integrating crawler correctness and politeness, caching mechanisms, resumable crawls, and enhanced data extraction capabilities.
Key Details
- Removed Tool Adapters: The update eliminates the tool adapters for the agents SDK and Vercel AI SDK, allowing users to define their tools independently.
- Updated Libraries: The package now includes the latest version of Zod for better data validation.
- Crawler Correctness: Robots.txt compliance is now opt-in and supports Disallow/Allow and Crawl-delay directives. Opt-in sitemap seeding from
/sitemap.xmlis also available. - URL Normalization: Improved URL normalization comprehensively strips tracking parameters and can handle canonical normalization.
- Throttling Options: The crawler supports per-host throttling with configurable
perHostConcurrencyandminDelayMs. - Caching: An opt-in disk HTTP cache for static fetches implements ETag and Last-Modified support. The system caches post-cleaning and markdown conversion of
ScrapedPageand can handle server responses with status 304 by serving cached bodies. - Resumable Crawls: A new opt-in crawlState persistence saves the crawl's frontier, including the queue, visited pages, queued items, errors, and max depth, which allows for resumable crawls without re-visiting pages.
- Data Extraction Improvements: The scraper now supports structured metadata extraction, including Canonical URL, OpenGraph, Twitter cards, and JSON-LD, kept in
metadata.structured. - Chunking for Agents: Opt-in chunking functionality returns
page.chunks[]with an approximate token size, heading path, and citation anchor, which is beneficial for RAG/tool loops.
Who It's For
This update is particularly beneficial for developers utilizing AI agents requiring efficient and structured web scraping capabilities.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Reddit user measures MCP token overhead: 67K tokens consumed before any question
A developer measured their MCP server token overhead at 67,000 tokens consumed before typing a single question, with Playwright MCP using 13,600 tokens and GitHub MCP using 18,000 tokens idle. They replaced MCP with skills and CLI tools for lower context costs.
SeeFlow: Architecture Diagrams That Are Wired to Your Live App
SeeFlow generates a flow canvas from your codebase and wires each node to your actual running app, with plugins for Claude Code, Codex, Cursor, Windsurf, and an MCP server.
Claudraband: Terminal Wrapper for Persistent Claude Code Sessions
Claudraband wraps the official Claude Code TUI in a controlled terminal to enable resumable workflows, remote session control via HTTP daemon, and ACP server integration for alternative frontends like Zed or Toad. It requires Node.js/Bun, authenticated Claude Code, and tmux for first-class local workflows.
Building a Coding Agent for 8k Context: Planner/Executor Split, Token Budgeting, and Parallel Execution
A detailed breakdown of building a CLI coding agent designed around 8k token limits, using a planner/executor architecture, strict token budgeting, and parallel task execution.