AgentCrawl Update Adds Critical Crawler Features and Enhancements
The latest update to AgentCrawl enhances its functionality as a TypeScript scraper/crawler, introducing several important features for developers using AI agents. This release focuses on production-readiness by integrating crawler correctness and politeness, caching mechanisms, resumable crawls, and enhanced data extraction capabilities.
Key Details
- Removed Tool Adapters: The update eliminates the tool adapters for the agents SDK and Vercel AI SDK, allowing users to define their tools independently.
- Updated Libraries: The package now includes the latest version of Zod for better data validation.
- Crawler Correctness: Robots.txt compliance is now opt-in and supports Disallow/Allow and Crawl-delay directives. Opt-in sitemap seeding from
/sitemap.xmlis also available. - URL Normalization: Improved URL normalization comprehensively strips tracking parameters and can handle canonical normalization.
- Throttling Options: The crawler supports per-host throttling with configurable
perHostConcurrencyandminDelayMs. - Caching: An opt-in disk HTTP cache for static fetches implements ETag and Last-Modified support. The system caches post-cleaning and markdown conversion of
ScrapedPageand can handle server responses with status 304 by serving cached bodies. - Resumable Crawls: A new opt-in crawlState persistence saves the crawl's frontier, including the queue, visited pages, queued items, errors, and max depth, which allows for resumable crawls without re-visiting pages.
- Data Extraction Improvements: The scraper now supports structured metadata extraction, including Canonical URL, OpenGraph, Twitter cards, and JSON-LD, kept in
metadata.structured. - Chunking for Agents: Opt-in chunking functionality returns
page.chunks[]with an approximate token size, heading path, and citation anchor, which is beneficial for RAG/tool loops.
Who It's For
This update is particularly beneficial for developers utilizing AI agents requiring efficient and structured web scraping capabilities.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Architor: Open-Source Tool for Phase-Gated Architecture Workflows with Claude Code
Architor is an open-source tool that structures Claude Code into a phase-gated architecture assistant with persistent design memory. It organizes system design into requirement evaluation, architecture decisions, component design, and validation phases, tracking decisions in an .arch workspace.
OpenClaw Shared Memory Plugin: SQLite-Based Multi-Agent Coordination
A developer built a plugin for OpenClaw multi-agent setups that enables agents to share memory using SQLite, eliminating the need for external services. The plugin allows explicit memory sharing via a tool, automatic context extraction, access control, entity tracking, and contradiction detection.
Claude Prototypes Real Estate Analysis App in 3 Hours Using Live Zillow Data via clawhub
A developer used Claude with the zillow-full clawhub tool to build a rental cash flow analysis app — pulling live Zillow API data, prototyping the UI around real JSON responses, and delivering a working prototype in one afternoon.
Khael AI Agent Shares Production Architecture Decisions for OpenClaw
Khael, an AI autonomous agent running on OpenClaw, details specific architectural decisions that have worked in production for months, including separate LAWS.md files, mode files, self-audit cron jobs, and specialized bot types.