Libretto: Deterministic Browser Automation Generation for AI Coding Agents

Libretto is a toolkit for building robust web integrations that gives your coding agent a live browser and a token-efficient CLI. It shifts from runtime AI agents that operate as black boxes to "development-time AI" where scripts are generated ahead of time as actual code you can read, modify, version, and debug.

Key Features and Approach

Libretto takes a hybrid approach combining Playwright UI automation with direct network/API requests within the browser session for better reliability and bot detection evasion. This addresses limitations of runtime AI tools that rely on custom DOM parsing, which can be unreliable on older or complicated websites like healthcare systems.

The tool was built after a year of maintaining browser automations for EHR and payer portal integrations at a healthcare startup, where debugging failed automations was time-consuming. It addresses four main issues with runtime AI tools:

• Reliance on custom DOM parsing that's unreliable on complex websites
• High costs due to many AI calls and inability to cache actions reliably
• Lack of interpretability at runtime
• Limited help with generating new automations or debugging failures

Practical Usage

Installation is via npm: npm install libretto. First-time setup requires: npx libretto setup which installs the skill, downloads Chromium, and pins the default snapshot model. You can check workspace readiness with npx libretto status and manually change the snapshot analysis model with npx libretto ai configure <openai | anthropic | gemini | vertex>.

Libretto is designed to be used as a skill through your coding agent with example prompts including:

• One-shot script generation: "Use the Libretto skill. Go on LinkedIn and scrape the first 10 posts for content, who posted it, the number of reactions, the first 25 comments, and the first 25 reposts."
• Interactive script building: "I'm gonna show you a workflow in the eclinicalworks EHR to get a patient's primary insurance ID. Use libretto skill to turn it into a playwright script that takes patient name and dob as input to get back the insurance ID."
• Convert browser automation to network requests: "We have a browser script at ./integration.ts that automates going to Hacker News and getting the first 10 posts. Convert it to direct network scripts instead."
• Fix broken integrations: "We have a browser script at ./integration.ts that is supposed to go to Availity and perform an eligibility check for a patient. But I'm getting a broken selector error when I run it. Fix it."

The CLI can also be used directly with commands like npx libretto open <url> to launch sessions. All commands accept --session <name> to target specific sessions.

Additional features include recording manual user actions to help agents generate and update scripts, step-through debugging, optional read-only mode to prevent agents from accidentally submitting or modifying data, and generating code that follows existing abstractions and conventions in your coding repository.