Understudy: A Teachable Desktop Agent That Learns Tasks by Demonstration
What Understudy Does
Understudy is a teachable desktop agent that operates your computer like a human colleague — handling GUI, browser, shell, file system, and messaging tools in one local runtime. The core innovation is teach-by-demonstration: you perform a task once, the agent records screen video plus semantic events, extracts the intent (not just coordinates), and turns it into a reusable skill.
Current Implementation Status
The system is designed as five layers, with current implementation status:
- Layer 1 (Operate Software Natively): Implemented today on macOS. Operates any macOS desktop app using 13 tools + screenshot grounding + native input.
- Layer 2 (Learn from Demonstrations): Implemented and usable today. User shows a task once — agent extracts intent, validates, learns.
- Layer 3 (Crystallized Memory): Partially implemented. Agent accumulates experience from daily use, hardens successful paths.
- Layer 4 (Route Optimization): Partially implemented. Automatically discover and upgrade to faster execution routes.
- Layer 5 (Proactive Autonomy): Still the long-term direction. Notice and act in its own workspace without disrupting the user.
Technical Capabilities
Understudy is a unified desktop runtime that mixes every execution route in one agent loop, one session, one policy pipeline:
- GUI: 13 tools + screenshot grounding + native input for any macOS desktop app
- Browser: Playwright managed + Chrome extension relay for any website with login sessions
- Shell: bash tool with full local access for CLI tools, scripts, file system
- Web: web_search + web_fetch for real-time information retrieval
- Memory: Semantic memory across sessions for persistent context and preferences
- Messaging: 8 channel support
How It Works in Practice
In the demo video, the creator teaches Understudy to: Google Image search → download a photo → remove background in Pixelmator Pro → export → send via Telegram. Then asks it to do the same for Elon Musk. The replay isn't a brittle macro — the published skill stores intent steps, route options, and GUI hints only as a fallback. It can prefer faster routes when available instead of repeating every GUI step.
Installation and Setup
Current platform: macOS only. Installation is via npm:
npm install -g @understudy-ai/understudy
understudy wizardThe published skill artifact from the showcase demo is available at examples/published-skills/taught-person-photo-cutout-bc88ec/SKILL.md for inspection.
Who It's For
Developers who work across multiple desktop applications and want to automate repetitive tasks without building custom integrations or workflow builders.
📖 Read the full source: HN AI Agents
👀 See Also
ViralCanvas.ai provides persistent context workspace for Claude models including Sonnet 4.5
ViralCanvas.ai is a visual workspace that sits on top of Claude's models, offering access to Sonnet 4.5, Sonnet 4.6, Opus 4.5, and Opus 4.6 with persistent context attachment. The tool addresses context degradation issues in long conversations by keeping connected documents actively weighted on every prompt.
Prompt-Master: Claude Skill for Generating Accurate AI Tool Prompts
Prompt-Master is a free Claude skill that writes accurate prompts for various AI tools including Cursor, Claude Code, GPT, Midjourney, Kling, and Eleven Labs. The tool has reached 600+ stars on GitHub and processes 4000+ traffic.
Exploring macOS's sandbox-exec for Secure Application Execution
sandbox-exec is a macOS command-line utility that allows applications to run in a restricted environment. Learn how to utilize it with custom sandbox profiles.
Layerkit: AI Image Editor with Editable Layers Built with Claude Code
A developer built Layerkit, a browser-based AI image editor that generates scenes with editable layers to avoid constant re-prompting. The tool uses a multi-stage AI pipeline where one LLM plans composition, an image model generates the scene, and another LLM analyzes the actual image to place readable text.