Understudy: A Teachable Desktop Agent That Learns Tasks by Demonstration
What Understudy Does
Understudy is a teachable desktop agent that operates your computer like a human colleague — handling GUI, browser, shell, file system, and messaging tools in one local runtime. The core innovation is teach-by-demonstration: you perform a task once, the agent records screen video plus semantic events, extracts the intent (not just coordinates), and turns it into a reusable skill.
Current Implementation Status
The system is designed as five layers, with current implementation status:
- Layer 1 (Operate Software Natively): Implemented today on macOS. Operates any macOS desktop app using 13 tools + screenshot grounding + native input.
- Layer 2 (Learn from Demonstrations): Implemented and usable today. User shows a task once — agent extracts intent, validates, learns.
- Layer 3 (Crystallized Memory): Partially implemented. Agent accumulates experience from daily use, hardens successful paths.
- Layer 4 (Route Optimization): Partially implemented. Automatically discover and upgrade to faster execution routes.
- Layer 5 (Proactive Autonomy): Still the long-term direction. Notice and act in its own workspace without disrupting the user.
Technical Capabilities
Understudy is a unified desktop runtime that mixes every execution route in one agent loop, one session, one policy pipeline:
- GUI: 13 tools + screenshot grounding + native input for any macOS desktop app
- Browser: Playwright managed + Chrome extension relay for any website with login sessions
- Shell: bash tool with full local access for CLI tools, scripts, file system
- Web: web_search + web_fetch for real-time information retrieval
- Memory: Semantic memory across sessions for persistent context and preferences
- Messaging: 8 channel support
How It Works in Practice
In the demo video, the creator teaches Understudy to: Google Image search → download a photo → remove background in Pixelmator Pro → export → send via Telegram. Then asks it to do the same for Elon Musk. The replay isn't a brittle macro — the published skill stores intent steps, route options, and GUI hints only as a fallback. It can prefer faster routes when available instead of repeating every GUI step.
Installation and Setup
Current platform: macOS only. Installation is via npm:
npm install -g @understudy-ai/understudy
understudy wizardThe published skill artifact from the showcase demo is available at examples/published-skills/taught-person-photo-cutout-bc88ec/SKILL.md for inspection.
Who It's For
Developers who work across multiple desktop applications and want to automate repetitive tasks without building custom integrations or workflow builders.
📖 Read the full source: HN AI Agents
👀 See Also
Chamber: AI Agent for GPU Infrastructure Management
Chamber is an AI agent that manages GPU infrastructure by handling tasks like provisioning clusters, diagnosing failed jobs, and managing workloads. It provides structured operations with validation and rollback, not just raw shell commands.
Found-Issues plugin logs bugs Claude ignores while working on other tasks
A Claude Code plugin that writes one-line entries to docs/found-issues.md when the agent spots bugs outside scope, with auto-closure on PR merge and tombstone detection.
Runtime: Sandboxed Coding Agents for Every Team Member
Runtime (YC P26) provides sandboxed coding agent infrastructure that lets non-engineers use Claude Code, Codex, and other agents safely. It snapshots multi-service environments (Docker, Kafka, Redis, seeded DBs) that boot in milliseconds, with guardrails at the infrastructure level.
Interfaze: New Model Architecture Beats Gemini-3-Flash and GPT-5.4-Mini on Deterministic Tasks
Interfaze, a new model architecture combining DNN/CNNs with transformers, outperforms Gemini-3-Flash, Claude-Sonnet-4.6, GPT-5.4-Mini, and Grok-4.3 across 9 benchmarks including OCR, vision, STT, and structured output.