Karpathy's autoresearch project: AI agents run overnight LLM training experiments

What Karpathy's autoresearch project does

Andrej Karpathy released a tiny repository called "autoresearch" that demonstrates an "AI researcher in a loop" concept. The system uses an AI agent to autonomously run LLM training experiments overnight on a single GPU.

How it works

The agent follows this workflow:

• Continuously edits the train.py file
• Runs 5-minute nanochat training experiments
• Checks whether the validation bits-per-byte (val_bpb) metric improved
• Repeats this cycle while you sleep

Setup and configuration

The project has a super minimal setup:

• Hardware: One GPU
• Files: One main file
• Metrics: One primary metric (val_bpb)

The human writes the research organization prompt in program.md, and the agent handles the code iteration.

Experiment throughput

With a fixed 5-minute budget per experiment, the system can run approximately 12 experiments per hour.

This approach demonstrates a practical implementation of automated research where AI agents can explore parameter spaces and training configurations autonomously, potentially accelerating experimentation cycles for developers working with language models.