Training GPT-1 on an RTX 2060 Super 8GB – Proof of Concept for Local Pretraining

A Reddit user (u/tevlon) successfully trained the original GPT-1 model (117M parameters) on an RTX 2060 Super with 8GB VRAM in about 1 hour. The project demonstrates that even budget gaming GPUs can pre-train sub-1B models, opening the door for hobbyists without cloud credits.

The author notes that while GPT-2 is too large for local training on such hardware, GPT-1 is 10x smaller and achievable. The key takeaway: every gamer with an RTX 2060 or similar can now pre-train a 1B-sized model—and models under 1B (e.g., Surya OCR from Datalab.to) are not garbage; they produce useful specialized models.

The code incorporated algorithmic improvements from the modded-nanogpt project while staying faithful to OpenAI’s original finetune-transformer-lm implementation. The codebase was written mostly by Claude 4.8 and Codex 5.5, guided by the user’s instructions.

Links:

• Code repository: github.com/epoyraz/modded-gpt-1
• Model weights: huggingface.co/epoyraz/modded-gpt-1

The author is not an ML researcher but a software engineer and gamer. The model is acknowledged as undertrained (1 hour only), but the principle stands: local pretraining is now accessible to anyone with a decent GPU.

This is a proof-of-concept that challenges the assumption that large-scale pretraining requires cloud clusters. If you have a gaming GPU with 8GB+ VRAM, you can experiment with pretraining—start with GPT-1 and scale up.