Cull: Open-Source Dataset Curation Engine for AI Image Pipelines
Cull is a machine curation engine for AI image datasets, built and maintained by u/Compunerd3. It automates the entire pipeline: scraping, classifying, captioning, and sorting — outputting a folder of triaged images with SD prompts ready for LoRA or finetune training.
End-to-End Pipeline
- Scraping: Supports Civitai (.com and .red), X/Twitter, Reddit, Discord, and any URL gallery-dl supports — Pixiv, DeviantArt, booru family, ArtStation, Tumblr, FurAffinity/e621, Imgur, Flickr, and ~340 others.
- Queue: Each image + source-side prompt dropped into a local queue. Per-source dedup, no database.
- Classification: Uses a vision-language model via multiple LM Studio instances (local) or Groq (cloud) — any OpenAI-compatible endpoint. Strict 17-field JSON schema ensures structured output.
- Sorting: Keepers go into category folders with a .txt prompt and a .vision.json audit record. Two score gates (quality + topic relevance) tunable in the UI.
- Dashboard: Flask + Alpine.js UI with start/stop, source toggles, gallery, prompt editor, ZIP export, and per-source stats.
Use Cases
The author used Cull for a 300-image LoRA and a 100,000-image finetune dataset. Set topic (e.g., "Female Influencer" or {artist} style art), toggle AUTO_CAPTION_ENABLED, walk away. For prompt-less archives, point LOCAL_IMPORT_DIR at a folder of JPEGs, toggle off prompt requirement, and turn on auto-captioning — each image gets an SD prompt, booru tags, or natural-language caption.
Technical Details
- Vision worker pluggable: Subclass
BaseVisionWorker, register. Two LM Studio endpoints run in parallel; keepalive worker pings every 15s to avoid idle-unload; optional idle-unloader to free VRAM. - AI assistant integration: Ships with Claude Code skill bundle in
.claude/skills/(cull-helper, lmstudio-vision, metadata-schema) and three sub-agents — works with Claude Code, Cursor, Aider, Codex. - Self-updater: Toast in dashboard, click Update, pulls from origin/main and relaunches.
- Stack: Python 3.10+, Flask, Alpine.js, Pillow, Playwright (X scraper), gallery-dl. Single machine, no Redis, no DB, no Docker.
- License: MIT.
Roadmap
Planned: more vision-worker backends, improved requeue UI, small headless CLI, video scraping and classification.
Repo: https://github.com/tlennon-ie/cull | Screenshots: https://imgur.com/a/kSvsAW9
📖 Read the full source: r/LocalLLaMA
👀 See Also
New Tool Injects Instructions into Claude Code Based on Context Usage
A developer created a tool that monitors context usage and injects custom instructions to prevent mid-task compaction disasters.
Developer Builds LibraHQ App to Solve AI Agent Memory Problem
A developer created LibraHQ, a free notes app that serves as a shared memory layer between chatbots and coding agents. The app records important notes and decisions from chats and stores them for future sessions, addressing the problem of AI agents forgetting previously made decisions.
Qhatu: Platform Turns GitHub Repos into Pay-Per-Use Micro SaaS with Claude
Qhatu is a platform that takes a GitHub repository and deploys it as a pay-per-use micro SaaS with a generated frontend and integrated payment processing. The system uses Anthropic APIs to analyze code, generate Dockerfiles, and create storefront UIs.
Qwen 3.6 27B Quantization Benchmark: Q4_K_M Beats Q8_0 on Practical Tradeoffs
Evaluated Qwen 3.6 27B across BF16, Q4_K_M, and Q8_0 GGUF quants on HumanEval, HellaSwag, and BFCL. Q4_K_M delivers near-BF16 scores with 48% less RAM, 1.45x speed, and 68.8% smaller file size.