Local semantic search for AI conversations with fastembed and LanceDB
A developer has implemented a local semantic search system for AI conversation history, processing 368K messages without cloud dependencies or API keys. The project uses fastembed with the BAAI/bge-small-en-v1.5 model for CPU-based embeddings and LanceDB as a vector store that operates as a single directory without a server process.
Technical Stack
- Embeddings: fastembed with BAAI/bge-small-en-v1.5 model (384 dimensions)
- Vector store: LanceDB - single directory, no server process, append-friendly
- Ingest: Pulls from JSONL session transcripts (Claude Code, any chat export)
- Embedding performance: ~500 docs/sec on M4 CPU
Key Implementation Details
The developer learned several practical lessons during the 4-month iteration:
- Selective embedding: Early versions embedded every message, which reduced signal-to-noise. The current implementation only embeds user messages and assistant messages with substance (skipping responses like "sure, here's that code"), cutting vector count by 60% while improving search quality.
- Chunking strategy: Switching from fixed-size chunks to conversation-turn chunks made a massive difference in retrieval relevance. Model choice (tried nomic-embed-text, bge-large, all-MiniLM) showed marginal differences compared to chunking approach.
- LanceDB advantages: The developer found LanceDB "stupidly underrated for personal-scale" - no server, no Docker, just a directory with instant appending of new vectors, replacing an overengineered pgvector setup.
- Re-embedding workflow: The bge-small-en-v1.5 model at 384 dimensions is fast enough to re-embed hourly as a cron job. A full re-index of 117K vectors takes approximately 4 minutes on M2 hardware.
Performance Metrics
- Total messages ingested: 407K
- Vectors indexed: 87K
- Search latency (p50): 12ms across 117K vectors
- Full re-index time: ~4 minutes (M2)
- Storage: ~180MB on disk
- API keys needed: 0
The project is open source under MIT license and available at github.com/mordechaipotash/brain-mcp. Installation is via pipx install brain-mcp && brain-mcp setup.
📖 Read the full source: r/LocalLLaMA
👀 See Also
free-claude-code adds GLM-5 support via NVIDIA NIM, expands to OpenRouter and Discord
free-claude-code now supports GLM-5 through NVIDIA NIM's free tier (40 requests/min) and adds OpenRouter integration, Discord bot support, and LMStudio local provider compatibility. The tool converts Claude Code's Anthropic API requests to work with alternative model backends.
Sylve: A FreeBSD Management Plane for Virtualization, Containers, and Storage
Sylve is a BSD-2 licensed management plane for FreeBSD that provides unified control over Bhyve VMs, FreeBSD Jails, ZFS storage, and networking. It uses a RAFT consensus model for clustering and includes Samba share management with ZFS snapshot automation.
Knowledge Raven: A Model-Agnostic Knowledge Base Platform Built with Claude Code
Knowledge Raven is a knowledge base platform that lets any MCP-compatible LLM search and cite company documents. The entire platform was built with Claude Code by a solo founder, featuring Python/FastAPI backend, MCP tool layer, and agentic RAG pipeline.
Introducing cltree: A File Tree TUI for Claude Code
cltree is a split-pane TUI that displays your project file tree in real-time alongside Claude Code, showing the current working directory, hiding noise, and allowing all keystrokes to pass through.