Mesh LLM: Distributed AI Computing on Iroh – Run LLMs Across Your Own GPUs

Mesh LLM is a distributed AI compute platform that pools GPUs and memory across multiple machines and exposes the whole thing as one OpenAI-compatible API at http://localhost:9337/v1. You can start one node, add more later, and let the mesh decide whether a model runs locally, routes to a peer that already has it loaded, or splits across several machines.

How It Works

Under the hood, Mesh LLM uses iroh endpoints for identity (public key) and networking. There is no central server. iroh handles NAT traversal, hole-punching, and relay fallback via QUIC connections. The protocol defines three ALPNs:

• mesh-llm/1 – main mesh (gossip, routing, HTTP tunnels, plugin channels)
• mesh-llm-control/1 – owner control plane (config sync, ownership attestation)
• skippy-stage/2 – latency-sensitive activation transport for split models

Inside the main connection, all communication is multiplexed over bidirectional QUIC streams tagged by a single leading byte:

• 0x01 GOSSIP – peer announcements (models, GPU, RTT, capabilities)
• 0x04 TUNNEL_HTTP – inference requests proxied to a peer
• 0x05 ROUTE_REQUEST – query which models a peer hosts
• 0x06 PEER_DOWN – dead-peer notification
• 0x07 PEER_LEAVING – graceful shutdown
• 0x08 PLUGIN_CHANNEL – plugin RPC
• 0x0e DIRECT_PATH_REQUEST – share direct addresses for NAT traversal

Split Mode ("Skippy")

For models too big for any single GPU (e.g., 235B mixture-of-experts), Mesh LLM has a split mode that partitions a model by layer ranges into stages. Layers 0–15 run on one node, 16–31 on the next, and so on. Activations flow from one stage to the next via QUIC streams. Several modest machines can together run a model none could hold alone.

Pluggable Architecture

Plugins declare their capabilities in a manifest. The runtime starts them, routes calls, and exposes their capabilities over MCP, HTTP, inference, and mesh events. The catalog ships with 40+ models — from half-a-billion-parameter models that fit on a laptop to the 235B giants.

Who It's For

Teams that want to control their own AI infrastructure: share GPU compute privately or publicly, run bigger models without buying bigger hardware, and avoid vendor lock-in. Any OpenAI client can point at localhost:9337 and stop caring where the work actually happens.