mistral.rs now supports Gemma 4 12B with multimodal, agentic, and Multi-Turn Prediction (MTP) features. This release includes web search and sandboxed code execution for building agentic apps, plus audio, image, and video input.

Installation

Single-line install for Linux/macOS and Windows:

# Linux/macOS
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh

Windows
irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex

Running with Agent & Quantization

Launch an OpenAI- and Anthropic-compatible HTTP server with a built-in web UI at localhost:1234/ui:

mistralrs run --agent -m google/gemma-4-12B-it --quant 4

Enabling MTP (Multi-Turn Prediction)

To use MTP, add the --mtp-model flag with the assistant model:

mistralrs run --agent -m google/gemma-4-12B-it --quant 4 --mtp-model google/gemma-4-12B-it-assistant

Key Features

• Full multimodal support: audio, image, and video
• Web search and sandboxed code execution for agentic workflows
• OpenAI and Anthropic-compatible HTTP server
• Built-in web chat UI at localhost:1234/ui

For more details: GitHub | Documentation