Using Local LLMs with ADK-Rust: Ollama and mistral.rs

A guide on how to use local LLMs like Ollama and mistral.rs with ADK-Rust for privacy and cost-efficiency.

Posted on: 2026-03-23 by AI Assistant

Privacy and cost are two of the biggest concerns when building AI-powered applications. Local Large Language Models (LLMs) address both by running directly on your hardware. ADK-Rust fully supports this with built-in integrations for popular tools like Ollama and mistral.rs.

Why Choose Local LLMs?

Running LLMs locally offers several key advantages:

1. Complete Privacy: Your data never leaves your machine.
2. No API Costs: You aren’t charged per token.
3. Low Latency: No network round-trips to external API servers.
4. Offline Capability: Your AI agent works even without an internet connection.

1. Using Ollama with ADK-Rust

Ollama is the easiest way to get started with local LLMs. It provides a simple, unified interface for running various open-source models like Llama 3.2, Qwen 2.5, and Mistral.

Step 1: Install Ollama

Follow the instructions for your operating system at the Ollama website.

Step 2: Start and Pull a Model

Run the following commands in your terminal:

ollama serve
ollama pull llama3.2

Step 3: Configure Your Rust Project

Add the adk-model crate to your Cargo.toml with the ollama feature enabled:

[dependencies]
adk-model = { version = "0.2", features = ["ollama"] }
adk-rust = { version = "0.2", features = ["cli", "ollama"] }
tokio = { version = "1", features = ["full"] }
anyhow = "1.0"

Step 4: Use in Code

Instantiate an OllamaModel and pass it to your LlmAgentBuilder:

use adk_model::ollama::{OllamaModel, OllamaConfig};
use adk_agent::LlmAgentBuilder;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Specify the model name (e.g., "llama3.2")
    let model = OllamaModel::new(OllamaConfig::new("llama3.2"))?;

    let agent = LlmAgentBuilder::new("local_assistant")
        .instruction("You are a helpful assistant running locally.")
        .model(Arc::new(model))
        .build()?;

    // Use your local agent...
    Ok(())
}

2. Mistral.rs Integration

For high-performance Rust-native inference without external servers, ADK-Rust integrates with mistral.rs. This allows you to run models directly on your hardware (CPU or GPU) using a pure Rust engine.

To use mistral.rs, add adk-mistralrs to your project and follow the configuration guide in the official documentation.

Conclusion

Whether you prefer the ease of use of Ollama or the performance of a native Rust engine like mistral.rs, ADK-Rust makes it simple to build private, cost-effective AI agents that run entirely on your local machine.