Local-First AI: Ollama, llama.cpp, and On-Prem Inference

Every call to a hosted model ships your data to someone else’s data center and shows up on a monthly invoice. For code on a laptop, internal tools, or anything handling sensitive data, “local-first AI” is increasingly the default — and it’s never been easier, because Ollama and llama.cpp turned a fiddly ML deployment problem into a two-command install.

In this post, you will learn how to run open models locally with Ollama, drop down to llama.cpp when you need maximum control, and pick a sensible deployment strategy (laptop, server, or Docker).

The stack in one picture

Layer	Tool	What it does
Runtime	llama.cpp	C/C++ inference engine — the same backend Ollama uses
Orchestration	Ollama	Model registry, CLI, and REST API on top of llama.cpp
Access	ollama-python / OpenAI-compatible API	Talk to models from your code

Ollama’s supported backends are built on llama.cpp, so understanding the lower layer explains what Ollama is doing for you — quantization, KV-cache management, and hardware acceleration (Metal, CUDA, ROCm).

Get started with Ollama

Install and run a model in three commands:

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
# Windows
irm https://ollama.com/install.ps1 | iex

ollama run gemma4

ollama run starts an interactive chat. But the real value for developers is the daemon: Ollama serves a REST API on http://localhost:11434 the moment it’s installed.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{"role": "user", "content": "Why is the sky blue?"}],
  "stream": false
}'

Using it from Python

pip install ollama gives you a first-party SDK:

from ollama import chat

response = chat(model="gemma4", messages=[
    {"role": "user", "content": "Why is the sky blue?"},
])
print(response.message.content)

Because Ollama also exposes an OpenAI-compatible endpoint at /v1, you can point any tool that speaks OpenAI — LangChain, LlamaIndex, LiteLLM — at your local models without code changes:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
r = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(r.choices[0].message.content)

Running in Docker for on-prem

For a shared team box or a containerized service, the official image is ollama/ollama. Warm a model on startup so the first request isn’t slow:

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
volumes:
  ollama_data:

docker compose up -d
docker exec ollama ollama pull gemma4

Dropping down to llama.cpp directly

Ollama handles most cases, but when you need a specific quantization, a GGUF you built yourself, or bare-metal control, use llama.cpp directly:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release

# server mode with an OpenAI-compatible API
./build/bin/llama-server \
  -m models/gemma-4b-it-q4_k_m.gguf \
  -c 8192 \
  --port 8080

The model file is a GGUF — llama.cpp’s quantized format. q4_k_m is a good default: ~4-bit quantization that keeps most of the quality at a fraction of the memory. Ollama handles GGUF for you; with llama.cpp you manage it yourself.

Putting It All Together

A realistic local-first setup for a dev team: Docker Compose runs Ollama with a couple of pinned models, your FastAPI app talks to localhost:11434/v1 through the OpenAI SDK, and CI uses the same image so tests run identically on a laptop or a server. The complete compose file and app wiring are in this gist.

Conclusion & Next Steps

You can now run open models locally via Ollama, reach them with an OpenAI-compatible client, and drop to llama.cpp for fine-grained control. Next steps: pin model versions for reproducibility, add a Modelfile to tune system prompts and parameters per model, and measure throughput (tok/s) with OLLAMA_NUM_PARALLEL before scaling to a GPU server.

References / Sources

• Ollama: install, REST API, and model library. https://github.com/ollama/ollama
• Ollama CLI and API reference. https://docs.ollama.com
• llama.cpp: the inference engine underneath Ollama. https://github.com/ggml-org/llama.cpp