Mastering Edge AI on Raspberry Pi with LiteRT and Gemma

Imagine building a fully autonomous robot that can see, hear, and react to its environment in real time—completely offline, on a single compact device like a Raspberry Pi. Edge AI unlocks exactly this autonomy. It enables developers to build secure, self-contained systems like intelligent robotics and local AI agents with zero cloud dependencies, ultra-low latency, and total data privacy.

With Google AI Edge’s LiteRT, running edge AI on a Raspberry Pi is now genuinely easy. LiteRT is a high-performance, production-proven on-device inference runtime that deploys everything from classical ML models to state-of-the-art LLMs out of the box, across platforms. Paired with Gemma, Google’s family of lightweight open models, the Raspberry Pi 5 becomes a serious edge-AI platform—capable of powering the Reachy Mini robot to perceive and react to its environment entirely locally in real time.

In this tutorial, you will learn how to deploy Gemma on a Raspberry Pi 5 with LiteRT, run inference on both CPU and GPU, and architect a real-time multimodal pipeline.

Prerequisites

• A Raspberry Pi 5 (4GB+ recommended) running Raspberry Pi OS (64-bit)
• A Hugging Face account and token (to download model files)
• Python 3.10+ and pip on the Pi

Choosing the Right Gemma Model

Gemma models are well suited for autonomous agents, smart cameras, and social robotics that reason and execute multi-step workflows on a Pi. Pick the variant that fits your hardware:

Model	Size	Best for
Gemma 3 270M	Tiny	Sentiment analysis, entity extraction on constrained devices
EmbeddingGemma 300M	Tiny	On-device RAG, semantic search, classification
Gemma 3 1B	Small	Summarization, content creation, multilingual text
Gemma 4 E2B	Medium	Continuous monitoring, fast text/image/audio inference, low RAM
Gemma 4 E4B	Medium+	Complex multi-step planning, strongest edge reasoning

For real-time speech and translation tasks, Gemma 4 E2B is the standout: its efficient tokenizer packs ~4.2 characters per token, producing an end-to-end generation speed of ~27.3 chars/sec (~300 words per minute)—twice the speed of normal human speech.

Gemma Performance on the Pi’s CPU

Through LiteRT-LM (the orchestration layer on top of LiteRT), deploying Gemma is a one-liner. Under the hood, sophisticated CPU acceleration via XNNPACK keeps resource efficiency and low latency:

Metric	Gemma 4 E2B on Pi 5
Prefill	99 tokens/sec
Decode	9 tokens/sec
Peak memory	1432 MB

That brings Gemma’s responsive, general-purpose intelligence to a $80 computer—enough to run voice assistants, smart cameras, and robots.

Installing the LiteRT CLI

The fastest way to get started is the LiteRT CLI, which aggregates conversion, quantization, benchmarking, and inference into one tool. Install it in a virtual environment:

python -m venv .venv
source .venv/bin/activate
pip install litert-cli

Running Your First Model

Download and run any compatible model directly from the LiteRT Hugging Face community. Set your token and run Gemma 4 E2B with a multimodal prompt:

export HUGGING_FACE_HUB_TOKEN=<your_hugging_face_token_here>

litert lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --attachment=image.jpg \
  --prompt="You are Reachy Mini. Identify the main object in front of you, \
    state its location (Left/Right/Center), and suggest head action in \
    10 words or less."

In one command, the model performs object identification, spatial reasoning, and action planning—a microcosm of a robotic agent.

GPU Inference with the WebGPU (Vulkan) Backend

The Pi 5’s quad-core ARM Cortex-A76 CPU delivers ~153.6 GFLOPS (FP32) and ~2.0 TOPS (INT8). The integrated Broadcom VideoCore VII GPU is smaller—~76.8 GFLOPS FP32, ~0.24 TOPS INT8—but introduces heterogeneous parallel execution, which is what matters for real-time edge apps.

Instead of saturating the CPU, delegate continuous vision or audio models to the VideoCore VII GPU, preserving CPU cycles for system monitoring, pipeline orchestration, and demanding LLM inference. LiteRT enables GPU inference on the Pi through its WebGPU (Vulkan) backend via ML Drift, letting you run computer vision, audio, and embedding models—including popular MediaPipe solutions and Ultralytics YOLO models—from the LiteRT Hugging Face community.

import litert_llm

# Route vision workloads to the GPU, text inference to the CPU
engine = litert_llm.create_engine(
    model_path="gemma-4-E2B-it.litertlm",
    backend="cpu",
)

vision_delegate = litert_llm.create_delegate(
    model_path="yolo26n.tflite",
    backend="gpu",  # Vulkan via ML Drift
)

Deep Dive: The Reachy Mini Pipeline

The Reachy Mini demo is a showcase of low-latency, real-time edge AI inference running entirely on a Raspberry Pi 5, splitting vision and language workloads into a concurrent, dual-processing architecture across CPU and GPU:

Camera frames ──► YOLO object detection ──► [GPU]
User speech ────► Moonshine ASR ──────────► [CPU]
                             │
                   combined context
                             ▼
              Gemma 4 E2B reasoning ──────► [CPU]
                             │
                             ▼
                 TTS synthesis ──────────► [CPU]
                             │
                             ▼
             Speech replies + robotic gestures

Each stage runs where it fits best:

1. Object Detection (YOLO on GPU): continuous detection avoids resource contention and frees the CPU.
2. Speech Recognition (Moonshine on CPU): transcribes audio locally.
3. Reasoning & Action (Gemma 4 E2B on CPU): processes the transcript with visual metadata to generate low-latency, streaming responses—speech replies and physical gestures.
4. Text-to-Speech (TTS on CPU): synthesizes audio in a stream.

The full source is in the LiteRT Samples repo.

Agentic Coding with the LiteRT CLI

The LiteRT CLI isn’t just for humans. You can add the LiteRT CLI skill and other advanced LiteRT skills to your AI coding agent (such as Google Antigravity), empowering agents to autonomously orchestrate multi-stage ML workflows—convert, quantize, benchmark, infer—on your behalf.

This is how a fully offline voice translator on a Raspberry Pi becomes a one-command build. See the Gemma Translator repo for the complete implementation.

An Ultra-Lean Footprint for IoT

For resource-constrained IoT devices, storage and memory overhead matter. Generic AI runtimes bundle heavy desktop or server dependencies; LiteRT is engineered specifically for on-device deployment. On ARM64 Linux, running LLM inference on a Pi through LiteRT uses a dramatically smaller download footprint than, say, Ollama—one of the reasons it fits where other stacks can’t.

Building an Offline Voice Translator

Let’s wire together the full stack. First, run TTS and ASR on the Pi:

# Transcribe speech to text with a local ASR model
litert lm run \
  --from-huggingface-repo=litert-community/moonshine-tiny \
  moonshine-tiny.tflite \
  --audio=input.wav

# Translate with Gemma
litert lm run \
  --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
  --prompt="Translate the following to Japanese: ..."

Chain these in a Python script for an end-to-end offline translator:

import subprocess

def translate_offline(audio_path: str, target_lang: str) -> str:
    # 1. ASR
    text = subprocess.run(
        ["litert", "lm", "run", "--from-huggingface-repo=litert-community/moonshine-tiny",
         "moonshine-tiny.tflite", f"--audio={audio_path}"],
        capture_output=True, text=True,
    ).stdout.strip()

    # 2. LLM translation
    result = subprocess.run(
        ["litert", "lm", "run",
         "--from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm",
         "gemma-4-E2B-it.litertlm",
         f'--prompt=Translate to {target_lang}: {text}'],
        capture_output=True, text=True,
    )
    return result.stdout.strip()

No cloud. No API keys beyond your Hugging Face token. Full privacy.

What’s Next for Pi Edge AI

LiteRT integration and Gemma models are coming soon to Hailo AI accelerators, so you’ll be able to offload inference to the Raspberry Pi AI HAT+ and AI HAT+ 2 through the same familiar LiteRT workflows—delivering massive hardware acceleration on top of what the Pi alone already achieves.

Conclusion & Next Steps

You’ve learned how to run Gemma on a Raspberry Pi 5 with LiteRT: choosing the right model, running it via the CLI, exploiting CPU/GPU heterogeneous parallelism, and architecting a real-time robotics pipeline.

Next steps:

• Build the Reachy Mini demo from the LiteRT Samples repo.
• Add the LiteRT CLI skill to your AI coding agent to automate your ML workflow.
• Benchmark E2B vs E4B on your Pi to balance reasoning and speed.
• Try the litert lm run command with your own images and prompts.

Edge AI on the Pi is here, and it’s fast, private, and cheap. Your robots are waiting.

References

• Mastering Edge AI on Raspberry Pi with LiteRT and Gemma
• LiteRT Developer Site
• LiteRT-LM
• LiteRT CLI
• Gemma 4 documentation