Gemma 4 on Arm: SME2 and the Optimized On-Device AI Stack

Real-time assistance, seamless communication, and greater personalization are now baseline expectations for billions of smartphone users. Delivering those experiences at global scale hinges on the compute foundation under them—and one constant is ubiquitous across the entire Android ecosystem: Arm.

Google’s launch of Gemma 4 accelerated the shift to on-device AI, and Arm’s latest CPU architecture innovations are what make it fast enough to feel native. At the center is SME2 (Scalable Matrix Extension 2), a set of advanced CPU instructions in the Armv9 architecture that accelerates the matrix-heavy workloads at the heart of every neural network—within the power envelope of a smartphone.

In this tutorial, you’ll understand the Arm + Gemma 4 stack, the real numbers behind it, and how to build apps that benefit from these optimizations with zero code changes.

Prerequisites

• Basic familiarity with Android and on-device ML
• A device built on Armv9 with SME2 (e.g., latest Android flagships with Arm C1 CPUs)
• Optionally, LiteRT and XNNPACK for hands-on experimentation

Why Arm is the Right Place for Gemma 4

Gemma 4 advances on-device AI with improved performance and efficiency while expanding support for the multimodal experiences that matter most on Arm-based devices: reasoning, agentic workflows, and vision-and-audio-enabled use cases. It delivers enhanced capabilities across text, audio, and image, broader language support, and a foundation for real-time assistive experiences—more responsive and context-aware, without increasing memory footprint.

But capability is only half the story. On-device AI has a hard budget: it must operate within the power envelope of modern smartphones. That’s exactly what Armv9’s SME2 was built to solve.

SME2: Matrix Math, In Hardware

Transformer inference is dominated by matrix multiplication. Historically that happened on the CPU with generic SIMD instructions—fast, but not specialized. SME2 adds dedicated matrix-extension instructions that accelerate the multiply-accumulate operations LLMs depend on, all within the power budget of a phone.

The key architectural facts:

• SME2 is built into the Armv9 ISA, and already shipped in Arm C1 CPUs integrated into the latest Android devices.
• Acceleration happens transparently: existing code, models, and deployment pipelines run faster with no changes.
• KleidiAI—Arm’s software acceleration layer—is integrated into leading runtimes like Google’s XNNPACK, and frameworks like LiteRT and MediaPipe.

In early Arm engineering tests, SME2 showed promising gains for Gemma 4 workloads. On the Gemma 4 E2B (Effective 2 Billion) model:

Operation	Speedup with SME2
Prefill (processing user input)	5.5x
Decode (generating responses)	1.6x

Prefill is the latency users feel when they hit send; a 5.5x speedup is the difference between a visible pause and an instant response.

The Software Stack: Where SME2 Reaches Your App

The optimizations flow down to developers for free because of how the stack is layered:

Your app (Gemma 4 model)
        │
Google LiteRT / MediaPipe
        │  ← KleidiAI integrated
Arm XNNPACK kernels
        │  ← SME2 matrix instructions
Armv9 CPU (Arm C1)

Because KleidiAI plugs into XNNPACK and LiteRT, apps targeting Arm-based Android devices with SME2 automatically access out-of-the-box performance optimizations. No changes to existing code, models, or deployment pipelines.

In Practice: The Envision Accessibility Use Case

The clearest illustration is Envision, an accessibility-focused app for blind and low-vision users. Historically, Envision’s scene interpretation relied on cloud connectivity. In an early prototype with Gemma 4 running locally on Arm CPUs with SME2, users can now:

1. Capture a photo of their surroundings.
2. Receive a detailed scene description directly on-device.
3. Do it without a network connection—and without sending sensitive data off-device.

The result: lower latency, stronger privacy, and more consistent experiences regardless of connectivity. This shift from cloud dependency to local inference is critical for mobile applications. It reduces infrastructure costs for developers, improves reliability for users, and unlocks new categories of real-time applications.

What the Speedups Mean for Users

In practice, software-level gains translate directly into better on-device experiences:

• Faster responses: prefill speedups make first-token latency feel instant.
• Smoother sustained interactions: decode improvements keep streaming replies flowing.
• More reliable on-device AI: performance headroom means the model can run longer without thermal throttling.
• Battery and thermal stability: SME2 does matrix work with better efficiency per watt.

All of this matters as models grow more capable. The power-efficient acceleration is exactly what lets Gemma 4 handle reasoning, agentic workflows, and vision-and-audio use cases on a phone without melting it.

Building for the Optimized Stack

You don’t need to write assembly to benefit—but you should make sure your pipeline uses the accelerated path. Here’s how to stay on the fast track:

1. Use the accelerated runtimes

Target LiteRT (and its XNNPACK kernels) rather than rolling your own inference. KleidiAI integration means SME2 devices get faster automatically:

# LiteRT with XNNPACK delegate on by default on supported CPUs
import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="gemma4.tflite")
interpreter.allocate_tensors()

2. Keep models on supported formats

LiteRT and MediaPipe formats flow through XNNPACK. Exotic formats may bypass the optimized kernels entirely.

3. Measure on real hardware

Baseline your app on an SME2 device to capture the 5.5x/1.6x deltas and confirm the accelerated path is being hit:

adb shell dumpsys cpuinfo
# or use the LiteRT benchmark tool on-device

Mobile Deployments: Android and Beyond

The Arm + Gemma 4 collaboration extends across the Android ecosystem, and the benefits aren’t limited to phones. As on-device AI becomes the default architecture rather than the exception, we’re seeing the same optimized stack appear in:

• Tablets and foldables (larger surfaces for agentic UI)
• Smart home and edge gateways
• Automotive infotainment
• IoT and robotics where power envelopes are even tighter

The economics are compelling: shifting workloads from cloud to local inference reduces infrastructure costs, improves reliability, and strengthens privacy.

Putting It All Together

A practical checklist for shipping Gemma 4 on Arm devices:

def on_device_recommendation() -> dict:
    return {
        "runtime": "LiteRT + XNNPACK",      # gets KleidiAI + SME2 for free
        "model": "Gemma 4 E2B/E4B (.tflite/.litertlm)",
        "target": "Armv9 with SME2 (Arm C1)",
        "expected": "5.5x prefill, 1.6x decode vs pre-SME2",
        "code_changes": "none required",
        "user_value": "instant responses, privacy, offline, lower cost",
    }

The future of mobile AI won’t be defined solely by larger models, but by how efficiently, securely, and pervasively they run at scale across the Android ecosystem.

Conclusion & Next Steps

You now understand the Arm + Gemma 4 stack: SME2 matrix instructions, the KleidiAI software layer, and the transparent speedups that arrive in LiteRT and XNNPACK. The net effect is faster, more private, more reliable on-device AI that reduces developer cost and unlocks real-time applications.

Next steps:

• Test your Gemma 4 app on an Armv9/SME2 device and measure prefill vs decode.
• Review your inference stack for XNNPACK/KleidiAI compatibility.
• Explore on-device accessibility features (like Envision) that become feasible with local inference.
• Keep an eye on Arm KleidiAI updates for new kernel optimizations.

The collaboration between Arm and Google is making it easier for developers to bring fast, responsive, privacy-preserving AI experiences to users without modifying existing applications. That’s the promise of on-device AI—and it starts with the silicon.

References

• Gemma 4 on Arm: Accessible, immediate, optimized on-device AI
• Gemma 4 documentation
• LiteRT Developer Site
• XNNPACK on GitHub