Running Gemma 4 On-Device in a Multi-Agent Flutter App

On-device inference changes the privacy calculus of AI apps. When your app handles personal records—text, photos, voice memos—even sending prompts to a trusted cloud provider is a compromise for privacy-conscious users. Running the model on the phone means the data never leaves, and the app works offline.

Gemma 4 E2B/E4B is the right fit for this: multimodal input, function calling, and small enough to run on consumer hardware. But integrating it into a multi-agent Flutter app is where the real lessons live. This post shares hard-won production patterns for running Gemma 4 E4B (~3.7GB) on mid-range Android devices with parallel agents—including the undocumented constraints we discovered in LiteRT-LM.

In this tutorial, you will learn the architecture that makes concurrent on-device agents stable, plus the pitfalls that will crash your app if you ignore them.

Prerequisites

• Flutter SDK 3.19+ and an Android device (4GB+ RAM)
• A Gemma 4 E4B .litertlm model file
• Basic Kotlin (for the platform channel bridge) and Dart

Architecture Overview

The core insight from production: for on-device LLM inference, call the native API directly. We initially evaluated flutter_gemma, a third-party Flutter plugin. It proved unstable—crashes severe enough to occasionally reboot the device. Google’s own Edge Gallery app, which calls the LiteRT-LM Kotlin API directly, ran the same model without issues on the same hardware.

So we built a thin platform-channel bridge between Dart and Kotlin:

┌──────────────────────────────┐
│        Dart layer            │
│  GemmaLocalClient (LLMClient)│
│  AgentPool (mutex + engine)  │
└──────────────┬───────────────┘
               │ MethodChannel / EventChannel
┌──────────────┴───────────────┐
│        Kotlin layer          │
│  init │ close │ infer │ cancel│
│  (reverse callback: tokens)  │
│        LiteRT-LM Engine      │
└──────────────┬───────────────┘
               │ QNN / GPU / CPU
               ▼
          Hexagon NPU

The Dart side exposes GemmaLocalClient implementing the same LLMClient interface as our cloud providers. The agent system doesn’t know or care whether it’s talking to GPT-4 or a local model.

The Kotlin Bridge

Kotlin side, lifecycle control through a MethodChannel with a reverse callback channel for streaming tokens:

class GemmaEngineBridge {
    private val engine: Engine = LiteRtEngineFactory.create(...)

    private val methodChannel = MethodChannel(
        flutterEngine.dartExecutor.binaryMessenger,
        "gemma_local/engine"
    ).apply {
        setMethodCallHandler { call, result ->
            when (call.method) {
                "init" -> { engine.initialize(); result.success(true) }
                "infer" -> { result.success(infer(call.argument("prompt"))) }
                "cancel" -> { engine.cancel(); result.success(true) }
                "close" -> { engine.close(); result.success(true) }
                else -> result.notImplemented()
            }
        }
    }

    private val tokenStream = EventChannel(
        flutterEngine.dartExecutor.binaryMessenger,
        "gemma_local/tokens"
    ).apply {
        setStreamHandler(object : EventChannel.StreamHandler {
            override fun onListen(args: Any?, events: EventChannel.EventSink) {
                tokenSink = events
            }
            override fun onCancel(args: Any?) { tokenSink = null }
        })
    }

    private fun infer(prompt: String): String {
        val conversation = engine.createConversation()
        val sb = StringBuilder()
        conversation.send(prompt) { token, _ ->
            sb.append(token)
            tokenSink?.success(token)
        }
        return sb.toString()
    }
}

The Dart Client Interface

The Dart side wraps this bridge behind a common interface so agents stay provider-agnostic:

abstract class LLMClient {
  Future<String> complete(String prompt);
  Stream<String> stream(String prompt);
  Future<void> cancel();
}

class GemmaLocalClient implements LLMClient {
  static const _channel = MethodChannel('gemma_local/engine');
  static const _tokens = EventChannel('gemma_local/tokens');

  @override
  Future<String> complete(String prompt) async {
    return await _channel.invokeMethod<String>('infer', {'prompt': prompt}) ?? '';
  }

  @override
  Stream<String> stream(String prompt) {
    return _tokens.receiveBroadcastStream({'prompt': prompt});
  }

  @override
  Future<void> cancel() => _channel.invokeMethod('cancel');
}

Now any agent—a card generator, a knowledge extractor, an asset analyzer—can be pointed at either the cloud or the local model by swapping one object.

The Engine Singleton Pattern

The critical design pattern is Engine singleton, Conversation per request. Loading a multi-GB model into GPU memory takes seconds. Creating a Conversation takes milliseconds. The Engine lives for the app’s lifetime; Conversations are created and disposed per request.

Serializing Concurrent Access

Here’s the constraint that bites: LiteRT-LM enforces one active Conversation per Engine. If multiple agents call the LLM simultaneously, violating this causes native crashes.

The fix: serialize all access behind a Dart-side global mutex. And critically, the lock must cover engine initialization, not just inference. Different request types need different engine configurations—image analysis requires a CPU vision backend, audio needs an audio backend. Without the lock covering initialization, concurrent backend switches crash the native layer.

import 'package:synchronized/synchronized.dart';

class EnginePool {
  static final _lock = Lock();

  static Future<T> run<T>(Future<T> Function() task) => _lock.synchronized(() async {
    // Acquire BEFORE engine init, hold until the inference stream closes.
    await _ensureEngineInitialized();
    return task();
  });

  static bool _initialized = false;

  static Future<void> _ensureEngineInitialized() async {
    if (_initialized) return;
    final client = GemmaLocalClient();
    await client.complete('warm up'); // forces model load
    _initialized = true;
  }
}

Every agent request flows through EnginePool.run(...). Serialization adds a small queueing delay but eliminates the native crashes entirely.

Multimodal Constraints You Will Hit

Multimodal support works, but comes with undocumented constraints we discovered through testing:

Images

• Only JPEG and PNG are accepted—WebP is silently rejected.
• There’s a 2520 image-patch ceiling. Large images cause segfaults during prefill. Cap the longest side at 896px.
• On MediaTek chipsets, the GPU vision backend crashes during decode. The CPU backend is stable for vision while the GPU handles text inference.

Future<Uint8List> normalizeImage(File image) async {
  final decoded = await decodeImageFromList(await image.readAsBytes());
  final side = decoded.width > decoded.height ? decoded.width : decoded.height;
  if (side <= 896) return image.readAsBytes();

  // Resize the longest side to 896px, preserving aspect ratio.
  final targetW = decoded.width > decoded.height
      ? 896
      : (decoded.width * 896 / decoded.height).round();
  final targetH = decoded.width > decoded.height
      ? (decoded.height * 896 / decoded.width).round()
      : 896;
  return encodeJpeg(await resize(decoded, targetW, targetH));
}

Audio

• Only WAV/PCM is supported. M4A, AAC, and MP3 all fail at the decoder level.
• Transcode on the Kotlin side using MediaExtractor + MediaCodec, resampling to 16kHz mono 16-bit PCM.

Thinking mode

• Improves reasoning for text-only tasks, but crashes when combined with vision input on some devices.
• Auto-detect multimodal content and disable thinking for those requests.

final wantsThinking = !hasImage && !hasAudio;

Never Trust Structured Output

Function-call JSON from Gemma 4 E4B is malformed often enough that you need a parse-error fallback. IDs, file paths, and references are routinely hallucinated. Two rules:

1. Catch parse errors at the Kotlin layer and return raw text so the agent can retry.
2. Validate every structured field against ground truth from agent state. Never round-trip IDs or paths through the model.

private fun extractToolCall(text: String): ToolCall? {
    return try {
        gson.fromJson(text, ToolCall::class.java)
    } catch (e: Exception) {
        null // malformed JSON — return raw text and let the agent retry
    }
}

Realistic Performance Numbers

Measured on a Redmi Pad (Dimensity 8100) with Gemma 4 E4B (~3.7GB):

Operation	Performance
Text inference (GPU)	~15–20 tokens/sec
Image analysis (CPU vision)	5–8 seconds per image
Audio transcription (CPU)	~0.3x realtime
Engine initialization	8–10s first load, cached after

For a fully offline use case this is acceptable—and it only gets faster as NPU support matures. But be honest in your design: on-device inference is not yet a replacement for cloud models when you need sustained throughput or perfect structured output.

Thermal Throttling Is Real

Sustained inference generates significant heat, triggering system-level CPU/GPU frequency reduction. Design for bursts: batch heavy work, add cooldown periods, and surface “device is warming up” states in the UI rather than letting token rates silently tank.

class ThermalAwareQueue {
  final _queue = <Future<void> Function()>[];
  DateTime _lastRun = DateTime.now();

  Future<void> enqueue(Future<void> Function() task) async {
    final wait = const Duration(seconds: 1);
    final elapsed = DateTime.now().difference(_lastRun);
    if (elapsed < wait) {
      await Future.delayed(wait - elapsed);
    }
    _lastRun = DateTime.now();
    await EnginePool.run(task);
  }
}

Putting It All Together

Your app now has three agents—card generation, knowledge extraction, asset analysis—all sharing one on-device engine:

final pool = EnginePool();

Future<void> generateCard(context) => pool.run(() {
  return cardAgent.generate(context);
});

Future<void> extractKnowledge(context) => pool.run(() {
  return extractorAgent.analyze(context);
});

Future<void> analyzeAsset(asset) => pool.run(() {
  return assetAgent.describe(asset);
});

All three serialize behind the mutex, share the engine singleton, and never crash the native layer.

Conclusion & Next Steps

Running Gemma 4 on-device in a multi-agent Flutter app is real and usable today for the right tasks. The architecture is clear: native API via a thin platform bridge, engine singleton with per-request conversations, global serialization including initialization, and defensive handling of multimodal and structured-output quirks.

Next steps:

• Persist KV-cache sessions so returning users skip the prefill phase.
• Add MTP drafters for a ~2.2x decode speedup.
• Gate “thinking mode” behind a per-request capability check.
• Consider Gemma 4 E2B if 2.4GB of model weight is more comfortable for your target devices.

The constraints are tractable once you know to design around them. Data that must never leave the device now can stay exactly where it belongs.

References

• Lessons Running Gemma 4 On-Device in a Multi-Agent Flutter App
• Bring state-of-the-art agentic skills to the edge with Gemma 4
• LiteRT-LM
• Gemma 4 documentation