Building a High-Performance On-Device LLM Client in Flutter with Qualcomm QNN

Running LLMs directly on a smartphone used to be a demo you showed once and never used again. Models were too large, inference too slow, and mobile hardware wasn’t ready. That’s over. Modern Snapdragon platforms ship dedicated NPUs capable of running sophisticated generative AI workloads locally, and with the Qualcomm QNN stack on top of LiteRT-LM, consumer devices like the Galaxy S24 and S25 can stream tokens at genuinely useful speeds.

Building an on-device LLM client, though, means navigating a maze: multi-gigabyte model downloads, Android’s scoped storage, choosing between MediaPipe delegates and Dart FFI, native library loading, streaming reasoning UX, and debugging the seams between Flutter, native Android, and the NPU.

In this tutorial, you will learn the architecture and development workflow for a high-performance on-device LLM client in Flutter, based on real production patterns for running Gemma 4 on the Hexagon NPU.

Prerequisites

• Flutter SDK 3.19+
• A Snapdragon device (Galaxy S24/S25 series recommended) or emulator
• The Gemma 4 E2B model in .litertlm format
• Android NDK and a willingness to write a little Kotlin

Understanding the Execution Pipeline

The first lesson: your app doesn’t “talk directly” to the NPU. Inference travels through several layers of abstraction, and each layer is responsible for orchestration, native dispatch, and hardware acceleration:

Prompt → Flutter UI → Dart FFI → LiteRT-LM Runtime → Qualcomm QNN → Hexagon CDSP

Getting this right starts with a model-format insight that trips up almost everyone: model file extensions determine the entire execution pipeline.

• .task models (Gemma 3 1B, SmolLM) go through the MediaPipe Java wrapper via EngineFactory.
• .litertlm models (Gemma 4, Qwen3) require the Dart FFI client (LiteRtLmFfiClient).

Passing a .litertlm file through the MediaPipe delegate fails immediately, because the delegate rejects the model header. Treat the file extension as a contract:

ModelInfo(
  id: 'gemma4_e2b',
  name: 'Gemma 4 E2B',
  family: 'Gemma 4',
  url: 'https://.../gemma-4-E2B-it.litertlm',
  sizeGB: 2.4,
  modelType: ModelType.gemma4,
  fileType: ModelFileType.litertlm, // Routes to LiteRtLmFfiClient
  supportsThinking: true,
)

Why You Should Stop Downloading Models Inside the App

A single Gemma 4 model exceeds 2GB. Re-downloading it after every test cycle destroys iteration speed. Instead, push the model to the device once and register it:

adb push gemma-4-E2B-it.litertlm /data/local/tmp/

Then copy it into the app sandbox from the app’s own context. Android’s scoped storage blocks adb push directly into the sandbox, so you work around it:

adb shell "run-as com.example.app cp /data/local/tmp/gemma-4-E2B-it.litertlm /data/data/com.example.app/files/"

What used to take fifteen minutes now takes about thirty seconds. Automate this in a script and your team will actually enjoy iterating on on-device AI.

Building a Self-Healing Model Registry

Manual model transfers break: files get moved, interrupted, or metadata goes stale. Build a self-healing registry that scans the local directory at startup, verifies file sizes, and re-registers anything missing:

Future<bool> isModelInstalled(ModelInfo model) async {
  final filename = model.url.split('/').last;
  final isInstalled = await FlutterGemma.isModelInstalled(filename);
  if (isInstalled) return true;

  // Self-healing: if the file exists locally, register it from the filesystem.
  final fileExists = await _checkFileExists(filename);
  if (fileExists) {
    await _registerLocalModel(
      filename,
      model.url,
      (model.sizeGB * 1024 * 1024 * 1024).toInt(),
    );
    return true;
  }
  return false;
}

Now the app is resilient to manual transfers, crashes, and interrupted installs. If the file is there, the app finds it and marks it ready.

Streaming Reasoning Without Breaking the UI

Models like Gemma 4 don’t just emit answers—they stream intermediate reasoning first. If you render those tokens directly into a markdown view, you get constant layout shifts: text jumps, responses reflow, the interface feels unstable.

Solution: separate thinking from answering. Buffer reasoning tokens independently and stream response tokens through the main chat:

class ReasoningParser {
  final StringBuffer thinking = StringBuffer();
  final StringBuffer answer = StringBuffer();

  void onChunk(StreamChunk chunk) {
    if (chunk.isThinking) {
      thinking.write(chunk.text);
    } else {
      answer.write(chunk.text);
    }
  }
}

Present reasoning inside a collapsible “Thinking Process” panel. Users get full transparency into the model’s plan without the main conversation jumping around.

Loading the Right Hardware Libraries

Running on the Snapdragon NPU isn’t as simple as shipping an APK. Your app must load the QNN delegate libraries at runtime:

// Kotlin side — initialize QNN delegation
System.loadLibrary("libQnnHtp.so")
System.loadLibrary("libLiteRtDispatch_Qualcomm.so")

Then build a fallback chain. If the NPU is unavailable, fall back to GPU via OpenCL, and finally to CPU:

val backend = when {
  hasQnnSupport() -> Backend.QNN_NPU
  hasOpenClSupport() -> Backend.GPU_OPENCL
  else -> Backend.CPU
}

Supporting multiple execution paths adds complexity, but it makes the app portable across the fragmented Android hardware landscape instead of tying you to one chipset.

The Platform Channel Bridge

The most stable production pattern is a thin Dart↔Kotlin bridge: a MethodChannel for lifecycle control (init, close, infer, cancel) with a reverse callback channel streaming tokens back to Dart. This gives you full control over the native side and avoids the crash-proneness of third-party Flutter wrappers.

class GemmaLocalClient implements LLMClient {
  static const _channel = MethodChannel('gemma_local/engine');

  @override
  Future<String> complete(String prompt) async {
    final answer = await _channel.invokeMethod<String>('infer', {
      'prompt': prompt,
      'maxTokens': 2048,
    });
    return answer ?? '';
  }

  @override
  Stream<String> stream(String prompt) async* {
    const stream = EventChannel('gemma_local/tokens');
    await for (final token in stream.receiveBroadcastStream(prompt)) {
      yield token as String;
    }
  }

  @override
  Future<void> cancel() => _channel.invokeMethod('cancel');
}

Because GemmaLocalClient implements the same interface as your cloud provider clients, your agent system never knows—or cares—whether it’s talking to GPT-4 or a local model. Swap the backend at runtime:

final client = await _chooseClient(); // cloud or local
final agent = Agent(client: client);

Engine Singleton, Conversation Per Request

Loading a multi-GB model into GPU memory takes seconds. Creating a Conversation takes milliseconds. The Engine lives for the app’s lifetime; Conversations are created and disposed per request. Moreover, LiteRT-LM enforces one active Conversation per Engine. If you run multiple agents in parallel, serialize everything behind a global mutex—including engine initialization, because different requests may need different backends (vision on CPU, text on GPU), and concurrent backend switching crashes the native layer.

class EnginePool {
  static final _lock = Mutex();
  static bool _ready = false;

  static Future<T> withEngine<T>(Future<T> Function() body) => _lock.protect(() async {
    if (!_ready) {
      final model = await FlutterGemma.getActiveModel(maxTokens: 4096);
      await model.createConversation().sendMessage('warm up');
      _ready = true;
    }
    return body();
  });
}

Handling Multimodal Constraints

Gemma 4 E2B is multimodal, but with constraints you’ll discover by testing:

• Images: only JPEG/PNG—WebP is silently rejected. There’s a ~2520 image-patch ceiling; cap the longest side at 896px or the prefill segfaults. On MediaTek, the GPU vision backend crashes during decode; use the CPU backend for vision and GPU for text.
• Audio: only WAV/PCM. Transcode M4A/AAC/MP3 on the Kotlin side using MediaExtractor + MediaCodec, resampling to 16kHz mono 16-bit PCM.
• Thinking mode: improves text-only reasoning but crashes with vision input on some devices. Auto-detect multimodal content and disable thinking for those requests.

final wantsThinking = !hasImageInput && !hasAudioInput;

Putting It All Together

A minimal production-shaped startup sequence:

Future<void> initOnDeviceLLM() async {
  await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]);

  const model = ModelInfo(
    id: 'gemma4_e2b',
    name: 'Gemma 4 E2B',
    family: 'Gemma 4',
    url: 'https://.../gemma-4-E2B-it.litertlm',
    sizeGB: 2.4,
    modelType: ModelType.gemma4,
    fileType: ModelFileType.litertlm,
    supportsThinking: true,
  );

  if (!await isModelInstalled(model)) {
    await FlutterGemma.installModel(
      modelType: ModelType.gemma4,
      fileType: ModelFileType.litertlm,
    ).fromNetwork(model.url).install();
  }
}

Lessons for Production

• Treat model formats as execution contracts. File extensions decide the runtime. Don’t guess.
• Automate local model discovery. Developers shouldn’t repair metadata by hand after moving files.
• Design for reasoning-first models. Streaming raw chain-of-thought into the chat is a UX regression.
• Optimize iteration speed relentlessly. The faster your team can test, the faster on-device AI improves.
• Call native APIs directly. A thin platform channel beats a heavy third-party wrapper that hides bugs.

Conclusion & Next Steps

You now have the architecture for a high-performance on-device LLM client: correct model routing, a fast model delivery workflow, a self-healing registry, streaming reasoning UX, QNN library loading with fallback chains, and a thread-safe engine pool.

Next steps:

• Add MTP (multi-token prediction) for a ~2.2x decode speedup.
• Persist sessions so returning users skip the prefill phase.
• Profile decode tokens/sec on your exact device and compare CPU/GPU/NPU.
• Ship a fallback path to cloud models for requests that exceed local capability.

The hardware ecosystem for on-device generative AI is maturing fast. The next challenge isn’t making LLMs run on phones—it’s making them pleasant to build. With this workflow, you’re ready for both.

References

• Inside-Out: Building a High-Performance On-Device LLM Client in Flutter with Qualcomm’s QNN
• LiteRT-LM
• flutter_gemma
• Gemma 4 documentation