The Future of Mobile Dev: On-Device Agents and Edge AI

Mobile development is entering its biggest architectural shift since the move from webviews to native toolkits. For a decade, “AI features” in mobile apps meant a round trip to the cloud: your prompt, your photos, and your documents were shipped to a datacenter, processed, and the answer came back over the network. That model worked, but it brought with it latency, privacy concerns, per-token API bills, and a hard dependency on connectivity.

The future of mobile development is on-device agents and edge AI. With Google’s Gemma 4 model family, the LiteRT-LM runtime, and frameworks like Flutter, developers can now build apps where an AI agent lives entirely on the phone, plans multi-step tasks, calls tools, and acts—all without a single byte leaving the device.

In this tutorial, you will learn how the on-device agent stack works, what it means for your app architecture, and how to write the code that powers a fully local, agentic Flutter app.

Prerequisites

• A recent Flutter SDK (3.19+)
• An Android device or emulator running Android 11+ with a GPU
• Basic familiarity with Dart and platform channels
• A Gemma 4 E2B or E4B model file in .litertlm format (download from the LiteRT Hugging Face community)

Why On-Device Agents Matter

Cloud AI solved the “can AI reason at all?” problem. On-device AI solves the problems that cloud AI can’t touch:

• Privacy by design: data never leaves the device, which removes most consent and compliance overhead.
• Zero latency: no network hop, no cold-start, no token streaming over flaky mobile connections.
• Deterministic cost: the model is a fixed binary on the device; inference is free and unlimited.
• Offline resilience: apps keep working in planes, tunnels, and the field.
• Agentic capability: Gemma 4’s multi-step planning and function calling mean the phone can execute real workflows, not just answer questions.

The catch is compute. Phones don’t have A100s, so every layer of the stack has been redesigned for efficiency. Understanding that stack is the key to building a good on-device agent.

The On-Device Agent Stack

An on-device agent is not a model—it’s a layered system. Here’s the mental model:

┌─────────────────────────────────────────────┐
│              Flutter UI / Agent Loop         │
├─────────────────────────────────────────────┤
│          Dart FFI / Platform Channel         │
├─────────────────────────────────────────────┤
│             LiteRT-LM Runtime                │
│   (KV cache, MTP drafting, session mgmt)     │
├─────────────────────────────────────────────┤
│      LiteRT Kernels (XNNPACK / ML Drift)     │
├─────────────────────────────────────────────┤
│       CPU (Arm SME2) │ GPU │ NPU (QNN)       │
└─────────────────────────────────────────────┘

Each layer exists to solve a specific problem:

1. The model (Gemma 4 E2B/E4B) is small enough to fit in the device memory but large enough to reason, call functions, and handle multimodal input.
2. LiteRT-LM provides the agentic runtime: session persistence, multi-token prediction, and constrained decoding for reliable tool calls.
3. LiteRT gives you hardware acceleration through XNNPACK and ML Drift across CPU, GPU, and NPU.
4. Your app owns the agent loop: it feeds context, interprets tool-call requests, runs tools, and decides when the task is complete.

Setting Up the Runtime

Start by adding the flutter_gemma package to your pubspec.yaml and registering a LiteRT-LM inference engine:

dependencies:
  flutter:
    sdk: flutter
  flutter_gemma: ^0.4.0

import 'package:flutter_gemma/flutter_gemma.dart';

Future<void> initOnDeviceAgent() async {
  await FlutterGemma.initialize(
    inferenceEngines: [LiteRtLmEngine()],
  );
}

Next, install a function-calling-capable model. Gemma 4 E2B is the sweet spot for mobile—it supports text, image, and audio input while staying under ~1.5 GB of working memory thanks to memory-mapped per-layer embeddings:

const gemma4E2BUrl =
    'https://www.kaggle.com/models/google/gemma-4/liteRt/gemma-4-E2B-it-litertlm/1';

Future<void> installModel() async {
  await FlutterGemma
      .installModel(
        modelType: ModelType.gemma4,
        fileType: ModelFileType.litertlm,
      )
      .fromNetwork(gemma4E2BUrl)
      .install();
}

The Agent Loop

The heart of an on-device agent is the loop: prompt the model, inspect its output for a tool-call request, execute the tool, feed the result back, and repeat until the model produces a final answer.

import 'dart:async';

typedef ToolHandler = Future<String> Function(Map<String, dynamic> args);

class OnDeviceAgent {
  final Map<String, ToolHandler> _tools = {};

  void registerTool(String name, ToolHandler handler) {
    _tools[name] = handler;
  }

  Future<String> run(String task) async {
    final model = await FlutterGemma.getActiveModel(maxTokens: 4096);
    final conversation = model.createConversation(
      systemInstruction: '''
You are an on-device assistant. You can call these tools: ${_tools.keys.join(', ')}.
When a task needs a tool, respond with EXACTLY a JSON object:
{"tool": "<name>", "args": {<arguments>}}
When the task is complete, respond with your final answer in plain text.
''',
    );

    String currentPrompt = task;
    const maxIterations = 5;

    for (var i = 0; i < maxIterations; i++) {
      final response = await conversation.sendMessage(currentPrompt);
      final text = response.text;

      final toolCall = _parseToolCall(text);
      if (toolCall == null) {
        return text;
      }

      final handler = _tools[toolCall['tool']];
      if (handler == null) {
        currentPrompt = 'Error: unknown tool "${toolCall['tool']}". Try again.';
        continue;
      }

      final result = await handler(toolCall['args'] as Map<String, dynamic>);
      currentPrompt = 'Tool "${toolCall['tool']}" returned:\n$result\n'
          'Continue with the task. If done, give the final answer.';
    }

    return 'Max iterations reached without a final answer.';
  }

  Map<String, dynamic>? _parseToolCall(String text) {
    final match = RegExp(r'\{.*\}', dotAll: true).firstMatch(text);
    if (match == null) return null;
    try {
      final decoded = jsonDecode(match.group(0)!) as Map<String, dynamic>;
      return decoded.containsKey('tool') ? decoded : null;
    } catch (_) {
      return null;
    }
  }
}

Note the pattern: the model never executes code. It requests tool execution by emitting structured JSON, and your Dart code is the only thing that can touch the OS. This is the security boundary that makes on-device agents safe.

Registering Real Tools

Let’s wire up two practical tools: one that reads a local file, and one that schedules a notification. Both are pure Dart, so they inherit all of Flutter’s platform reach.

import 'dart:convert';
import 'package:flutter_local_notifications/flutter_local_notifications.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await initOnDeviceAgent();
  await installModel();

  final agent = OnDeviceAgent();

  agent.registerTool('read_file', (args) async {
    final path = args['path'] as String;
    final file = File(path);
    if (!await file.exists()) return 'File not found: $path';
    return file.readAsString();
  });

  agent.registerTool('schedule_notification', (args) async {
    final plugin = FlutterLocalNotificationsPlugin();
    await plugin.initialize(const InitializationSettings(
      android: AndroidInitializationSettings('@mipmap/ic_launcher'),
    ));
    await plugin.show(
      args['id'] as int? ?? 0,
      args['title'] as String,
      args['body'] as String,
      const NotificationDetails(
        android: AndroidNotificationDetails('agent', 'Agent Tasks'),
      ),
    );
    return 'Notification scheduled';
  });

  runApp(const MyApp(agent: agent));
}

Handling the Thinking Mode

Gemma 4 models support “thinking mode”—a scratchpad where the model reasons step-by-step before committing to an action. Stream these tokens separately so the UI stays responsive and doesn’t reflow constantly.

class ThinkingAwareResponse {
  final String reasoning;
  final String answer;

  const ThinkingAwareResponse(this.reasoning, this.answer);
}

Stream<ThinkingAwareResponse> streamAgentOutput(
  Conversation conversation,
  String prompt,
) async* {
  final buffers = {'reasoning': StringBuffer(), 'answer': StringBuffer()};
  String current = 'answer';

  await for (final chunk in conversation.streamMessage(prompt)) {
    if (chunk.reasoning != null) {
      current = 'reasoning';
      buffers['reasoning']!.write(chunk.reasoning);
    } else {
      current = 'answer';
      buffers['answer']!.write(chunk.text);
    }
    yield ThinkingAwareResponse(
      buffers['reasoning']!.toString(),
      buffers['answer']!.toString(),
    );
  }
}

In the UI, render reasoning inside a collapsible panel instead of dumping raw chain-of-thought into the chat window. Users get the transparency without the layout instability.

Engine Singleton, Conversation Per Request

The single most important performance pattern for on-device LLMs: load the engine once, create and dispose conversations per request. Loading a multi-GB model into GPU memory takes seconds; creating a conversation takes milliseconds.

class EnginePool {
  static Future<void> acquire() async {
    await _mutex.acquire();
    await _ensureEngineReady();
  }

  static void release() => _mutex.release();

  static bool _engineReady = false;

  static Future<void> _ensureEngineReady() async {
    if (_engineReady) return;
    final model = await FlutterGemma.getActiveModel(maxTokens: 4096);
    // Force initialization and model warm-up.
    await model.createConversation().sendMessage('Hello');
    _engineReady = true;
  }
}

If you run multiple agents in parallel (a content generator, a summarizer, an entity extractor), remember that LiteRT-LM enforces one active conversation per engine. Serialize all access behind a single mutex that spans both engine initialization and inference—switching hardware backends mid-flight under concurrent load is a reliable recipe for a native crash.

Putting It All Together

A minimal, fully offline agentic app:

import 'package:flutter/material.dart';

void main() => runApp(const OfflineAgentApp());

class OfflineAgentApp extends StatelessWidget {
  const OfflineAgentApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Offline Agent',
      theme: ThemeData(colorSchemeSeed: Colors.teal, useMaterial3: true),
      home: const AgentScreen(),
    );
  }
}

class AgentScreen extends StatefulWidget {
  const AgentScreen({super.key});

  @override
  State<AgentScreen> createState() => _AgentScreenState();
}

class _AgentScreenState extends State<AgentScreen> {
  final _controller = TextEditingController();
  final _messages = <String>[];
  bool _busy = false;

  Future<void> _run() async {
    final task = _controller.text.trim();
    if (task.isEmpty) return;

    setState(() {
      _messages.add('You: $task');
      _busy = true;
    });
    _controller.clear();

    final agent = OnDeviceAgent();
    agent.registerTool('read_file', (args) async => 'Sample file content');
    final result = await agent.run(task);

    setState(() {
      _messages.add('Agent: $result');
      _busy = false;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Offline Agent')),
      body: Padding(
        padding: const EdgeInsets.all(16),
        child: Column(
          children: [
            Expanded(
              child: ListView.builder(
                itemCount: _messages.length,
                itemBuilder: (_, i) => Padding(
                  padding: const EdgeInsets.symmetric(vertical: 4),
                  child: Text(_messages[i]),
                ),
              ),
            ),
            Row(
              children: [
                Expanded(
                  child: TextField(
                    controller: _controller,
                    enabled: !_busy,
                    decoration: const InputDecoration(hintText: 'Ask anything offline...'),
                  ),
                ),
                IconButton(
                  onPressed: _busy ? null : _run,
                  icon: _busy
                      ? const SizedBox(
                          width: 20,
                          height: 20,
                          child: CircularProgressIndicator(strokeWidth: 2),
                        )
                      : const Icon(Icons.send),
                ),
              ],
            ),
          ],
        ),
      ),
    );
  }
}

The Tradeoffs You Must Know

On-device agents are not a drop-in replacement for cloud models. Be honest about the constraints:

• Structured output reliability: function-call JSON from small models is malformed often enough that you need parse-error fallbacks and retries.
• Hallucinated identifiers: never round-trip IDs or file paths through the model. Validate against ground truth from app state.
• Thermal throttling: sustained inference generates real heat. Design for bursts, not 24/7 generation.
• Modality quirks: image input has patch ceilings (cap the longest side at ~896px), and audio requires specific formats (WAV/PCM).

A sound architecture treats the model as a smart-but-unreliable teammate and puts every correctness guarantee in your code.

Conclusion & Next Steps

You’ve built a fully offline, agentic mobile app: an engine initialized once, a tool-calling loop with a security boundary, streaming reasoning support, and a responsive UI. This is the architectural template for the coming wave of on-device agents.

Next steps:

• Add MCP skills so the agent can call remote tools when connectivity exists, then gracefully degrade offline.
• Implement session save/restore so long conversations survive app restarts without a full re-prefill.
• Profile with the litert-lm CLI to compare CPU, GPU, and NPU backends on your target hardware.
• Explore Gemma 4 E4B for tasks that need noticeably stronger reasoning at the cost of a larger footprint.

The future of mobile development isn’t apps that call AI—it’s apps that are AI, living entirely on your users’ devices.

References

• Gemma 4 documentation
• Bring state-of-the-art agentic skills to the edge with Gemma 4
• LiteRT-LM
• flutter_gemma