On-Device Agentic Skills in Flutter with flutter_gemma_agent

An LLM that only answers questions is a toy. An LLM that picks a skill, runs it, and acts on the result is an agent. The flutter_gemma_agent package turns Flutter’s on-device Gemma inference into exactly that: a fully offline agent that maintains a SKILL.md catalog, decides which skill to invoke via function calling, runs it, and feeds the result back—all without a network connection.

This is the same pattern that powers the Google AI Edge Gallery’s Agent Skills, and it’s now available as an opt-in satellite package over flutter_gemma.

In this tutorial, you will learn how to install the package, define skills, register executors, and run a tool-calling agent loop entirely on-device.

Prerequisites

• Flutter SDK 3.19+
• A function-calling-capable model: Gemma 4 E2B or E4B (.litertlm format) recommended
• An Android, iOS, macOS, or Windows device for full agent support

Note: Web is not supported yet. Browser LLM runtimes don’t reliably emit tool calls, so the agent loop is disabled there.

How Agent Skills Work

A skill is a Markdown file (SKILL.md) with YAML frontmatter and one of four execution mechanisms:

Skill type	What it does	Android	iOS	macOS	Windows	Linux
text-only	A persona/instruction the model follows	✅	✅	✅	✅	✅
MCP	Calls remote MCP tools over Streamable HTTP	✅	✅	✅	✅	✅
native-intent	Opens an OS surface (mail, calendar, etc.)	✅	✅	✅	✅	✅
JS	Runs skill JavaScript in a sandboxed webview	✅	✅	✅	✅¹	❌²

¹ Windows JS skills need the WebView2 Runtime. ² Linux has no embeddable webview, so JS skills return an ErrorResult.

The model stays small and general; the skills give it a growing toolbox.

Installation

Add the package to your pubspec.yaml:

dependencies:
  flutter_gemma: ^0.4.0
  flutter_gemma_agent: ^0.1.0

Registering the Engine and Model

Initialize the inference engine and load a function-calling model:

import 'package:flutter_gemma/flutter_gemma.dart';

const gemma4E2BUrl =
    'https://www.kaggle.com/models/google/gemma-4/liteRt/gemma-4-E2B-it-litertlm/1';

Future<void> setup() async {
  await FlutterGemma.initialize(
    inferenceEngines: [LiteRtLmEngine()],
  );

  await FlutterGemma
      .installModel(modelType: ModelType.gemma4, fileType: ModelFileType.litertlm)
      .fromNetwork(gemma4E2BUrl)
      .install();

  final model = await FlutterGemma.getActiveModel(maxTokens: 4096);
  // ... build the agent session
}

Loading Skills and Building the Agent

The package ships a SkillRegistry (holds available/selected skills and builds the discovery string) and executors for each mechanism. Load the bundled starter skills from assets:

import 'package:flutter_gemma_agent/flutter_gemma_agent.dart';

final source = AssetSkillSource();
final registry = SkillRegistry()
  ..addAll(await source.load(), selected: true);

final session = await AgentSession.fromModel(
  model,
  registry: registry,
  executors: [
    TextSkillExecutor(),
    JsSkillExecutor(sourceFor: source.jsSkillSourceFor),
    NativeIntentExecutor(),
    // McpSkillExecutor(...) to call remote MCP tools.
  ],
);

The starter catalog ships seven skills covering all four mechanisms:

Skill	Type	What it does
calculate-hash	JS	Hash a piece of text
qr-code	JS (image)	Generate a QR code
query-wikipedia	JS (data)	Summarize a Wikipedia topic
interactive-map	JS (webview)	Show a location on an embedded map
send-email	intent	Open the OS mail composer
create-calendar-event	intent	Open the calendar event editor
kitchen-adventure	text-only	A text-adventure dungeon-master persona

Two-Stage Skill Discovery

Dumping every skill description into the system prompt wastes tokens and confuses small models. Instead, SkillRegistry builds a cheap name + description discovery string for the system prompt; the model picks the skills it needs, and only those get their full detail loaded.

class SkillRegistry {
  String buildDiscoveryString() {
    return _skills.values
        .where((s) => s.selected)
        .map((s) => '- ${s.name}: ${s.description}')
        .join('\n');
  }

  Skill? resolve(String name) => _skills[name];
}

Small prompts mean more reliable tool calls—essential on 2–4 billion parameter models.

The Agent Loop

AgentSession orchestrates the loop over flutter_gemma’s function calling, emitting a Stream with skill loads, tool calls, inline results, and streamed text:

final stream = session.send('Calculate the hash of "hello world"');

await for (final event in stream) {
  switch (event.type) {
    case AgentEventType.skillLoad:
      print('Loading skill: ${event.skillName}');
    case AgentEventType.toolCall:
      print('Calling ${event.toolName} with ${event.arguments}');
    case AgentEventType.result:
      print('Skill returned: ${event.text}');
    case AgentEventType.text:
      print(event.text); // streamed final answer
  }
}

Under the hood the loop is a bounded iteration: prompt → parse tool call → execute → feed result back → repeat until the model stops emitting tool calls.

Writing Your Own SKILL.md

Skills are simple to author. Here’s a text-only skill:

---
name: friendly-narrator
description: Tells a short story about a given topic
type: textOnly
---

You are a friendly narrator. When asked, tell a three-sentence story
about the given topic. Keep it warm and playful.

A JS skill runs in a sandboxed webview and returns results through a window contract:

<!-- skill/scripts/index.html -->
<script>
  window.ai_edge_gallery_get_result = async (data, secret) => {
    // data: arguments from the model
    // secret: injected, NEVER placed in the model prompt
    const result = `Echo: ${data.message}`;
    return JSON.stringify({ result });
  };
</script>

For native intents, define the OS action in frontmatter:

---
name: send-email
description: Opens the OS email composer for the given address
type: intent
intent: mailto
---

# Send Email

Opens the mail composer addressed to the given recipient.

Secrets are injected as the JS secret argument and never enter the model prompt—so a prompt-injection attack can’t exfiltrate them.

Building the Chat UI

Drop in the adaptive AgentChatView for a complete agent interface in one widget:

import 'package:flutter/material.dart';
import 'package:flutter_gemma_agent/flutter_gemma_agent.dart';

class AgentScreen extends StatelessWidget {
  final AgentSession session;

  const AgentScreen({super.key, required this.session});

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('On-Device Agent')),
      body: AgentChatView(session: session),
    );
  }
}

Or compose your own UI from SkillManagerView, McpManagerView, SecretEditorDialog, and SkillTesterView.

Platform Setup Notes

Most skills need no setup. For the platform-specific bits:

• iOS: create-calendar-event needs a usage description in ios/Runner/Info.plist:

  <key>NSCalendarsUsageDescription</key>
  <string>Create calendar events from the agent.</string>

• Android: schedule_notification requires core-library desugaring in android/app/build.gradle(.kts):

  android {
    compileOptions { isCoreLibraryDesugaringEnabled = true }
  }
  dependencies {
    coreLibraryDesugaring("com.android.tools:desugar_jdk_libs:2.1.4")
  }

• Windows: JS skills require the WebView2 Runtime (pre-installed on Windows 11).

Safety Rules

Adding a skill grants the model the ability to run that skill’s code or open OS surfaces. Follow these rules:

1. Only load skills you trust. Treat skill files like dependencies.
2. Require secrets judiciously. require-secret keys are stored in memory and passed to the skill, never to the model prompt.
3. Whitelist intents. NativeIntentExecutor runs behind user/OS confirmation for a reason—keep that guard in place.

Putting It All Together

Assemble the complete app:

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await setup();

  final model = await FlutterGemma.getActiveModel(maxTokens: 4096);
  final source = AssetSkillSource();
  final registry = SkillRegistry()
    ..addAll(await source.load(), selected: true);

  final session = await AgentSession.fromModel(
    model,
    registry: registry,
    executors: [
      TextSkillExecutor(),
      JsSkillExecutor(sourceFor: source.jsSkillSourceFor),
      NativeIntentExecutor(),
    ],
  );

  runApp(MaterialApp(home: AgentScreen(session: session)));
}

Now your Flutter app can respond to “show me Paris on a map,” “send an email to the team,” or “hash this text”—decided and executed entirely on-device, fully offline.

Conclusion & Next Steps

You’ve turned a Flutter app into an on-device agent with a real toolbox: skills defined as SKILL.md files, four execution mechanisms, two-stage discovery, and a streaming tool-calling loop.

Next steps:

• Author your own skill catalog and register a custom AssetSkillSource.
• Add an McpSkillExecutor so the agent can reach remote MCP tools when connectivity exists.
• Swap Gemma 4 E4B in for stronger multi-step planning.
• Build an AgentChatView customization with your own event rendering.

The on-device agent is verified on Android, iOS, macOS, and Windows. Your apps can now act, not just answer.

References

• flutter_gemma_agent package
• Agent Skills documentation
• Google AI Edge Gallery
• Gemma 4 documentation