Skip to content
Blog

On-Device AI in Flutter with Gemini Nano and GenKit

How to run Gemini Nano on-device inside a Flutter app — via ML Kit GenAI and the AICore system service — with capability gating, model fallback, and a hybrid local/cloud strategy.

Published on • August 10, 2026

AI Assistant

Sending every prompt to the cloud isn’t always what you want. On-device AI gives you privacy (data never leaves the device), speed (no network round-trip), offline capability, and zero per-request cost. In 2026, on-device models have become genuinely useful for summarization, classification, and short rewrites. Gemini Nano — Google’s smallest Gemini model — runs directly on Android devices through the AICore system service, and Flutter can reach it through ML Kit’s GenAI APIs.

In this post, you will learn how Gemini Nano works on Android, how to gate on availability, how to call it from a Flutter app, and how to build a hybrid local-then-cloud strategy so your app degrades gracefully on unsupported devices.

Gemini Nano and AICore

Gemini Nano runs in Android’s AICore system service, which leverages device hardware (NPUs on Google Tensor, Samsung Exynos/Snapdragon) for low inference latency and keeps the model up to date. The critical architectural point: AICore manages the model for you. You don’t embed weight files in your APK or handle downloads — AICore distributes and updates Gemini Nano, keeping your app’s disk and memory footprint tiny.

Two API layers exist:

  • ML Kit GenAI APIs — high-level, out-of-the-box quality for popular use cases built on top of AICore (image description, summarization, speech recognition).
  • AICore on-device inference APIs — lower-level access with more control, including LoRA support.

For a Flutter app, the practical path is a Flutter plugin that wraps the ML Kit GenAI / AICore interface on Android.

The availability problem: not every device has it

Gemini Nano isn’t available everywhere. Availability depends on the device’s hardware and whether the model is downloaded. This shapes the whole architecture: always check availability before using it, and provide a fallback.

Check model status and gate your UI on the result instead of a device allowlist:

import 'package:flutter_local_ai/flutter_local_ai.dart';

final localAI = FlutterLocalAI();

Future<void> checkCapabilities() async {
  final status = await localAI.modelStatus();  // available | downloading | unavailable
  final ready = status == ModelStatus.available;
  setState(() => isLocalAIReady = ready);
}

The plugin exposes isAvailable() and getModelStatus(); on unsupported hardware these return unavailable cleanly. Declare AICore as an optional system component so your app installs and runs even without it:

<!-- AndroidManifest.xml -->
<queries>
  <package android:name="com.google.android.aicore" />
</queries>

Running inference locally

Once the model is available, running a generation is a direct call:

final results = await localAI.generate(
  prompt: 'Summarize this article in three bullet points:\n$articleText',
);
final summary = results.first;

Latency for on-device inference runs around 50–300ms — no network round-trip at all. That makes it the right tool for tasks like draft generation, sentiment analysis, and structured data formatting.

The hybrid pattern: local first, cloud fallback

On-device models have limitations — smaller context windows and reduced reasoning power. The production pattern is a routing engine: try Gemini Nano first (fast, free, private), and fall back to cloud (Gemini via Firebase GenKit or Vertex AI) when the local model isn’t available, the task is complex, or the device is offline-but-cloud-reachable.

import 'package:flutter_local_ai/flutter_local_ai.dart';

final localAI = FlutterLocalAI();

Future<String> smartProcess(String prompt) async {
  // 1. Try on-device first (fast, free, private)
  try {
    if (await localAI.modelStatus() == ModelStatus.available) {
      final results = await localAI.generate(prompt: prompt);
      return results.first;
    }
  } catch (e) {
    debugPrint('Local inference failed: $e');
  }
  // 2. Fall back to cloud (more powerful, costs money, requires internet)
  return await generateInCloud(prompt);
}

Routing more intelligently means considering task complexity, network telemetry (connectivity_plus), and device state. A 3.2B-parameter model handles lightweight tasks; prompts needing deep reasoning, multi-turn context, or multimodal input go to the cloud. Battery-aware routing is also common: skip local inference below ~20% battery or during thermal throttling.

GenKit for the cloud tier

The cloud side of the hybrid pattern is where Firebase GenKit shines. GenKit provides structured LLM integration — telemetry, schema enforcement, tool calling, and caching — inside a Cloud Function, so API keys never reach the client and you can swap models without an app-store release:

// Firebase Cloud Function (TypeScript)
import { onRequest } from 'firebase-functions/v2/https';
import { initializeGenkit } from '@genkit-ai/firebase';

const ai = initializeGenkit();

export const generate = onRequest(async (req, res) => {
  const prompt = req.body.prompt;
  // Check a Firestore cache by prompt hash before calling the live API
  const cached = await getCachedResponse(prompt);
  if (cached) { res.json({ text: cached }); return; }

  const llm = ai.generate({ prompt, model: 'gemini-1.5-flash' });
  const text = (await llm).text;
  await cacheResponse(prompt, text);  // cuts repeat API costs
  res.json({ text });
});

A response cache can eliminate up to 90% of duplicate API fees for repetitive commands — an important cost control once your hybrid app routes the complex tasks to the cloud.

Putting It All Together

A complete hybrid Flutter AI feature: check Gemini Nano availability on launch and show a capability badge; route simple prompts (summarize, classify, rephrase) to the local model; route complex or multimodal requests to a GenKit Cloud Function with a Firestore cache; and always fall back to cloud when local inference fails. The app stays responsive offline for the tasks that matter, keeps private data on-device, and only spends API dollars where it adds value.

Conclusion & Next Steps

You now understand how Gemini Nano runs via AICore, why availability gating is non-negotiable, how to call it from Flutter, and how to build a local-first/cloud-fallback hybrid with GenKit. Next steps: add flutter_local_ai to a test device with a supported chip, verify modelStatus() gating, and wire the GenKit fallback tier with caching.

References / Sources

flutter gemini-nano on-device-ai genkit mobile ml-kit