On-Device AI in the Browser: LiteRT-LM with WebGPU and WebAssembly

Run Gemma 4 LLM inference in the browser with LiteRT-LM, WebGPU, and WebAssembly. Learn the JavaScript API, serverless architecture, and how to build privacy-preserving web apps with zero backend.

Published on • August 19, 2026

AI Assistant

The most interesting place to run an LLM in 2026 isn’t a datacenter—it’s the browser tab already open on your user’s machine. LiteRT-LM, the same production-proven runtime that powers Google’s on-device AI across Chrome, ChromeOS, and the Pixel Watch, is now fully accessible on the web through its JavaScript API.

Powered by WebGPU and WebAssembly, LiteRT-LM delivers lightning-fast LLM routing and execution client-side. This unlocks web applications that are serverless, secure, and completely privacy-preserving: no model API keys, no backend infrastructure, no data leaving the browser.

In this tutorial, you will learn how to run Gemma 4 directly in the browser with LiteRT-LM, and how to build a complete client-side AI web app.

Prerequisites

A WebGPU-capable browser (latest Chrome/Edge, or Safari with WebGPU enabled)
Node.js for serving your app during development
A Gemma 4 model in a web-deployable format (LiteRT-LM web builds)

How LiteRT-LM Runs in the Browser

LiteRT-LM builds on the foundational success of the MediaPipe LLM Inference engine’s web solution. The web runtime provides:

WASM-based execution: production-proven inference pipelines compiled to WebAssembly.
WebGPU acceleration: GPU compute for prefill and decode, bypassing the CPU entirely.
No server required: model weights are fetched once and cached client-side.
Privacy by default: prompts never leave the device.

Decode speeds are genuinely usable—on a MacBook Pro via WebGPU, expect up to 76 tokens/sec decode, making real-time chat interfaces feel native.

Setting Up the Web Runtime

Install the LiteRT-LM JavaScript API:

npm install @litert-lm/core

Import the runtime and load a model:

import { LiteRtLmWeb } from '@litert-lm/core';

const engine = new LiteRtLmWeb({
  modelUrl: '/models/gemma-4-E2B-it-litertlm.bin',
  backend: 'webgpu', // or 'wasm' for CPU fallback
  multiTokenPrediction: true,
});

Loading the model is an explicit step, so you control the UX while large weights download:

const session = await engine.createSession({
  maxTokens: 4096,
  onProgress: (loaded, total) => {
    console.log(`Loading model: ${(loaded / total * 100).toFixed(0)}%`);
  },
});

Streaming a Response

The key to a good chat UX is streaming tokens as they’re generated:

const stream = session.sendStreaming('Explain speculative decoding simply.');

const output = document.getElementById('output');
let text = '';

for await (const chunk of stream) {
  text += chunk.token;
  output.textContent = text; // render incrementally
}

With multiTokenPrediction enabled, you get speculative decoding in the browser—draft and verify multiple tokens per pass for a significant speedup, exactly as on mobile.

Building a Serverless AI App

Because the model runs entirely client-side, your “backend” is a static file server. Here’s a complete minimal app:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>In-Browser AI</title>
</head>
<body>
  <h1>In-Browser Gemma 4</h1>
  <div id="output" class="chat"></div>
  <textarea id="prompt" rows="3" placeholder="Ask anything..."></textarea>
  <button id="send" disabled>Send</button>

  <script type="module">
    import { LiteRtLmWeb } from '@litert-lm/core';

    const output = document.getElementById('output');
    const prompt = document.getElementById('prompt');
    const send = document.getElementById('send');

    const engine = new LiteRtLmWeb({
      modelUrl: '/models/gemma-4-E2B.bin',
      backend: 'webgpu',
    });

    const session = await engine.createSession({ maxTokens: 4096 });
    send.disabled = false;

    send.addEventListener('click', async () => {
      const text = prompt.value.trim();
      if (!text) return;

      output.insertAdjacentHTML('beforeend', `<p><b>You:</b> ${text}</p>`);
      const reply = document.createElement('p');
      reply.innerHTML = '<b>Gemma:</b> ';
      output.appendChild(reply);

      for await (const chunk of session.sendStreaming(text)) {
        reply.textContent += chunk.token;
      }
      prompt.value = '';
    });
  </script>
</body>
</html>

Serve it with any static server and you have a fully working, fully offline AI app:

npx serve .   # or: python -m http.server 8080

Session Management in the Browser

Long conversations re-prefill on every reload—unless you persist the session. LiteRT-LM’s web runtime supports session save/restore, serializing the KV cache to IndexedDB:

// Save the session's KV cache to IndexedDB
const snapshot = await session.save();
await idbKeyVal.set('chat-session', snapshot);

// On return visit, restore instead of re-prefilling
const restored = await engine.restoreSession(await idbKeyVal.get('chat-session'));
const reply = await restored.send('Continue where we left off.');

This is the same technique that powers Google AI Edge Gallery’s extended skills on mobile, now available in the browser.

Privacy and Security Model

In-browser inference has a security story that cloud APIs can’t match:

Data never leaves the device. No prompts transmitted, no logs retained server-side.
No API keys. Nothing to leak, nothing to rotate, nothing to store in env vars.
Serverless by construction. A static host can’t be breached for model access—there is no model endpoint.
Sandboxed execution. The browser isolates the WASM/WebGPU runtime from the rest of the system.

The tradeoffs are equally real: model weights must be served (typically 1–3GB cached client-side), performance depends on the user’s GPU, and very large context windows may strain device memory.

Constrained Decoding for Tool Calls

Even in the browser, agentic patterns matter. LiteRT-LM’s constrained decoding gives you structured output for reliable tool calls:

const response = await session.send('Add a meeting at 9am', {
  responseFormat: {
    type: 'json_schema',
    schema: {
      type: 'object',
      properties: {
        tool: { type: 'string' },
        args: { type: 'object' },
      },
      required: ['tool'],
    },
  },
});

Guaranteed-valid JSON means your client-side agent loop can trust the model’s structured output instead of praying the parser doesn’t throw.

Performance Tips

To get the most out of the browser runtime:

Prefer WebGPU over WASM when available—expect up to 76 tokens/sec decode on a MacBook Pro.
Enable multi-token prediction for a ~2.2x throughput boost.
Persist sessions to skip prefill on return visits.
Warm the model during an idle moment (after first paint) to hide load time.
Consider a CPU fallback via the wasm backend for machines without WebGPU.

Putting It All Together

You now have the complete recipe for browser-native AI:

Serve model weights from static storage (or a CDN) alongside your app.
Instantiate LiteRtLmWeb with webgpu and multi-token prediction.
Stream responses token-by-token into the UI.
Persist sessions in IndexedDB to avoid re-prefill.
Use constrained decoding for agentic tool calls.

No backend. No API bills. No privacy compromises. Just an LLM running where the user is.

Conclusion & Next Steps

LiteRT-LM brings state-of-the-art on-device LLM inference to the open web. You’ve learned how to run Gemma 4 entirely in the browser with WebGPU and WebAssembly, stream responses, manage sessions, and build serverless AI apps.

Next steps:

Compare webgpu vs wasm backends on your target hardware.
Build an agentic web app with client-side tool calling and constrained decoding.
Add offline support with a service worker caching the model weights.
Explore MediaPipe integration for browser-based vision and audio alongside the LLM.

Building for the AI web has never been more aligned with privacy and cost. The server is optional—and so is the cloud.

References

webgpu webassembly litert-lm browser-ai on-device-ai gemma