ONNX Runtime Web: Running Models in a Flutter Web App

ONNX is the export format that ends the framework argument: PyTorch, TensorFlow, scikit-learn — export once to .onnx, and onnxruntime-web runs it in the browser. For a Flutter web app that means one JS dependency, one session created at startup, and a thin dart:js_interop layer. This post walks the path end to end, from install to reading output tensors in Dart.

Install and create the session

Add the package with npm, because onnxruntime-web ships JS plus .wasm binaries that your build must copy:

npm install onnxruntime-web

Session creation is where most setup lives. Point wasmPaths at the bundled .wasm files and decide the thread count before the first session exists:

// web/onnx.js
import * as ort from 'onnxruntime-web';

ort.env.wasm.wasmPaths = '/ort/';           // where the .wasm files were copied
ort.env.wasm.numThreads = self.crossOriginIsolated ? 4 : 1;

const session = await ort.InferenceSession.create('/models/classifier.ort', {
  executionProviders: ['webgpu', 'wasm'],
});

Create the session once and reuse it — initialization is expensive, run() is the hot path. Two rules on wasmPaths: the .wasm files must come from the same build version as the JS bundle (a mismatch fails to initialize), and numThreads above 1 only engages when the page is cross-origin isolated (COOP and COEP headers); otherwise ONNX Runtime falls back to single-threaded execution on its own, and forcing 1 just makes that explicit.

Prefer the ORT format for the web

ONNX files work, but the ORT format exists for size-constrained targets like web: it is optimized at conversion time, parses faster, and peaks at lower memory. Convert with the script shipped in the Python package:

pip install onnxruntime
python -m onnxruntime.tools.convert_onnx_models_to_ort models/classifier.onnx
# models/classifier.ort + required_operators.config

The .ort extension is enough — InferenceSession.create infers the format from the file name or from a marker in the bytes, so no extra option is needed.

Choose execution providers deliberately

executionProviders is an ordered list, and ONNX Runtime walks it until a provider accepts an op. The four web providers are wasm (the CPU baseline), webgl, webgpu, and webnn. WASM supports every ONNX operator; the GPU providers only cover a subset, so always keep wasm last in the list as the safety net:

async function pickProviders() {
  if (navigator.gpu && (await navigator.gpu.requestAdapter())) {
    return ['webgpu', 'wasm'];
  }
  return ['webgl', 'wasm'];
}

The requestAdapter() check matters: navigator.gpu existing does not guarantee an adapter, and a WebGPU session that fails at creation should degrade rather than break the feature. One more trade-off: ort.env.wasm.proxy = true offloads compute to a Web Worker and keeps the UI responsive, but it cannot be combined with the WebGPU EP (GPU buffers are not transferable) and it does not work under a strict CSP.

Feed tensors and read outputs

Input names come from the model, not from you — check session.inputNames once and build tensors to match, with the exact dims:

const tensor = new ort.Tensor('float32', inputFloat32Array, [1, 3, 224, 224]);
const results = await session.run({ input: tensor });
const probs = results.output.data;   // Float32Array, read-only view
console.log(session.outputNames, probs[0]);

run() resolves with an object keyed by output name; results.output.data is typed-array backed, so copying into a Dart Float32List is a straight sublistView. Preprocessing — resize, normalize, NCHW layout — belongs on the JS side next to the tensor, exactly as in the TensorFlow Lite post.

Reach it from Dart

Inference is headless, so no platform view is involved: one bridge global, one async call, typed arrays crossing the boundary:

import 'dart:js_interop';
import 'dart:js_interop_unsafe';
import 'package:web/web.dart' as web;

Future<Float32List> classify(Float32List pixels, List<int> dims) async {
  final bridge = web.window.getProperty('onnxBridge'.toJS)! as JSObject;
  final pending =
      bridge.callMethod('run'.toJS, pixels.toJS, dims.toJS)! as JSPromise;
  final output = await pending.toDart as JSFloat32Array;
  return output.toDart;
}

From here the rest of the ML inference post applies unchanged: quantize the model, cache the .ort file under a versioned URL with a long-lived Cache-Control, warm it up after load, and move repeated calls into a worker when latency shows up in your frame timings.

Takeaway: Export to ONNX, convert to the .ort format, create one session with an ordered provider list ending in wasm, and drive session.run from Dart — detection and fallback decide between WebGPU speed and WASM compatibility.

Sources

• https://onnxruntime.ai/docs/tutorials/web/
• https://onnxruntime.ai/docs/api/js/
• https://onnxruntime.ai/docs/performance/model-optimizations/ort-format-models.html
• https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API