GenUI SurfaceController: Driving Dynamic UI from LLM Output in Flutter

Most AI-powered Flutter apps render a chat transcript: bubbles of text, maybe a Markdown body, the occasional card hand-written by the developer. The genui package (published by labs.flutter.dev, currently at 0.10.x on pub.dev) takes a different bet — the model returns structured UI instructions, and Flutter renders real widgets from them. The component that turns those instructions into live UI is the SurfaceController.

This post walks through what the SurfaceController actually does in the pipeline, how A2UI messages map onto widget state, and how to teach the model new widgets via a custom catalog.

The pipeline, end to end

genui is built around a small set of cooperating objects:

• Conversation — the facade and entry point. It owns the SurfaceController and a Transport, and orchestrates the whole loop.
• A2uiTransportAdapter — implements Transport, wrapping an A2uiParserTransformer that parses raw text chunks from an LLM stream into structured generation events.
• SurfaceController — the runtime engine. It manages the lifecycle of UI surfaces, applies incoming A2UI messages, and owns the DataModel.
• Catalog / CatalogItem — the vocabulary of widgets the model is allowed to emit, each with a name, a JSON schema for its data, and a builder function.
• Surface — the widget you place in your tree; it listens to the controller and rebuilds when its surface changes.

The interaction cycle runs like this: the user types a prompt and the app calls conversation.sendRequest(). The conversation triggers the transport’s onSend, where your code calls whichever LLM backend you’ve chosen and pipes response chunks into transport.addChunk(). The parser transformer turns those chunks into text events or A2UI message events. The SurfaceController handles each message, updating the DataModel and surface tree. Surface widgets rebuild automatically. When the user taps something, the controller captures the event and emits a ChatMessage on onSubmit, which the Conversation feeds back into a new request — closing the loop.

Wiring it up:

class _MyHomePageState extends State<MyHomePage> {
  late final SurfaceController _controller;
  late final A2uiTransportAdapter _transport;
  late final Conversation _conversation;

  @override
  void initState() {
    super.initState();

    // The controller gets a widget catalog up front — the model can only
    // emit widgets that exist in it.
    _controller = SurfaceController(
      catalogs: [CoreCatalogItems.asCatalog()],
    );

    _transport = A2uiTransportAdapter(onSend: _onSendToLLM);

    _conversation = Conversation(
      controller: _controller,
      transport: _transport,
    );

    _conversation.events.listen((event) {
      if (event is ConversationSurfaceAdded) {
        setState(() => _surfaceIds.add(event.surfaceId));
      } else if (event is ConversationSurfaceRemoved) {
        setState(() => _surfaceIds.remove(event.surfaceId));
      }
    });
  }

  Future<void> _onSendToLLM(ChatMessage message) async {
    // Backend-agnostic: call your AI client of choice and feed chunks
    // back into the transport.
    final responseStream = myLlmClient.streamGenerateContent(message);
    await for (final chunk in responseStream) {
      _transport.addChunk(chunk);
    }
  }
}

Rendering the surfaces the controller creates is straightforward — each surface ID from a ConversationSurfaceAdded event becomes a Surface in your widget tree:

return Surface(host: _conversation.host, surfaceId: id);

What the controller does with A2UI messages

The model doesn’t send Flutter widgets — it sends A2uiMessages, each an instruction to perform one of a small set of actions: createSurface, updateComponents, updateDataModel, or deleteSurface. The SurfaceController is the component that interprets these:

• createSurface allocates a new surface with a stable ID and an initial component tree. The conversation surfaces this as a ConversationSurfaceAdded event so your app can render it.
• updateComponents mutates the component tree of an existing surface — swapping widgets, changing their static properties.
• updateDataModel writes to the shared DataModel. This is where the reactive magic lives: widgets are bound to paths in the model rather than holding their own state, and when data at a path changes, only the widgets depending on that path rebuild.
• deleteSurface tears a surface down, raising ConversationSurfaceRemoved.

That data model is the architectural centerpiece. Instead of the model emitting a whole new tree every time something changes, it emits a data patch, and the controller updates the store. Input widgets like a TextField bound to /user/name write straight into the model; any other widget bound to the same path rebuilds. The result is a high-bandwidth, low-churn interaction loop between user, UI, and model — no full-tree regeneration per keystroke.

Building custom surfaces: extending the catalog

The shipped CoreCatalogItems gives you a starter vocabulary, but the real power is defining your own CatalogItems so the model can emit domain-specific widgets. Each item combines a name (the string the model references), a schema (via json_schema_builder), and a builder:

final _schema = S.object(
  properties: {
    'question': S.string(description: 'The question part of a riddle.'),
    'answer': S.string(description: 'The answer part of a riddle.'),
  },
  required: ['question', 'answer'],
);

final riddleCard = CatalogItem(
  name: 'RiddleCard',
  dataSchema: _schema,
  widgetBuilder: (context) {
    final questionNotifier = context.dataContext.subscribeToString(
      context.data['question'] as Map<String, Object?>?,
    );
    final answerNotifier = context.dataContext.subscribeToString(
      context.data['answer'] as Map<String, Object?>?,
    );

    return ValueListenableBuilder<String?>(
      valueListenable: questionNotifier,
      builder: (context, question, _) {
        return ValueListenableBuilder<String?>(
          valueListenable: answerNotifier,
          builder: (context, answer, _) {
            return Container(
              constraints: const BoxConstraints(maxWidth: 400),
              padding: const EdgeInsets.all(16),
              child: Column(
                crossAxisAlignment: CrossAxisAlignment.start,
                children: [
                  Text(question ?? ''),
                  const SizedBox(height: 8),
                  Text(answer ?? ''),
                ],
              ),
            );
          },
        );
      },
    );
  },
);

Register it by extending the default catalog, then point the controller at it:

_controller = SurfaceController(
  catalogs: [CoreCatalogItems.asCatalog().copyWith([riddleCard])],
);

Add json_schema_builder to your dependencies (flutter pub add json_schema_builder) to get the S schema builders.

Gotchas

The model has to be told about your widgets. Adding a CatalogItem isn’t enough — the agent only uses widgets it knows about. Use prompt engineering (one-shot or few-shot examples in the system instruction) that names the catalog item and shows when to emit it. This is the least automatic part of the whole system.

You own the LLM integration. genui is deliberately backend-agnostic; onSend is where you plug in firebase_ai, a custom HTTP client, or anything else. There’s no built-in provider.

Network entitlements on Apple platforms. For iOS/macOS builds, add com.apple.security.network.client with a true value to your entitlements files or outbound requests will fail.

Version churn. At 0.10.x, the API is still evolving — pin your version and check the changelog when upgrading.

Debugging the stream. Enable logging to watch app↔agent traffic: configureLogging(level: Level.ALL) and listen to the returned logger’s onRecord stream.

When to use it

Reach for genui when the UI genuinely can’t be predetermined — travel planners, dashboards assembled from natural-language queries, form-heavy agents. If your UI is a fixed set of screens with an LLM narrating between them, a conventional chat UI plus hand-built widgets will be simpler and more predictable. The SurfaceController earns its keep exactly when the shape of the UI is the model’s output.

Sources

• genui on pub.dev — package overview, core concepts, getting-started guide, and custom catalog tutorial (BSD-3-Clause, published by labs.flutter.dev).
• flutter/genui on GitHub — DESIGN.md and the simple_chat example referenced by the package README.