Agentic Game Design: Procedural World Building with Gemini 3 Multimodal Guidance

A game world is not a text document. It’s a 3D scene where geometry, lighting, placement, and physics must agree — and that’s exactly where most AI code-generation pipelines break down. Leaky design doc? Fixable. A castle that clips through the terrain? Not so much.

Agentic game design solves this by treating the LLM as a multimodal director: the model plans the world in text, then looks at the rendered result and corrects its own spatial mistakes. With Gemini 3’s vision and spatial reasoning, an agent can go from a prompt to a playable, coherent 3D world.

Why a Schema Compiler Beats Raw Code Generation

When you ask an LLM to emit Three.js code for a 3D object, the results are wildly inconsistent — invalid coordinates, broken hierarchies, deprecated APIs. The reliable pattern flips the responsibility: the model describes geometry in a constrained JSON vocabulary; a deterministic compiler does the wiring.

The LLM never writes executable code directly; it describes what it wants in a constrained format, and deterministic code handles the how. — elsewhere, a text-to-3D studio (https://dev.to/rawr/elsewhere-a-text-to-3d-studio-3bif)

{
  "name": "castle",
  "materials": [
    { "color": [0.6, 0.55, 0.5], "roughness": 0.8, "metalness": 0.1 }
  ],
  "parts": [
    {
      "geometry": "Box",
      "parent": null,
      "position": [0, 2, 0],
      "scale": [6, 4, 6]
    },
    {
      "geometry": "Cylinder",
      "parent": "tower1",
      "priority": 2
    }
  ]
}

The compiler parses, validates, budgets (pruning parts when the scene exceeds a triangle or draw-call budget), auto-snaps disconnected pieces to parents, and emits the final scene graph. Failure modes become predictable and fixable, because the model’s output space is constrained.

The Generate, See, Correct Loop

Planning a world via text and guessing coordinates fails. The breakthrough is turning composition into a vision feedback loop:

1. Gemini plans the full scene through text.
2. Deterministic code builds every asset and places them.
3. The system captures screenshots from multiple angles (overview, ground level, top-down).
4. It packs those images with a prompt and asks Gemini to score the layout and list specific problems (“the fountain clips through the stall,” “the trees are all the same scale”).
5. Gemini proposes fixes; the system applies them and re-evaluates, looping up to five times or until scores plateau.

from google import genai

def evaluate_scene(client, screenshots):
    parts = [
        ("You are a 3D layout critic. Return a JSON score "
         "and a list of specific spatial problems from these three camera angles."),
    ]
    for img in screenshots:
        parts.append(img)
    return client.models.generate_content(
        model="gemini-3-flash-preview",
        contents=parts,
        config={"response_mime_type": "application/json"},
    )

Each camera angle catches a different class of bug — the overview exposes composition gaps, the ground view catches scale mismatches, and the top-down reveals clipping and overlap. The model becomes its own art director.

System Instructions Give One Model Many Roles

A single gemini-3-flash-preview instance can play four different roles in one generation run just by swapping systemInstruction:

• Asset designer — turns a prompt into a geometry schema.
• Spatial planner — arranges assets across a terrain grid.
• Visual critic — evaluates the screenshot set.
• Layout editor — translates criticism into concrete fixes.

This is far more effective than one giant prompt trying to do everything at once: each system instruction constrains output format and evaluation criteria for that specific phase.

Orchestrating the World-Building Pipeline

A production pipeline orchestrates the multimodal steps:

flowchart LR
    A["Story / prompt"] --> B["Narrative spec"]
    B --> C["World plan & graph"]
    C --> D["Asset schema<br/>(Gemini 3)"]
    D --> E["Schema compiler"]
    E --> F["3D scene"]
    F --> G["Screenshots"]
    G --> H["Vision eval"]
    H --> I["Apply fixes"]
    I --> G
    H --> J["Sky / audio / NPC logic"]
    J --> K["Export game bundle"]

Community projects show the full pattern end-to-end. sanchitsingh001/Gemini_Live_Agent_Challenge turns a plain-text story into a structured narrative spec and world graph, then builds a Godot 3D world with generated sky, voiceover, and music on Google Cloud (https://github.com/sanchitsingh001/Gemini_Live_Agent_Challenge). Separate services handle narrative generation, world compilation, and a live game server for NPC dialogue.

Thinking Budget Matters

For well-constrained schema output, turn thinking off. A scene generation run makes 10–15 Gemini calls across planning, asset creation, evaluation, and refinement; disabling extended reasoning on the structured-output calls keeps the whole loop interactive.

For structured JSON output, Flash is fast, cheap, and proved to be very reliable. Disabling thinking for structured-output calls and keeping it for vision evaluation cut per-call latency without sacrificing quality. — elsewhere dev blog (https://dev.to/rawr/elsewhere-a-text-to-3d-studio-3bif)

Putting It All Together

The reusable recipe for agentic world building:

1. Constrained output — the model emits JSON schema, never raw engine code.
2. Deterministic compiler — handle geometry, transforms, and hierarchy in code.
3. Vision feedback — render, screenshot from multiple angles, and let the model critique its own work.
4. Role-per-instruction — use system prompts to split planning, building, and evaluating.
5. Budget your thinking — cheap for structured output, expensive where reasoning matters.

This is the same recipe behind GameDevBench, which evaluates agents that solve real game-development tasks inside an engine (https://arxiv.org/abs/2602.11103), and it’s why treating the agent as a dev-team member rather than an autocomplete produces coherent, playable results.

Conclusion & Next Steps

You’ve learned to build an agentic game-design pipeline: plan with text, build with a schema compiler, and correct with the generate-see-iterate vision loop, all powered by Gemini 3’s multimodal sense.

To go further:

• Preserve narrative coherence — let the vision loop also judge mood and density so a “battlefield aftermath” reads differently from a “peaceful garden.”
• Add NPC behaviors — script branching dialogue and utility-based AI for the world’s inhabitants.
• Build a flywheel — share worlds so the community can fork and extend each other’s creations.

Procedural worlds are becoming a conversation. Type a theme, watch the agent lay out the terrain, place the castle, and then — by looking at its own render — fix what it got wrong. That generate-see-correct loop is what makes agentic game design real.