Agents in VR/AR: Navigating 3D Spaces with Gemini 3 Native Spatial Reasoning

For most of the agent era, AI agents lived in a two-dimensional world. They read text, clicked buttons, and reasoned over files — but ask one to “grab the blue mug on the shelf to your left” and it would freeze. Spatial reasoning is the missing ingredient: the ability to understand objects in three-dimensional space, their positions, orientations, and relationships to each other.

With Gemini 3’s native spatial understanding, agents can now point to objects, track them over time, generate movement trajectories, and act inside immersive environments. In this tutorial, you’ll learn how to build an agent that perceives a 3D scene through the Gemini API, grounds its understanding with bounding boxes and trajectories, and controls a WebXR scene in a closed perception-action loop.

Why Spatial Reasoning Is Hard for AI

Text-only models treat space as a vocabulary problem. Ask one to describe a room and it produces plausible prose — but the moment it has to act on that room (reach an object, avoid an obstacle, follow a path), prose fails. The geometry is missing.

Spatial reasoning enables AI to understand the physical world and interact with it. — Google AI for Developers, Spatial reasoning (https://ai.google.dev/gemini-api/docs/robotics-spatial)

The breakthrough is that Gemini 3 reasons about space directly from multimodal input — video, depth, and 3D coordinates — rather than guessing from text descriptions.

The Perception-Action Loop for 3D Agents

A spatial agent runs a tight cycle:

flowchart LR
    A["Capture scene<br/>(video / depth / mesh)"] --> B["Perceive<br/>(Gemini 3 multimodal)"]
    B --> C["Ground<br/>(points, boxes, trajectories)"]
    C --> D["Plan action<br/>(agent reasoning)"]
    D --> E["Execute in XR<br/>(WebXR / Unity / Godot)"]
    E --> A

Each iteration tightens the loop. This is the same ReAct-style cycle used by text agents, but the observations and actions are now spatial.

Step 1 — Send spatial observations to Gemini

Gemini Robotics ER models (and Gemini 3’s multimodal API) accept video and images directly. To let the model “see” the space, send frames or a video clip and ask for grounded outputs:

from google import genai

client = genai.Client()

prompt = (
    "You are looking through the camera of an AR agent in a room. "
    "Find the red mug on the table. Return its 2D bounding box "
    "in image coordinates and the 3D direction the agent should reach toward."
)

response = client.models.generate_content(
    model="gemini-3-flash-preview",
    contents=[prompt, video_bytes],
    config={"response_mime_type": "application/json"},
)

Step 2 — Ground with bounding boxes and points

Raw prose isn’t actionable. Ask the model to return structured spatial primitives:

• Points — (x, y) locations the agent can point at or reach toward.
• Bounding boxes — (x_min, y_min, x_max, y_max) for object detection and grasping targets.
• Trajectories — ordered sequences of points that define a path for movement or a robot arm.

{
  "objects": [
    {
      "label": "red_mug",
      "bbox": [320, 210, 390, 300],
      "center": [355, 255],
      "reach_direction": [0.2, -0.1, 0.9]
    }
  ]
}

The closed-loop part matters: the agent serves its controller onto the detected target, then re-captures and re-verifies, iterating until the alignment error is below a threshold.

Step 3 — Track objects over time

Static detection isn’t enough for a moving world. Gemini Robotics ER 2 can analyze video frames and track objects over time, which is what turns a one-shot locator into a persistent navigator.

Gemini Robotics ER models can point to objects, track them in video, detect them with bounding boxes, and generate movement trajectories. — Gemini API Spatial reasoning docs (https://ai.google.dev/gemini-api/docs/robotics-spatial)

Step 4 — Act through WebXR

Now wire perception to a real 3D runtime. A WebXR scene gives you a headset pose and lets you place or move objects relative to the user. The agent decides what to do based on the scene graph, then calls a tool to mutate it.

// agent-tools.js — tools the Gemini agent can call
const xrTools = {
  locateObject: async (label) => { /* query scene graph */ },
  reachToObject: async (objectId) => { /* servo ray onto object */ },
  grabObject: async (objectId) => { /* perform grab gesture */ },
  moveObject: async (objectId, position) => { /* place in 3D */ },
};

Register these tools with the Gemini API and the agent orchestrates them — locate, reach, grab, place — exactly as a text agent orchestrates file tools.

Grounding Instead of Guessing

The most important lesson from real spatial-agent projects: don’t make the model reason about space through raw coordinates. When one team tried asking Gemini to place assets at precise 3D coordinates from text alone, overlaps, wrong distances, and nonsense scale ratios resulted. The fix was to let the model see the result and judge it visually.

Text is terrible for spatial evaluation. Switching from “describe what’s wrong with the layout” to “here are three camera angles, what do you see?” changed everything. — elsewhere, a text-to-3D studio (https://dev.to/rawr/elsewhere-a-text-to-3d-studio-3bif)

Adopt the same pattern:

1. Render the current state of your scene.
2. Capture screenshots from multiple angles.
3. Send them to Gemini and ask for spatial feedback.
4. Apply corrections and repeat until scores plateau.

Putting It All Together

A complete spatial agent ties these pieces into one loop:

• Perception: Gemini 3 turns video frames into object detections and bounding boxes.
• Grounding: structured outputs turn detections into actionable coordinates.
• Action: WebXR tools execute reaches, grabs, and placements.
• Verification: a second multimodal pass confirms the action succeeded.

Google’s Vibe Coding XR project demonstrates this at scale: Gemini acts as an expert XR designer, translating natural-language prompts into physics-aware Android XR apps in under a minute using the open-source XR Blocks framework (https://research.google/blog/vibe-coding-xr-accelerating-ai-xr-prototyping-with-xr-blocks-and-gemini/). Community builds go further — agents that type on virtual keyboards by grounding characters visually, navigate for blind users with 360° scans, and visually servo controller rays onto objects with sub-pixel precision.

Conclusion & Next Steps

You’ve learned how spatial reasoning lets Gemini 3 agents perceive, ground, and act in 3D space: sending video observations, requesting structured spatial primitives, tracking objects over time, and executing through WebXR tools.

To go further:

• Add depth: stream depth data so the agent understands occlusion and distance, not just 2D positions.
• Long-horizon tasks: let the agent plan a multi-step spatial sequence (“set the table”) rather than single reaches.
• Evaluation: build a benchmark with a fixed scene and scored outcomes, the way GameDevBench scores agentic game-development tasks.

Spatial reasoning turns agents from text processors into inhabitants of the 3D web. The loop — perceive, ground, act, verify — is the same one robotics has used for decades. Now a single multimodal model provides the perception, and your XR runtime provides the hands.