Google Gemini API Computer Use: Build Agents That See, Reason, and Act

Google has officially launched Computer Use as a built-in tool in Gemini 3.5 Flash, making it easier than ever for developers to build AI agents that can see, reason about, and interact with computer screens. Previously available only as a standalone Gemini 2.5 preview model, computer use is now natively integrated into Google’s main Flash model — opening the door to powerful browser, mobile, and desktop automation.

What is Gemini Computer Use?

Computer Use lets developers build agents that interact with computer interfaces the same way a human would — by looking at screenshots and performing actions like clicks, typing, and scrolling.

The core loop works like this:

1. Your application captures a screenshot of the screen
2. The model analyzes the screenshot and the user’s prompt
3. The model returns a function_call with a suggested UI action (e.g., “click at coordinate (x, y)”)
4. Your client-side code executes the action using automation tools like Playwright
5. A new screenshot is captured and fed back to the model

Each action also includes an intent field explaining the model’s reasoning behind the step — making the agent’s decision-making process transparent.

Three Environments: Browser, Mobile, and Desktop

Gemini Computer Use supports three distinct environments:

Browser (ENVIRONMENT_BROWSER)

The default environment. Supports actions like:

• click, double_click, triple_click, right_click
• type — Type text with optional Enter key
• scroll — Scroll in any direction at a coordinate
• drag_and_drop — Drag items across the screen
• navigate, go_back, go_forward — Browser navigation
• press_key, hotkey — Keyboard shortcuts
• take_screenshot, wait

Mobile (ENVIRONMENT_MOBILE)

Android-optimized actions:

• open_app — Open an app by name
• click, type, go_back
• list_apps — List available applications
• long_press — Long press at coordinates

Desktop (ENVIRONMENT_DESKTOP)

Full desktop control with all mouse and keyboard actions, plus:

• mouse_down, mouse_up — Press and hold / release mouse
• move — Move cursor without clicking
• key_up — Release a pressed key

How to Get Started

1. Enable Computer Use in Your Code

from google import genai
from google.genai import types

client = genai.Client()

# Add computer use to the list of tools
generate_content_config = types.GenerateContentConfig(
    tools=[
        types.Tool(
            computer_use=types.ComputerUse(
                environment=types.Environment.ENVIRONMENT_BROWSER,
            )
        ),
    ]
)

2. Send a Prompt with a Screenshot

Include the user’s goal and an initial screenshot:

response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents=[
        types.Part.from_image(screenshot),
        "Click the login button"
    ],
    config=generate_content_config,
)

3. Execute the Action

Parse the response coordinates, scale them from the 1000x1000 normalized grid to your actual viewport, and execute using Playwright:

# Parse and execute
for action in response.function_calls:
    if action.name == "click":
        x = scale_x(action.args["x"])
        y = scale_y(action.args["y"])
        page.mouse.click(x, y)

4. Loop Until Task Complete

Capture a new screenshot after each action and send it back to continue the workflow.

Safety Built In

Gemini Computer Use includes several safety layers:

• Safety decisions — Each action is classified as regular, require_confirmation, or blocked
• Configurable safety policies — Fine-tune behavior with built-in policy categories and overrides
• Prompt injection detection — Opt-in feature (Gemini 3.5 Flash+) that scans screenshots for hidden adversarial instructions like “Ignore previous commands”

⚠️ As a Preview capability, Computer Use may contain errors. Google recommends supervising closely for important tasks and avoiding use for critical decisions or sensitive data.

Gemini Enterprise Agent Platform

For enterprise teams, Google Cloud offers Computer Use Sandboxes — secure, isolated browser environments provisioned as containerized containers. You can:

• API requests — Send navigation, click, and type commands directly
• CDP connection — Connect via Chrome DevTools Protocol using Playwright
• Live streaming — Monitor agent actions in real-time via VNC/noVNC

Use Cases

Computer Use unlocks automation for:

• Continuous software testing — Agents that test UIs across environments
• Knowledge work — Automating form filling, data entry, and report generation
• Customer support — Bots that navigate apps on behalf of users
• Research — Agents that browse, extract data, and synthesize findings

Availability

Computer Use in Gemini 3.5 Flash is available via the Gemini API and the Gemini Enterprise Agent Platform. Developers can start building today.

💡 Try it out in the demo environment hosted by Browserbase or dive into the reference implementation.