Skip to content
Blog

Choosing the Right Model Size for Each Subtask

A routing strategy for agent pipelines: classify subtasks by reasoning depth, measure quality-per-dollar with per-tier evals, and cascade from small models to frontier models only when confidence is low.

Published on • November 16, 2026

AI Assistant

Most teams wire the same frontier model into every step of their agent: intent classification, field extraction, one-line summaries — all routed to the most expensive brain on the shelf. The bill says $0.008 per classification; a smaller tier does the same job for a fraction of that, usually faster. The hard part isn’t finding cheaper models. It’s knowing which steps can downgrade without quality falling off a cliff.

In this post, you will learn how to build a routing strategy on the Gemini API: classify subtasks by reasoning depth, measure quality-per-dollar with an eval harness per task tier, respect latency budgets, and implement a cascade that starts on a small model and escalates to a larger one only when confidence is low — including a decision table and a runnable Python router built with the google-genai SDK.

Prerequisites

  • A Google AI Studio API key (GEMINI_API_KEY) with paid-tier access.
  • Python 3.10+ and pip install google-genai pydantic. The Interactions API surface used below needs google-genai ≥ 2.3.0.
  • A labeled eval set per task tier — even 30–50 examples per tier is enough to start.
  • Familiarity with JSON-schema-constrained outputs.

Classify Subtasks by Reasoning Depth

Before choosing a model, choose a tier for the task. Sort every step in your pipeline by how much reasoning it actually requires:

  1. Mechanical — classification, extraction, summarization, entity linking, routing decisions. Pattern matching with a short answer. Minimal or zero thinking.
  2. Tactical — planning the next tool call, drafting a response under constraints, answering questions that need moderate context synthesis. Some thinking, bounded depth.
  3. Strategic — hard debugging, architecture trade-offs, multi-step proofs, long-horizon agent decisions. Deep reasoning, long decodes, high stakes.

The Gemini model catalog maps cleanly onto these tiers, and the pricing page makes the spread concrete (standard tier, per 1M tokens, November 2026):

TierSubtask examplesDefault modelInput / OutputWhy this rung
Mechanicalintent classification, field extraction, chunk summariesgemini-3.5-flash-lite$0.30 / $2.50Cost-efficient, optimized for high-volume agentic tasks
Tacticalplan next tool call, constrained draft, moderate QAgemini-3.8-flash$0.75 / $3.75”Most intelligent Flash” — agentic and enterprise workflows
Strategichard debug, design plan, multi-step reasoninggemini-4-hydrogen$1.50 / $7.50Efficient frontier: speed and efficiency for everyday agentic work
Frontierrare, high-stakes, failure-is-expensivegemini-4-carbon$2.00 / $10.00Most capable reasoning model; not on the free tier

Two notes on the table. First, prices move — Carbon doubles to $4/$20 on December 1, 2026, and the 3.8 Flash line moves January 1, 2027, so treat these numbers as the shape of the ladder, not a contract. Second, gemini-4-argon sits alongside Carbon as an “efficient frontier” option with lower token spend per turn; if your strategic tier rarely needs Carbon’s ceiling, Argon at the same $2/$10 price may deliver better throughput per dollar on your specific tasks — that’s exactly what the eval harness below exists to decide.

Pair each tier with a thinking level. The Gemini docs expose configurable thinking levels — minimal, low, medium, high, xhigh — so a mechanical task should never pay for a high reasoning pass it doesn’t use.

Measure Quality-Per-Dollar With Per-Tier Evals

Never compare models globally. Compare them within a tier, on that tier’s data, and score them on cost per correct answer — the only metric that combines quality and price:

# eval_tiers.py — per-tier quality-per-dollar harness.
import json
from google import genai

client = genai.Client()  # reads GEMINI_API_KEY

PRICES = {  # (input, output) USD per 1M tokens, standard tier, Nov 2026
    "gemini-3.5-flash-lite": (0.30, 2.50),
    "gemini-3.8-flash": (0.75, 3.75),
    "gemini-4-hydrogen": (1.50, 7.50),
    "gemini-4-carbon": (2.00, 10.00),  # becomes 4.00 / 20.00 on Dec 1, 2026
}


def evaluate(model: str, dataset: list[dict]) -> dict:
    """Run every example on one model; return accuracy and cost metrics."""
    in_price, out_price = PRICES[model]
    correct = spent = 0.0
    for ex in dataset:
        resp = client.models.generate_content(
            model=model,
            contents=ex["prompt"],
            config={"response_mime_type": "application/json",
                    "response_schema": ex["schema"]},
        )
        usage = resp.usage_metadata
        spent += (usage.prompt_token_count * in_price
                  + usage.candidates_token_count * out_price) / 1e6
        prediction = json.loads(resp.text)
        correct += int(prediction == ex["expected"])
    correct = int(correct)
    return {
        "model": model,
        "accuracy": correct / len(dataset),
        "avg_cost": spent / len(dataset),
        "cost_per_correct": spent / correct if correct else float("inf"),
    }


for candidate in ("gemini-3.5-flash-lite", "gemini-3.8-flash", "gemini-4-hydrogen"):
    print(evaluate(candidate, MECHANICAL_DATASET))

What it does: each labeled example carries its own JSON schema, so every candidate is judged on identical, machine-checkable outputs. usage_metadata gives real token counts — no estimates. What to look for: the cheapest model whose accuracy clears your bar for that tier. A typical result reads “flash-lite: 94.2% at $0.00004 per case; hydrogen: 96.1% at $0.00061” — the small model buys 94% of the quality at 6% of the cost, and the 1.9-point gap goes to escalation rather than a blanket upgrade.

The Router: Small Model First, Escalate on Confidence

With tier assignments validated, the router is a lookup plus a cascade. Start at the tier your classifier assigns; if the response is low-confidence or malformed, step up one rung and try again:

# router.py — depth-tiered routing with confidence cascade.
import json
import time
from enum import Enum
from google import genai

from eval_tiers import PRICES  # shared (input, output) price table

client = genai.Client()


class Depth(str, Enum):
    MECHANICAL = "mechanical"
    TACTICAL = "tactical"
    STRATEGIC = "strategic"


ROUTES = {
    Depth.MECHANICAL: {"model": "gemini-3.5-flash-lite", "thinking": "minimal", "min_conf": 0.85},
    Depth.TACTICAL:   {"model": "gemini-3.8-flash",      "thinking": "low",      "min_conf": 0.80},
    Depth.STRATEGIC:  {"model": "gemini-4-hydrogen",     "thinking": "high",     "min_conf": 0.0},
}
LADDER = [Depth.MECHANICAL, Depth.TACTICAL, Depth.STRATEGIC]

ANSWER_SCHEMA = {
    "type": "object",
    "properties": {
        "answer": {"type": "string"},
        "confidence": {"type": "number"},
    },
    "required": ["answer", "confidence"],
}


def route(task: str, depth: Depth) -> dict:
    """Run the assigned tier, then escalate on low confidence or bad JSON."""
    result = None
    for rung in LADDER[LADDER.index(depth):]:
        spec = ROUTES[rung]
        started = time.perf_counter()
        resp = client.models.generate_content(
            model=spec["model"],
            contents=task,
            config={
                "response_mime_type": "application/json",
                "response_schema": ANSWER_SCHEMA,
                "thinking_config": {"thinking_level": spec["thinking"]},
            },
        )
        latency = time.perf_counter() - started
        usage = resp.usage_metadata
        cost = (usage.prompt_token_count * PRICES[spec["model"]][0]
                + usage.candidates_token_count * PRICES[spec["model"]][1]) / 1e6
        try:
            data = json.loads(resp.text)
        except json.JSONDecodeError:
            data = None  # schema violation also escalates
        result = {"answer": data and data["answer"], "confidence": data and data["confidence"],
                  "model": spec["model"], "escalated": rung != depth,
                  "latency_s": round(latency, 2), "cost_usd": round(cost, 6)}
        if data is not None and data["confidence"] >= spec["min_conf"]:
            return result
    return result  # frontier rung already consumed — return its best effort

Three design decisions are doing the heavy lifting. Confidence thresholds differ per tier — mechanical tasks need a high bar because they’re cheap to escalate; the strategic rung has no rung above it, so its threshold is 0 and it returns whatever it produced. Schema violations escalate instead of crashing: a malformed JSON answer is the small model telling you it’s out of its depth. Escalation is capped at the ladder’s end — an agent that can retry forever is an agent that can bill forever. Log every escalated: True result; the escalation rate per tier is your ongoing signal for whether a threshold is calibrated or merely comfortable.

Latency: The Other Half of the Equation

Right-sizing isn’t only about dollars. Every escalation adds a second round trip, and thinking levels trade latency for depth. If the first rung takes T_s, the larger rung takes T_L, and the escalation rate is e, expected latency is:

E[T] = T_s + e * T_L

Plug in realistic numbers: T_s = 0.4s, T_L = 3.0s, e = 0.12 gives 0.4 + 0.36 = 0.76s — versus 3.0s for always calling the large model. You get better latency than the frontier-only baseline while paying full price only on the 12% of traffic that needed it. That’s the rare engineering trade where both sides win, provided you keep e honest: an escalation rate that drifts to 50% means your thresholds are wrong or your tier assignment is wrong, and both are fixable from the eval data.

Interactive paths should also avoid high thinking on the tactical rung — the docs describe thinking levels from low-latency low to deep xhigh, and the wrong level on a chat path will blow your p95 even when the model tier is right.

Putting It All Together

A support triage pipeline wires the pieces into four steps: classify the ticket (gemini-3.5-flash-lite, mechanical), extract structured fields (gemini-3.5-flash-lite), draft a resolution plan (gemini-3.8-flash, tactical), and route hard failures to gemini-4-hydrogen only when the plan’s confidence falls below 0.80. Each night, the eval harness re-runs every tier’s dataset and prints the cost-per-correct table; each request logs its tier, model, escalation flag, latency, and cost. A day of traffic looks like this:

task                       depth        model                   escalated  latency  cost_usd
classify_intent            mechanical   gemini-3.5-flash-lite    no         0.31     0.00004
extract_order_fields       mechanical   gemini-3.5-flash-lite    no         0.28     0.00003
draft_refund_reply         tactical     gemini-3.8-flash         no         0.92     0.00021
diagnose_flaky_test        strategic    gemini-4-hydrogen        yes        4.10     0.00610
plan_migration_strategy    strategic    gemini-4-hydrogen        no         5.72     0.00880

The escalated column is the one to watch: it tells you how much work the cascade is quietly redoing, and it’s the input to next week’s threshold tuning.

Conclusion and Next Steps

You now have a complete right-sizing loop: a depth-based tiering of subtasks, a decision table grounded in the current Gemini lineup and pricing, an eval harness that scores candidates on cost per correct answer, and a router that starts small and escalates on low confidence or malformed output — with the latency math to prove the cascade pays for itself.

Homework, in order of return:

  • Build the eval harness first, even before the router — you cannot right-size what you haven’t measured.
  • Log escalation rates per tier and alert when they cross 20%; recalibrate thresholds against the eval set, not vibes.
  • Re-run evals on a schedule and on deprecation news — gemini-2.5-flash auto-routes to gemini-3.5-flash-lite on December 1, 2026, and today’s model table will look different next quarter.
  • Add latency p95 per tier to the harness so quality-per-dollar never silently becomes quality-per-dollar-per-ten-seconds.
  • For offline subtasks (batch enrichment, nightly evals), move that tier to the Gemini Batch API and halve the token cost again.