MLOps for LLMs: Versioning Prompts, Data, and Weights

LLM applications drift for reasons that have nothing to do with weights: someone edits a production prompt, the retrieval corpus gets re-synced, or a fine-tune silently changes behavior. When an incident strikes, “what changed?” should have a one-command answer, not an archaeology dig through chat history. That is what MLOps gives you: versioned prompts, versioned data, and versioned models, all tracked against the metrics that decide whether a change is an improvement. — “Track prompt versions, maintain cost dashboards, log every call, trace workflows, and keep the system observable. You cannot manage what you cannot measure.”

MLflow is the canonical open-source answer — a tracking server plus a model registry. We’ll version each of the three artifacts the brief names: prompts, datasets, and model weights.

Prerequisites

• pip install mlflow google-genai pandas and a running mlflow tracking server (mlflow server or local file: backend).
• MLflow logs runs with mlflow.start_run(), params with log_params(), metrics with log_metric(s), and models with log_model(registered_model_name=...). The Model Registry versions registered models and supports aliases; load via models:/<name>/<version> or an alias.

Step 1: Version the prompt

Treat a prompt like code: immutable, tagged, hashed. Store it as a file and log its digest + content as an artifact so the version is unmistakable later.

import mlflow, hashlib

PROMPTS = {
    "triage": {
        "text": "You are a support triage agent. Return JSON with severity and owner.",
        "model": "gemini-2.5-flash",
        "temperature": 0.2,
    }
}

def version_prompt(name: str) -> str:
    return hashlib.sha256(PROMPTS[name]["text"].encode()).hexdigest()

with mlflow.start_run(run_name=f"prompt:triage:v1"):
    mlflow.log_params(PROMPTS["triage"])                 # text + var params
    mlflow.log_param("prompt_sha256", version_prompt("triage"))
    mlflow.log_text(PROMPTS["triage"]["text"], "prompt.txt")  # artifact

Step 2: Version the data (evaluations + fine-tune corpus)

Your eval set is an artifact. If it changes and scores go up, is the model better — or the set easier? Log the dataset hash and the metrics against it:

def eval_and_log(prompt_key, dataset_uri, dataset_hash, judge):
    scores = run_offline_eval(prompt_key, dataset_uri)   # faithfulness, answer_relevancy...
    with mlflow.start_run(run_name=f"eval:{prompt_key}") as run:
        mlflow.log_params({
            "dataset_hash": dataset_hash,                # compare apples-to-apples
            "dataset_uri": dataset_uri,
            "prompt_key": prompt_key,
            "prompt_sha": checkpoint_version(prompt_key),
        })
        mlflow.log_metrics(scores)                       # faithfulness, hit_rate...
        mlflow.log_artifact("eval_results.csv")

Recording the dataset_hash alongside scores is the “knowledge, not tools” discipline.

Step 3: Version the model / weights

Fine-tunes and weight snapshots go through the Model Registry, which version-increments and tracks lineage:

import mlflow

model_info = mlflow.pyfunc.log_model(
    artifact_path="llm-app",
    python_model=MyPromptWrapper(prompt_version="v1"),   # wraps model + prompt together
    registered_model_name="triage-agent",
)
mlflow.set_registered_model_alias("triage-agent", "champion", model_info.model_version)

You can later load by alias (champion/candidate) rather than a hard version, which cleanly supports canary or A/B rollout.

Step 4: Compare runs and promote

df = mlflow.search_runs(experiment_names=["triage-evals"], order_by=["metrics.answer_relevancy DESC"])
print(df[["run_id", "params.dataset_hash", "params.prompt_sha",
          "metrics.faithfulness", "metrics.answer_relevancy"]])

# Promotion rule: candidate beats champion on the KPI *and* shows cost within budget.
if candidate.relevance >= champion.relevance and candidate.cost <= champion.cost:
    mlflow.set_registered_model_alias("triage-agent", "champion", candidate.model_version)

Putting It All Together: the three-part commit

prompt v17 (text + params)  ──logged──┐
dataset v9  (hash + metrics)         ├──▶ run_id ──▶ registered model "triage-agent"
model v2   (weights/wrapper)───────┘         (alias: champion)

Any production issue reads as: “champion = model v2 + prompt v17 + eval-on-dataset v9.” Roll it back by shifting the alias.

MLOps for the brief’s “knowledge before tools” framing

• Version the corpus: retrieval data is knowledge; it must be versioned or you cannot audit claims.
• Track cost per run: log usage_metadata totals aligned with metrics.
• Human oversight: promotions to champion require an accountable human reviewer — the registry’s alias mechanics build the “who approved” audit trail.

Conclusion & Next Steps

MLOps for LLMs is version persistence for the three things that change: prompt, data, and weights. Next: land a remote tracking server + object-store artifacts; wire the RAG-eval harness to log into the registry on every corpus sync; and add a CI gate that refuses to promote a candidate that regresses against the held-out eval slice.

References / Sources

• MLflow docs. https://mlflow.org/docs
• MLflow Model Registry. https://mlflow.org/docs/latest/model-registry.html