AgentOps: running AI agents in production
Shipping an agent is the start, not the end. AgentOps is the operational discipline that keeps agents reliable, affordable and accountable after launch.
By Diego Bianqui · Updated
Why agents need their own operations practice
Traditional software is deterministic: the same input gives the same output, and tests that pass today pass tomorrow. Agents are not. The same request can take a different path, call different tools and cost ten times more. Behaviour changes when a model is updated, a prompt is edited, a tool's API changes or the data behind a retrieval index drifts.
AgentOps applies the ideas of DevOps and MLOps to that reality: you can't prevent every failure in advance, so you make every run observable, measure quality continuously, release changes carefully, and respond fast when something goes wrong.
The AgentOps lifecycle
- Build — version prompts, tools, model choices and configuration together, so any run can be traced back to exactly what produced it.
- Evaluate — run a suite of representative tasks before every release and compare against the current version.
- Release — roll out gradually: shadow runs, a small share of traffic, then wider, with a fast rollback.
- Observe — trace every run in production: steps, tool calls, tokens, latency, cost and outcomes.
- Improve — turn failures and user feedback into new evaluation cases, so the same failure can't ship twice.
Tracing: the foundation
A trace records the full path of one agent run: the input, each model call with its context and output, each tool call with its arguments and result, each decision and the final outcome. Without traces you can't debug, evaluate or audit an agent. With them, most other AgentOps practices become possible.
OpenTelemetry has generative-AI semantic conventions for this kind of telemetry (still evolving), and most agent frameworks and observability tools can emit or ingest it. Using an open standard keeps your traces portable across vendors — the same argument as keeping your harness model-agnostic.
Evaluation: evals, not vibes
A deeper guide: evaluating agents and RAG with evals and RAGAS.
- Offline evals: a curated set of tasks with expected outcomes, run before every change. Start small — a few dozen real cases beat hundreds of synthetic ones.
- Graders: exact checks where possible (did the right record change?), model-based graders where needed, calibrated against human judgement on a sample.
- Online evals: score a sample of production runs continuously, and alert when quality drops.
- Safety evals: adversarial cases for prompt injection, tool misuse and data leakage — the risks in the OWASP Agentic Top 10.
- Regression discipline: every incident becomes a test case.
Metrics that matter
| Metric | Why it matters |
|---|---|
| Task success rate | The only metric the business cares about: did the agent do the job? |
| Escalation rate | How often the agent hands over to a human — too high wastes the investment, too low may hide risk. |
| Tool-call error rate | Failed or rejected tool calls point to bad tool design, permissions or context. |
| Cost per successful task | Tokens and tool costs divided by successes, not by runs. |
| Latency (p50 / p95) | Long tails frustrate users and often signal loops or retries. |
| Guardrail and approval triggers | How often policies block or pause actions — and whether those were true positives. |
| Steps per task | Rising step counts usually mean the agent is struggling or looping. |
Cost and latency control
- Budgets in the harness: maximum steps, tokens, time and spend per run, enforced in code.
- Model routing: a smaller, faster model for simple steps; the strongest one only where it pays off.
- Caching: stable prompt prefixes and repeated tool results, cached where your platform allows.
- Context discipline: shorter, better-selected context is cheaper and usually more accurate — see context engineering.
Incidents and accountability
Agents will misbehave. Plan for it: a documented way to stop an agent and revoke its credentials, an owner on call, traces kept long enough to investigate, and a post-incident review that adds tests and controls. This operational evidence is exactly what an AI management system under ISO/IEC 42001 or the NIST AI RMF expects to see.
Related guides
- Harness engineering: the system around the model
- Context engineering: what the model sees, and when
- Governing AI agents with ISO/IEC 42001 and the NIST AI RMF
Sources
Working on agents in production? Get in touch.