--- title: "Agent Evals Belong in Production, Not Just CI" description: "Google’s generally available agent-evaluation stack connects offline experiments, simulators, production traces, online monitors, and versioned metrics. The practical lesson is broader than one platform: an eval program should follow the agent through its full operating lifecycle." canonical: "https://www.symbaiex.com/blog/agent-evals-belong-in-production-not-just-ci" last-updated: "2026-08-24T00:00:00.000Z" --- # Agent Evals Belong in Production, Not Just CI > Google’s generally available agent-evaluation stack connects offline experiments, simulators, production traces, online monitors, and versioned metrics. The practical lesson is broader than one platform: an eval program should follow the agent through its full operating lifecycle. Published: 2026-08-24 Reading time: 7 minutes Tags: ai agents, evaluation, observability, production ai, reliability A pre-release evaluation suite can prove that an AI agent passed the cases a team thought to write. Production reveals the cases nobody anticipated. That gap is not a reason to abandon offline testing. It is a reason to connect offline and online evaluation into one operating loop. Google’s July 31 general-availability announcement for agent and model evaluations in Gemini Enterprise Agent Platform makes that loop concrete. The service uses the same versioned metrics across local experiments and production traffic, adds user and environment simulators, grades trajectories, clusters failures, and supports online monitoring from live traces. The product details are useful. The larger lesson is more durable: **An agent eval is not a score attached to a model. It is a versioned claim about system behavior under a specific task, tool environment, policy, and trajectory.** ## The trajectory is the unit that matters A conventional model evaluation often compares an input with an output. An agent adds planning, tool selection, arguments, side effects, retries, memory, and recovery. The final answer can be correct even when the path was unsafe. It can be wrong even when the tool actions produced the intended artifact. A successful outcome can hide an unnecessary permission escalation or a failed action that happened to be repaired later. That means an agent evaluation should inspect at least: - whether the task goal was satisfied - which tools were selected - whether tool arguments matched the schema and authority boundary - what external state changed - whether failures were detected and handled safely - whether the final response accurately represented the result - whether policy, privacy, and grounding constraints held across the run Google’s evaluation service exposes metrics for task success, tool-use quality, trajectory quality, safety, grounding, hallucination, and final-response quality. The exact metric names are less important than the scope: the system is judged as a sequence, not only as prose. ## Keep the metric definition stable while the environment changes Teams often run one framework in development and a different dashboard in production. When the numbers diverge, nobody knows whether the agent changed or the measurement changed. A versioned metric registry creates a cleaner comparison. The same code-based check, rubric, or judge configuration can run against a local experiment and a sampled production trace. Versioning should include more than the metric name. Preserve: - metric code or rubric text - judge model and parameters - task and tool schemas - system and policy instructions - dataset or trace selector - agent and model versions - environment configuration - expected evidence and pass threshold Without that record, a green dashboard may only mean the evaluator drifted with the system. ## Simulators are for negative paths, not fake confidence An agent that calls external systems is difficult to test safely. The most important cases often require a dependency to fail, return malformed data, add latency, or deny permission. Google describes both user and environment simulators. A user simulator can play a multi-turn persona. An environment simulator can intercept a tool call and return a forced error, slow response, or mocked payload without touching production. Those mechanisms are valuable when used to create adversarial operating conditions: - the approval arrives after the run times out - a tool returns success without the expected artifact - a dependency is slow enough to trigger a retry - the retry risks duplicating an external write - a user changes the goal after partial completion - a permission is revoked between planning and execution - memory contains a stale or malicious instruction - the recovery path receives incomplete state A simulator should expand the failure surface. It should not generate thousands of easy cases and turn coverage volume into false assurance. ## Production monitoring needs sampling and routing ![Trace-slice diagram separating task, tool, route, policy, recovery, and outcome evidence instead of relying on one average.](https://strong-bee-384.convex.cloud/api/storage/5a9cf628-475b-48ec-882b-7e5a44604d3c) *Image: SYMBiEX editorial system* Evaluating every trace with a large judge model can be expensive and slow. Evaluating none of them leaves the team blind to drift. Google’s online-monitoring design supports sampling and targeted filters over collected traces. That suggests a practical monitoring hierarchy: 1. Deterministic checks on every run where possible: schema validity, required approvals, state transitions, artifact existence, and policy invariants. 2. Targeted evaluation for high-risk or unusual trajectories: privileged tools, external writes, repeated failures, long duration, or customer escalation. 3. Random samples for broad quality and regression detection. 4. Full investigation for alerts, incidents, or statistically meaningful drift. The monitor should route evidence to an owner. A chart that becomes red without a triage path is decoration. For every production metric, define the threshold, owner, response window, investigation evidence, rollback option, and condition for updating the offline suite. ## Production failures should become regression cases ![Four-stage evaluation flywheel connecting offline cases, production samples, independent monitoring, and regression cases.](https://strong-bee-384.convex.cloud/api/storage/95f7e6cc-a75d-47ae-925b-6e766e955f02) *Image: SYMBiEX editorial system* The evaluation loop compounds only when observed failures change future tests. A useful incident-to-eval pipeline looks like this: 1. Capture the production trajectory and external artifacts. 2. Classify the failure against a stable taxonomy. 3. Reduce it to the smallest reproducible case without losing the mechanism. 4. Add a deterministic assertion or reviewed rubric. 5. Reproduce the relevant tool environment with a simulator or isolated test system. 6. Run the case against the current and proposed agent versions. 7. Keep it in the regression suite after the incident closes. Google’s failure clustering can help group large evaluation jobs into actionable categories. Teams still need their own taxonomy. Useful top-level classes include authority, planning, tool use, state, grounding, policy, recovery, and reporting. A taxonomy turns a pile of red traces into a decision about where the system is weak. ## Keep the monitor outside the agent’s authority An evaluation surface becomes part of the attack surface when the system being judged can rewrite the test, alter the trace, influence the grader, or choose which evidence is visible. Separate the agent’s runtime identity from the evaluation pipeline. Store traces and metric versions in a location the agent cannot modify. Use hidden tests for critical behavior. Verify external artifacts independently rather than trusting the agent’s final message. LLM judges can be useful, especially for open-ended trajectories, but they should not be the only source of truth. Pair them with deterministic checks, ground-truth artifacts, human review for consequential cases, and disagreement analysis across evaluator versions. The principle is straightforward: **The system can produce evidence. It should not control the record used to judge that evidence.** ## A minimal lifecycle for agent evaluation A durable program does not need every feature on day one. It needs one connected loop. Start with: - a small versioned dataset of critical tasks - deterministic checks for permissions, artifacts, and state transitions - trajectory capture in development and production - one simulator for a high-cost failure mode - sampled online evaluation using the same metric definitions - an owner and response rule for every alert - a process that converts real failures into regression cases Then expand where the evidence shows a gap. Add adaptive rubrics when fixed criteria are brittle. Add more simulation when external dependencies dominate failures. Add issue clustering when the case volume becomes hard to interpret. Do not begin with a giant metric catalog. Begin with the operating claim the agent must keep. ## The durable pattern Offline tests answer: *Can this version pass known cases under controlled conditions?* Online monitors answer: *What is this version doing under real traffic, tools, data, and failures?* A mature agent-evaluation system connects those answers with shared metrics, versioned evidence, simulators, independent monitoring, and a regression loop. That is the difference between an eval dashboard and an engineering control plane. ## Primary sources - [Google Developers Blog: Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA](https://developers.googleblog.com/agent-and-model-evaluations-in-gemini-enterprise-agent-platform-are-now-ga/) - [Google Cloud documentation: Evaluate your agents](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/evaluate-agents) - [Google Cloud documentation: Continuous evaluation with online monitors](https://docs.cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/evaluate-online)