Evidence note: This article rests on primary vendor and project sources — Google’s developer blog, Microsoft’s developer blog, project repositories, a Hugging Face research write-up, and general reference entries. Those are the publishers’ own accounts of their own work, which is a different thing from independent corroboration. Product capabilities described below have not been verified against hands-on testing, and no benchmark figures are reproduced here because the sources cited do not supply ones that could be checked. Where a claim rests on a single vendor’s description, that is marked at the claim itself.
Google’s Agent Development Kit adds native live evaluation
Google’s developer blog has published guidance on evaluating live voice agents in the Agent Development Kit (ADK), describing native live evaluation aimed specifically at live and voice agents (source). Evidence level: single vendor source. Google’s own post is the only account of this capability referenced here; its scope, availability tier, and current release status have not been independently confirmed, and readers building against it should check the ADK documentation directly rather than treating the description below as a specification.
The gap this addresses is real enough to state plainly, independent of any product. A text agent can be tested by comparing strings. A voice agent’s output is audio produced under timing pressure, in a conversation whose next turn depends on what the agent just said. The usual test harness has nothing to assert against.
What a live evaluation run is said to contain
According to the same Google post, a live evaluation run exercises graph-based agent workflows against an LLM-driven simulated user, and that simulated user generates actual audio through Gemini text-to-speech rather than passing text directly into the agent (source). Evidence level: vendor description, not independently verified.
Three design decisions are worth separating, because each carries a different weight:
- Graph-based workflows as the unit under test. The thing being scored is a multi-step agent structure, not a single model call.
- A model playing the caller. The counterparty is generated rather than scripted, so the conversation is not fixed in advance.
- Real synthesized audio in the loop. The agent receives speech, not a transcript — which means the speech recognition path is inside the measurement rather than stubbed out around it.
That third point is the one with the most consequence and the least outside confirmation. Putting synthesized audio through the pipeline means the evaluation includes transcription error, and transcription behavior is its own moving target — a subject covered separately in our explainer on Gemini 3.5 Transcribe and its live variant. Whether ADK’s harness isolates transcription failures from reasoning failures is not addressed in the material available, and remains an open question.
Why voice agents need this before production
The argument that live voice agents require rigorous automated testing before production — because real multi-turn conversations are unpredictable in ways a scripted demo is not — is advanced in Google’s post and is not independently corroborated here. Evidence level: vendor-stated rationale.
The reasoning is nonetheless easy to follow. A demo runs one path. Production runs every path: interruptions, silence, background noise, a caller who changes their mind three turns in, a caller who answers a question that was never asked. A scripted test suite encodes the paths the author already thought of, which is exactly the set of paths that were never going to be the problem. A generated counterparty widens that set — though by how much, and with what coverage guarantees, the sources do not say.
How the scoring is described
Google’s post describes developers defining evaluation scenarios along with natural-language rubrics, which are then applied automatically to score both audio responses and tool executions (source). Evidence level: single vendor source; no scoring accuracy or agreement figures are published in the material cited.
Two halves of that deserve different scrutiny. Scoring tool executions is mechanically checkable: a tool was called or it was not, with arguments that either match or do not. Scoring audio responses against a natural-language rubric is a judgment call delegated to a model, and the reliability of that delegation is not established by the announcement. A rubric that reads "the agent confirmed the booking politely" is a prompt, and prompts have failure modes.
Where the result is read
Per the same source, evaluation transcripts can be inspected in ADK Web, and the evaluation command-line tool can be run inside a continuous integration pipeline (source). Evidence level: vendor description.
This is the part with the clearest practical shape, and also the part that raises the sharpest question. A non-deterministic evaluation wired into CI produces a pass/fail signal that can vary between runs on unchanged code. The sources do not describe how run-to-run variance is handled, whether scores are thresholded, or how a team is meant to distinguish a regression from noise. For anyone planning to gate a merge on this, that is the first thing to establish empirically.
When the sandbox leaks the answer
A sharper failure mode comes from a different vendor entirely. Microsoft’s developer blog argues that an AI coding agent evaluation is only as good as the sandbox it runs in, and describes a case where an agent passes by finding the answer somewhere on the machine rather than knowing it (source). Evidence level: single vendor source; the specific incident described has not been independently reproduced or corroborated here.
If that account holds, the implication is uncomfortable in a useful way: a correct answer can invalidate the measurement. The output is right. The score is green. The thing the score was supposed to measure — the model’s capability — was never exercised, because a leftover test fixture, a cached result, or a git history made the answer retrievable. Correctness and validity come apart, and the usual signal for "the test worked" no longer distinguishes them.
The environment is an instrument, not a backdrop
The general form of that argument, also from Microsoft’s post, is that evaluation quality is bounded by sandbox quality (source). Evidence level: vendor-stated principle, not independently tested.
Stated that way it generalizes past coding agents. In the ADK voice case, the "sandbox" is the simulated caller, the text-to-speech voice, the audio channel, and the tool environment the agent acts against. Each is a piece of measuring apparatus. A simulated user that is too agreeable produces agreeable transcripts. A synthetic voice with no disfluency measures an agent against a speaker who does not exist. None of this is a claim about ADK specifically — the available material does not characterize the fidelity of its simulated users, and that gap is open.
The transferable idea is that environments should be audited the way instruments are: what is reachable inside them, what is absent from them, and what that absence quietly rules out of the measurement.
Reading a score after that
The technique underneath most rubric-based scoring is LLM-as-a-Judge, described as the use of a large language model as an evaluator (reference), and a benchmark in computing is described as a standardized performance evaluation (reference). Large language models themselves are characterized as machine learning models built for natural language tasks such as generation (reference). Evidence level: general reference entries, offered as definitions rather than as findings.
Standardization is what makes a benchmark comparable, and it is also what makes it gameable — the same fixed shape that lets two systems be compared lets a system be fitted to the shape. Allen AI’s BenchMIRT is published as an analysis of what LLM benchmarks are actually measuring (source); that description comes from the write-up itself. Its findings are examined in our piece on what LLM benchmarks are actually measuring, and the broader ecosystem context sits in our survey of the 2026 AI evaluation landscape.
Put beside the sandbox-leak problem, the two concerns stack rather than cancel: a benchmark may measure something other than the capability named on it, and the environment may hand the system the answer. A green number survives both.
The tooling around the problem
Several projects address adjacent pieces, each described here according to its own published material:
- lmms-eval is presented by its repository as a multimodal evaluation toolkit covering text, image, video, and audio tasks (project repository). Audio coverage is the relevant part for voice work, though the sources give no basis for comparing its audio task suite against ADK’s live evaluation.
- easy-dataset is described by its repository as a tool for building datasets for LLM fine-tuning, retrieval-augmented generation, and evaluation (project repository). Evaluation sets are an input to every technique above, and where the set comes from determines what the score can mean.
Evidence level for both: each project’s own description; no independent assessment of capability or maturity is offered.
Adjacent automation in the same space
Google’s developer blog also describes Tunix on TPUs supporting autonomous LLM post-training driven from a single Markdown specification (source). Evidence level: single vendor source, not independently verified. Separately, EVAL — Elastic Versatile Agent with Langchain — is published as an agent that executes user requests (project repository), per its own repository.
The shared direction is worth naming even without endorsing any of the implementations: specification-driven automation, where a document becomes an executable plan. It applies to training runs and to evaluation runs alike, and it moves the decisive judgment upstream into whoever writes the specification. The same pattern shows up wherever evaluation meets the physical or perceptual world, including the simulation questions raised in our coverage of Gemini Robotics 2 and whole-body intelligence.
Developer takeaway: what to ask before trusting a green run
None of the capabilities above have been tested here, so the useful output is a set of questions rather than a recommendation:
- What is reachable from inside the sandbox? List it explicitly — repository history, cached artifacts, prior run outputs, network. Anything reachable is a candidate answer leak.
- Could this task be passed by retrieval instead of capability? If yes, the score measures the environment, not the system.
- Who or what is playing the counterparty, and how varied is it? A simulated user that never interrupts, never mishears, and never changes its mind is a specification of the easy case.
- Is the judge itself evaluated? Rubric scoring by a model is a prompt with failure modes; agreement against human labels is the check, and the sources cited publish none.
- How much does the score move on unchanged code? Establish that variance before gating CI on it, or the gate will be measuring noise.
- Where did the evaluation set come from? A set generated from the same family of models being tested is not a neutral ruler.
A proposed follow-up, not an experiment performed: running the same agent build through an ADK live evaluation under two sandbox configurations — one with the working tree and prior artifacts reachable, one stripped — and recording whether scores diverge. That comparison would turn the sandbox-leak argument from a vendor assertion into a measurement. Until it is run, it stays exactly what it is here: a question worth asking before a green run is believed.