The question
How do success criteria determine what a simulation considers a successful outcome?
What the Review Identified
One finding illustrates the implementation risk examined in this area.
A simulation can pass because its configured success criteria were satisfied. The important question is what those criteria actually measure: an assistant's response, a tool call, a reported result, or the underlying external outcome.
What we investigated
We traced how the platform converts a simulation's conversation and observable results into a pass or fail outcome, including structured outputs, comparators, expected values, and required versus optional criteria.
What we found
A simulation passes according to the specific conditions defined in its success criteria.
A passing result therefore establishes that the configured criteria were satisfied. It does not, by itself, establish that the complete real-world outcome occurred.
Why it matters
A criterion can evaluate what the assistant reported without verifying the underlying external system.
For example, the platform's appointment example explicitly distinguishes between reporting appointment details and verifying calendar state.
Implementation implication
Developers need to inspect what each required success criterion actually measures before treating a passing simulation as evidence that the underlying workflow succeeded.
What This Demonstrates
A passing simulation establishes that the configured success criteria were satisfied. It does not automatically establish that the complete real-world outcome occurred.
In practice: a team can treat a passing evaluation as proof of a successful workflow when the criterion only verified what the assistant reported.