The question
How do simulation webhooks affect what happens after a simulation run, and what can a developer reasonably conclude from the webhook data about the simulation's outcome?
What the Review Identified
One finding illustrates the implementation risk examined in this area.
A simulation webhook can tell a system that a simulation run started or ended and provide information about that run. That lifecycle event is separate from determining whether the simulation actually satisfied its success criteria.
What we investigated
We examined the simulation lifecycle events sent to a configured webhook, including what information is provided when a simulation starts and ends, and how that information relates to the simulation's evaluation.
What we found
Simulation webhooks provide lifecycle and run information.
A simulation.run.ended event establishes that the simulation run ended and provides information about that run. It does not, by itself, establish that the simulation passed its success criteria or that the intended external outcome occurred.
Why it matters
A developer could receive a successful webhook delivery and treat it as evidence that the simulation succeeded.
The webhook and the simulation evaluation answer different questions: the webhook reports what happened to the run, while the evaluation determines whether the configured success conditions were satisfied.
Evidence
A controlled review of the simulation lifecycle and evaluation path showed that the ended event can be received separately from inspecting the criteria used to determine whether the simulation passed.
Result
The webhook provides lifecycle/run evidence, not an independent pass/fail or business-outcome guarantee.
Implementation implication
Developers need to distinguish between receiving confirmation that a simulation run ended and verifying what the simulation actually established. Evaluation results, transcript or message data, tool execution, and external system state may require separate inspection.
What This Demonstrates
A simulation webhook provides lifecycle and run evidence. It does not automatically establish that the simulation passed or that the intended external outcome occurred.
In practice: a system can react to a completed simulation run as though the underlying test succeeded even though its success criteria or external outcome have not been verified.