Agent Eval Flow
Evaluation → evolutionOpenKritt / Case study 02
Found the bug. Verified the impact.
A billing user can refund another tenant’s invoice. Does the security report prove what it claims?
Grounding passes · severity still unresolved
Native agent runs on synthetic casesSame test. Same measurement.BEFOREAFTER
Submitted proof reproducesIndependent replay of the seeded flaw
1 / 1
1 / 1
Grounding contract passesDemo rule; baseline types and sign implicit
0 / 1
1 / 1
Severity matches policyDeclared demo rule; a separate requirement
0 / 1
0 / 1
What the report showed
Stage 0 said “demonstrated” without execution. Source-only review was allowed; final fields missed our implicit contract.
What we changed
Require local execution and a typed handoff. Clarify the grading contract; separate measured facts from inference.
Evidence
Native request: HTTP 200 · balance 25,000 → 24,000 cents. Final claim: −1,000 cents · disclosure = true. Independent replay confirms those effects.