Agent Eval Flow
Evaluation → evolution

OpenKritt / Case study 02

Found the bug. Verified the impact.

A billing user can refund another tenant’s invoice. Does the security report prove what it claims?

Grounding passes · severity still unresolved
Native agent runs on synthetic cases
Same test. Same measurement.BEFOREAFTER
Submitted proof reproducesIndependent replay of the seeded flaw
1 / 1
1 / 1
Grounding contract passesDemo rule; baseline types and sign implicit
0 / 1
1 / 1
Severity matches policyDeclared demo rule; a separate requirement
0 / 1
0 / 1
What the report showed

Stage 0 said “demonstrated” without execution. Source-only review was allowed; final fields missed our implicit contract.

What we changed

Require local execution and a typed handoff. Clarify the grading contract; separate measured facts from inference.

Evidence

Native request: HTTP 200 · balance 25,000 → 24,000 cents. Final claim: −1,000 cents · disclosure = true. Independent replay confirms those effects.