LLMCheck and repeatable AI evaluation
AI Enablement · Open source
A support assistant promises an immediate refund even though the policy requires approval for that amount. An engineer needs a way to check the same failure after the application changes.
My contribution
I built LLMCheck’s Python capture, review, and replay workflow. I separated review from replay so an engineer can inspect the expected behavior before it becomes a test. I also packaged a runnable offline example.
What it enabled
A reviewed failure becomes a repeatable check. The application adapter produces a fresh answer, and a judge evaluates it against the reviewed criteria. The offline example uses a scripted answer and a test judge, so a teammate can try the process without an API key.
Why repeatability mattered
As prompts, tools, or application logic change, the same failure can recur. A saved case gives the team a way to check the behavior again and inspect the verdict against the criteria for that run.
Run the offline example · Read the full walkthrough