What an Eval Is
How you measure an AI system before shipping it, so 'is it better?' stops being a vibe.
Normal software is predictable: same input, same output, so you write a test that checks for the exact answer. AI is not like that. Ask the same question twice and you can get two different wordings. "Is it right?" stops being a simple check, and that is the problem an eval solves.
What an eval is
An eval is a test built for systems that do not give the same answer twice. It has two parts: a set of cases (inputs plus what a good answer looks like), and a way to score the output. Sometimes that score is an exact match, sometimes a rubric, sometimes another model grading against criteria. Run it and you get a number: how often the system does the right thing.
Why you need it
Without an eval, "is the new version better?" is a vibe. You tweak a prompt, it feels better, you ship it, and you have quietly broken three other cases you were not looking at. An eval turns that hunch into a measurement, so you can change things and actually know whether you improved them. This matters even more for agents, where a single change ripples through every step of the loop.
Why it matters
Evals are how AI work stops being arts-and-crafts and becomes engineering. Before you ship a change to something that talks to real users, you want a number that says it got better, not a feeling. That number comes from an eval - and once you have it, you can put a real cost next to a real quality number and make actual trade-offs.