Always Be Evaluating

Compared to the previous generation of AI and ML algorithms, generative AI models like LLMs or multimodal models capable of open-ended responses with unstructured text, image, or video have introduced new industry challenges for quality evaluations.

When considering an image classifier, determining "correctness" is a straightforward application of simple logic: did you predict the ground-truth image label? When evaluating an LLM, responses are more varied, evaluations are more subjective, and you're likely tasking the model to balance different capabilities in your application.

For reproducibility in research, the community organizes around open benchmarks designed to assess specific capabilities. As foundation models evolve from the labs into production software systems, the capabilities that address customer needs are prioritized.

Benchmarks are the easiest way to rein in the scope of candidate AI models from the 1.5 million open-weight models on the hub to something more manageable for consideration in your next experiment.

However, open research benchmarks are subject to "benchmark hacking," where unprincipled model providers release weights developed to take advantage of publicity by topping the leaderboards. All this is even though the AI research community likely does not represent your user base.

One of the most reliable ways to use benchmarks is as a "guardrail metric" for your AI model treatments. After comparing the base model with your finetuning or inference optimizations, do you observe a significant degradation of a key skill or capability as indicated by a dip in a benchmark score?

Recent flagship AI models like Grok 3, GPT-4.5, and now Lama 4 were released despite lackluster research benchmark performance. Each emphasizes "better vibes," conversational skills, and a neutralized moral/political bias.

What matters for users of your AI application?

It's never been easier to assess your models for your application as data synthesis and auto evaluators have reduced custom evaluation to prompting a tool with a bit of context about your application and data.

To consistently succeed with your AI initiatives, it's essential to recognize that any benchmark, even one customized with your application in mind, will still only be a proxy to the metrics you care to optimize, related to customer satisfaction and business value. Thus, these offline metrics offer a skewed view of AI model fitness. Unfortunately, they are often poor predictors of what will delight your users.

So, how do you make sense of this to build great AI products?

I advocate for evaluating every step of the experiment! Review the reported benchmarks, check the model's Elo ratings on the chatbot arena, and see what developers report on r/localllama. Run your custom fine-grained skills assessments, compare the loss amongst your finetunes, and score and filter your training samples for quality. Measure latency and any other metrics that will help you to confidently deploy your best treatments for users to assess in an online experiment.

With millions of open-weight models and thousands of benchmarks, it's easy to find false positives in your search for the best AI. Don't just use metrics that support a narrative created after the fact; choose your metrics ahead of time with a hypothesis in mind. Your defense against institutionalizing "bad intel" into your AI initiative is to raise your scientific standards in AI evaluation.

Discovering AI models that excel across several unrelated evaluation methods, you can reason with more confidence and explainability to get consensus around your next AI experiment. The online, end-to-end evaluation helps you build knowledge about what delights users of your AI. The final launch/no-launch decisions around the next AI update should be data-driven after proving to impact meaningful business metrics positively.

So, what's your hypothesis?