Why I Built Our Own LLM Benchmark, and How You Can Too
11:35 - 12:05, 18 Listopada 2026 / AI & ARCHITECTURE
Generic benchmarks tell us which model is best overall, not which one is best for our work. So at Towards AI, I built a benchmark (now public!) and evaluation pipeline around the tasks our team actually performs. In this talk, I’ll break down how we turned real workflows and quality expectations into test cases, rubrics, automated checks, repeated runs, and cost and latency tracking and how we now use the results to choose models and catch regressions.
I’ll also share what I learned building most of the system quickly with AI itself, what was worth automating, and what still requires human judgment.
The goal is to show why every team building with LLMs should own an evaluation pipeline, and give you a blueprint for building one that reflects your users, standards, and constraints rather than someone else’s leaderboard.
POZIOM:
ŚCIEŻKA:
