Can I rely on public benchmark scores?

Use them to build a shortlist only. Benchmarks may not resemble your task, and models can be tuned toward them. Your own test set is the better predictor of real results.

How big should the test set be?

Enough to cover the main cases and edge cases, often 30 to 100 examples to start. Grow it with real failures you see in production.

Choosing and Evaluating an LLM for Your Use Case

Introduction

There is no single best model. The right choice depends on your task, budget, and speed needs, and the landscape changes often. Public leaderboards are a starting point, but the reliable method is to test candidates on examples from your own use case. This lesson shows how.

Prerequisites

Calling an LLM API lesson
A task you want to automate, such as classifying support tickets

Step-by-Step Guide

Step 1: List your requirements

Write down what matters: answer quality, response speed, cost per request at your expected volume, context length, support for images or tools, data privacy and hosting requirements, and the reliability you need. Rank them, because you will have to trade some off.

Step 2: Shortlist candidates in different tiers

Providers typically offer a range: larger models for the hardest reasoning, mid-size models for everyday production work, and small fast models for simple high-volume tasks. Pick a few across tiers, including at least one cheaper option, since many tasks do not need the most capable model.

Step 3: Build a small golden test set

Collect 30 to 100 realistic inputs with the correct answer for each, including edge cases and a few examples the model should refuse or say it does not know. This set is more valuable than any public benchmark because it reflects your real task.

cases = [
    {"input": "I was charged twice for my order", "expected": "billing"},
    {"input": "The app crashes when I open settings", "expected": "bug"},
    {"input": "How do I export my data?", "expected": "how_to"},
]

Step 4: Run each candidate on the same set

Use identical prompts and settings, score the results automatically where you can, and record cost and latency too. For free-text tasks, use a rubric or have a stronger model grade answers, but spot-check the grader by hand because it can be biased or wrong.

import anthropic

client = anthropic.Anthropic()
MODELS = ["claude-opus-5-5", "claude-sonnet-5-5", "claude-haiku-5-5"]

def classify(model: str, text: str) -> str:
    response = client.messages.create(
        model=model,
        max_tokens=16000,
        system="Classify the ticket as billing, bug, or how_to. Reply with one word.",
        messages=[{"role": "user", "content": text}],
    )
    return next(b.text for b in response.content if b.type == "text").strip().lower()

for model in MODELS:
    correct = sum(classify(model, c["input"]) == c["expected"] for c in cases)
    print(model, correct, "/", len(cases))

Step 5: Decide on cost per completed task

Compare total cost to get a correct result, not just price per token. A cheaper model that needs retries, or a bigger one that finishes in one step, can change the answer. Re-run your test set whenever you change prompts or when a provider releases new models.

FAQs

Can I rely on public benchmark scores?: Use them to build a shortlist only. Benchmarks may not resemble your task, and models can be tuned toward them. Your own test set is the better predictor of real results.
How big should the test set be?: Enough to cover the main cases and edge cases, often 30 to 100 examples to start. Grow it with real failures you see in production.

Summary

Define your requirements, shortlist models across tiers, and compare them on a golden test set built from your real task. Choose by cost per correct result, and re-test when anything changes.