How do I avoid overfitting the prompt to my test set?

Keep a separate group of held-out cases you do not look at while tuning, and check it only occasionally. If the tuned prompt does well on both groups, the improvement is more likely real.

Is a model-graded evaluation reliable?

It can be useful for scale, but treat it as an estimate. Spot-check its grades against human review and prefer objective checks where the task allows.

Evaluating and Iterating on Prompts Like Code

Introduction

Changing a prompt and eyeballing one answer is how teams ship regressions. A prompt is part of your software, so it deserves tests, versions, and measurements. This lesson shows a simple evaluation loop you can set up in an afternoon.

Prerequisites

Structured prompts and few-shot lessons
A task with answers you can check

Step-by-Step Guide

Step 1: Create a test set before tuning

Collect realistic inputs with expected results, including edge cases and past failures. Keep them in a file under version control so every change is tested against the same cases.

# tests/cases.json
[
  {"input": "Refund for order 1042", "expected": "refund"},
  {"input": "Where is my package?", "expected": "tracking"},
  {"input": "Cancel my subscription", "expected": "cancellation"}
]

Step 2: Write a small evaluation harness

Run every case through the current prompt, compare the output with the expected answer, and print a pass rate plus the failures. Automated checks work best for classification and extraction; use exact or structured comparisons wherever possible.

import json
import anthropic

client = anthropic.Anthropic()
PROMPT = "Classify the request as refund, tracking, or cancellation. Reply with one word."

def run(text: str) -> str:
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        system=PROMPT,
        messages=[{"role": "user", "content": text}],
    )
    return next(b.text for b in response.content if b.type == "text").strip().lower()

cases = json.load(open("tests/cases.json"))
failures = []
for case in cases:
    got = run(case["input"])
    if got != case["expected"]:
        failures.append((case["input"], case["expected"], got))

print("Pass rate:", (len(cases) - len(failures)) / len(cases))
for item in failures:
    print("FAIL:", item)

Step 3: Change one thing at a time

Alter a single element, such as an instruction, an example, or the effort setting, then re-run the set. If several things change together you will not know which one helped or hurt. Keep a short log of each version and its score.

Step 4: Grade open-ended answers carefully

For free text, define a rubric of specific, checkable criteria and score against it, either by hand on a sample or with a model acting as a grader. Calibrate any model grader against human judgments, because it can favor longer answers or its own style.

Step 5: Keep testing after launch

Re-run your set when you change models, tools, or data. Add real failures from production as new test cases so the same mistake cannot return. Track cost and speed alongside quality.

FAQs

How do I avoid overfitting the prompt to my test set?: Keep a separate group of held-out cases you do not look at while tuning, and check it only occasionally. If the tuned prompt does well on both groups, the improvement is more likely real.
Is a model-graded evaluation reliable?: It can be useful for scale, but treat it as an estimate. Spot-check its grades against human review and prefer objective checks where the task allows.

Summary

Treat prompts as code: keep a versioned test set, run an automated harness, change one thing at a time, and add production failures back as tests. That turns prompt tuning into measurable engineering.