Why does the same prompt cost more in a long chat?

The full conversation history is sent as input on every turn, so each new message pays for all the earlier ones too, unless you summarize, trim, or cache the history.

Does a bigger context window mean I should put everything in?

Not necessarily. More context costs more, can slow responses, and can dilute focus. Include what is relevant, and use retrieval to pick the right passages from large collections.

LLM Tokens, Context Windows, and Sampling Explained

Introduction

Tokens, context windows, and sampling settings come up in every LLM project because they drive cost, limits, and output variety. This lesson explains each in plain terms and shows how to count tokens before you spend money on a request.

Prerequisites

The LLM introduction lesson
Python and the anthropic package if you want to run the code

Step-by-Step Guide

Step 1: Estimate and count tokens

A rough rule for English is that a token is a bit under a word, but the exact count depends on the model. Use your provider's token counter instead of guessing. Counting tokens does not generate any text.

import anthropic

client = anthropic.Anthropic()

count = client.messages.count_tokens(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": "Explain recursion in one paragraph."}],
)
print(count.input_tokens)

Step 2: Understand the context window

The context window is the maximum number of tokens the model can handle in one request, covering your prompt, the conversation so far, any documents, and the response it writes. Some models also use tokens for internal reasoning, which counts toward the output limit. Leave generous room for output so responses are not cut off.

Step 3: Know how cost is calculated

Providers usually price input tokens and output tokens separately, and output tokens cost more. Long conversations resend history on every turn, so cost grows with conversation length. Techniques such as prompt caching reduce the cost of repeated context.

Step 4: Learn what sampling settings do

Temperature controls randomness: low values make the model pick the most likely tokens, giving consistent answers, and high values allow more varied, creative output. Top-p (nucleus sampling) limits choices to the smallest set of tokens whose probabilities add up to p. Top-k limits choices to the k most likely tokens.

Step 5: Check what your model allows

These settings exist across the industry, but some recent models fix them and reject requests that set temperature, top-p, or top-k. If you need consistency on such models, use clear instructions, structured output formats, and test sets rather than tuning randomness. Always check the documentation for your specific model.

FAQs

Why does the same prompt cost more in a long chat?: The full conversation history is sent as input on every turn, so each new message pays for all the earlier ones too, unless you summarize, trim, or cache the history.
Does a bigger context window mean I should put everything in?: Not necessarily. More context costs more, can slow responses, and can dilute focus. Include what is relevant, and use retrieval to pick the right passages from large collections.

Summary

Tokens measure everything, the context window caps what fits in one request, and cost follows tokens. Sampling controls variety where the model allows it; where it does not, rely on clear prompts and testing.