From probabilities to a token: decoding

Recall the loop: at each step the model outputs a probability for every vocabulary token. Turning that distribution into one chosen token is decoding (a.k.a. sampling). The strategy you pick controls how "safe" vs. "varied" the output is.

Greedy decoding — always take the single highest-probability token. Repeatable and focused, but can be bland or get stuck in loops.

Sampling — draw a token according to its probability (a token with p=0.3 is picked ~30% of the time). Introduces variety — and the possibility of a less obvious, more creative continuation.

Two knobs shape the distribution before sampling:

Temperature

Temperature flattens or sharpens the probability distribution before sampling.

• Low (→0): sharpen toward the top token → near-greedy, focused, repeatable. Good for extraction, classification, code, anything where you want the "obvious" answer.
• High (→1 and above): flatten the distribution → lower-probability tokens get a real chance → more varied, creative, and more prone to going off the rails.

Probabilities for the next token, before vs. after temperature:

              " Paris"  " a"   " located"  …
raw            0.80     0.10    0.05
temp = 0.2  →  0.97     0.02    0.01     (sharpened — almost always " Paris")
temp = 1.2  →  0.55     0.20    0.12     (flattened — " a" and " located" now plausible)

Top-p (nucleus sampling)

Top-p keeps only the smallest set of tokens whose probabilities sum to p, and samples from that "nucleus" (renormalized). top_p = 0.9 means "consider only the most likely tokens that together cover 90% of the probability mass, ignore the long tail." It caps how weird the choice can get without hard-coding a count.

Temperature and top-p are often used together, but change one at a time when tuning — moving both makes the effect hard to reason about.

"Temperature 0" ≠ deterministic (or correct)

A common trap. Setting temperature to 0 makes decoding greedy, which is much more repeatable — but not a guarantee of identical output across calls, because:

• Floating-point non-determinism, hardware/batching differences, and provider-side changes can break exact ties differently.
• Model updates change the weights entirely.

And crucially, repeatable ≠ correct. Greedy decoding just takes the most probable token; if the model's most-probable answer is wrong, temperature 0 returns that wrong answer every time. Use low temperature for consistency, not as a correctness or determinism guarantee.

Stopping: stop tokens, stop sequences, and max_tokens

Generation runs the loop until something tells it to stop:

• End-of-turn token — the model predicts a special stop token on its own (normal completion).
• Stop sequences — you supply strings that force a halt when generated (e.g. stop at "\n\n" or a closing tag). Useful for structured output where you know the boundary.
• max_tokens — a hard cap on output tokens. If the model hits it mid-sentence, output is truncated and the response reports a max_tokens stop reason — that's your signal to raise the cap or stream.

from anthropic import Anthropic
client = Anthropic()

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,                       # HARD cap on OUTPUT tokens; too low → truncated answer
    messages=[{"role": "user", "content": "List three RAG evaluation metrics."}],
)
resp.stop_reason                            # "end_turn" (finished) or "max_tokens" (was truncated) — always check this

max_tokens, cost, and latency

Two knobs, two cost/latency effects:

• max_tokens is a ceiling, not a target — the model stops when done; you're billed for actual output, not the cap. But set it high enough that legitimate answers aren't cut off. For long outputs, stream the response (token-by-token) so a slow generation doesn't hit an HTTP timeout, and so users see progress.
• Every output token is one forward pass — so latency scales with output length far more than input length. Shorter, more focused answers are faster and cheaper. (Input tokens are cheaper per token than output tokens on most providers, and can be discounted further by prompt caching.)

# Streaming: yields text as it's generated — avoids timeouts on long outputs and shows progress.
with client.messages.stream(
    model="claude-opus-4-8", max_tokens=8000,
    messages=[{"role": "user", "content": "Explain RAG in detail."}],
) as stream:
    for text in stream.text_stream:        # iterate the text deltas as they arrive
        print(text, end="", flush=True)    # flush so tokens appear immediately (like process.stdout.write)
    final = stream.get_final_message()      # the complete assembled message, plus usage stats
final.usage.output_tokens                   # what you were actually billed for on output

Choosing settings — a practical default

Task	Temperature	Notes
Extraction, classification, structured output	low (0–0.3)	Want the obvious, consistent answer
Coding, factual Q&A, RAG	low (0–0.4)	Consistency + fewer creative detours
Brainstorming, copywriting, variety	higher (0.7–1.0)	Want range across runs
"Determinism" for tests	cache the result yourself	Don't rely on temperature 0

Newer model families may replace these knobs with an effort/verbosity control and heavier reliance on prompting — the goal (focused vs. varied, short vs. thorough) is the same; the lever changes.

Recap

• Decoding turns next-token probabilities into a chosen token: greedy (top token) or sampling (by probability).
• Temperature sharpens (low) or flattens (high) the distribution; top-p samples from the smallest set covering probability mass p. Tune one at a time.
• Temperature 0 is repeatable, not truly deterministic, and never a correctness guarantee — cache results yourself if you need determinism.
• Control length with stop sequences and max_tokens; always check stop_reason, and stream long outputs. Output tokens dominate cost and latency.

from anthropic import Anthropic
client = Anthropic()

prompt = "Write a one-sentence tagline for a coffee shop on Mars."

for t in [0.1, 0.7, 1.2]:
    print(f"\n=== temperature {t} ===")
    for _ in range(3):
        resp = client.messages.create(
            model="claude-sonnet-5", max_tokens=60, temperature=t,
            messages=[{"role": "user", "content": prompt}],
        )
        print("-", resp.content[0].text.strip())

from anthropic import Anthropic
client = Anthropic()

def answer(question: str, max_tokens: int):
    resp = client.messages.create(
        model="claude-opus-4-8", max_tokens=max_tokens,
        messages=[{"role": "user", "content": question}],
    )
    return resp

resp = answer("Explain how RAG works, step by step, in detail.", max_tokens=40)
print("stop_reason:", resp.stop_reason)         # -> "max_tokens" : the answer was cut off
if resp.stop_reason == "max_tokens":
    print("Truncated — retrying with more room.")
    resp = answer("Explain how RAG works, step by step, in detail.", max_tokens=1500)
print(resp.content[0].text)

import time
from anthropic import Anthropic
client = Anthropic()

q = "Write a detailed 300-word explanation of vector databases."

# Non-streaming: you wait for the whole thing before seeing anything.
t = time.perf_counter()
resp = client.messages.create(model="claude-opus-4-8", max_tokens=800,
                              messages=[{"role": "user", "content": q}])
print(f"non-streaming: first output at {time.perf_counter()-t:.1f}s (the whole response)")

# Streaming: first tokens arrive almost immediately.
t = time.perf_counter()
first = None
with client.messages.stream(model="claude-opus-4-8", max_tokens=800,
                            messages=[{"role": "user", "content": q}]) as stream:
    for _ in stream.text_stream:
        if first is None:
            first = time.perf_counter() - t
            break
print(f"streaming: first token at {first:.1f}s")