The one thing an LLM does

Strip away the chat interface and a large language model does exactly one thing: given a sequence of tokens, predict the probability of every possible next token. That's it. Everything else — answering questions, writing code, "reasoning" — emerges from doing this well, repeatedly.

To generate text, the model runs a loop:

1. Take the tokens so far ("The capital of France is")
2. Predict a probability for every possible next token   →  " Paris" 0.91, " a" 0.02, " located" 0.01, …
3. Pick one (usually the/a high-probability one)          →  " Paris"
4. Append it, and go back to step 1                        →  "The capital of France is Paris"

It is autoregressive: each new token is fed back in to predict the next. A chat "response" is just this loop running until the model predicts a special stop token. There is no plan, no database lookup — only next-token prediction, learned from a very large amount of text.

Tokens, not words

The model doesn't see characters or words — it sees tokens: chunks of text from a fixed vocabulary (~100k–200k entries). A token is often a word, but common sub-word pieces and spaces are tokens too.

• "cat" → 1 token
• "unbelievable" → maybe un + believ + able (3 tokens)
• " Paris" (with the leading space) → 1 token; the space matters
• Code, punctuation, and non-English text tend to use more tokens per character.

Rough rule of thumb for English: ~4 characters ≈ 1 token, or ~0.75 words per token. But it's only a rule of thumb — always measure with the provider's tokenizer when it matters.

Why you care:

• Cost is billed per token (input + output), not per word.
• Limits — max_tokens, the context window — are token counts.
• Latency scales with tokens generated (remember: one forward pass per output token).

# Anthropic ships a token counter — never estimate with OpenAI's tiktoken (it's a different vocab).
from anthropic import Anthropic          # `import` ~ JS import
client = Anthropic()                       # reads ANTHROPIC_API_KEY from the environment

resp = client.messages.count_tokens(
    model="claude-opus-4-8",              # count is model-specific — pass the model you'll actually call
    messages=[{"role": "user", "content": "How many tokens is this sentence?"}],
)
resp.input_tokens                          # -> an int; the exact token count for that input

Embeddings: meaning as vectors

Before a token can be processed it's turned into an embedding — a vector of numbers (hundreds to thousands of dimensions) that positions the token in a space where meaning is geometric. Tokens with similar meaning or usage sit near each other. (This is the same idea you met in RAG, applied at the token level inside the model.) The model learns these vectors during training; they're the raw material attention operates on.

Attention and the transformer

The breakthrough architecture behind modern LLMs is the transformer, and its core mechanism is attention. Intuitively:

For each token, attention lets the model look at every other token in the context and decide how much each one matters for predicting what comes next.

In "The trophy didn't fit in the suitcase because it was too big", to resolve what "it" refers to, the model attends strongly to "trophy" and "suitcase" and weighs them. Attention is how the model routes information across the sequence — it's why LLMs handle long-range dependencies that older models couldn't.

A transformer stacks many layers of attention (plus feed-forward networks). Each layer refines the representation; by the top layer, the model has enough context to predict the next token well. The number of these parameters (the learned weights) is what "billions of parameters" refers to.

The context window

The context window is the maximum number of tokens the model can attend to at once — prompt plus generated output. Current frontier models are large (e.g. ~200K to 1M tokens), but it is still a hard ceiling:

• Everything the model "knows" in the moment lives in the context window. It has no memory between separate API calls — each request is stateless; you resend the whole conversation every time.
• When you approach the limit you must drop, summarize, or retrieve older content (the subject of the Context Engineering topic).
• Cost and latency grow with how full the window is, so "just stuff everything in" is rarely optimal.

Training vs. inference

• Training (done once, by the model provider): the model reads enormous text corpora and adjusts its billions of parameters to get better at next-token prediction. Expensive, slow, frozen at a cutoff date.
• Inference (what you do on every API call): the trained model runs the prediction loop on your input. Fast(er), cheap(er), and the only part you control — via the prompt and decoding settings.

This split is why the model's knowledge is frozen at its training cutoff, and why RAG (feeding fresh text at inference) and fine-tuning (a light re-training) exist.

Why models hallucinate

Because the model is optimizing next-token plausibility, not truth. Asked something it doesn't "know," it still produces the most plausible-sounding continuation — which can be confidently wrong. It has no built-in notion of "I'm not sure." This isn't a bug you can fully prompt away; it's inherent to next-token prediction. The mitigations you'll learn — grounding via RAG, retrieval, tool use, asking it to cite or abstain — all work by changing what's plausible (making the grounded answer the likely one), not by giving the model a fact-checker.

Pitfalls / misconceptions

• "It looks things up." No — with no tools/RAG, it generates from parametric memory only.
• "It remembers our last chat." No — each API call is stateless; memory is something you build by resending history.
• "Longer prompt = always better." More context can help, but it costs tokens, adds latency, and can dilute the signal ("lost in the middle"). Relevance beats volume.
• "Temperature 0 makes it deterministic and correct." It makes it more repeatable, not guaranteed identical, and never guarantees correct (see the Generation topic).

Recap

• An LLM predicts the next token over its vocabulary, then loops (autoregressive generation).
• It operates on tokens, not words — tokens drive cost, limits, and latency; measure them.
• Embeddings turn tokens into meaning-vectors; attention (the transformer's core) lets every token weigh every other — powerful but quadratic in length.
• The context window is a hard token ceiling; the model is stateless between calls.
• Knowledge is frozen at training; hallucination is inherent to plausibility-maximizing generation, mitigated by grounding, not eliminated by it.

from anthropic import Anthropic
client = Anthropic()

def tokens(text: str) -> int:
    return client.messages.count_tokens(
        model="claude-opus-4-8",
        messages=[{"role": "user", "content": text}],
    ).input_tokens

for s in [
    "The quick brown fox jumps over the lazy dog.",
    "def add(a, b): return a + b  # sum two ints",
    "🚀🔥✨🤖🧠",
    "L'intelligence artificielle transforme les entreprises.",
]:
    print(f"{tokens(s):3d} tokens / {len(s):3d} chars  = {tokens(s)/len(s):.2f}  | {s}")

from anthropic import Anthropic
client = Anthropic()

prompt = """Answer ONLY from the context. If the context doesn't contain the answer, say you don't know.

Context: Marta Vellini is a fictional person invented for a test. She has never won any award.

Question: What year did the economist Marta Vellini win the Nobel Prize?"""

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=200,
    messages=[{"role": "user", "content": prompt}],
)
print(resp.content[0].text)   # should now decline rather than invent a year