Calling LLM APIs

Making real API calls to LLM providers: the messages/chat request shape, system vs user roles, multi-turn conversations (stateless — you resend history), structured (JSON-schema) outputs, function/tool calling, streaming, and robust error handling.

~3hllm-apisapi
Learning objectives
  • Make a basic API call and read the response, with roles (system/user/assistant).
  • Maintain a multi-turn conversation given that the API is stateless.
  • Get reliable structured (JSON-schema) output instead of parsing free text.
  • Use function/tool calling so the model can invoke your code.
  • Stream responses and handle errors with typed exceptions.
Note

Examples use the Anthropic Python SDK (pip install anthropic) with current model IDs (claude-opus-4-8, claude-sonnet-5, claude-haiku-4-5). OpenAI, Mistral, and Gemini SDKs have the same shape — a chat/messages endpoint, roles, tools, streaming — so the concepts transfer; only the method names and exact fields differ. Always check the provider's current docs for specifics.

The basic call

The core endpoint is "messages" (or "chat completions"): you send a list of role-tagged messages, you get a response. Set your key in the environment (ANTHROPIC_API_KEY) — never hard-code it.

from anthropic import Anthropic

client = Anthropic()                       # reads ANTHROPIC_API_KEY from the environment

resp = client.messages.create(
    model="claude-opus-4-8",              # which model to use
    max_tokens=1024,                       # cap on output tokens (required on Anthropic)
    messages=[                              # the conversation so far, as a list of role/content dicts
        {"role": "user", "content": "Explain what an embedding is in one sentence."},
    ],
)

# response.content is a LIST of content blocks (text, tool_use, …) — not a bare string.
# Guard on block.type before reading .text (a thinking or tool block would have no .text):
for block in resp.content:
    if block.type == "text":
        print(block.text)
Watch out

response.content is a list of blocks, not a string. Reading resp.content[0].text blindly breaks the moment the first block isn't text. Filter by block.type == "text".

Roles: system, user, assistant

  • system — instructions that set behavior/persona for the whole conversation. On Anthropic it's a top-level system= parameter; on OpenAI-style APIs it's a message with role: "system".
  • user — the human's messages.
  • assistant — the model's previous replies (you include these to give it conversational memory).
resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024,
    system="You are a concise senior AI engineer. Answer in at most 3 sentences.",  # persona/rules
    messages=[{"role": "user", "content": "When should I use RAG vs fine-tuning?"}],
)
Tip

Exercise 1 has you make your first call with a system prompt and safely extract the text — the foundation everything else builds on.

Multi-turn conversations (the API is stateless)

The model remembers nothing between calls. To continue a conversation you resend the whole history each time, appending the new turn. (This is why long conversations cost more — you pay to re-send prior tokens each request; prompt caching, covered later, reduces that.)

messages = []                                              # our running history (like an array)

def ask(user_text: str) -> str:
    messages.append({"role": "user", "content": user_text})     # add the new user turn
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024, messages=messages)
    answer = next(b.text for b in resp.content if b.type == "text")   # first text block
    messages.append({"role": "assistant", "content": answer})   # store the reply so the next turn has context
    return answer

ask("My name is Pierre.")
ask("What's my name?")            # -> "Pierre" — only because we resent the prior turns
Tip

Exercise 2 has you build this tiny conversation loop and prove the statelessness: the model only "remembers" what you resend.

Structured outputs (JSON, reliably)

Parsing free-text answers is brittle. Instead, constrain the model to a JSON schema so you get valid, parseable data. The Anthropic SDK's messages.parse() validates the response against a schema (here a Pydantic model — like a typed shape / zod schema) and hands you a typed object:

from pydantic import BaseModel                # Pydantic = runtime-validated typed models (≈ zod)

class Extraction(BaseModel):                  # the shape you want back
    name: str
    company: str
    wants_demo: bool

resp = client.messages.parse(                 # parse() = create() + schema validation
    model="claude-opus-4-8", max_tokens=1024,
    messages=[{"role": "user",
               "content": "Jane Doe from Acme asked for a demo of our RAG product."}],
    output_format=Extraction,                 # constrain output to this schema
)

data = resp.parsed_output                     # a validated Extraction instance (or None on refusal)
data.name, data.company, data.wants_demo      # "Jane Doe", "Acme", True — typed, no regex parsing
Tip

Exercise 3 has you extract structured fields from messy text with a schema and get back a typed object — the pattern for every "turn text into data" task.

Function / tool calling

Tool calling lets the model ask your code to run a function and use the result. You describe the tool (name, description, input schema); the model, when it decides it needs it, returns a tool_use block with arguments; you run the function and send the result back; the model continues. The SDK's tool runner automates that loop:

from anthropic import Anthropic, beta_tool

client = Anthropic()

@beta_tool                                     # decorator: turns a typed function into a tool
def get_weather(city: str) -> str:
    """Get the current weather for a city."""  # the docstring becomes the tool description
    return f"18°C and sunny in {city}"          # your real implementation would call an API

runner = client.beta.messages.tool_runner(     # drives the ask→call→result→continue loop for you
    model="claude-opus-4-8", max_tokens=1024,
    tools=[get_weather],
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
)
for message in runner:                          # iterate until the model stops calling tools
    for block in message.content:
        if block.type == "text":
            print(block.text)                   # final answer uses the tool's result

The mental model: the model decides when and with what arguments; your code decides what actually happens. This is the seed of agents (next track) — an agent is a model in a loop with tools.

Tip

Exercise 4 has you define a simple tool (e.g. a calculator or a fake lookup) and let the model call it — your first taste of the agent loop.

Streaming and error handling

For long outputs, stream (see the Generation topic). And wrap calls with the SDK's typed exceptions rather than string-matching error text — handle retryable errors (rate limits, 5xx) differently from client errors (bad request, auth):

import anthropic

try:
    resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024,
                                  messages=[{"role": "user", "content": "hi"}])
except anthropic.RateLimitError as e:          # 429 — back off and retry (SDK also auto-retries some)
    retry_after = e.response.headers.get("retry-after", "60")
    print(f"Rate limited; retry after {retry_after}s")
except anthropic.APIStatusError as e:          # any other non-2xx — inspect status/message
    print(f"API error {e.status_code}: {e.message}")
except anthropic.APIConnectionError:           # network failure before a response
    print("Network problem; check connectivity")
Note

The SDKs auto-retry 429/5xx with backoff (a couple of times by default) — only add custom retry logic when you need behavior beyond that. Catch specific exceptions, most-specific first.

Recap

  • The messages/chat call takes role-tagged messages (system/user/assistant) and returns a list of content blocks — filter by type == "text".
  • The API is stateless: resend the full history each turn for a multi-turn conversation.
  • Prefer structured outputs (JSON schema / messages.parse) over parsing free text.
  • Tool calling lets the model invoke your functions — the model chooses when/args, your code executes; the tool-runner automates the loop. This is the basis of agents.
  • Stream long outputs and handle errors with typed exceptions, distinguishing retryable from non-retryable.

Calling LLM APIs — Exercises

pip install anthropic pydantic and set ANTHROPIC_API_KEY. The patterns transfer to other providers' SDKs.


Exercise 1 — First call with a system prompt

Make a call with a system prompt that constrains the answer (e.g. "answer in exactly one sentence"), and extract the text safely (guarding on block type).

Reference solution
from anthropic import Anthropic
client = Anthropic()

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=200,
    system="You are a senior AI engineer. Answer in exactly one sentence.",
    messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)

text = next((b.text for b in resp.content if b.type == "text"), "")   # safe extraction
print(text)
print("stop_reason:", resp.stop_reason)      # sanity-check it finished ('end_turn')

The next(... for ... if ...) with a default is the idiomatic safe way to pull the first text block — never index resp.content[0].text blindly.


Exercise 2 — A stateless conversation loop

Build a small loop that keeps history and prove statelessness: tell the model a fact in turn 1, ask it back in turn 2 (works), then start a fresh history and ask again (fails).

Reference solution
from anthropic import Anthropic
client = Anthropic()

def make_asker():
    messages = []
    def ask(text: str) -> str:
        messages.append({"role": "user", "content": text})
        resp = client.messages.create(model="claude-opus-4-8", max_tokens=300, messages=messages)
        answer = next(b.text for b in resp.content if b.type == "text")
        messages.append({"role": "assistant", "content": answer})
        return answer
    return ask

ask = make_asker()
print(ask("My favorite tool is LangGraph."))
print(ask("What's my favorite tool?"))        # -> LangGraph : it saw the resent history

fresh = make_asker()
print(fresh("What's my favorite tool?"))      # -> it has no idea : new history, nothing resent

The contrast makes statelessness concrete: memory is entirely the history you resend, not something the API holds.


Exercise 3 — Structured extraction

Extract structured fields from a messy sentence into a typed object using a schema.

Reference solution
from pydantic import BaseModel
from anthropic import Anthropic
client = Anthropic()

class Lead(BaseModel):
    name: str
    company: str
    interest: str
    wants_demo: bool

resp = client.messages.parse(
    model="claude-opus-4-8", max_tokens=500,
    messages=[{"role": "user", "content":
        "Hi, I'm Tom Bernard at Mirakl — really interested in your agent-building tooling, "
        "can we set up a demo next week?"}],
    output_format=Lead,
)

lead = resp.parsed_output
assert lead is not None                        # None would mean a refusal
print(lead.name, lead.company, lead.interest, lead.wants_demo)
# Tom Bernard  Mirakl  agent-building tooling  True

No regex, no fragile string parsing — the schema guarantees a typed, validated object (or None on refusal, which you should handle).


Exercise 4 — Let the model call a tool

Define one tool the model can call (a calculator or a fake data lookup) and let the tool runner drive the loop. Confirm the final answer used the tool's result.

Reference solution
from anthropic import Anthropic, beta_tool
client = Anthropic()

@beta_tool
def order_status(order_id: str) -> str:
    """Look up the shipping status of an order by its ID."""
    fake_db = {"A100": "shipped, arrives Tuesday", "A200": "processing"}
    return fake_db.get(order_id, "unknown order")

runner = client.beta.messages.tool_runner(
    model="claude-opus-4-8", max_tokens=500,
    tools=[order_status],
    messages=[{"role": "user", "content": "Where is my order A100?"}],
)

for message in runner:
    for block in message.content:
        if block.type == "text":
            print(block.text)      # final answer mentions 'shipped, arrives Tuesday'

You never called order_status yourself — the model decided it needed it, chose the argument ("A100"), the runner executed your function, and the model wove the result into its answer. That model-in-a-loop-with-tools is exactly what an agent is.