Calling LLM APIs
Making real API calls to LLM providers: the messages/chat request shape, system vs user roles, multi-turn conversations (stateless — you resend history), structured (JSON-schema) outputs, function/tool calling, streaming, and robust error handling.
- Make a basic API call and read the response, with roles (system/user/assistant).
- Maintain a multi-turn conversation given that the API is stateless.
- Get reliable structured (JSON-schema) output instead of parsing free text.
- Use function/tool calling so the model can invoke your code.
- Stream responses and handle errors with typed exceptions.
Examples use the Anthropic Python SDK (pip install anthropic) with current model IDs
(claude-opus-4-8, claude-sonnet-5, claude-haiku-4-5). OpenAI, Mistral, and Gemini SDKs have the
same shape — a chat/messages endpoint, roles, tools, streaming — so the concepts transfer; only the
method names and exact fields differ. Always check the provider's current docs for specifics.
The basic call
The core endpoint is "messages" (or "chat completions"): you send a list of role-tagged messages,
you get a response. Set your key in the environment (ANTHROPIC_API_KEY) — never hard-code it.
from anthropic import Anthropic
client = Anthropic() # reads ANTHROPIC_API_KEY from the environment
resp = client.messages.create(
model="claude-opus-4-8", # which model to use
max_tokens=1024, # cap on output tokens (required on Anthropic)
messages=[ # the conversation so far, as a list of role/content dicts
{"role": "user", "content": "Explain what an embedding is in one sentence."},
],
)
# response.content is a LIST of content blocks (text, tool_use, …) — not a bare string.
# Guard on block.type before reading .text (a thinking or tool block would have no .text):
for block in resp.content:
if block.type == "text":
print(block.text)
response.content is a list of blocks, not a string. Reading resp.content[0].text blindly
breaks the moment the first block isn't text. Filter by block.type == "text".
Roles: system, user, assistant
- system — instructions that set behavior/persona for the whole conversation. On Anthropic it's a
top-level
system=parameter; on OpenAI-style APIs it's a message withrole: "system". - user — the human's messages.
- assistant — the model's previous replies (you include these to give it conversational memory).
resp = client.messages.create(
model="claude-opus-4-8", max_tokens=1024,
system="You are a concise senior AI engineer. Answer in at most 3 sentences.", # persona/rules
messages=[{"role": "user", "content": "When should I use RAG vs fine-tuning?"}],
)
↳ Exercise 1 has you make your first call with a system prompt and safely extract the text — the foundation everything else builds on.
Multi-turn conversations (the API is stateless)
The model remembers nothing between calls. To continue a conversation you resend the whole history each time, appending the new turn. (This is why long conversations cost more — you pay to re-send prior tokens each request; prompt caching, covered later, reduces that.)
messages = [] # our running history (like an array)
def ask(user_text: str) -> str:
messages.append({"role": "user", "content": user_text}) # add the new user turn
resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024, messages=messages)
answer = next(b.text for b in resp.content if b.type == "text") # first text block
messages.append({"role": "assistant", "content": answer}) # store the reply so the next turn has context
return answer
ask("My name is Pierre.")
ask("What's my name?") # -> "Pierre" — only because we resent the prior turns
↳ Exercise 2 has you build this tiny conversation loop and prove the statelessness: the model only "remembers" what you resend.
Structured outputs (JSON, reliably)
Parsing free-text answers is brittle. Instead, constrain the model to a JSON schema so you get
valid, parseable data. The Anthropic SDK's messages.parse() validates the response against a
schema (here a Pydantic model — like a typed shape / zod schema) and hands you a typed object:
from pydantic import BaseModel # Pydantic = runtime-validated typed models (≈ zod)
class Extraction(BaseModel): # the shape you want back
name: str
company: str
wants_demo: bool
resp = client.messages.parse( # parse() = create() + schema validation
model="claude-opus-4-8", max_tokens=1024,
messages=[{"role": "user",
"content": "Jane Doe from Acme asked for a demo of our RAG product."}],
output_format=Extraction, # constrain output to this schema
)
data = resp.parsed_output # a validated Extraction instance (or None on refusal)
data.name, data.company, data.wants_demo # "Jane Doe", "Acme", True — typed, no regex parsing
↳ Exercise 3 has you extract structured fields from messy text with a schema and get back a typed object — the pattern for every "turn text into data" task.
Function / tool calling
Tool calling lets the model ask your code to run a function and use the result. You describe the
tool (name, description, input schema); the model, when it decides it needs it, returns a tool_use
block with arguments; you run the function and send the result back; the model continues. The SDK's
tool runner automates that loop:
from anthropic import Anthropic, beta_tool
client = Anthropic()
@beta_tool # decorator: turns a typed function into a tool
def get_weather(city: str) -> str:
"""Get the current weather for a city.""" # the docstring becomes the tool description
return f"18°C and sunny in {city}" # your real implementation would call an API
runner = client.beta.messages.tool_runner( # drives the ask→call→result→continue loop for you
model="claude-opus-4-8", max_tokens=1024,
tools=[get_weather],
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
)
for message in runner: # iterate until the model stops calling tools
for block in message.content:
if block.type == "text":
print(block.text) # final answer uses the tool's result
The mental model: the model decides when and with what arguments; your code decides what actually happens. This is the seed of agents (next track) — an agent is a model in a loop with tools.
↳ Exercise 4 has you define a simple tool (e.g. a calculator or a fake lookup) and let the model call it — your first taste of the agent loop.
Streaming and error handling
For long outputs, stream (see the Generation topic). And wrap calls with the SDK's typed exceptions rather than string-matching error text — handle retryable errors (rate limits, 5xx) differently from client errors (bad request, auth):
import anthropic
try:
resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024,
messages=[{"role": "user", "content": "hi"}])
except anthropic.RateLimitError as e: # 429 — back off and retry (SDK also auto-retries some)
retry_after = e.response.headers.get("retry-after", "60")
print(f"Rate limited; retry after {retry_after}s")
except anthropic.APIStatusError as e: # any other non-2xx — inspect status/message
print(f"API error {e.status_code}: {e.message}")
except anthropic.APIConnectionError: # network failure before a response
print("Network problem; check connectivity")
The SDKs auto-retry 429/5xx with backoff (a couple of times by default) — only add custom retry logic when you need behavior beyond that. Catch specific exceptions, most-specific first.
Recap
- The messages/chat call takes role-tagged messages (system/user/assistant) and returns a
list of content blocks — filter by
type == "text". - The API is stateless: resend the full history each turn for a multi-turn conversation.
- Prefer structured outputs (JSON schema /
messages.parse) over parsing free text. - Tool calling lets the model invoke your functions — the model chooses when/args, your code executes; the tool-runner automates the loop. This is the basis of agents.
- Stream long outputs and handle errors with typed exceptions, distinguishing retryable from non-retryable.
Calling LLM APIs — Exercises
pip install anthropic pydantic and set ANTHROPIC_API_KEY. The patterns transfer to other
providers' SDKs.
Exercise 1 — First call with a system prompt
Make a call with a system prompt that constrains the answer (e.g. "answer in exactly one sentence"), and extract the text safely (guarding on block type).
Reference solution
from anthropic import Anthropic
client = Anthropic()
resp = client.messages.create(
model="claude-opus-4-8", max_tokens=200,
system="You are a senior AI engineer. Answer in exactly one sentence.",
messages=[{"role": "user", "content": "What is retrieval-augmented generation?"}],
)
text = next((b.text for b in resp.content if b.type == "text"), "") # safe extraction
print(text)
print("stop_reason:", resp.stop_reason) # sanity-check it finished ('end_turn')
The next(... for ... if ...) with a default is the idiomatic safe way to pull the first text block —
never index resp.content[0].text blindly.
Exercise 2 — A stateless conversation loop
Build a small loop that keeps history and prove statelessness: tell the model a fact in turn 1, ask it back in turn 2 (works), then start a fresh history and ask again (fails).
Reference solution
from anthropic import Anthropic
client = Anthropic()
def make_asker():
messages = []
def ask(text: str) -> str:
messages.append({"role": "user", "content": text})
resp = client.messages.create(model="claude-opus-4-8", max_tokens=300, messages=messages)
answer = next(b.text for b in resp.content if b.type == "text")
messages.append({"role": "assistant", "content": answer})
return answer
return ask
ask = make_asker()
print(ask("My favorite tool is LangGraph."))
print(ask("What's my favorite tool?")) # -> LangGraph : it saw the resent history
fresh = make_asker()
print(fresh("What's my favorite tool?")) # -> it has no idea : new history, nothing resent
The contrast makes statelessness concrete: memory is entirely the history you resend, not something the API holds.
Exercise 3 — Structured extraction
Extract structured fields from a messy sentence into a typed object using a schema.
Reference solution
from pydantic import BaseModel
from anthropic import Anthropic
client = Anthropic()
class Lead(BaseModel):
name: str
company: str
interest: str
wants_demo: bool
resp = client.messages.parse(
model="claude-opus-4-8", max_tokens=500,
messages=[{"role": "user", "content":
"Hi, I'm Tom Bernard at Mirakl — really interested in your agent-building tooling, "
"can we set up a demo next week?"}],
output_format=Lead,
)
lead = resp.parsed_output
assert lead is not None # None would mean a refusal
print(lead.name, lead.company, lead.interest, lead.wants_demo)
# Tom Bernard Mirakl agent-building tooling True
No regex, no fragile string parsing — the schema guarantees a typed, validated object (or None on
refusal, which you should handle).
Exercise 4 — Let the model call a tool
Define one tool the model can call (a calculator or a fake data lookup) and let the tool runner drive the loop. Confirm the final answer used the tool's result.
Reference solution
from anthropic import Anthropic, beta_tool
client = Anthropic()
@beta_tool
def order_status(order_id: str) -> str:
"""Look up the shipping status of an order by its ID."""
fake_db = {"A100": "shipped, arrives Tuesday", "A200": "processing"}
return fake_db.get(order_id, "unknown order")
runner = client.beta.messages.tool_runner(
model="claude-opus-4-8", max_tokens=500,
tools=[order_status],
messages=[{"role": "user", "content": "Where is my order A100?"}],
)
for message in runner:
for block in message.content:
if block.type == "text":
print(block.text) # final answer mentions 'shipped, arrives Tuesday'
You never called order_status yourself — the model decided it needed it, chose the argument
("A100"), the runner executed your function, and the model wove the result into its answer. That
model-in-a-loop-with-tools is exactly what an agent is.