What to test in an AI app

LLM output is non-deterministic, so you don't assert on exact model text. Instead you test the deterministic scaffolding around the model, which is where most bugs actually live:

• Parsing/validation — does structured output parse? does a bad payload get rejected?
• Routing/tool selection — given an intent, is the right tool/branch chosen?
• Retrieval/scoring logic — pure functions like the RAG scorer or a chunker.
• Prompt assembly — is the context injected correctly, within limits?
• Adapters — is the HTTP request to the provider shaped right (mock the network)?

pytest basics

A test is a function named test_* that uses assert. Run everything with pytest.

# file: test_scoring.py
def normalize(s: str) -> str:
    return s.strip().lower()

def test_normalize_trims_and_lowercases():
    assert normalize("  RAG ") == "rag"      # plain assert; pytest shows both sides on failure

def test_normalize_empty():
    assert normalize("   ") == ""

pytest            # discovers test_*.py files and test_* functions, runs them
pytest -v         # verbose: one line per test
pytest -k normalize   # only tests matching "normalize"

To assert that something raises, use pytest.raises:

import pytest

def test_rejects_bad_input():
    with pytest.raises(ValueError):     # passes only if the block raises ValueError
        parse_config("not json")

Fixtures: reusable setup

A fixture is setup that tests request by naming it as a parameter — pytest injects the return value (dependency injection for tests). Great for building sample data or a temp resource once.

import pytest

@pytest.fixture
def sample_chunks() -> list[str]:          # the fixture "provides" this value
    return ["RAG grounds answers.", "Embeddings are vectors.", "Chunk with overlap."]

def test_retrieval_uses_chunks(sample_chunks):   # name the fixture as a param → pytest injects it
    assert len(sample_chunks) == 3
    assert any("RAG" in c for c in sample_chunks)

pytest ships useful built-in fixtures too — e.g. tmp_path gives each test a fresh temp directory.

Parametrize: one test, many cases

Instead of copy-pasting a test, feed it a table of inputs — data-driven testing, like Jest's test.each.

import pytest

@pytest.mark.parametrize("text, expected", [
    ("  RAG ", "rag"),        # each tuple is one case: (input, expected)
    ("Agent", "agent"),
    ("", ""),
])
def test_normalize_cases(text, expected):
    assert normalize(text) == expected      # runs once per row, reported separately

Mocking the LLM

Tests must not hit a real API — it's slow, costs money, and is non-deterministic. Mock the call so you control its return value. The lightweight way in pytest is monkeypatch:

# file: assistant.py
async def call_llm(prompt: str) -> str:
    ...                                   # real network call in production

async def summarise(text: str) -> str:
    raw = await call_llm(f"Summarise: {text}")
    return raw.strip()

# file: test_assistant.py
import assistant

async def test_summarise_strips(monkeypatch):
    async def fake_call_llm(prompt: str) -> str:   # a stand-in with a fixed answer
        return "  a summary  "
    monkeypatch.setattr(assistant, "call_llm", fake_call_llm)   # swap the real fn for the fake

    result = await assistant.summarise("long text")
    assert result == "a summary"          # we test OUR logic (the strip), not the model

The equivalent with the standard library is unittest.mock.patch:

from unittest.mock import patch

@patch("assistant.call_llm")              # replaces assistant.call_llm with a Mock
def test_with_patch(mock_llm):
    mock_llm.return_value = "hello"
    ...                                    # assert your code handled "hello" correctly

Static typing with mypy

Type hints are checked by mypy (a separate tool, like tsc over JS). It catches type mismatches before runtime without changing how the code runs.

pip install mypy
mypy .            # type-check the project; reports mismatches, missing returns, None misuse

def top_k(scores: list[float], k: int) -> list[float]:
    return sorted(scores, reverse=True)[:k]

top_k([0.1, 0.9], "two")   # mypy error: Argument 2 has incompatible type "str"; expected "int"

Structuring an AI app

Keep layers separated so business logic doesn't depend on frameworks or vendors — the same "depend on abstractions" idea as clean architecture elsewhere:

myapp/
  domain/        # pure logic: scoring, chunking, prompt assembly (no FastAPI, no OpenAI imports)
  services/      # orchestration: "answer a question" ties domain + adapters together
  adapters/      # the outside world: LLM client, vector DB, HTTP — swappable, mockable
  api/           # FastAPI routes: thin, just parse input → call a service → return
  tests/

The win: domain/ is trivially unit-testable (pure functions), and you can mock adapters/ to test services/ without a network. Routes stay thin.

JSON & light SQL for AI apps

Two persistence tools you'll reach for constantly.

JSON — the json module (parse/stringify, like JSON.parse/JSON.stringify):

import json

data = {"question": "what is RAG?", "top_k": 4}
text = json.dumps(data)             # dict → JSON string   (JSON.stringify)
back = json.loads(text)             # JSON string → dict   (JSON.parse)

with open("cfg.json") as f:         # read a file (context manager auto-closes it)
    cfg = json.load(f)              # parse straight from the file object

SQLite — a zero-config file database in the standard library; perfect for local AI-app state:

import sqlite3

conn = sqlite3.connect("app.db")                    # a file-backed DB (no server needed)
conn.execute("CREATE TABLE IF NOT EXISTS runs (id TEXT, score REAL)")
conn.execute("INSERT INTO runs VALUES (?, ?)", ("r1", 0.87))   # ? placeholders prevent SQL injection
conn.commit()                                        # persist the write
rows = conn.execute("SELECT id, score FROM runs").fetchall()   # [('r1', 0.87)]
conn.close()

Recap

• Test the deterministic scaffolding around the model (parsing, routing, scoring, adapters), not the model's exact words.
• pytest: plain assert, pytest.raises for errors, fixtures for setup, parametrize for data-driven cases.
• Mock external LLM/HTTP calls (monkeypatch or unittest.mock.patch) so tests are fast and deterministic.
• mypy is your static type checker (the tsc of Python).
• Separate domain / services / adapters / api so logic is pure and testable; persist with JSON and SQLite (always ? placeholders).

def keyword_match(answer: str, keywords: list[str]) -> bool:
    if not keywords:
        raise ValueError("need at least one keyword")
    text = answer.lower()
    return any(k.lower() in text for k in keywords)

import pytest
from grader import keyword_match   # assuming the function lives in grader.py

def test_matches_when_keyword_present():
    assert keyword_match("It uses RAG grounding", ["rag"]) is True

def test_no_match_when_absent():
    assert keyword_match("fine-tuning only", ["rag", "retrieval"]) is False

def test_empty_keywords_raises():
    with pytest.raises(ValueError):
        keyword_match("anything", [])

import pytest
from grader import keyword_match

@pytest.mark.parametrize("answer, keywords, expected", [
    ("It uses RAG grounding", ["rag"], True),
    ("RETRIEVAL augmented", ["retrieval"], True),      # case-insensitive
    ("fine-tuning only", ["rag", "retrieval"], False),
    ("embeddings and vectors", ["vector"], True),      # substring match
])
def test_keyword_match_cases(answer, keywords, expected):
    assert keyword_match(answer, keywords) is expected

# assistant.py
async def call_llm(prompt: str) -> str: ...       # real network call
async def summarise(text: str) -> str:
    raw = await call_llm(f"Summarise: {text}")
    return raw.strip()

# test_assistant.py  (run with: pytest;  needs pytest-asyncio or anyio for async tests)
import assistant

async def test_summarise_trims(monkeypatch):
    async def fake_call_llm(prompt: str) -> str:
        assert "Summarise:" in prompt        # you can also assert HOW your code called the dependency
        return "   a tidy summary   "
    monkeypatch.setattr(assistant, "call_llm", fake_call_llm)

    result = await assistant.summarise("some long text")
    assert result == "a tidy summary"        # tests YOUR logic, not the model

import json, sqlite3

# (a) JSON round-trip
record = {"id": "run1", "score": 0.87, "topic": "rag-fundamentals"}
with open("run.json", "w") as f:
    json.dump(record, f)                 # write dict → file as JSON
with open("run.json") as f:
    loaded = json.load(f)                # read back
assert loaded["score"] == 0.87

# (b) SQLite
conn = sqlite3.connect("runs.db")
conn.execute("CREATE TABLE IF NOT EXISTS runs (id TEXT, score REAL)")
conn.executemany(
    "INSERT INTO runs VALUES (?, ?)",    # ? placeholders — never f-string values into SQL
    [("run1", 0.87), ("run2", 0.42)],
)
conn.commit()
rows = conn.execute("SELECT id, score FROM runs ORDER BY score DESC").fetchall()
print(rows)                              # [('run1', 0.87), ('run2', 0.42)]
conn.close()