Capstone: Ship an End-to-End Agent

The synthesis project: take one real problem from discovery to a deployed, evaluated, guardrailed, cost-justified AI system — spanning every track — and package it as a portfolio case study that proves you can deliver business value, not just call an API.

~12hportfoliocapstoneconsulting
Learning objectives
  • Take one real problem from discovery to a deployed, evaluated, guardrailed AI system.
  • Deliberately exercise every track — and prove it with a track→capstone checklist.
  • Scope a slice that is complete end-to-end, not broad and half-built.
  • Package the result as a portfolio case study that leads with business value, not tech.
Note

This is the whole course, applied once. Everything so far built a skill; the capstone integrates them into a single shipped artifact and the story you tell about it. Its output is two things: a working system, and a case study that becomes your portfolio — the proof a client or employer actually evaluates. Budget real time (≈12h+); this is a build, not a reading.

The brief

Build and ship one complete AI system that solves a real, narrow problem — end to end. Pick a domain you know or care about (support assistant over docs, a research agent, an internal-tool automation, a domain-specific Q&A). The requirement isn't scale or novelty; it's completeness across the lifecycle: a real input produces a real, grounded, evaluated output that a user would act on, running in a deployed environment, with the business case written up.

Good capstone shapes (all doable as a narrow slice):

  • A RAG assistant over a real document set, with citations, abstention, and an eval suite.
  • A tool-using agent that automates a specific multi-step task with human approval on consequential actions.
  • A hybrid (Track 4) — a no-code trigger calling your deployed code service — mirroring real client delivery.

The lifecycle — every track, once

The capstone is the course in sequence. Walk each phase, pulling the specific track:

Phase 1 — Discover & scope (Track 8.1). Write the problem, its cost, success metrics, and a narrow first slice. Even if it's a personal project, do this honestly — it's what a client engagement starts with.

Phase 2 — Build the core (Tracks 1–4).

  • A clean Python service (FastAPI), tested, typed (Track 1).
  • The LLM integration — prompts, structured outputs, the right model (Track 2).
  • If knowledge-based: the RAG pipeline — chunk, embed, vector store, hybrid retrieval, reranking (Track 3).
  • If multi-step: the agent — tool calling, the loop, memory, orchestration; and the honest chain-vs-agent call (Track 4).

Phase 3 — Evaluate & harden (Tracks 5–6).

  • A golden set + eval suite with real metrics (faithfulness, task success) and a threshold (Track 5).
  • Observability — traces, cost, latency (Track 5).
  • Guardrails, safety, and data protection — input/output filtering, PII handling, and (if multi-tenant) isolation; injection-aware design (Track 6).

Phase 4 — Deploy (Track 7).

  • Containerize it; run it somewhere real (a managed container platform is plenty) (Track 7).
  • A minimal CI/CD pipeline with the eval gate (Track 7.2).

Phase 5 — Package & present (Track 8).

  • The ROI/business case — value, honest TCO, payback (Track 8.2).
  • The stakeholder-ready narrative and a demo on realistic inputs (Track 8.3).
  • The case study write-up (below).
Tip

Exercise 1 has you run Phase 1 for your chosen capstone — the discovery + scope doc that everything else builds on.

The track → capstone checklist

Use this to verify you actually exercised the whole course. Aim to check every row (skip a row only with a reason — e.g. "no multi-tenancy, single user"):

TrackCapstone evidence
1 PythonTested, typed FastAPI service
2 LLM FoundationsDeliberate model choice; structured outputs; tuned prompts
3 RAGChunking + vector store + hybrid retrieval + citations (if knowledge-based)
4 AgenticTool calling / agent loop with a justified chain-vs-agent decision
5 Eval/OpsGolden set + eval suite + threshold; traces + cost/latency
6 SecurityGuardrails, PII handling, injection-aware design, tenant isolation (if applicable)
7 DeployContainerized + deployed + CI with an eval gate
8 DeliveryDiscovery doc, ROI/business case, case study, realistic demo
Tip

Exercise 2 has you build the core (Phases 2–3) for your slice, and Exercise 3 has you deploy it with an eval-gated pipeline (Phase 4) — the parts that turn a notebook into a system.

Scope discipline (the thing that makes it finishable)

The capstone fails the same way real projects do: over-scoping. Apply Track 8.1 ruthlessly — one workflow, one user, one document set, one channel. A RAG assistant over 200 pages of one team's docs, fully evaluated and deployed, is a far stronger capstone than a half-built platform over 5,000. Depth across the lifecycle beats breadth of features. Build the riskiest part (usually retrieval quality or a flaky integration) first so you learn early whether the slice is viable.

Watch out

"Complete and narrow" beats "broad and impressive-looking." A reviewer trusts a small thing that genuinely works end-to-end — deployed, evaluated, cost-justified — far more than a big demo that only runs on the presenter's three inputs. The capstone proves you can ship, which is the rare skill.

The case study (your actual deliverable)

The system is the work; the case study is the asset. It's how a client or employer evaluates you, and for a consultancy it's a reusable sales artifact. Structure it as the story of value delivered, not a tech tour:

  1. The problem — what it was and what it cost (Track 8.1 framing).
  2. The approach — the solution and why this scope (and what you deliberately deferred).
  3. How it works — a brief, honest architecture (RAG/agent/deploy), at a stakeholder's altitude.
  4. Results — eval scores tied to a business metric, cost/latency, and the ROI/payback (Track 8.2).
  5. Honesty — limitations, what you'd do next, what you learned. (Naming limits builds credibility — Track 8.3.)
  6. Demo — a link or recording on realistic inputs, including an abstention and a graceful error.

Lead with the problem and the outcome; the technology is supporting evidence. That framing — value first, tech as proof — is exactly what separates a consultant from a coder, and it's the note the whole curriculum ends on.

Tip

Exercise 4 has you write the case study for your capstone — the portfolio artifact that proves the whole course.

Pitfalls

  • Over-scoping. The classic capstone killer — narrow to one complete slice; depth over breadth.
  • Skipping the unglamorous tracks. No evals, no guardrails, no deploy = a demo, not a system. The checklist rows are the point.
  • A notebook, not a system. If it only runs locally in a notebook, you skipped deploy — ship it somewhere real.
  • A tech-tour case study. Leading with architecture instead of problem/outcome reads as a coder, not a consultant.
  • Demoing on easy inputs. Show it on realistic and hard cases, including an abstention — the demo trap from Track 8.3.

Recap

  • The capstone is the entire course applied once: one real, narrow problem taken from discovery → build → evaluate/harden → deploy → package.
  • Deliberately exercise every track and verify with the track→capstone checklist — evals, guardrails, and deploy are what make it a system, not a demo.
  • Scope ruthlessly: one complete, deployed, evaluated slice beats a broad half-built platform; build the riskiest part first.
  • The case study is the real deliverable — lead with problem and outcome, tie eval results to a business metric and ROI, and be honest about limits.
  • Value first, technology as proof — the stance that turns everything you learned into delivered value.
  • Worked examples map to the exercises: scope your capstone (Ex 1), build the core (Ex 2), deploy with an eval gate (Ex 3), write the case study (Ex 4).

Capstone: Ship an End-to-End Agent — Exercises

These four exercises ARE the capstone project, run in phases. Unlike other topics, expect this to take many hours across sessions. Pick one real, narrow problem and carry it all the way through.


Exercise 1 — Discover & scope your capstone (Phase 1)

Choose your problem and write the discovery + scope doc: the problem and its cost, success metrics, the narrow first slice (one workflow / user / doc set / channel), and what you're explicitly deferring.

Reference solution (worked example)

Example capstone: a RAG assistant over a nonprofit's grant-guidelines docs.

  • Problem & cost: Staff spend ~6 hrs/week hunting through 180 pages of funder guidelines to answer applicant questions; slow and error-prone.
  • Success metrics: ≥85% of a 30-question golden set answered correctly with citation; correct abstention on out-of-scope questions; answer in under 5s.
  • First slice (in): the 180 guideline pages; a web query box; grounded RAG with citations + abstention; single user group (grants team).
  • Deferred: other document sets, Slack integration, multi-step agent actions, multi-tenant.
  • Riskiest part to build first: retrieval quality on dense policy text (chunking + hybrid).

The point: even for a personal project, do discovery honestly — it's the doc a real engagement opens with, and it defines "done" so scope can't creep. Yours will differ; keep it narrow and end-to-end.


Exercise 2 — Build the core + harden (Phases 2–3)

Implement the end-to-end slice (Tracks 1–4) and add evals + guardrails (Tracks 5–6). Produce a runnable system and an eval score against your golden set.

Reference solution (checklist)

For the grant-guidelines RAG example, "done" for this exercise means:

  • Track 1: a tested FastAPI service with a /ask endpoint.
  • Track 2: a deliberate model choice; a grounded prompt with abstention; structured output where useful.
  • Track 3: guideline pages chunked (recursive, token-budgeted), embedded, in a vector store; hybrid retrieval; citations returned.
  • Track 4: likely a chain not an agent (fixed retrieve→generate) — and you can justify that call.
  • Track 5: a 30-case golden set; an eval script reporting faithfulness + citation-correctness + abstention, with a threshold; basic tracing of prompt/response/tokens/latency.
  • Track 6: input/output guardrails, PII handling if any, injection-aware handling of retrieved text.

Deliverable: the running service + a printed eval scorecard (e.g. faithfulness 0.9, citation 0.87, abstention 1.0). If a metric is below target, that's your next iteration — which is the real work.


Exercise 3 — Deploy with an eval gate (Phase 4)

Containerize the system, deploy it somewhere real (a managed container platform is fine), and set up a minimal CI pipeline that runs the eval suite as a gate.

Reference solution (checklist)
  • Containerize (Track 7.1): a slim, multi-stage Dockerfile; config/secrets via env; a .dockerignore.
  • Deploy (Track 7.4): push the image to a registry and run it on a managed container platform (Azure Container Apps / Cloud Run / App Runner) — a public or auth-gated URL you can demo.
  • CI + eval gate (Track 7.2): a workflow that lints, tests, builds the SHA-tagged image, and runs run_eval.py --min-score 0.85 — failing the build on regression.

Deliverable: a live URL (or a docker run that anyone can reproduce) and a CI run showing the eval gate passing. If it only runs in your notebook, this phase isn't done — the capstone proves you can ship, and shipping means it runs somewhere other than your laptop.


Exercise 4 — Write the case study (Phase 5)

Write the portfolio case study for your capstone: problem, approach, how it works, results (tied to a business metric + ROI), honest limitations, and a demo on realistic inputs.

Reference solution (structure + example)

Case study: Grant-guidelines assistant

  1. Problem: Grants team spent ~6 hrs/week searching 180 pages of funder guidelines — slow, inconsistent answers to applicants.
  2. Approach: A grounded RAG assistant over the guidelines with citations and abstention; scoped to one team and one doc set first (deferred integrations).
  3. How it works: brief, honest architecture — chunked guidelines in a vector store, hybrid retrieval, an LLM answering only from retrieved context with sources. (Stakeholder altitude, not a code dump.)
  4. Results: 89% golden-set accuracy with citations, correct abstention on out-of-scope questions, ~3s answers, ~€40/mo run cost → 5 hrs/week saved (€600/mo value), payback in weeks. (Eval tied to a business metric + ROI.)
  5. Honesty: limits — struggles on cross-referenced clauses; next step is parent-child chunking. What I'd add: feedback loop, Slack surface.
  6. Demo: a link/recording showing a real question answered-with-citation, and an out-of-scope question correctly refused.

Lead with problem and outcome; the tech is supporting evidence. This case study — not the code — is the asset a client or employer evaluates, and for a consultancy it's a reusable sales piece. That "value first, tech as proof" framing is the whole point of the course.