Business metrics vs. technical metrics

An eval score of 0.92 (Track 5) is a technical metric — it means nothing to a CFO. Business value is measured in the client's terms: time, money, revenue, risk. Every project needs both, with a clear line from one to the other:

Technical metric	Business metric it should ladder up to
Answer faithfulness / accuracy	Fewer escalations, lower error-handling cost, less risk
Retrieval quality / deflection	Support tickets deflected → agent hours saved → €
Latency (p95)	User satisfaction, task completion, conversion
Automation success rate	Manual hours eliminated → labor cost saved

The discipline: for every technical metric you optimize, name the business metric it moves. If you can't draw that line, you're optimizing something the client doesn't value. This is the bridge from Track 5's eval work to the boardroom.

Calculating ROI

ROI = (value delivered − total cost) / total cost. Both sides need honest numbers. Value is usually one of a few forms — quantify it in the client's units:

• Labor saved — hours eliminated × loaded hourly cost. (E.g. 200 support hrs/month × €30 = €6,000/month.)
• Revenue gained — more conversions, faster response, upsell.
• Cost avoided — fewer errors, less rework, avoided penalties.
• Risk reduced — harder to price, but real (compliance, churn).

# A simple monthly ROI sketch — the numbers you'd put in front of a client.
value_per_month = 200 * 30          # 200 support hours saved × €30/hr loaded cost = €6,000
run_cost_per_month = 800            # inference (tokens) + hosting + monitoring
build_cost = 15000                  # one-time build (or the diagnostic + build fee)
maint_per_month = 1000              # the part people forget: prompt/model upkeep, evals, incidents

monthly_net = value_per_month - run_cost_per_month - maint_per_month   # 6000 - 800 - 1000 = €4,200/month
payback_months = build_cost / monthly_net                             # 15000 / 4200 ≈ 3.6 months to break even
# Present BOTH: monthly net benefit AND payback period — a CFO reads payback fast.

Present payback period (how long until it pays for itself) and ongoing monthly net — those are the numbers a non-technical decision-maker acts on. Be conservative on value and generous on cost; an ROI case that survives scrutiny wins more than an optimistic one that collapses.

Total cost of ownership — the part people miss

The number-one ROI mistake is counting only the build cost. An AI system's real cost is dominated by what comes after launch:

• Inference cost at scale — tokens are cheap per call and expensive per million. A demo costs cents; the same app at production volume can cost thousands/month. Model it with real volume × per-token pricing (Track 5 observability gives you the numbers).
• The maintenance treadmill — prompts drift, models get deprecated, evals must be re-run, incidents happen (all of Track 5's LLMOps). This is ongoing engineering time, not a one-off.
• Human-in-the-loop cost — if the design needs human review (Track 6), that reviewer time is a running cost.
• Data & integration upkeep — the RAG corpus, connectors, and permissions need maintaining.

A project with a great build-cost ROI can be net-negative once TCO is honest — especially high-volume, low-value-per-call ones. Surfacing TCO protects the client and your reputation (an AI system that quietly costs more than it saves ends relationships).

The business case

A business case assembles discovery + ROI into a decision document a stakeholder can approve. Keep it tight:

1. The problem & its cost — from discovery, in the client's numbers ("support spends 200 hrs/month on X").
2. The proposed solution — briefly, and why this scope (Track 8 Topic 1).
3. The value — quantified benefit, conservatively.
4. The cost — build + honest TCO (run + maintenance).
5. ROI & payback — the headline numbers.
6. Risks & assumptions — what could change the math (volume, model pricing, adoption), and how you'll measure.

The best business cases are honest about uncertainty — a range with stated assumptions beats a single confident number that reality breaks.

Track value after launch

The business case is a prediction; value realization is the proof. Instrument the business metric (using the observability/eval loop from Track 5) so you can show the client the actual hours saved / tickets deflected / revenue moved after launch. This turns "we think it helped" into "here's the measured benefit" — which is what renews contracts and justifies the next phase. Predicted ROI wins the project; measured ROI wins the relationship.

Pitfalls

• Reporting only technical metrics. A 0.92 eval score means nothing to a buyer — ladder it to a business KPI.
• Counting only build cost. TCO (inference at scale + maintenance + HITL) often dominates — model it honestly.
• Demo-volume cost estimates. Costs that look tiny in a demo explode at production volume — use real volume.
• Optimistic single-number ROI. A conservative, assumption-stated case survives scrutiny; a rosy one collapses.
• Never measuring realized value. Predicted ROI isn't proof — instrument the business metric and report actuals.

Recap

• Pair every technical metric with the business metric it moves (time, money, revenue, risk) — clients buy outcomes, not eval scores.
• ROI = (value − total cost) / total cost; present payback period and monthly net in the client's units.
• Account for the total cost of ownership people miss — inference at scale and the LLMOps maintenance treadmill — and model cost at production volume.
• Assemble a tight business case (problem, solution, value, cost, ROI, risks) that's honest about uncertainty.
• Track realized value after launch — measured ROI wins the relationship.
• Worked examples map to the exercises: metric mapping (Ex 1), ROI/payback (Ex 2), TCO estimate (Ex 3), a business case (Ex 4).

value = 150 * 28          # €4,200/month saved
run = 600
maint = 900
build = 12000
monthly_net = value - run - maint      # 4200 - 600 - 900 = €2,700/month
payback = build / monthly_net          # 12000 / 2700 ≈ 4.4 months

volume = 800_000
value = volume * 0.004        # €3,200/month of work replaced
inference = volume * 0.0006   # €480/month tokens
maint = 1200
monthly_net = value - inference - maint   # 3200 - 480 - 1200 = €1,520/month