The pipeline

CI/CD automates commit → production: CI (continuous integration) runs checks on every change; CD (continuous delivery/deployment) ships the artifact that passed. For an AI app the pipeline is:

push/PR → lint → unit tests → build & tag image → EVAL GATE → push to registry → deploy (canary → full)

Everything before deploy is a gate: fail any step and the change doesn't ship. The AI-specific insight is that unit tests are necessary but not sufficient — they verify your code, not your model's output quality. That's what the eval gate adds.

Here's a CI workflow (GitHub Actions) for the lint/test/build stages:

# .github/workflows/ci.yml
name: ci
on: [push, pull_request]                 # run on every push and PR
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4          # get the code
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: ruff check .                  # lint (fast, catches style/errors) — a gate
      - run: mypy .                        # type-check — a gate
      - run: pytest                        # unit tests (Track 1) — a gate
      - run: docker build -t app:${{ github.sha }} .   # build image, tagged by the commit SHA

The eval gate

This is what makes it AI CI/CD. Run the offline eval regression suite (Eval Design, Track 5) as a pipeline step, and fail the build if quality drops below threshold — including on weak slices. A prompt or model change that passes unit tests but tanks quality is caught here, not by users:

  eval:
    runs-on: ubuntu-latest
    needs: test                          # only run if unit tests passed
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python run_eval.py --dataset golden_set.jsonl --min-score 0.85   # the eval gate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}   # secret injected by the CI platform, never in code
# run_eval.py exits non-zero (fails the job) if overall or any weak-slice score is below threshold.

# run_eval.py — the script the gate runs
import sys
result = run_eval(feature, load_dataset())          # from Track 5
if result["overall"] < 0.85 or result["by_tag"].get("adversarial", 1) < 0.70:
    print(f"EVAL REGRESSION: {result}"); sys.exit(1)  # non-zero exit → the CI job fails → merge blocked
print(f"eval passed: {result}")

Versioning images & the registry

Every build produces an immutable image tagged to its git commit (app:<sha>), pushed to a container registry (Docker Hub, GitHub Container Registry, Azure Container Registry). Tagging by SHA means every deployed artifact traces back to exact code, and any past version can be redeployed — the basis of rollback:

  build-push:
    runs-on: ubuntu-latest
    needs: [test, eval]                  # only build the deployable image if BOTH gates passed
    steps:
      - uses: actions/checkout@v4
      - run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
      - run: |
          docker build -t ghcr.io/acme/app:${{ github.sha }} -t ghcr.io/acme/app:latest .
          docker push ghcr.io/acme/app:${{ github.sha }}     # immutable, SHA-tagged
          docker push ghcr.io/acme/app:latest                # moving 'latest' pointer

Tag by immutable SHA for anything you deploy or roll back to; latest is a convenience pointer, never the thing you pin in production (same lesson as pinning dated model snapshots in LLMOps).

Deploy strategies & rollback

CD ships the passed artifact. Two safe strategies (both from the LLMOps rollout discipline):

• Blue-green — run two environments; deploy to the idle one (green), test it, then switch traffic from blue to green all at once. Instant rollback = switch back to blue.
• Canary — route a small % of traffic to the new version, watch metrics (quality/cost/latency from Observability), then ramp up or roll back.

  deploy:
    runs-on: ubuntu-latest
    needs: build-push
    if: github.ref == 'refs/heads/main'   # only deploy from main, and only after all gates passed
    steps:
      - run: ./deploy.sh ghcr.io/acme/app:${{ github.sha }} --canary 10   # deploy this SHA to 10% first
      # promotion to 100% (or rollback) is gated on the canary's live metrics

The through-line with LLMOps (Track 5): the pipeline encodes the discipline — gates block bad changes, SHA-tagged images make rollback instant, and canary/blue-green contain a bad deploy. CI/CD is where those principles become automation.

Pitfalls

• Unit tests as the only gate. They check code, not model quality — add the eval gate or you ship silent regressions.
• Deploying latest. Pin the SHA-tagged image so you know (and can roll back to) exactly what's running.
• Secrets in the repo. Use the CI platform's secret store; never commit keys or bake them into images.
• Flaky eval gate. Non-deterministic evals with too-tight thresholds block good changes — curate the set, pin the model, add margin.
• No rollback path. A pipeline that can deploy but not instantly roll back turns a bad release into an outage.

Recap

• CI/CD automates commit → production through gates: lint → tests → build/tag image → eval gate → registry → deploy.
• The eval suite is a first-class gate — unit tests verify code, evals verify model quality; fail the build on regression (incl. weak slices).
• Version images by git SHA and push to a registry so every deploy is traceable and any version is redeployable.
• Deploy with canary or blue-green and keep rollback instant (switch back / redeploy the prior SHA).
• CI/CD is where the LLMOps discipline becomes automation — gates, versioning, and safe rollout, enforced.
• Worked examples map to the exercises: a CI workflow (Ex 1), an eval gate (Ex 2), SHA image versioning (Ex 3), a deploy/rollback strategy (Ex 4).

name: ci
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: ruff check .
      - run: mypy .
      - run: pytest
      - run: docker build -t app:${{ github.sha }} .

  eval:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python run_eval.py --min-score 0.85
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

# run_eval.py
import sys
r = run_eval(feature, load_dataset())
if r["overall"] < 0.85 or r["by_tag"].get("adversarial", 1) < 0.70:
    print("EVAL REGRESSION:", r); sys.exit(1)   # non-zero -> job fails -> merge blocked
print("eval passed:", r)

docker build -t ghcr.io/acme/app:$GIT_SHA -t ghcr.io/acme/app:latest .
echo "$REGISTRY_TOKEN" | docker login ghcr.io -u "$USER" --password-stdin
docker push ghcr.io/acme/app:$GIT_SHA
docker push ghcr.io/acme/app:latest
docker pull ghcr.io/acme/app:$GIT_SHA        # redeploy any exact past version