Scale by staying stateless

One container serves limited traffic. To handle more, run many identical replicas behind a load balancer. The prerequisite: your app must be stateless — no per-user data held in process memory — so any replica can serve any request. Everything that must persist (conversation history, vector store, sessions) lives in external services (the DB/vector store/memory-store from earlier tracks), not in the app instance.

             ┌── replica 1 ──┐
load balancer ├── replica 2 ──┤ → all share external state: vector DB, Postgres, Redis (memory), object store
             └── replica 3 ──┘

Because you already externalized state (RAG store, LLMOps config, memory), scaling is mostly run more copies. The AI-specific wrinkle: your throughput is often bounded not by your CPU but by the model provider's rate limits (tokens/requests per minute). Scaling replicas doesn't raise that ceiling — handle it with request queuing, backoff/retry, and provider quota increases, and use async I/O (Track 1) so each replica handles many concurrent in-flight LLM calls while waiting on the provider.

Kubernetes basics

Kubernetes (K8s) runs and scales containers across a cluster of machines. You declare desired state and K8s continuously reconciles reality to match (restarting crashed containers, replacing failed nodes). The core objects you must recognize:

• Pod — the smallest unit: one (or a few) containers running together. Usually you don't manage pods directly.
• Deployment — declares "run N replicas of this image, keep them healthy"; handles rolling updates and self-healing.
• Service — a stable network endpoint load-balancing across a deployment's pods (they come and go; the service is fixed).
• HorizontalPodAutoscaler (HPA) — adds/removes replicas based on load (CPU, or custom metrics like queue depth).

apiVersion: apps/v1
kind: Deployment                    # "keep N healthy replicas of this image"
metadata: { name: llm-app }
spec:
  replicas: 3                        # desired replica count (HPA can override)
  selector: { matchLabels: { app: llm-app } }
  template:
    metadata: { labels: { app: llm-app } }
    spec:
      containers:
        - name: app
          image: ghcr.io/acme/app:SHA        # the SHA-tagged image from CI/CD (Topic 2)
          ports: [{ containerPort: 8000 }]
          env:
            - name: OPENAI_API_KEY
              valueFrom: { secretKeyRef: { name: app-secrets, key: openai-key } }  # secret, not inline
          readinessProbe: { httpGet: { path: /health, port: 8000 } }   # don't route traffic until healthy
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler        # scale replicas 3..20 based on CPU load
metadata: { name: llm-app-hpa }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: llm-app }
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

The readinessProbe and self-healing give you resilience; the HPA gives elastic scale. This is a lot of power — and a lot of complexity.

Don't reach for Kubernetes too early

Kubernetes is powerful and operationally heavy — a cluster to run, upgrade, secure, and debug. Most AI apps, especially a consultant's SMB clients, don't need it. The pragmatic ladder (echoing the code-vs-no-code judgment from Track 4):

Option	Fits when
Single container / VPS	low traffic, one service, simplest ops (your n8n-style VPS)
Managed container platform (Azure Container Apps, Google Cloud Run, AWS App Runner)	you want autoscaling + rolling deploys without running a cluster — the sweet spot for most apps
Kubernetes (AKS/EKS/GKE)	many services, complex networking, an existing K8s org, or scale/control a managed platform can't meet

Managed container platforms give you most of Kubernetes' benefits (autoscaling, health checks, rolling updates, secrets) with a fraction of the ops burden — they run your container image and scale it, no cluster to babysit. Start there; graduate to Kubernetes only when its complexity is justified by real need.

Secrets at scale & Infrastructure as Code

Secrets don't belong in YAML or images (Topics 1–2). At scale, a secret store (Azure Key Vault, AWS Secrets Manager, or Kubernetes Secrets backed by a vault) holds them, and the platform injects them at runtime — the secretKeyRef above pulls from a managed secret, never a literal.

Infrastructure as Code (IaC) provisions all this — clusters, databases, secret stores, networking — declaratively in version-controlled files, so environments are reproducible and reviewable (the same reproducibility principle as containers and LLMOps, applied to infrastructure):

# Terraform: provision infra declaratively (Bicep is the Azure-native equivalent)
resource "azurerm_container_app" "llm_app" {
  name = "llm-app"
  template {
    container {
      image = "ghcr.io/acme/app:SHA"        # same image; infra defined as reviewable, versioned code
      cpu   = 0.5
      memory = "1Gi"
    }
    min_replicas = 2                          # autoscaling bounds, declared in code
    max_replicas = 10
  }
  secret { name = "openai-key" }              # wired to a Key Vault reference, not a literal
}

IaC means "spin up an identical staging environment" is a command, not a week of clicking — and every infra change is a reviewable diff.

Pitfalls

• Stateful app instances. Per-user state in process memory breaks horizontal scaling — externalize all state.
• Scaling into a rate limit. More replicas don't raise the provider's token/request cap — queue, back off, raise quota, go async.
• Premature Kubernetes. A cluster you don't need is pure ops overhead — prefer a managed container platform until you truly need K8s.
• Secrets in YAML/images. Use a vault + runtime injection (secretKeyRef), never literals in manifests.
• Click-ops infrastructure. Manually configured infra isn't reproducible or reviewable — use IaC.

Recap

• Scale LLM services by keeping them stateless and running horizontal replicas behind a load balancer, with all state in external services.
• Throughput is often capped by the provider's rate limits, not your compute — handle with queuing, backoff, quota, and async.
• Kubernetes (pods/deployments/services/HPA) gives elastic, self-healing scale — but is heavy; most apps are better on a managed container platform (Azure Container Apps, Cloud Run).
• Keep secrets in a vault injected at runtime, and provision everything with IaC (Terraform/Bicep) for reproducible, reviewable infrastructure.
• Don't over-reach — match the platform to real need, the same judgment as code-vs-no-code.
• Worked examples map to the exercises: statelessness + rate limits (Ex 1), reading K8s objects (Ex 2), platform choice (Ex 3), secrets flow + IaC (Ex 4).

Deployment(replicas: 3, image: app:SHA, readinessProbe: /health)
Service(selects app=llm-app)
HorizontalPodAutoscaler(min: 3, max: 20, cpu target: 70%)