Building the Lab: safe-by-construction infra demos

The Lab (/lab) lets a visitor press a button that actually deletes a running pod and watch Kubernetes heal it — plus a bounded load test, a live event stream, the real rate limiter, and an API playground. The design constraint was blunt: never execute untrusted visitor input, and never let a demo threaten the site it lives on.

Isolation first

Every demo workload lives in a disposable demo namespace that is NetworkPolicy-default-deny in both directions — a demo pod can reach DNS and its own namespace, nothing else. The core service's new Kubernetes power is a namespaced Role (list/delete pods in demo only), never a ClusterRole, proven scoped with kubectl auth can-i:

auth can-i delete pods --as=system:serviceaccount:gipc:core-lab -n demo   # yes
auth can-i delete pods --as=system:serviceaccount:gipc:core-lab -n gipc   # no

Stdlib-only Kubernetes

The Go core talks to the Kubernetes API over plain net/http with the mounted ServiceAccount token and CA — no client-go, keeping the zero-dependency invariant. The chaos button kills one Running pod under a Deployment; the ReplicaSet recreates it and kube-state-metrics shows the dip and recovery.

Hard-capped, fail-closed

The mutating endpoints are bounded server-side so they can't be weaponised:

• Chaos: per-IP cooldown, single-flight, a fixed selector, a LabEnabled gate.
• Load test: a FIXED internal target (no user URL → no SSRF), code-clamped duration/concurrency/total, one active run per IP.
• Everything streams over Server-Sent Events (the existing hub) — no WebSocket.

A dedicated red-team pass then tried to break each invariant — SSRF, privilege escalation, DoS, blast radius, secret/token leakage — and every one held. The interactive visitor shell was the one deliberate cut: too dangerous on a single-node host, honestly deferred.