Context
“We are resilient” is not an experiment unless you can replay the outage and measure recovery. LLM agents and automation platforms inherit this layer: a model call that succeeds while the database is partitioned will lie with confidence.
The Cloud Reliability Lab is where those breakages are recorded so they can be replayed.
Decision
Profile query plans with EXPLAIN (ANALYZE, BUFFERS). Put a circuit breaker in front of Redis. Serve a time-bounded stale response only when the database is down. Use Toxiproxy to inject latency and outages. Remediation is narrow: only idle transactions whose application_name starts with a lab prefix can be terminated, and the action is audited. Backup and restore verification compares row counts on an isolated restore.
What I rejected
Advertising self-healing you have not injected, and a remediation job that kills arbitrary backends. Unbounded kill is an outage you caused.
Failure scenario
Toxiproxy blackholes Redis. The circuit opens. The service continues on the documented fallback. When the proxy heals, Prometheus shows how long recovery took.
Evidence
PostgreSQL/Redis stack, Toxiproxy scenarios, Prometheus metrics, and replayable remediation steps.
Inspect: Repository · Run locally