# Why reliability claims have to be replayable

Canonical: https://stevenmoriasi.com/blog/cloud-reliability-lab
By [Steven Moriasi](https://www.linkedin.com/in/steven-moriasi/).
Source: https://github.com/steven-moriasi/cloud-reliability-lab
Date: 2025-07-03

## Context

“We are resilient” is not an experiment unless you can replay the outage and measure recovery. LLM agents and automation platforms inherit this layer: a model call that succeeds while the database is partitioned will lie with confidence.

The [Cloud Reliability Lab](https://github.com/steven-moriasi/cloud-reliability-lab) is where those breakages are recorded so they can be replayed.

## Decision

Profile query plans with `EXPLAIN (ANALYZE, BUFFERS)`. Put a circuit breaker in front of Redis. Serve a time-bounded stale response only when the database is down. Use Toxiproxy to inject latency and outages. Remediation is narrow: only idle transactions whose `application_name` starts with a lab prefix can be terminated, and the action is audited. Backup and restore verification compares row counts on an isolated restore.

## What I rejected

Advertising self-healing you have not injected, and a remediation job that kills arbitrary backends. Unbounded kill is an outage you caused.

## Failure scenario

Toxiproxy blackholes Redis. The circuit opens. The service continues on the documented fallback. When the proxy heals, Prometheus shows how long recovery took.

## Evidence

PostgreSQL/Redis stack, Toxiproxy scenarios, Prometheus metrics, and replayable remediation steps.

Inspect: [Repository](https://github.com/steven-moriasi/cloud-reliability-lab) · [Run locally](https://github.com/steven-moriasi/cloud-reliability-lab#readme)

_—by [Steven Moriasi](https://www.linkedin.com/in/steven-moriasi/)_