Chaos Engineering: A Practical Starting Point
Most teams don't know how their system behaves under failure until failure shows up uninvited. A database connection pool exhausts. A retry storm takes down a downstream service. Someone discovers, live, that the "redundant" load balancer config was never actually redundant.
Chaos engineering is the practice of injecting failure into a system on purpose, in a controlled way, so you find these gaps on a Tuesday afternoon instead of during a customer-facing outage. It has a reputation for being something only Netflix-scale companies do. It isn't. You can start small, on a single service, with almost no tooling.
It's not the same as a game day
A game day is a human exercise. You schedule it, gather the on-call team, and have them practice responding to a simulated incident. The goal is to test people and process.
Chaos engineering is a technical practice. You inject a real fault (kill a process, add latency, drop a percentage of network packets) and observe what the system does, often without a human in the loop at all. The goal is to test the system itself: does it degrade gracefully, does it retry sanely, does it alert correctly. You can and should do both, but they answer different questions.
Start with a hypothesis, not a tool
The instinct is to reach for a chaos engineering platform first. Resist it. Start with a hypothesis about your system that you're not confident about.
Good hypotheses sound like this:
- "If the recommendations service is unavailable, checkout should still complete without recommendations."
- "If our primary database read replica goes down, traffic should fail over within 30 seconds without manual intervention."
- "If the payments provider times out, we retry with backoff instead of hammering it."
Write the hypothesis down before you touch anything. It defines what "pass" and "fail" look like, and it keeps the experiment from turning into aimless poking.
Run it somewhere safe first
Don't run your first chaos experiment in production. Run it in staging, or in a production-like environment with synthetic traffic. You're not trying to prove you're brave, you're trying to learn something about your system's behavior with an acceptable blast radius.
A reasonable progression:
- Staging, manual fault injection. Kill a process with
kill -9, add latency withtc(Linux traffic control), or block a port with a firewall rule. No tooling required. - Staging, automated fault injection. Use a tool to run the same experiment repeatedly and check the result is consistent.
- Production, small blast radius. Run the experiment against a single instance, a single availability zone, or a small percentage of traffic, with a fast abort mechanism.
- Production, broader scope. Only once you've built confidence at each earlier stage.
Most teams get real value and stop around stage 2 or 3. You don't need to run chaos experiments company-wide across every service to benefit from the practice.
Failure modes worth testing first
If you're not sure where to start, these tend to surface the most useful findings for the least effort:
Dependency failure. Kill or block access to a downstream service your application depends on. Does your code time out sanely, or does it hang until the client gives up? Do you have a circuit breaker, or does one slow dependency take down the whole request path?
Resource exhaustion. Fill up disk space, exhaust a connection pool, or spike CPU on an instance. Does your monitoring catch it before customers do? Does the service fail loudly, or does it silently start dropping requests?
Network partition. Block traffic between two nodes that normally talk to each other. This is especially valuable for anything using leader election or consensus, since split-brain scenarios are notoriously hard to reason about on paper.
Process kill. Send SIGKILL to a running process and see what happens to in-flight requests, whether your orchestrator restarts it fast enough, and whether anything downstream needed graceful shutdown that it didn't get.
Clock skew. If you have anything relying on time synchronization (TLS certs, token expiry, distributed locks), test what happens when a node's clock drifts.
Always have a kill switch. Before you run any experiment against production, know exactly how you'll stop it and roll back, and test that rollback path separately from the experiment itself.
Tooling, if and when you need it
For manual experiments, standard Linux and cloud tools get you further than people expect:
# Add 300ms of latency to all outbound traffic on eth0
tc qdisc add dev eth0 root netem delay 300ms
# Drop 10% of packets to simulate a flaky network
tc qdisc add dev eth0 root netem loss 10%
# Remove the delay when you're done
tc qdisc del dev eth0 root netemFor anything you plan to run repeatedly, purpose-built tools remove a lot of the manual setup:
| Tool | Good for |
|---|---|
| Chaos Monkey / Chaos Toolkit | Random instance termination, general-purpose experiments |
| Gremlin | Managed chaos experiments with a UI and safety controls |
| Litmus | Kubernetes-native chaos experiments |
| AWS Fault Injection Simulator | Native AWS fault injection (EC2, ECS, RDS) |
None of these matter if you haven't done the manual version first. The tool automates the injection, it doesn't tell you what's worth injecting.
Make the results actionable
An experiment that doesn't change anything wasn't a waste, but it also isn't done. If you find that killing a dependency causes a full outage instead of graceful degradation, that's an action item with an owner, same as anything coming out of a postmortem.
Track chaos experiments the same way you'd track incidents: what you tested, what you expected, what actually happened, and what you're doing about the gap. Over time, this becomes a record of resilience work that's easy to point to when someone asks what the on-call team has been doing besides responding to pages.
Connect it to your alerting
The most underrated part of chaos engineering isn't the failure injection, it's checking whether your monitoring caught it. If you kill a critical process and nobody gets paged within your target detection time, you've found a monitoring gap that's arguably more valuable than the resilience finding itself. A system that fails ungracefully but pages the right person immediately is in better shape than one that fails gracefully but leaves you blind to it.
Before you scale up your chaos practice, make sure the alerts that are supposed to fire during these experiments actually do. If they don't, fix that first. An experiment is only as good as your ability to observe its outcome.
Where to go from here
Pick one dependency your team is least confident about. Write a one-sentence hypothesis about how the system should behave if it fails. Run the simplest possible version of that experiment in staging this week. That's the whole starting point. The elaborate chaos engineering programs you read about at large companies all started with someone doing exactly this, once.