Root Cause Analysis Methods That Actually Work for Incident Response
Most postmortems have a "root cause" section that names one thing: a bad config push, a database index that went missing, a dependency that timed out. That's rarely the whole story. The config push wouldn't have mattered if there was a review step, or if the deploy pipeline had a canary stage, or if the on-call engineer had a runbook that caught it in two minutes instead of forty.
Root cause analysis (RCA) is the practice of working backward from a failure to find the conditions that let it happen, not just the trigger that set it off. Teams that skip this step keep fixing the same category of incident with a different name every quarter.
There isn't one correct RCA method. Below are three that show up most often in engineering orgs, with the tradeoffs of each.
5 Whys
The simplest method, and the one most teams start with. You ask "why" repeatedly, each answer becoming the next question, until you hit something you can actually act on.
Example incident: checkout API returned 500s for 12 minutes.
- Why did checkout return 500s? The payment service pool exhausted its connections.
- Why did the pool exhaust? A downstream fraud-check API started responding slowly.
- Why did slow responses exhaust the pool? Connections weren't released until the request timed out, and the timeout was set to 30 seconds.
- Why was the timeout 30 seconds? It was the client library default, never changed for this integration.
- Why wasn't it changed? No one owned reviewing third-party client defaults during the integration.
The fifth answer is where the real fix lives: a checklist item for reviewing timeout and retry defaults on every new external dependency. Fixing only the immediate symptom (lowering the timeout) prevents this exact incident but leaves the process gap that will produce the next one.
Where it breaks down: 5 Whys assumes a single causal chain. Real incidents usually have two or three contributing factors running in parallel, and forcing them into one line of questioning either flattens the analysis or leads you down a single branch while ignoring the others. It also depends heavily on who's in the room. Different people asking "why" at each step will land in different places, since there's no structure forcing you to consider alternate branches.
Best for: small to medium incidents with a fairly linear cause. A single service degrading because of one clear trigger. Not great for incidents involving multiple systems failing together.
Fishbone diagrams (Ishikawa)
Fishbone analysis forces you to look at multiple categories of contributing cause at once instead of following one thread. You draw a spine (the incident) with branches for categories like:
- Code: bugs, missing validation, bad defaults
- Infrastructure: capacity, network, hardware
- Process: missing review steps, unclear ownership
- People: training gaps, unclear on-call expectations
- Tools: monitoring gaps, alerting misconfiguration
- External: third-party outages, vendor changes
For each category, you ask what factors in that bucket contributed to the incident, even partially. You're not looking for the single root cause here, you're mapping everything that had to be true for the incident to happen the way it did.
Take a database outage caused by a migration that locked a table during peak traffic. A fishbone breakdown might surface:
- Code: the migration used an ALTER TABLE that requires an exclusive lock on that database version
- Process: no policy requiring migrations to run during low-traffic windows
- Tools: no pre-deploy check that flags locking migrations
- People: the engineer who wrote the migration hadn't worked with this table before and didn't know its traffic pattern
None of these alone caused the incident. Together, they did. A fix that only addresses one branch (say, adding a migration linter) is real progress but leaves the others as open risk.
Fishbone analysis works best as a group exercise, not something one engineer fills out alone. The value comes from people with different context (the on-call responder, the code author, someone from platform) filling in branches independently before comparing notes. Doing it solo tends to produce a diagram that just reflects whatever the facilitator already believed going in.
Where it breaks down: it's slower than 5 Whys and can produce a sprawling list of contributing factors with no clear priority. Without a facilitator willing to cut off branches that aren't productive, sessions run long and produce diagrams nobody references again.
Best for: larger incidents with multiple teams involved, or incidents where the postmortem group suspects "there's more to this than the obvious trigger."
Fault tree analysis
Fault tree analysis (FTA) comes from reliability engineering in aerospace and industrial systems, and it's more rigorous than either method above. You start with the failure (the "top event") and build a tree downward, connecting causes with AND/OR logic gates.
An AND gate means all the child conditions had to be true simultaneously for the parent to occur. An OR gate means any one of them was sufficient.
Example: "Users couldn't check out for 20 minutes" (top event)
This could happen via an OR gate between two paths:
- Payment service down, OR
- Inventory service down
Under "payment service down," you might have an AND gate:
- Primary payment region failed, AND
- Failover to secondary region did not trigger automatically
That AND gate is the interesting part. Primary region failures happen occasionally and are expected. The incident only became user-facing because the automatic failover, which should have absorbed the failure silently, also didn't work. A postmortem that only addresses "why did the primary region fail" misses that the actual gap is in the failover logic, which is the thing meant to make primary failures a non-event.
FTA is the right tool when you suspect an incident happened because multiple independent safeguards failed at once, not because of one broken thing. It's especially useful for incidents your team considered "impossible" beforehand, since those usually involve two or three protective layers failing together.
Where it breaks down: it takes real effort to build correctly, and it's overkill for straightforward incidents. Teams that try to FTA every P3 config typo burn a lot of time for no extra insight. It also requires someone comfortable with the notation, or the tree turns into an unreadable mess.
Best for: high-severity incidents where redundant safeguards existed and still failed together. Payment systems, data integrity incidents, anything with a designed failover that didn't fire.
Picking a method for a given incident
| Method | Time investment | Best for | Weak point |
|---|---|---|---|
| 5 Whys | 10-15 minutes | Linear causes, single service | Misses parallel contributing factors |
| Fishbone | 30-60 minutes, group session | Multi-team incidents, unclear scope | Can sprawl without a firm facilitator |
| Fault tree | Half a day or more | Redundant safeguards failing together | High effort, wrong tool for simple incidents |
A reasonable default: run 5 Whys during the incident review meeting for anything Sev3 or below. Reserve fishbone sessions for Sev2 incidents that touched more than one service. Pull out fault tree analysis only for Sev1 incidents, especially ones where a safeguard that was supposed to prevent exactly this kind of failure didn't work.
Don't treat these as mutually exclusive either. It's common to start with 5 Whys to get oriented, realize the causes fork into multiple branches, and switch to a fishbone mid-session. The method is a tool for structuring the conversation, not a ritual you have to follow to the letter.
The output matters more than the method
Whichever method you use, the postmortem is only useful if it produces specific, ownable action items. "Improve monitoring" is not an action item. "Add a pre-deploy check that flags migrations requiring an exclusive lock, owned by the platform team, due in two weeks" is.
Teams that get good at RCA tend to have one habit in common: they track whether the action items from past incidents actually shipped. If your last five postmortems all listed "add better alerting" as a follow-up and none of them shipped an actual alert, the root cause analysis wasn't the problem. The follow-through was.
Tie RCA output back into whatever system tracks engineering work, not a postmortem doc that gets read once and archived. The analysis is only worth the time it took if it changes what happens the next time something starts to fail the same way.