How I Triage a Codebase That Is on Fire
When a system is failing and nobody knows why, the order of questions matters. This is my triage list, from is it running to is something contaminating the data.
When someone hands me a system that is failing, I do not start by reading code. I start with a series of questions, ordered from most critical to least. They apply to nearly all software.
Start with the most critical question
Is it even running? Everything else depends on the answer, and it is surprising how often nobody has checked.
If it is not running, the work is to get it running and stable. Cleanup can wait.
Can I see what happened?
Next, can I read the logs, and is there a trail of everything? If I cannot see what the system did, every other step is guessing.
A system with no logs or audit trail turns every problem into a debate. Adding visibility early pays back through the whole job.
Is anything exposed or unprotected?
Then I look at risk that is not about the bug at hand.
Is sensitive data hashed or encrypted? How are tokens and authentication handled? Is anything publicly reachable that should not be?
I ask these early because a failing system is often leaving something open. Finding that before someone else does is the cheaper path.
Is it doing its job?
After that, is the application functioning the way it is supposed to? A running system can still be wrong, so I compare actual behavior with what people expect.
Is something corrupting the data?
Last on the list, but never skipped: is something contaminating the data? Bad data spreads quietly, and fixing a bug does not clean up what it already wrote.
Why a fixed order helps
A list keeps panic from setting the order. It also gives the client something concrete, because each answer is either yes, no, or not known yet.
Not known yet is a useful answer. It shows exactly where the next hour should go.
Related
This is how I approach code rescue work. Two concrete examples are Fixing Prometheus High-Cardinality Metrics and Rebuild Staging From Production Backups.