Runbooks for the things that go wrong at 3am
The value of a runbook is not that it contains knowledge. It is that it works when the person reading it is tired and alone.
In a distributed team, the person who encounters a problem is frequently the only one awake. Whatever they can find in writing is the entire available support, and prose written for a calm reader does not survive that context. For broader context on maintaining shared technical documentation, GitHub Docs is a useful reference.
Write for someone who is not you
Runbooks written by an expert tend to assume the reader knows why each step exists, which is exactly what a stressed reader at three in the morning does not have.
Numbered steps. One action each. Expected result stated after each step so the reader can tell whether it worked. No prose paragraphs in the middle of a sequence. For a practical software-side reference on ownership in repeatable processes, this guide adds useful detail.
The step that goes wrong is the reason someone is reading. A runbook that only describes the happy path abandons the reader at the exact point they needed it.
Start with diagnosis, not with the fix
A runbook should open with how to confirm this is actually the problem it describes. Otherwise it gets applied to a superficially similar situation and makes things worse.
Include what the symptoms look like, what to check first, and — importantly — what it might be instead.
State the blast radius
Before any destructive step, say what it affects and whether it is reversible. A tired person needs to know that restarting this service drops connections for four minutes, before they do it rather than after.
Say when to wake someone
Every runbook needs an escalation point: the condition under which the reader should stop and call for help, and who to call.
Without it, people either escalate immediately, defeating the purpose, or persist far too long alone, which is how a small incident becomes a large one. Naming the threshold explicitly — if this has not resolved in twenty minutes, wake the on-call lead — gives permission that people otherwise will not take.
Test them by having someone else follow them
A runbook validated by its author is validated by the one person who cannot see its gaps. Have someone unfamiliar follow it exactly, in a drill, and note every point where they hesitate or improvise.
Every hesitation is a defect. This exercise takes an hour and finds problems that would otherwise be found during an actual incident.
Update them immediately after use
The moment of maximum knowledge is just after an incident. That is when the gaps are obvious and the correct steps are fresh.
Adding the update to the incident review — as a required output rather than a good intention — is what keeps runbooks from decaying into descriptions of a system that no longer exists.