When systems fail, the speed and calm with which an organisation responds depends almost entirely on how well its incident command and on-call arrangements are designed. Done well, they resolve issues quickly while protecting the people who run them. Done poorly, they produce slow, chaotic responses and a steady erosion of the very engineers you most need. For enterprise leaders, the design of on-call is not merely an operational detail: it is a retention, resilience, and reputation issue all at once. This article sets out how to design both for faster resolution and for sustainable, humane operation.
Separate the roles in an incident
In a serious incident, asking one person to diagnose the problem, coordinate the response, and communicate with stakeholders guarantees that all three are done badly. Effective incident command separates these roles. An incident commander coordinates and decides, freeing technical responders to focus on the problem itself, while a communications role keeps stakeholders informed so engineers are not interrupted by anxious questions. The commander does not need to be the most senior person or the deepest expert; they need to keep the response organised and decisive.
Make these roles explicit and rehearsed so that when an incident begins, people know who is doing what without negotiation. Clarity of role under pressure is what separates a controlled response from a scramble, and it is far easier to establish in calm conditions than to invent in the middle of an outage.
Design rotations that people can sustain
On-call burnout is rarely caused by a single bad night. It accumulates from rotations that are too frequent, too disruptive, and never compensated by recovery. Design rotations with enough engineers that no one carries the pager too often, and ensure follow the sun or sensible handovers so that nobody is woken repeatedly across consecutive nights. Build in recovery time after a heavy shift, and recognise on-call work explicitly through compensation, time off, or both. Treating on-call as an unpaid expectation breeds resentment and attrition.
Watch the leading indicators of strain: how often people are paged out of hours, how many alerts turn out to be noise, and how often the same person is disturbed. These metrics tell you whether the rotation is sustainable long before your best engineers quietly decide to leave.
Attack alert noise relentlessly
Nothing burns out an on-call engineer faster than being woken for alerts that did not require human action. Every page should correspond to a real problem that a person can and must address now. Audit your alerts regularly and remove or downgrade anything that is routinely ignored, auto resolves, or does not need immediate attention. Route lower priority signals to a queue reviewed in working hours rather than to the pager. A quieter pager is not a sign of complacency; it is a sign of a well tuned system and a respected team.
Treat recurring alerts as defects to be fixed, not conditions to be endured. If the same page fires every week, the right response is to address the underlying cause, not to train people to tolerate the interruption.
Make response fast through preparation, not heroics
Fast resolution comes from preparation, not from individual brilliance under pressure. Maintain runbooks for common failure modes so responders are not improvising at three in the morning. Ensure access, tooling, and dashboards are ready before they are needed, because fumbling for credentials during an outage wastes precious minutes. Practise through game days and simulated incidents so that the response is familiar rather than novel when it matters. Organisations that rely on heroics are one resignation away from losing the ability to respond at all.
Equally, empower responders to act. If every significant decision requires waking a senior leader for approval, resolution slows and the on-call engineer is set up to fail. Define clearly what responders may do on their own authority, including taking disruptive actions to protect the wider system, and trust them to use it.
Learn from every incident without blame
The value of an incident is not fully realised until you have learned from it. Run blameless reviews that focus on the conditions and systems that allowed the failure rather than on individual error. Blame drives information underground, so people hide mistakes and the same failures recur. A culture where engineers can speak openly about what went wrong is the foundation of genuine improvement and of a team that trusts its leadership.
Turn the findings into action. A review that produces insights but no changes is theatre. Track the improvements identified, assign them owners, and follow them through, because the credibility of the whole process rests on incidents actually leading to a more resilient system over time.
- Separate incident commander, technical responder, and communications roles, and rehearse them in advance.
- Staff rotations with enough engineers to avoid frequent disruption, and recognise on-call work explicitly.
- Audit alerts regularly, routing anything that does not need immediate human action away from the pager.
- Maintain runbooks, ready access, and game days so response is prepared rather than improvised.
- Empower responders to take significant action without waiting for layers of approval.
- Run blameless reviews and track the resulting improvements through to completion.
What good looks like
A healthy incident operation feels calm even under pressure. Roles are clear, the pager is quiet because alerts are meaningful, and responders have the runbooks, access, and authority to act decisively. Engineers are not afraid of being on-call because the rotation is sustainable, the noise is low, and their effort is recognised. When incidents happen, and they always will, the organisation responds quickly, communicates honestly, and learns genuinely, so each failure leaves the system a little more resilient than before.
Crucially, good design protects people as well as systems. The engineers who run your platform are a finite and valuable resource, and an on-call model that grinds them down is a false economy. Design for both fast resolution and human sustainability, and you get a response capability that lasts.
Leaders set the tone here more than any process document. When senior management treats on-call burden as a serious operational concern, funds adequate staffing, recognises the work openly, and insists on blameless learning, the whole organisation responds in kind. When they treat it as an unfunded expectation and look away from the strain, no rota design will compensate. The most resilient organisations understand that incident response is ultimately a human capability supported by good systems, and they invest accordingly in both. Need support applying this approach? Email sales@halfteck.com.