Production Readiness: Fighting Fires or Building Better Systems?

In 2018 Tanya Reilly gave a talk called ‘The History of Fire Escapes’ in which she argues that we need to ‘focus on better software, not better incident response’. When I was recently asked how much time SREs should spend firefighting, that talk came to mind. The ideal amount of time to spend responding to incidents is 0 (which we never achieve, but that’s the goal). A healthy SRE team spends most of its time improving the systems it is responsible for.

Reilly also argues that ‘software needs a fire code’, in other words, a set of best practices to prevent or mitigate failures - just as building regulations have evolved over time to reduce the incidence and impact of building fires.

We don’t have fire codes for software. But many organisations use Production Readiness Reviews (PRRs) as a model to improve reliability of software systems. This talk will discuss why we don’t have a fire code for software; what PRRs can and cannot achieve in terms of reliability; the difference between PRRs run by a team that is onboarding a service and PRRs run by consulting teams; and what to do when all your team does is fight fires.