Rethinking Reliability: What You Can (and Can't) Learn From Incidents

This talk presents research collected from the VOID—an open database of public incident reports. Containing over 2,000 reports for almost 700 organizations, the database allows for more structured review and research about software-related incident reporting. Key results from our research challenge standard industry practices for incident response and analysis, like tracking Mean Time To Resolve (MMTR) and using Root Cause Analysis (RCA) methodology. In particular, we demonstrate how unreliable MTTR can be, and how RCA can lead to environments where people are less likely to admit mistakes and speak up about things that could lead to future incidents. We propose alternate metrics (SLOs and cost of coordination data), practices (Near Miss analysis), and mindsets (humans are the solution, not the problem) to help organizations better learn from their incidents, and make their systems safer and more resilient.


Speaker

Courtney Nash

Internet Incident Librarian & Senior Research Analyst @Verica

Courtney Nash is a researcher focused on system safety and failures in complex sociotechnical systems. An erstwhile cognitive neuroscientist, she has always been fascinated by how people learn, and the ways memory influences how they solve problems. Over the past two decades, she’s held a variety of editorial, program management, research, and management roles at Holloway, Fastly, O’Reilly Media, Microsoft, and Amazon. She lives in the mountains where she skis, rides bikes, and herds dogs and kids.

Read more
Find Courtney Nash at:

Date

Tuesday Dec 6 / 09:00AM PST ( 50 minutes )

Topics

SRE Reliability Open Database MTTR RCA SLOs Incidents Engineering

Share

From the same track

Session SRE

Did the Chaos Test Pass?

Tuesday Dec 6 / 10:10AM PST

People used to ask me all the time how to figure out if their chaos test has “passed,” and I’d always say “well, that’s a loaded question.” To confirm that a chaos test “passed,” we need to do verification of hypotheses - sometimes you’re trying to prove some system behavior occurred in response

Speaker image - Christina Yakomin

Christina Yakomin

Senior Site Reliability Engineering Specialist @Vanguard_Group

Session SRE

The Endgame of SRE

Tuesday Dec 6 / 11:20AM PST

The containers are deployed and the builds are green. Yaml flows through the system, linted, reviewed, tested, and shipped with ease and regularity. Our intrepid SRE finds themself at a crossroads. The infrastructure is great but teams still struggle to maintain error budgets.

Speaker image - Amy Tobey

Amy Tobey

Senior Principal Engineer and SRE Practice Leader @Equinix

Session SRE

The Eternal Sunshine of the Toil-Less Prod

Tuesday Dec 6 / 12:30PM PST

One of the most important decisions in building an SRE practice is what kind of work should be assigned to the SRE team, and in what percentages.

Speaker image - Sasha Rosenbaum

Sasha Rosenbaum

Director of the Cloud Services Black Belt Team @RedHat