You are viewing content from a past/completed QCon Plus - November 2020

Session

User Simulation for Rapid Outage Mitigation

Uber operates in over 10,000 cities across the world, with different offerings and features in each market. People all over the world rely on Uber for critical aspects of their daily lives. Uber can be their source of income, their commuting strategy, or their ride to the hospital. In COVID times, our role is all the more critical with Uber Eats acting as the central source of income for an increasing number of restaurants and couriers. With so much at stake, we don’t have the luxury of waiting for aggregate production tracking metrics to notice that some discernible percentage of users are experiencing an issue -- the changes made to our 4000+ microservice architecture are rolled out by both engineering and operations teams leveraging multiple different tools at the city, zone, region and global levels at a frequency that is impossible to coordinate. In this environment, how does Uber maintain a high level of reliability and prevent outages before they are felt by users?    

In this talk, I walk through the independent, external monitoring service that Uber developed to identify issues in production at the individual city level all across the globe, and how we leveraged composable integration tests simulating thousands of diverse, test users to cut our time to mitigation in half. Attendees will also learn how Uber surfaces and predicts its most dire outages and how the combination of machine learning and tracing enables us to reliably narrow down the root cause of an outage. 


Speaker

Carissa Blossom

Member of Production Engineering team for Eats & Delivery @Uber

Carissa Blossom is a member of the Production Engineering team for Eats & Delivery at Uber. She is also an Incident Commander for Ring0, Uber Engineering’s primary task force for critical outage mitigation. In her four years at Uber, she has served as Production Engineer for Eats,...

Read more

From the same track

Session

Less Mess, Less Stress: The Reliability Benefits of Custom Tools

Wednesday Nov 4 / 01:00PM EST

Tooling is an often overlooked, yet critical, component of infrastructure for most engineering teams. Adopting new infrastructure is easier than ever thanks to the cloud-native movement, open-source community, and the proliferation of Infrastructure-as-a-Service (IaaS) providers. However, each...

Daniel Hochman

Platform Engineer @Lyft

Session

A Sticky Situation: How Netflix Gains Confidence in Changes

Wednesday Nov 4 / 01:50PM EST

How do you know whether a change will affect end users in a negative way? As interactions in distributed systems grow increasingly complex, it can be challenging to get an answer to this question. One approach is to use a canary in which we introduce a new service into the environment, users...

Haley Tucker

Senior Software Engineer, Resilience Team @Netflix

PANEL DISCUSSION

Operational Excellence Panel

Wednesday Nov 4 / 03:30PM EST

Being on call for a production system can be stressful whether it is your first time or you have been carrying a pager for years. When that alert goes off, will you be prepared? Will your system reliability mechanisms behave as intended? If not, are you able to debug and understand what’s...

Carissa Blossom

Member of Production Engineering team for Eats & Delivery @Uber

Tammy Bryant Butow

Principal Site Reliability Engineer @Gremlin

Suudhan Rangarajan

Senior Software Engineer @Netflix

View full Schedule

Less than

16

weeks until QCon Plus May 2022

Level-up on the emerging software trends and practices you need to know about.

Deep-dive with world-class software leaders at QCon Plus (Nov 1-12, 2021).

Save your spot for $549 before February 7th

Register