Sharma Podila

(He / him / his)

Software Engineer @Netflix

Software Engineering leader, system builder, collaborator, mentor. Deep expertise in cloud resource management, distributed systems, data infrastructure. Proven track record of delivering impactful large scale distributed systems of cross functional scope.

Find Sharma Podila at:

Session

Microservices to Async Processing Migration at Scale

Netflix creates and analyzes operational and analytical data associated with playback of thousands of titles by over 200 Million members worldwide. The data powers product features such as members’ ability to see and manage their viewing history. The data also feeds into the core business analytics as well as to the personalization and recommendation engines.

Previously built systems utilized microservices to ingest playback data in a synchronous manner, which potentially propagates any intermittent back pressure to the edge, and sometimes all the way to the clients on member devices. Utilizing an asynchronous processing model for the playback data, with a durable queue to absorb intermittent back pressure, we have migrated the systems seamlessly with no interruption to, and no changes required from, both the upstream and the downstream services.

We share our experience from the migration along with our design and implementation choices. Asynchronous processing at scale requires attention to managing any data loss with highly available infrastructure, elasticity to handle bursts without a high latency, fault tolerance with graceful degradation, as well as handling out of order and duplicate data. Specifically, we share the lessons learned from migrating the viewing history subsystem that provides the product feature as well as powers the personalization and recommendation engines.

Date

Monday Nov 8 / 09:10AM PST (40 minutes)

Track

Modern Data Architectures, Pipelines, & Streams

Topics

Data StreamsAsync ProcessingMicroservicesData EngineeringDatabaseStreaming Data

Add to Calendar

Add to calendar

Share

Less than

2

weeks until QCon Plus Nov 2021

Level-up on the emerging software trends and practices you need to know about.

Deep-dive with world-class software leaders at QCon Plus (Nov 1-12, 2021).

Save your spot for $799 before November 12th

Register