Distributed Systems for Developers

Essential Complexity in Systems Architecture

Simplicity is a virtue - simple systems should be easier to understand, easier to work on, easier to operate, and thus, more reliable. Everyone agrees on this in principle, to the extent that the ideal of promoting simplicity now appears in industry job ladder descriptions.    

But what does simplicity actually look like in terms of distributed systems design? A naive approach might suggest that it means removing components of your system, reducing it to the essential business logic, and as little else as possible. However, by attempting to minimize the number of boxes on an architecture diagram we can create significant complexity elsewhere, particularly as regards understanding and controlling system behavior.    

This talk will look at some real distributed system architectures and examine the tradeoffs that they make between the number of moving parts and operability. By the end of the talk, attendees should appreciate the subtle ways that apparently simple systems can create complex and difficult to understand behaviors.

Change Data Capture for Distributed Databases @Netflix

At Netflix, we have hundreds of microservices that rely on hybrid backends ranging from RDS or NoSQL to ElasticSearch or Iceberg. This necessitates hybrid backends for distributed high throughput applications since no single database can handle all the access patterns. A classic example of this pattern is Apache Cassandra for robustness and resilience and Elastic Search for search capabilities. Keeping data in sync among these data stores is a challenging problem usually solved by individual teams building sync processes and audits. This adds complexity and operational overhead to the teams. Change Data Capture (CDC) provides an optimal solution for receiving all changes seen on a database. 

CDC events from NoSQL databases with Active-Active setups like Apache Cassandra have unique challenges due to data partitioning and replication. Current CDC solutions for this rely on running within the database cluster and providing a stream with duplicated events. Our solution takes a more efficient approach by de-duping the stream in a stream processing framework. This involves having a distributed copy of the source DB in a stream processing system like Apache Flink. This enables better handling of the CDC stream since we can have before and after images of row changes. 

In this talk, we will cover the challenges associated with capturing CDC events from Cassandra (C*) and how we efficiently provide a data stream of change events while maintaining minimal load on the C* cluster. We will discuss the Flink ecosystem and how the backing store of RocksDB is used to make sure the data stream has full row changes instead of differences between rows which is provided by C* CDC.

Greenwater, Washington: An Availability Story

System availability is a personal experience. If you’re having a bad time, it’s not much comfort if everybody else is doing fine. Unfortunately, the traditional ways - and even many of the modern ways - of thinking about availability don’t take this into consideration, and instead treat availability like a system property. This talk is about defining, and designing for, availability that takes people into account, including examples of massive-scale cloud systems designed using these principles.

Distributed Systems Panel

The Speaker is preparing the presentation abstract. More details will be available soon.