Gwen Shapira

Software Engineer @Confluent, PMC Member @Kafka, & Committer Apache Sqoop

Gwen is an engineering leader at Confluent, managing the Cloud-Native Kafka team. She has 15 years of experience working with code and customers to build scalable data architectures, integrating microservices, relational and big data technologies. She currently specializes in building real-time reliable data processing pipelines using Apache Kafka. Gwen is an author of “Kafka - the Definitive Guide”, "Hadoop Application Architectures", and a frequent presenter at industry conferences. Gwen is also a committer on the Apache Kafka and Apache Sqoop projects. When Gwen isn't coding or arguing about protocols, you can find her peddling on her bike exploring the roads and trails of California, and beyond.

Find Gwen Shapira at:

Session

Panel: Event Driven Architectures of Scale

Event-driven architectures (EDA) have been around for decades but are back in vogue, driven in part by the shift to the cloud and multicore architectures, data-driven applications, and related trends such as microservices and distributed computing.  

Our expert panel will explore what we’ve learned as an industry about event-driven architectures over the decades, the advantages - both business and technical - that EDA offers, and thoughts for the future with a particular focus on messaging and Serverless systems. Likely topics include dealing with complexity, debugging/troubleshooting, and common day 2 issues.

Date

Tuesday May 18 / 10:00AM PDT (40 minutes)

Track

Architectures You've Always Wondered About

Topics

ArchitectureScalabilityEvent Driven ArchitectureMicroservices

Add to Calendar

Add to calendar

Share

Session

Building and Scaling a Control Plane for 1000s of Kafka Clusters

At Confluent Cloud we deploy, manage, maintain and scale 1000s of Kafka, Kafka Connect and KSQL clusters. In order to create a cloud-native experience for our users, we've built a control plane that provides both self-serve functionality for our users and automation for Confluent engineers. We ask for a lot from our control plane - it has to support all the services that are part of Confluent Platform, it has to give each service owner significant control over their service configuration and the ability to extend the functionality for their service independently, it has to abstract away the differences between the various clouds and network architectures. Most importantly - it has to scale on multiple dimensions such as number of clusters, API keys, high rate of configuration changes, etc.    

In order to support all these requirements, we've been practicing what we preach - our cloud control plane uses an event-driven architecture based on Apache Kafka. In this talk, we'll cover some of our architecture highlights, the interesting challenges we've encountered as we scaled our cloud usage, and how we evolved a control plane for Kafka while evolving Kafka for the cloud.

Date

Tuesday May 18 / 09:00AM PDT (40 minutes)

Track

Architectures You've Always Wondered About

Topics

KafkaApacheArchitectureCloud NativeStreaming DataClustering & CachingClusters

Add to Calendar

Add to calendar

Share

Logo

Build your learning journey and level-up on the skills most in-demand in 2021. Attend QCon Plus (Nov 1-5, 2021).

Save your spot for $549 before August 31st

Register