Gwen Shapira

Software Engineer @Confluent, PMC Member @Kafka, & Committer Apache Sqoop Gwen is an engineering leader at Confluent, managing the Cloud-Native Kafka team. She has 15 years of experience working with code and customers to build scalable data architectures, integrating microservices, relational and big data technologies. She currently specializes in building real-time reliable data processing pipelines using Apache Kafka. Gwen is an author of “Kafka - the Definitive Guide”, "Hadoop Application Architectures", and a frequent presenter at industry conferences. Gwen is also a committer on the Apache Kafka and Apache Sqoop projects. When Gwen isn't coding or arguing about protocols, you can find her peddling on her bike exploring the roads and trails of California, and beyond.

Find Gwen Shapira at:

Event Driven Architectures of Scale Panel

The speakers are currently updating the session details. The details for this session will be added soon.


Tuesday May 18 / 01:00PM EDT (40 minutes)

TRACK Architectures You've Always Wondered About TOPICS ArchitectureScalabilityEvent Driven ArchitectureMicroservices ADD TO CALENDAR Calendar IconAdd to calendar

Building and Scaling a Control Plane for 1000s of Kafka Clusters

At Confluent Cloud we deploy, manage, maintain and scale 1000s of Kafka, Kafka Connect and KSQL clusters. In order to create a cloud-native experience for our users, we've built a control plane that provides both self-serve functionality for our users and automation for Confluent engineers. We ask for a lot from our control plane - it has to support all the services that are part of Confluent Platform, it has to give each service owner significant control over their service configuration and the ability to extend the functionality for their service independently, it has to abstract away the differences between the various clouds and network architectures. Most importantly - it has to scale on multiple dimensions such as number of clusters, API keys, high rate of configuration changes, etc.    

In order to support all these requirements, we've been practicing what we preach - our cloud control plane uses an event-driven architecture based on Apache Kafka. In this talk, we'll cover some of our architecture highlights, the interesting challenges we've encountered as we scaled our cloud usage, and how we evolved a control plane for Kafka while evolving Kafka for the cloud.


Tuesday May 18 / 12:00PM EDT (40 minutes)

TRACK Architectures You've Always Wondered About TOPICS KafkaApacheArchitectureCloud NativeStreaming DataClustering & CachingClusters ADD TO CALENDAR Calendar IconAdd to calendar

Build your learning journey and level-up on the skills most in-demand in 2021. Attend QCon Plus (May 17-28, 2021).

Save your spot for $599 before May 28th