Building and Scaling a Control Plane for 1000s of Kafka Clusters

At Confluent Cloud we deploy, manage, maintain and scale 1000s of Kafka, Kafka Connect and KSQL clusters. In order to create a cloud-native experience for our users, we've built a control plane that provides both self-serve functionality for our users and automation for Confluent engineers. We ask for a lot from our control plane - it has to support all the services that are part of Confluent Platform, it has to give each service owner significant control over their service configuration and the ability to extend the functionality for their service independently, it has to abstract away the differences between the various clouds and network architectures. Most importantly - it has to scale on multiple dimensions such as number of clusters, API keys, high rate of configuration changes, etc.    

In order to support all these requirements, we've been practicing what we preach - our cloud control plane uses an event-driven architecture based on Apache Kafka. In this talk, we'll cover some of our architecture highlights, the interesting challenges we've encountered as we scaled our cloud usage, and how we evolved a control plane for Kafka while evolving Kafka for the cloud.


Gwen Shapira

Software Engineer @Confluent, PMC Member @Kafka, & Committer Apache Sqoop
Gwen is an engineering leader at Confluent, managing the Cloud-Native Kafka team. She has 15 years of experience working with code and customers to build scalable data architectures, integrating microservices, relational and big data technologies. She currently specializes in building real-time... Read more Find Gwen Shapira at:


Vivek Sharma

Senior Engineer @Confluent
Vivek is a Senior Software Engineer at Confluent where he applies 10+ years of experience with the world's largest distributed systems to the task of creating a fully self-serve control plane that orchestrates thousands of mission critical clusters. He is a fan of decoupled distributed... Read more

Tuesday May 18 / 12:00PM EDT (40 minutes)

TRACK Architectures You've Always Wondered About TOPICS KafkaApacheArchitectureCloud NativeStreaming DataClustering & CachingClusters ADD TO CALENDAR Calendar IconAdd to calendar

From the same track

View full Schedule