Vivek Sharma

Senior Engineer @Confluent

Vivek is a Senior Software Engineer at Confluent where he applies 10+ years of experience with the world's largest distributed systems to the task of creating a fully self-serve control plane that orchestrates thousands of mission critical clusters. He is a fan of decoupled distributed architectures, low latency systems and long distance running.


Building and Scaling a Control Plane for 1000s of Kafka Clusters

At Confluent Cloud we deploy, manage, maintain and scale 1000s of Kafka, Kafka Connect and KSQL clusters. In order to create a cloud-native experience for our users, we've built a control plane that provides both self-serve functionality for our users and automation for Confluent engineers. We ask for a lot from our control plane - it has to support all the services that are part of Confluent Platform, it has to give each service owner significant control over their service configuration and the ability to extend the functionality for their service independently, it has to abstract away the differences between the various clouds and network architectures. Most importantly - it has to scale on multiple dimensions such as number of clusters, API keys, high rate of configuration changes, etc.    

In order to support all these requirements, we've been practicing what we preach - our cloud control plane uses an event-driven architecture based on Apache Kafka. In this talk, we'll cover some of our architecture highlights, the interesting challenges we've encountered as we scaled our cloud usage, and how we evolved a control plane for Kafka while evolving Kafka for the cloud.


Tuesday May 18 / 09:00AM PDT (40 minutes)


Architectures You've Always Wondered About


KafkaApacheArchitectureCloud NativeStreaming DataClustering & CachingClusters

Add to Calendar

Add to calendar



Build your learning journey and level-up on the skills most in-demand in 2021. Attend QCon Plus (Nov 1-12, 2021).

Save your spot for $699 before October 9th