Modern Data Engineering

Building Latency Sensitive User Facing Analytics via Apache Pinot

Real-time analytics has become the need of the hour for modern Internet companies. The ability to derive internal insights around business metrics, user growth & adoption as well as security incidents from all the raw logs is crucial for day to day operation. Even more critical is enabling access to usage analytics for the millions of customers which is non-trivial to achieve.

A good example of this is LinkedIn’s ‘Who Viewed My Profile’ which allows all 700 million+ users to slice and dice their page view data. Another example is Uber’s Restaurant Manager which enables restaurant owners across the globe to gain insights around menu preference, sales metrics, busy hours and so on. All such user facing applications need an analytical store that can support 1000s of queries per second at a millisecond response time granularity while ingesting millions of events/second.

In this talk, we will elaborate on how this is made possible using Apache Pinot - a popular, open source, distributed OLAP store. Specifically, we will talk about how to maintain the p99th latency SLA in the presence of organic data growth and concurrent queries.

Modern Data Engineering Panel

The Speaker is preparing the presentation abstract. More details will be available soon.

Serverless Search for My Blog With Java, Quarkus, & AWS Lambda

A Serverless app? With Java?! Absolutely!

We’ll discuss when Serverless is a great fit (and when it isn’t!) and why you don’t need to leave the Java platform when going Serverless. Based on the real-world example of a Serverless blog search, you’ll learn how Quarkus and GraalVM help you with building apps for AWS Lambda, which are blazing fast to start up and are memory-efficient at the same time. We’ll explore

• Which options Quarkus provides for creating AWS Lambda apps, making noticeable cold-start delays a thing of the past 
• What it takes to enable existing libraries like Apache Lucene to be used in GraalVM native binaries
• How to build Serverless apps which are portable across platforms, avoiding lock-in into specific cloud providers

Join us for this session to stop worrying about cold-starts and exploding costs, and learn how to love Serverless -- Java style!

Designing IoT Data Pipelines for Deep Observability

Millions of IoT devices emitting trillions of events per day enable us to track the health of the Tesla fleet. From a data engineering perspective, it's a challenging scale, but what makes it unique is how naturally fragile the data pipeline is. The physical world is full of chaos: what if network partitions lasted days, not seconds? Ever suffered data loss due to sub-zero temperatures? As these faults propagate, they threaten the quality of data-driven insights.

In this talk, we will share recipes that immunize IoT data pipelines against these faults from the get-go. We have built these recipes around a toolchain called dataflow. While designed for IoT, dataflow concepts generalize well. Learn how to apply these techniques to draw insights such as a drop in coverage, anomalous patterns & end to end processing latencies.