You are viewing content from a past/completed QCon - May 2021

Session

Data Mesh in the Real World: Lessons Learnt From the Financial Markets

CMC Markets, a FTSE 250 financial services company, has been running a successful trading platform for more than 30 years and is now undertaking a broad and ambitious transformation to take advantage of new technologies and ways of working. One vital aspect of a successful transformation to a company like CMC Markets is an approach to data that can scale. Traditionally, many organizations have handled data in a way that is too siloed, centralized, or detached from the core of the business. Therefore we have embraced the Data Mesh approach, which we believe will enable an organization like ours to deliver data-based products rapidly and successfully, and more broadly to reposition data at the center of IT and the business.

We have expectedly run into a number of interesting architectural, technical, and organizational challenges while identifying the best approach to shift the organization towards a Data Mesh and to build the required capabilities (e.g. Data Discovery and bridging the gap between cloud/on-premise data systems).

This is the story of our journey so far, at the coalface. We hope that the challenges we will highlight and the learnings we will share will be of interest to anyone else embarked on a similar journey or considering it.

Main Takeaways

1 Hear about the transformation a company goes through, having data at the center of that transformation.

2 Learn about data mesh and the challenges of implementing one.


Tareq, what is the focus of your work these days?

I work for CMC Markets, which is an online financial trading business. CMC Markets is undergoing a very ambitious transformation. I'm part of the Core Data team, trying to introduce a data approach to make sure that data is at the center, and the organization can achieve the transformation goals, which are about rapid innovation, delivery, and so on. So that's in a nutshell what I'm focusing on.

What is the motivation for your talk?

I've been interested in data and I have worked on different data-related aspects in I.T., technologies and databases and big data and cloud, and so on. As part of my previous work, I have been able to observe different data models and how they work and to observe the limitations of these. When the data mesh concept emerged, we recognized many of the challenges we saw in previous jobs and were very excited about the potential of this. The data mesh doesn't give you exact recipes for everything but it’s the foundational concept that underpins our work. We’ve learned a lot about the practicalities of introducing data mesh and the challenges that needed to be overcome. Data mesh at CMC Markets has its own particular context, but I think it’s actually interesting to contrast with some of the other things you see out there. So we thought that some of the learnings we have are definitely worth sharing and contributing to the community and seeing where things are, where we fit also in terms of the data mesh ecosystem. We're genuinely excited about the opportunity to share that with the broader community.

Considering the other big trends of these days, like microservices, service mesh, cloud serverless, where does the data mesh fit into all these?

I think data mesh shares the same ethos as other progressive models, so to speak, like microservices, which is this decentralized, autonomous, collaborative way of working. I think in the past the focus has been on, for example, microservices. But now what is exciting about data mesh is that this is bringing this focus, this ability to be able to innovate in a decentralized and scale around data. This is very exciting because data is the hard bit in all these problems. Also, data mesh does not dictate any technology. Actually, the use case we have in my company is hybrid in that we have to reconcile on-premise infrastructure with new cloud infrastructure, existing stable products that are data-heavy with new emerging products. And we need to be able to allow stability and also evolution for these things to coexist. Hence the interesting data mesh where these things come together. 

How would you describe the persona and the level of the target audience for this presentation?

I think the talk would be really interesting for anyone who is interested in concrete experiences on data mesh. That would be data architects, data engineers, enterprise architects, technical leaders. Also, aside from the technology aspect, I think for those who are interested in organizational transformation, which puts data at the center, would also find it useful as well, because that is the broader context of what my team is working on now.


Speaker

Tareq Abedrabbo

Core Data Principal Engineer @CMCMarkets

Tareq Abedrabbo is a Core Data Principal Engineer at CMC Markets. His main interests are in designing and building data systems that can scale, technically and organisationally. During his career, Tareq worked with a variety of organizations, from startups to large enterprises, on complex...

Read more
Find Tareq Abedrabbo at:

Date

Thursday May 27 / 06:10AM PDT (40 minutes)

Track

Modern Data Pipelines and Data Mesh

Topics

ArchitectureData MeshDatabase

Add to Calendar

Add to calendar

Share

From the same track

Session Architecture

Data Mesh: An Architectural Deep Dive

Thursday May 27 / 08:10AM PDT

Data Mesh is a paradigm shift in how we imagine and build big data management solutions, and most importantly is a shift in how we form our teams around these solutions and govern them. Data Mesh departs from half a century old assumptions on how we need to manage analytical data to be useful. It...

Zhamak Dehghani

Director of Emerging Technologies @thoughtworks & Creator of the Data Mesh concept

Session Architecture

Robust Foundation for Data Pipelines at Scale - Lessons From Netflix

Thursday May 27 / 07:10AM PDT

At Netflix, Data/ML pipelines are widely used and have become central for the business. A very wide scenario presents diverse use cases that go beyond recommendations, predictions and data transformations. As big data and ML gains presence and becomes more impactful, the scalability and stability...

Jun He

Sr. Software Engineer in the Big Data Orchestration Team @Netflix

Harrington Joseph

Sr. Software Engineer @Netflix Data Platform Orchestration Team

Session Architecture

Data Pipelines & Data Mesh: Where We Are and How the Future Looks Like

Thursday May 27 / 09:10AM PDT

In the panel, experts with different backgrounds will discuss the current challenges for building Modern Data Pipelines and applying Data Mesh in the real world. We will also discuss how the future looks like in terms of new techniques, architectures, and tools to make effective data-based projects.

Zhamak Dehghani

Director of Emerging Technologies @thoughtworks & Creator of the Data Mesh concept

Tareq Abedrabbo

Core Data Principal Engineer @CMCMarkets

Jacek Laskowski

IT freelancer, Java Champion & Author of "The Internals Of"

View full Schedule

Logo

Build your learning journey and level-up on the skills most in-demand in 2021. Attend QCon Plus (Nov 1-12, 2021).

Save your spot for $699 before October 9th

Register