Unknown: Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
Have you ever woken up to a crisis because a number on a dashboard is broken and no 1 knows why? Or sent out frustrating Slack messages trying to find the right dataset? Or tried to understand
They started building Outland as an internal tool for themselves. They started building Outland as an internal tool for themselves.
Outland is a collaborative workspace for data driven teams, like GitHub for engineering or Figma for design teams.
By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets and code, Atlan enables teams to create a single source of truth for all of their data assets
and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker, and more.
Go to dataengineeringpodcast.com/outland
today. That's a t l a n, and sign up for a free trial.
If you're a data engineering podcast listener, you get credits worth
$3, 000 on an annual subscription.
Check out our friends
over at Linode.
With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Pacaderm, and Dagster.
With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.
Go to data engineering podcast.com/linode
today. That's
l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host is Tobias Macy. And today, I'm interviewing Kishore Gopalakrishna
and Shang Fu about Apache Pinot and its applications for powering user facing analytics. So, Kishore, can you start by introducing yourself?
Unknown: Definitely. Thank you so much, Tobias, for inviting us here. Super excited to be on this show. My name is Kishore Gopalakrishna, and I'm the CEO and cofounder of Startree. I'm also the cocreator of Apache Pinot.
Before Strattery, I was at LinkedIn for 8 years. I built a multiple a lot of distributed systems such as Espresso, ThirdEye, Apache Helix, which is a cluster management framework,
which help build a lot of distributed systems.
Also involved in a lot of Apache projects like Goblin, s 4,
Unknown: Dataverse, and many others. And, Sean, how about yourself? Hi. This is Sean Fu, and, super excited
to join this.
So for my background, I'm currently a cofounder at StarTree,
and I'm also a cocreator of a project.
Before StarTree,
I was a architecture at Uber's streaming data team. So build this onto a streaming solution
for data infrastructure.
Unknown: And before Uber, I was at same thing to build Apache Pino, basically. Going back to you, Kishore, do you remember how you first got involved in the area of working on data? Yeah. It actually goes
Unknown: a long way back, even before Hadoop became a thing. Back in Yahoo, I was working in this forecasting team. We needed data to do the forecasting as usual. But the funny thing that happened at that time was we were to fill up the form,
and then we were supposed to wait for a month before the data shows up because all this data was in Oracle,
and we never had access to this. So this was a time when data democratization was not unit so when it was more like a dream for us. That's when Hadoop came in at Yahoo and everything changed. And I was so
excited with this change, and I thought I'm more
directly diving to that ecosystem.
That's when I started building
a lot of distributed systems. Started with Apache s 4, which is a streaming system.
I also worked in ad indexing and, lot
of data related work that got me excited.
What really changed something for me was
LinkedIn being a huge proponent of open source. So I was able to join LinkedIn very early on in their journey of going from single monolith of Oracle
to building all these distributed systems such as Espresso, Kafka,
etcetera.
The first system that we built where we built is a NoSQL
distributed systems database. And as part of that, I also got ability to build opportunity to build Apache Helix, which is a cluster management framework.
So from then, there was no turning around for me. It was all data, data, data. And I've been super excited and super fortunate to get this opportunity at LinkedIn,
build a bunch of systems.
So overall for me, just that ability to have access to this data and get all these insights
is super exciting and also super challenging, and I love all the challenges there. So that's kind of when I started, but I haven't really gotten out of the data management yet.
Unknown: And, Sean, do you remember how you first started working with data? Yeah. So for me, my story was pretty simple. I was just a graduate from the school, and
I joined the LinkedIn. So my first project, though, was actually the Pinot project. It was, like, in the ads team and then building, like, those ads reporting things around the p and o and for serving as customers.
This project was, like, very challenging. And later on, we got a lot more use cases
on Pino, and, all those things are, like, super, super, super challenging. And you learned a lot from the data and from working with the engineers. Those things are pretty fun, and I actually continue my career on the data
world for, like, past 10 years. And after I left the LinkedIn and joined the Uber, at Uber, we actually build the entire streaming
data infrastructure there. We first stabilized
the Kafka infrastructure at
Uber, took a lot of time to, like, get things
being successfully
produced to Kafka
for, like,
half a year, and then we build a streaming processing,
infrastructure using, like, SAMSA and, Flink.
After that,
the story
comes to, like, the real time analytics infrastructure. So we have Pinot being adopted at Uber, and we
also build a lot of use cases on top of Pinel.
Later on, we use
Presto to provide
better
kind of full SQL support and the joins with different
data sources across, like, the entire Uber.
And, also, I've been involved to, like, the data visualization
layer, like Supersat,
and also, like, the data management
layer, like, metadata stores kind of things.
Unknown: In terms of the Pino project, I'm wondering if you can describe a bit about what it is and some of the story behind
how it came to be and why it was necessary and why you've decided that this is something that you wanted to spend your time and focus on in the form of StarTree.
Unknown: So just just a brief intro in in terms of what is Apache Pinot. So it's a real time OLAP data store specifically designed for
interactive user facing applications. Right? I mean, that's a lot of words right there in that statement.
The key aspect of these applications, user facing applications, is that they demand
very, very low latency, even at high throughput. Right? Another way that people internally
at LinkedIn used to describe this describe, you know, as was queryable Kafka. So you can pretty much point Pinot to any stream, and then you can start querying. That's the right way you can think of what Pinot is. And
in terms of how this happened at LinkedIn,
it was actually a very exciting story. There are 2 parts to that. The first part is, like, when
what do you call as p no 1 dot of in fact, I was not even part of that project. It was really Sean who built the first version of p no that's
which was really built on top of a Lucene. Right? And then we had an internal framework called Bobo and Sensei, which was also built on Lucene, and that's what we use to power some of our initial versions of who viewed my profile. If you're familiar with that product on LinkedIn where you go to your profile, who viewed my profile page, and you get to know everyone that viewed your profile and where did they come from, which company,
what is the skill set. And this was a very important application for LinkedIn. Because what happened in 2011, 2012
is many would come to LinkedIn only for when they're changing their job. It was not really the vibrant
social network that it is today. Right? And that was super important for LinkedIn to change that paradigm and then where the engagement had to increase drastically. So what this app, who in my profile,
allowed LinkedIn to do was
a lot of people would come to this page, know who are actually looking at their profile, and then lot of connections
and invites would actually originate from this page. So it's a very, very important thing for, for LinkedIn that changed the whole dynamics of how many people who came to LinkedIn.
And the professional network that LinkedIn is today wouldn't be possible without this app. Yeah. So the engagement metric had to increase a lot for LinkedIn. And this is when COVID my profile was built as 1 of the first applications to change that. And we were, like, super surprised with the success of this application. At any
given time, there were, like, some thousands of people looking at this application. Right? And the first version that we built
was really not scaling. I remember when I joined the the p note team, we had almost 1, 000 notes serving just who will make profile. And it was crazy to operate this because it was not built to operate at that scale,
and we were expecting less than 100 millisecond latency. So we used to break it up into vertical charts.
We had, like, multiple clusters to just serve 1 application.
It was a huge challenge, and it was a nightmare, to be honest, for all of us at LinkedIn. In fact, we were internally joking that we should just kill this product, go back to batch mode, and just show static graphs to the end users. But the metrics really showed a told a very different story, right, in terms of the engagements, the metrics, and the number of people actually coming back to LinkedIn was really, really high.
And that got us that got the product management super excited, and they said, hey. We need to build a new system that can actually handle the scale. And that's where
we built Pino from scratch. And we learned a lot from these issues that we had seen with the Pino 1.0.
Once we rebuild it, the 1, 000 nodes came down to, like, 75 nodes. And by when we'd actually
along with that change, the traffic had grown to 5 x. So overall, we saw almost, like, 45 x improvement in efficiency, and
that was really the game changer for LinkedIn as well. So what happened after that was
every entity on LinkedIn, call it, a post that you put it on LinkedIn
or a jobs or ad or even a company profile. Any entity had events associated with that, and those events would flow into Kafka. And Pinot would point at those events, and there would be an app built on top of it. So there are tons of apps, like, almost 70 to 100 apps on LinkedIn that you see today, and they're all being powered by Pinot in the back end. So any number that you see, any chart, most likely coming from Pinot. And today, if I remember the numbers correctly, it's serving around 200, 000 queries per second. This is really, really great achievement for us in terms of what we were able to do with Pinot and how we change the whole way of thinking about data at LinkedIn. And pretty much it became a de facto thing. Any page, any entity that LinkedIn created,
analytics came as along with it by default. So that's kind of how we started and where we are now today with Pino at LinkedIn.
Unknown: You mentioned that the primary focus of Pino has been around these
user facing analytics,
for instance, with LinkedIn being able to see, you know, who viewed your profile graph.
And I'm curious,
what are some of the
alternative approaches that people may have used for being able to power these user facing analytical use cases and some of the capabilities that Pino provides that
make it stand out as a method for being able to power these experiences?
Unknown: There are multiple
technologies that were used before Pinot. Right? I think 1 of the things that really changed at LinkedIn was we were providing these analytics to
100 of millions of members and being able to also keep the latency very, very low. Because
as you know, users,
they don't have the same patience as the external users. They don't have the same patience as the internal analyst or the prep managers within the company. So there was a huge difference in terms of the expectations from an external
user facing versus internal use users.
And
in the classic alternatives where, as I said, elastic search was something and we, in fact, use that. There are also key value stores that people use it where they precompute everything,
but that has the curse of dimensionality.
Right? As you add more dimensions,
the storage explodes and they are not able and there is also
lot of inflexibility in the query patterns when you use a p value store. So those those were some of the things. I think the key change,
even though it was user facing analytics,
what really
changed at LinkedIn for us was this notion of
building data products. Right? And if you look at, existing tools,
they always
existing technologies, the way people consume the insights from them were through SQL editors or internal dashboards. Right? And that never
will
generate the level of query that you can generate through an app. Like, if you look at
all the apps on LinkedIn or on Uber,
no 1 knows that they're actually running a query behind the scene. So it is really a simple click is running a query behind the scenes. So our motto and our philosophy at LinkedIn was really all these reports, custom reports that we were generating.
Can we actually
convert them into data products? I'll give you an example here. For
1 of the products that LinkedIn had was talent insights. Right? Which is
simply providing insights on how many people moved from 1 company to another company. So you can think of it from Amazon.
They can think, hey. There are so many people moving out of Amazon. Where did they go to? Did they go to Google? Did did they go to Facebook?
Which area are we losing people?
Which skill sets we need to improve on? So there are a lot of these insights. And before Pinot, the way this was delivered to the users was someone internally would generate a report every month and then send it out. Right? And then there are only few questions that these reports could answer. And every time they needed some change, again, there would be an interaction between
the Amazon folks and the LinkedIn folks, and then someone had to write a Spark job or a Hadoop job to generate these reports and send it back. Now fast forward few years, now you have a Talent Insights as a product on LinkedIn, right, where the entire data is just every profile changes just simply fed into Pinot. And the customers can now come in, and then they can ask the questions. And then right there, you are able to slice and dice and then generate insights that you want. And there is no middle person in involved in it. Right? It's a huge change in the way things are seen both from LinkedIn's perspective and also the amount of power given to the consumers.
So that's the whole idea in terms of the primary use cases that motivated us to build, you know, and we capitalized
heavily on that. And as it's really amazing to see all these different products using Pino as the back end. So as far as Pino itself, it seems to sit at kind of a cross section of technologies
Unknown: where, in 1 sense, it seems to fit in the OLAP use case that products such
as ClickHouse or numerous other database engines
work
for as well as sitting in this embedded
end user analytics use case, such as what cube JS or good data are trying to power.
And I'm wondering if you can just talk through some of the
sort of juxtapositions
of Pinot as compared to some of these other technologies and
some of the systems that Pino is
Unknown: maybe designed to work in conjunction with and is assuming in order to be able to get the best possible use out of it? Sure. I think I'll let Sean actually talk a little bit about Uber itself right now. Like, how, you know, evolved in Uber, especially
seeing Uber is actually a 3 sided marketplace.
So there are a lot of cool things that happen, and then we can probably get into the
different trade offs and comparing with the roll up systems as well. So I just want to talk about some, like, Uber stories.
Unknown: If you think of Uber, it's actually a marketplace. If you take,
like, Uber Eats as an example,
they are, like, service providers.
Those are, like, the restaurants, and then they are
consumers,
which are the riders
to deliver your order, and there are also eaters who will order the phone app, whatever things they want to eat. Right? There will be a third part to which are the internal
operations.
They actually support all the business, and they can give coupons to the eters if they don't want to other. So those kind of things are, like, form, like, 1 marketplace for Uber Eats.
For Uber's usage at
Pino,
we first start with this kind of city
life dashboard
for Uber Eats internal operations.
So this was actually a dashboard that's shown
every TV for, like, Uber Eats soft Office.
So all these operation people, they come actually look at this dashboard
to see how many live orders in my city, in my area,
and what are the long waiting orders
that they need to take actions on those things. Right? They probably need to call the riders
to say, like, are you in a traffic jam, or if you are waiting in the restaurant. So at the very beginning of the Uber is business. So this kind of dashboard has been, like, helped the business grows a lot to, like, greatly improve the customer satisfaction and improve the
cost efficiency
of the whole end to end pipeline
workflow.
So
we actually expand
keynote's usage
to more like user facing analytics.
If you think of, like, service provider, like restaurants, they are also, like, 1 part
of Uber Eats business.
We actually want to provide
the more insights on, like, how your business is doing. Right? So
hence, we built something called restaurant manager,
which is, like, the
analytics
dashboard for
the Uber Eats restaurant.
The restaurant's owners actually look at those dashboard
and find out, oh, this is my revenue so far, and this is my projected revenue
for today.
And here are all my upcoming orders.
Also, for existing,
like, orders, what are the reviews
for every dishes. Right? So they can get all these kind of insights
from the dashboard.
And that's actually greatly
improved,
restaurant engagement, because they can actually
find out how their business is doing, and they can actually get the real time feedback
from, like, how their business is doing. Because all those things, usually, they won't be able to get unless you
ask specifically to every customers in your restaurant. Right? So this kind of experience actually improved
in our restaurant
engagement a lot. And after that,
we started some more use cases on the consumer side on the eaters.
So if you think of Uber Eats, right, so the initial
Uber is app only shows you the restaurants nearby with some kind of heuristics
to
suggest, like, those are the restaurants you like. You'll probably want to order something from there.
Figuring out that people may just order from those
restaurants they like, and they may not
pick those new restaurants, new joining restaurant. Right? So we actually want to improve
those kind of engagement
for new restaurants
and also
let eaters to actually exploring
more kind of restaurants.
So here, we actually
use Pinel
to build a feature called the orders
near you
to power the Uber Eats home feed.
So
this kind of feature will actually show what are the current orders
nearby
you, and it's basically
whatever things your neighbors are actually ordering.
This kind of a social experience,
very unique for
the eaters.
This kind of thing also inspiring the food and the restaurant discovery,
and they're showing, like, what your neighbors are ordering. This feature is actually being
quite popular when you are looking at the dishes your neighbors are actually ordering. So since this feature is actually part of the Uber Eats home feed, If you think of, like, how many
users are actually being looking at their app at the lunch or dinner time, has to be, like, a super fast and personalized and scalable
system in order to support as a backbone for this kind of system. And, of course, all the geospatial
features index,
those things, so we need to build for that.
Soon, a lot of usage has been expanded to broad areas at Uber, so Uber later
on allows
ads system
allows, like, ads to be, like, placed at Uber app, so you can't so actually in Uber, we build the ad system. You can actually do ads bidding,
and those kind of things at the Uber app. For this kind of system, we actually need to provide a super accurate performance and the business metrics.
Don't count down any kind of impression
or, like, click events, or you don't want to miss any of this kind of events. Those things will actually
make the ads
customers to be like
having bad experience about ads performance, and you also don't want to overcharge
or mischarge
your ads customers.
So that we actually build the upsell feature in Pinot along with exact ones, Kafka and the Flink. We actually provide the exact ones guarantee for the whole n to n streaming data infrastructure at Uber. That was actually
pretty impressive and very hard to achieve across the industry. Of course, at Uber, there are, like, so many data scientists, and they actually
need kind of the full SQL
access to your data, and hence we
are building the Presto Pino connector,
which
really connect Presto to Pino. And data science team,
actually, they use full SQL to explore all the data in Pino,
along with, like, joining with their, like, hive table
and do more advanced
algorithms
and verifications,
a lot of things, build models as well. So all those things are, like, being build across.
Unknown: RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.
Their SDKs make event streaming from any app or website easy, and their state of the art reverse ETL pipelines enable you to send enriched data to any cloud tool.
Sign up for free or just get the free t shirt for being a listener of the data engineering podcast atdataengineeringpodcast.com/rudder.
1 of the interesting
complexities
of the
particular
problem that Pino is trying to solve for is
because you are exposing this to end users,
it needs to be able to support sort of high concurrency. And I'm wondering, what are some of the ways that that
influences the overall architecture of the system to be able to manage that type of workload as compared to what some of these other OLAP engines are optimized for?
Unknown: Yes. So I think 1 of the design principles that we had very early on, we looked at all these other OLAP systems and these databases.
So the typical way most databases
thought about this solving these problems was, hey. We have to
scan all these data. We have to do x numb amount of work. How can we do this work faster? You had Cindy. You had columnar store. Those are all the classic things that people adopted.
But we took a very different approach because we knew that we could only push them as fast as the SSDs or the CPUs
that were all already available.
So our philosophy
was to eliminate that work instead of trying to do the work faster. Right? That's where we build a ton of indexes in Pinot. So we have
wide range of, indexes like sorted index. We have the bitmap index, which is the inverted index concept as well. We have the range index.
We have JSON index. We have text index. We have geospatial.
So there are a lot of different indexing techniques that we built, which
eliminated the work that needs to be done. So we would use this index, get to the right
location. So we even added a lot of enhancement in terms of rearranging the data. So even though we ingest it while we actually write the segments, we would rearrange the data so that we get locality. So we could solve
the query with 1 seek instead of having 100 seeks. So we did all these low level optimizations and a lot of these happen automatically.
And then last but not the least, we went beyond just the filtering and the indexing. Right? And a lot of them also took a lot of time in terms of aggregation because a lot of these user facing applications needed aggregation queries. So we built powerful indexes such as StarTree index. In fact, that's 1 of the reasons why we named the company as well as start re because it's a unique index that's not available in any of these other systems.
It's something that allows you to get even scanning billions of rows. With starter index, you can actually get that in millisecond level latency.
Lot of cool things that have happened in in the indexing area within Pino,
and that kind of what makes it
very, very different from this other OLED engines.
And, also, another principle here is that you have all these different
features within, you know, and you can use
things when you want them. Right? You don't have to use all of these at once. But the idea is that you have the knobs that can take you all the way from
tens of QPS to thousands of QPS. Right? And that's the beauty of the system. It's 1 system that can get you started on a project
and get get can take you all the way if your project becomes very popular. So we have a lot of products at LinkedIn which started very small, but then as soon as they become popular, now you need to move that to into another system. So that's introduced a lot of operational overhead. Now you need to have multiple systems within the company.
1 for internal, 1 for external.
That's kind of what we were able to prove at LinkedIn and Uber. And now you see that that has
spread across all these other companies like Walmart, Stripe, Target,
Cisco. All these companies have started doing the same thing. So it becomes 1 platform, and you can solve both your internal and external use cases.
Unknown: For those types of use cases, I'm curious
how
you have seen these businesses approach the question of which database engine and query engine they should be using to power these analytics, whether it's ClickHouse or Snowflake or,
you know, Postgres and then some of the different additional systems that they're going to need to be able to complement and support and integrate with
the ultimate technology that they decide as sort of that core building block.
Unknown: It's interesting to see the journey from the company's point of view. Right?
How do they look at the evolution of their analytics? So most companies, if you look at their journey,
they start off with the business intelligence first. They then get into the operational analytics.
So that's their evolution. And most of these tools, like a data warehouse or the whole app tools, the prior ones, we're able to address these needs. But now companies are seeing the value of providing these insights to their users and their customers,
and that's when they get stuck. They hit a wall, and they are not able to take these existing systems.
So the way we see it is you need all these different tools. Right? It's not just 1 tool. It's data warehouse is not just going to solve all the problems, but it's about how do you actually integrate these systems easily. If you look at data warehouse, that has a very different set of needs and differ different persona. So you have this business analyst
who write, like, 100 will join and then lot of complex queries that they keep changing and they create now. So for them, data warehouse is a perfect tool where they can actually create dashboards on top of it and then share the results. But when you look at app based consumption
where the end user is not technically,
well versed in terms of writing a SQL SQL query, for them, it is through apps. So they need to simply be able to click. They need to be context aware. They need to know what tool they're using and how to use that UX. For them, it's a very different experience. But then they end up generating a lot more queries,
but the query patterns are kind of well known in advance because it's the app that's generating the query. So different purposes, different engines, and they're all made for different purpose. Just to give you an example, Snowflake,
across all their customers, is really running 1 query per second. Right? Whereas LinkedIn, you know, just at LinkedIn is running at 200, 000 queries. And the main reason for this is with the access pattern. So 1 on 1 side, it's
through SQL editors
and dashboards, which doesn't really generate a lot of QPS. And on the other side, it is apps which end up generating tons of QPS behind the So the workload characteristics is very different.
The freshness requirement is very different. Also, the latency requirement is very different. So depending on these, you pick 1 system versus the other. But in the end, you need to have
ability to solve all the different use cases.
Unknown: Yep. I can add something related to Uber.
So at Uber,
I can take the example of the others near you feature. Actually, when we build these applications,
the initial version
of this kind of implementation was actually a sender. Cassandra is actually a data back end, like, the permanent store for all the place, the Uber's orders, and all those, like, restaurants.
So at the first iteration,
we actually implemented by, like, first, retrieve
all the nearby
restaurants
for given user's location. Right? You have the latitude and the longitude.
And once you have all the nearby restaurants,
you actually fetch
all the active orders
from another service called the order gateway service.
Once you find out all those orders,
then you will try to rank all those results based on the residency and, merge those things and push it to the home fix. However, for this kind of application,
it actually translate
1 users
into, like, 100
Cassandra lookups.
Right? So this kind of
translation will actually
greatly
increase
the workload on Cassandra.
So based on the estimation
for the Cassandra team in order to support this kind of use cases, it'll need to 6 times
Cassandra
store because all those kind of grid workload. They didn't need, like, more applications, the failure to store in order to start that. And once we move those things
into PNO right? So if you think of, like, all the all their status change, those kind of things going to PNO, Pinot is also optimized
for
readability.
Right? And with
the geospatial indexing support,
we actually convert this kind of 1
user's page review
into just 1 query inside Pinot. Also, with the geospatial index, the core latency was actually
dropped from in Cassandra, it was, like, a couple of seconds. You actually finish this kind of 100 lookup round trip. The p 99 was less than 50 milliseconds for just the 1 core, which means this also greatly improves
the user experience in terms of when you open the Uber Race app, you don't need to wait the app logo to spinning multiple seconds in order to see everything you come. Immediately see what are the others nearby you. Right? So all those kind of user experience improvement are super critical for the product team as well. I just mentioned those are cost effective for Cassandra. You need so many
machines. Right? And for pinot side,
the this kind of use cases are just being supported with 10 regular machines.
And that's also, like, very impressive achievement from the data team side to actually being able to provide this kind of support with a very small, like, hardware footprint.
Unknown: Digging more into the architecture of P and O itself, can you talk through the different
components of the platform and some of the ways that you proved out some of those early design decisions and
some of the elements that were key to being able to support these low latency and high concurrency queries?
Unknown: So from the Pinot side, the key design principle is we want to provide a predictable core latency and high availability and the scalability system for our users.
So for our external clients, we need the the always on system.
So you don't have any luxury for downtime by saying, okay. We have a database maintenance. Right? So we we actually need to make sure all the replications are there. There's no single point of value of the system.
So for all the layers inside the panel, we are using Apache Helix and the zookeeper for the cluster management and handle all those node value, all the node upgrade
seamlessly.
Second part is, like, for a p node to be, like,
super operational
friendly.
So at the LinkedIn, there were only 3 SREs there. At Uber,
when we build a PNO from the very beginning and then till now, the Uber's PNO team size hasn't been grow significantly.
I think we only have, like, 3 to 5 people at Uber to support the entire PIN node cluster, and the PIN node cluster size has been grew from, initially, it's only 3 nodes to guarantee your applications.
And now in 1 data center, there are more than 500 nodes
to
serve all burst use cases. And the worst to mention that this kind of pinhole has been designed as, like, a highly configurable.
So we actually want 1 system and 1 cluster for multiple use cases. Right? So if you think of, like, at Uber, this 1 cluster
in 1 region has been serving all the use cases along with the internal dashboarding
with the user facing analytics,
data scientist team to do full table scan on some kind of the tables.
So we're actually building the multi tenancy model PNodes so you can achieve all those kind of resource isolation
Unknown: inside the PNodes. And for the sort of typical end to end use case of
I have this event. It's triggered by some user using the application
through to,
I have aggregated across this event and all similar events to be able to provide some useful insight to an end user.
What are all of the different
system components that are necessary
for Pino to be able to
incorporate and aggregate and deliver that information to the end user, whether that's
Kafka for managing the streaming event inputs through to some of the kind of UI and application development considerations for integrating with Pinot.
Unknown: I would like to see this whole stack as, like, track 1st is the tracking of click clients. Right? Do you need to somehow track all the activities that is done by the user?
And, typically, this is either
a Kafka client or it's some sort of an application in this d k that is embedded in the application itself. The second part is, as you mentioned, the collection. Right? So this is where something like Kafka
or Pulsar or PubSub or even have all these things come in as this. All these different systems come in play. The third 1 is really the store plus analyze. Right? And that's where something like either the data warehouse
or something like p no comes into picture, both storing and computing. So there are different applications here. Some of them just provide the storage. They don't provide the compute. Some of them just provide the compute. Some of them provide both compute and storage. So there is a lot of different systems that are getting that are being built in just the compute and storage part. It's complex, and it's also very important piece of the tech stack. And the last but not the least is, like, really your visualization layer or any sort of apps that you build on top of it. So in terms of, you know, we try to make it as simple as possible. There is an intermediate layer that I missed, which is the processing layer before it gets to the storage. So if you look at it, it's collection,
and then there is a processing layer like Flink, Samsa,
and other ETL systems, case streams, or even Spark and Huddl in the batch. Right? I think there are these processing system that just process and then put it back into the original store, be it Kafka or a Datalink.
So what we are doing is part of Pino is we want to simplify
the ability for any user to get started. So
when we initially started, you know, we
had this need for only having structured data. So that means you needed a processing
layer in between that would take the data from Kafka
and convert it into a structured form and then feed it into Pinot.
So off late, we are going from just having them supporting the structure
to even being able to support semi structured
and even getting all the way to the unstructured data as well. Now we can just
simply point at Kafka
in just the complex arrow format as it is in we convert internally into JSON.
We are able to add indexes on any nested complex JSON fields, and that the and the users can actually start querying.
And as they start querying, if they want to create new fields for some of the most commonly used ones, they can add derived columns on on the fly. So we are trying to change the workflow from the user and also the experience where they can simply get started very fast.
They don't have to worry about indexes. They don't have to worry about the e ETL step in between. But as they learn, as they build more and more applications, they can create the right columns on the fly without having to ingest the data. They can add indexes on the fly. They can configure pre in the pre aggregations using start re index. So we are trying to make it more iterative process, and this every iteration is dynamic and can be done on demand
without having to reboot stuff the data. That's the way we are going about because we don't want to have a lot of initial step for the users to actually get something out of Pinot. And the second 1 on the consumption side is also the need to whether you do a pre join
or you do a post join. Right? And, again, this depends on the application itself.
Sometimes the latency is they have a much bigger buffer for the latency requirements.
In that case, they can just ingest the data as it is, have a layer on top of it, which is Presto that can actually do the joints across data in p node and also in some other data sources.
Even trying to simplify that if the data is always in p node, then the joints can also be pushed down into p node. So we are adding joints as we speak, and we had the first version of join, PR go out last week. That's another capability that we are adding. So overall, we are trying to
reduce the amount of preprocessing
and the post processing that is needed
for someone to con build applications on top of Pinot. At the same time, we have all these other knobs
in case the workload requirement changes
and you have
a very popular product and then you need really, really low latency for the end users, you can still do it. If you want to run internal applications,
we want to use build applications such as anomaly detection and things like that. You can do that too. So overall,
we are trying to improve the coverage,
simplify the initial onboarding experience for Pino, and then but at the same time, have the ability that if they want to build, like, really, really powerful applications,
they still have the knobs to actually get there.
Unknown: As far as the data modeling considerations,
you mentioned that in order to be able to
run joins
across data that's stored in Pinot, you end up having to use Presto
for being able to support that type of syntax. And so I'm I'm wondering if you can talk through some of the ways that
users of P and O need to be thinking about the way that they lay out their data structures and data models to be able to power these
user facing analytics use cases
and some of the edge cases that you've run into as you start to design and build out these systems?
Unknown: Definitely.
So as I said, this was a big concern in terms of the usage of Pinot because they had to do a lot of thinking
upfront because we had only structured data support.
Right? And that means we could only support column types of int, long, all the primitives.
But now we have with the support of JSON and being able to even index fields within a JSON structure
and even unstructured,
This is reducing
drastically.
You can defer that thinking to a later time so that you can get started very quickly.
And the second, in terms of pre join versus the post join, this is a great thing that keeps coming up every now and then. It's just simply impossible to get the latency that you need for your user facing analytics when you have joins during the query time. So
you will be sort of forced to do the join up front, if you are going for user facing analytics
because the experience of the user is really, really critical.
Lot of times we have seen
companies try to avoid the join and then they try to put the do the join during the query time and the experience is so bad because you have this cursor spinning all the time and the experience is really bad from the user point of view. So it is worth paying that additional
overhead upfront in terms of doing the joins
before and then avoiding the joins during the real time. So push it. So it's really in
the overall scheme. It's where do you put the join? Where do you put the transformation?
That is really what someone needs to think about. You can always get started with
no joins, no transformation
as it is. Now as you see the product becoming more and more successful,
then you can introduce the pre join and also the pre transformation if needed. It's all about having that flexibility
that you can change this, but you don't have to change the system itself. That's the most important thing because changing the system means that you're trying to change the whole stack. Now you need to change your clients. You need to change
a bunch of things, and that is something that is not needed that we want to avoid as much as possible. Somewhere down to, like, from an Uber side, initially, we actually
Unknown: just try to do the very simple thing. We try to dump 1 Kafka
topic into 1 p note table. That's kind of the story you are like.
We allow user to do the self serve email onboarding. They can't just, say, okay. I have a 1 type of topic which has schema inside and can pick some columns and dump the data into Pinot. And then I can use, like, Presto for joining and for exploring,
then they will figure out, okay. So for my use cases,
I can't survey it from the Presto side for some kind of users.
And then they later on figuring out, okay. I need this kind of latency and the QPS
requirements to be supported. Then they actually move this kind of presto drawing or whatever the declaration queries or pre aggregations
into something like stream processing, framework like Flink. And they can write, like, stream SQL to actually create those kind of new tables, so flatten the tables. And then ingest that into Pinot. And in Pinot side, you can also build assorted index
or inverting index in terms of serving more, like, user facing analysis queries, like since those queries are, like, per user business thing, you can you can actually do more indexing and the query performance furthermore. So all those kind of are more, like, journey or, like, how a product or the data model has been built. People really, although they have the rough idea of their product, they don't really get the data modeling
first and make it right. It's always iterative.
If you build the entire stack, which from the Kafka to Presto,
it actually allows the product team or the application people to figuring out what they really need, and, also,
it greatly improves
the iterative time. Right? So if you want to upgrade
your system, add 1 more new feature. The turnaround time is actually much faster in terms of using this whole stack.
Unknown: Are you looking for a structured and battle tested approach for learning data engineering?
Would you like to know how you can build proper data infrastructures that are built to last? Would you like to have a seasoned industry expert guide you and answer all of your questions?
Join Pipeline Academy,
the world's first data engineering boot camp. Learn in small groups with like minded professionals for 9 weeks part time to level up in your career.
The course covers the most relevant and essential data and software engineering topics that enable you to start your journey as a professional data engineer or analytics engineer.
Plus, they have ask me anythings with world class guest speakers every week. The next cohort starts in April of 2022.
Visitdataengineeringpodcast.com/academytoday
and apply now.
Another interesting element of
this application space is the question of the data life cycle,
where when you're working on internal analytics for an organization, you're likely going to want to keep all of the data available for querying for a fairly long time horizon.
But for user facing applications, those time horizons can be highly variable depending on context.
And I'm curious
how you have approached that aspect of the problem in P and O to be able to support this
sort of management of time horizon and then data tiering so that you don't necessarily
have to have an all or nothing approach where maybe you just have to accept some level of latency for queries that are going further back in time.
Unknown: Absolutely. No. That's a great point. As you saw, we started off with the intent of user facing analytics where the retention span is much lower. Right? I mean, we see somewhere
between 30 days to 2 years. Right? That's where and do you know the value of data also decreases drastically as they age?
But 1 of the common things that we have been asked from the users is if we allow the speed of pino,
and we also want to use it for internal analytics as well, but it is not cost effective because now we need very tightly coupled architecture. So now we need to scale the compute and the storage together,
but we very infrequently
query the older
data. So that's 1 other key big change that we did in Pino. So we were able to add this concept of tiered storage.
So that again, kudos to all the engineers in the Pino
team. We built it in a way that there was a right abstraction in terms of the
file system API. We were able to move from the local file system to the remote file system. So now, you know, can actually run directly on s 3. That
changes the whole game because we are seeing
someone like Stripe, for example, has
the transactions from day 1. It's huge. They're already at, like, 2, 000 notes with petabyte of data and it's already
overtaken LinkedIn and Uber in terms of the scale.
And something like a tiered storage is a big game changer because now you can keep the data for as long as you want, and you can tier it to the s 3 or any other blob store as the data ages. It's a really big feature. And the best part about this, as with AD, everything else is, you know, has been designed where you can almost think of about this as a slider. You can actually decide at which point in the slider that you want to be. You can have 1 extreme of all of them are tightly coupled, local both the computer and storage,
and the other extremes, everything is decoupled. And you can also be in between. You can say for the 1st 10 days, I want it to be tightly coupled. I want the data to be local. And after this data ages more than 10, it has to it can go go to the s 3 tier. Those are all, again, configurable
not only at a cluster level, but you can even configure it at every table level. You can have some tables
have
moved the data after 30 days, some of them after 10 days. Again, it's a ability of not only providing you the trade off, but also the ability to provide that control
that you can decide the trade off point. And depending on the application and depending on the requirement,
you can change that. So tier 2 is 1 of the big big things for us. We have added the support for s 3. We plan to add for GCS and Azure as well very soon. Apart from that, for data life cycle management,
Unknown: you know, side that you can also use to
do, like, aggregations
oppose aggregations
for your real time data as well. Because typically,
when you are ingesting data from Kafka, all those things are event based. However, for your queries or your applications,
a lot of users, they they actually just to show the aggregation results. Right? So in terms of that, Pinel Millian can actually do the post aggregation
after 1 day is finished or the configurable time, actually. You can you can write in hour or in
1 day to
aggregate all those events and generate much compact data block and,
serve it in a more efficient way, which means for even for the same data,
you can trace many more years of the data, but with, like, a very low cost. You will also then come do this kind of aggregation.
Unknown: As far as being able to customize Pino, you mentioned that it was designed to be flexible and adaptable to the sort of end users' requirements. I'm wondering if you can talk through some of the
interfaces that are available for people to be able to extend and implement their own capabilities
on top of or within Pinot?
Unknown: So for the Pinot side, we basically make, like, every component
to be, like, extensible or pluggable. For example, at the LinkedIn time, LinkedIn has the stack of all the batch processing in Hadoop.
The LinkedIn has its building data format is is several, and the real time ingesting are coming always from Kafka.
So that's kind of, like, how the initial version of penal, which you only need to support this kind of system integrations.
And the later on, when we move to the open source, we've got many people has, like, different requirements, and they have different systems.
So we make a lot of things to be customized. The first is definitely the encoding and the file formats. So apart from the Averro side, we also support
straight for the above
and also like a or, say, all those kind of different
file format is already been built in inside the Pinot.
Execution framework. The batch execution framework, we have the support from
Hadoop, from Spark, from Flink.
So all those kind of batch processing are now available
from the open source side that you can actually run your, you know, segmented generation job in Spark or Flink as you wish. Also, for the
streaming data ingestion,
Pinocall actually
now, you know, abstract
all those kind of streaming interface, how you get data into Pinel. So from Kafka side, it's just like version upgrade. We have 2 Kafka interface, 1 4, Kafka point 9 and the 1 for point Kafka, kava 2.0 plus. So you have 2 Kafka connectors, and then we later on build the Kinesis port. But, also, we have the Google Pubs app and the Apache browser. Recently, there some more contribution
from the community for RabbitMQ.
So all those things are, like, all based on
the clean,
the right abstraction
for the interface. So it makes
the user easily
hook up their existing system.
And, also, they can contribute
effect to the Pino community.
Unknown: And in terms of the applications
of Pino, I'm wondering what have been some of the most interesting or innovative or unexpected ways that you've seen it used. I probably was definitely the 1 that came to my mind is
Unknown: the ability to just keep the raw data forever and at the same time be able to do aggregation also on top of it using start index. So now you can start at a very high level, and then you can start doing root cause analysis. You can narrow down, and you can get to all the way to a raw event. So that is a very interesting use case that we saw.
And in terms of just the ads itself, I think there is
the way people have started using the geospatial indexing
and the ad tracking and the ad accounting is also another
amazing thing that we saw where ads are shown on buses.
And then
you track the buses on 1 side, and then you track the users on the other side. And then you are able to
do the in intersection between the 2, and then you are now able to see how many people actually saw an ad on the bus. So that's an amazing use case that we saw. It's really cool to
see how many people
get inspired from things like Uber Eats. Like, you can think of a website now selling cars. Now they can provide analytics to their dealers
who can actually
how many people viewed your car page, how many people clicked on what, and you can kind of provide all these insights back to the dealer. So there is lot of these insights that can be provided.
So just a call, for example, another cool application that we saw is the transcripts of the call that we are talking about. Right? Like, that can actually be fed into Pinot. You can think about, like, how long did Tobias talk, how long did Sean talk about in this call? And how long did I talk? What was the tone? Who said hi first? Who said bye? All those things can actually be provided. It's like lot of these things that we never thought being in a company at LinkedIn and Uber,
but now it is lot of things in terms of both b to b and also b to b to c. Trading is another application. So there is tons of things that we are actually seeing in the wild that we had definitely not thought about before. But really good to see how people draw analogies between
the ones that we have already built and how they translate it to their use case. Education is another 1. They're looking at the videos. They're like, okay. How long did the users watch this video? When did they pause?
How many was there any section that was watched multiple times? So again, so a lot of analytics on pretty much every entity that you can see on the web, you can basically provide analytics around that. So, yeah, definitely
lot more things than
than what we expected. And, again, LinkedIn is a great example. We already have, like, 70 products on LinkedIn itself.
That itself is was a huge win for us in terms of seeing
how quickly the adoption grew and what kind of products they were able to build.
Unknown: And in terms of the kind of growth and adoption of Pino, I know you mentioned a number of different companies that have been relying on it. But
from when it first became public to where you are now, I'm wondering if you can talk to some of the ways that you're thinking about the
sort of community engagement, community involvement,
and areas of contribution that you're looking for.
Unknown: Yeah. Definitely. I remember when we started, we had a very small community. We had, like, a Slack channel of, like, 100 people in 2020, and then we were not even tracking their downloads. We had 0 downloads.
And fast forward 2 years, we have, like, more than 2, 000 people in the Slack community.
Very vibrant, asking a lot of different very, very exciting questions, and we get to know all the use cases that we had not thought about before.
And at the same time, we had, like, almost a 1000000 downloads more than a 1000000 downloads last year itself.
So huge huge growth in the community
across again, lot of help from LinkedIn, Uber, and Stripe, and all these other companies getting involved in the project as well. It keeps pushing the envelope further and further.
The bunch of things that we do want help from the community is extending the features at Uber, for example, helped us add the absurd feature, which is a big differentiator and something that a lot of other systems are not able to add. Then tiered storage is something that we from start re is contributing to that towards that. So join is another 1 that we did. We are adding that as well. To JSON storage. So there are a lot of these things, and we actually even publish the road map. And we also share the road map, and people community can actually work on that. Any members from the community are listening to this. Definitely encourage them to vote on these features because
take that very seriously. That's something that we consider for the prioritization of the work that we do. Again, it's it's community is our product management for us. They tell us what the priority is. So really looking forward, not just from contribution,
but also in terms of their requirements and where they want to take the system forward.
Unknown: In your own experience
of working on the Pino project and now your work at StarTree, what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
Unknown: I think what's the biggest challenging here is
when we were at LinkedIn and, Uber, we actually focused a lot to the speed of PayNow. Because for all those kind of well established companies,
you have a pretty good kind of ecosystem, which all the software foundation has been placed there. So you can you can leverage those. For example,
how you deploy Pinot. At the LinkedIn, they're, like, content management. At Uber, they're also, like, have all the Docker, Mesos,
and all those things there. But when we come to start with and we figure out, oh, this industry are using Docker, using Kubernetes,
They want
to have those kind of integrations. So we actually quickly build those
recipes on how you can simply deploy Payno. You can
also do things like the management. Right? So at Uber and LinkedIn, we have been build a lot of toolings.
Now we actually
make all those toolings to be actually built in p node APIs,
which you can easily do, let's say, able rebalance, those kind of things in p node itself.
Another thing here is we actually transformed
to make p node plug in model, and we will focus we will actually
focus the more focus the more on the right abstraction of the interfaces.
So panel itself can be easily extended
to, like,
suitable to, integrate with other systems.
So
at the very beginning, when we build, for example, the Kafka
connector, it's a chips like a couple of months since. The once we make the clean interface for that. So for the NIS
connector, the POC was down in 2 days, and it took, like, a week for us to really productionize it. But this kind of turnaround time are actually super impressive,
and you actually realize the power of you are doing the right thing and making the right app structure. And all those pluggable model
really makes the difference also for the community to contribute things into your system, and then you have the right capability for, like, unit test, integration test, all those things. Yeah. I think Sean pretty much touched on lot of different points here. Only thing that I would probably add there is
Unknown: kind of our thought process being that LinkedIn was always about speed. If you kind of look at the evolution of how you build a feature or a project is, like, make it work first and then make it right and make it fast. But at LinkedIn, we would always shoot for make it fast because that's kind of the scale at which we were operating. So speed was, like, really, really important for us. And I think as we see more in the open source, it's about the iteration speed is also very important. So we want to get the feature out. We get it in a good enough shape so that people can use it, but then constantly iterate on top of it. So being more focused on the iteration speed and then taking it to the finish lines
in multiple steps versus trying to get to the best in the first version. I think that's something that we are being aware of and then trying to change that.
Unknown: For people who are looking for a system that will provide low latency queries and potentially serve,
you know, multiple concurrent users for user facing analytics? What are the cases where Pinot is the wrong choice and they might be better served with a different architecture
or a different storage engine or, you know, maybe reconsidering their requirements?
Unknown: Yeah. Definitely. I mean, the first thing is it's definitely not a replacement for data warehouse. We have seen many cases where, you know, coexist with BigQuery, Snowflakes. They have all this data in Snowflake, but it doesn't allow get them the speed that they want to serve external applications.
So they take the data and then put it into p n o. So that's perfectly fine. This is probably not a tool where you give it to your business analyst and then ask them to write, like, a super complex 100 way joint query. So definitely not a replacement for data warehouse.
It's also not a replacement for, like, your traditional key value stores. So I wouldn't,
instead of using MongoDB, Cassandra, or any of these other systems, you don't wanna use Pino for that. There's no transactionality
that is there. It is not really built for asset compliance, so it's not an old TP store in that way. Definitely not a relational database as well like old TP, not MySQL, Postgres, So that's not a replacement for that.
And it's also not a replacement for such systems where you want
ranking relevance
like Lucene and Elasticsearch. They have the right systems.
Though we are seeing a lot of workloads
move from some of these systems into Pino. And the reason is just that something like p no was not available, so people ended up using whatever system that they had. But they hit the scale and they they hit a wall, and they're not able to get what they want from the system. So we do see a lot of use cases
move from Postgres, from Elasticsearch,
from Snowflake data warehouses, and also even from the key value stores. A lot of them do move to Pinot, but that's because they were not the right systems to use for analytics in the first place.
Unknown: As you continue to
build and iterate on the Pino project and grow the business around StarTree? What are some of the things you have planned for the near to medium term or any projects that you're excited to dig into?
Unknown: Yeah. Definitely. Eared storage is 1 thing that we all already released a preview version of that. So now we can we support s 3. We we will plan to continue iterate a lot on that. We see that,
our goal is to be able to provide almost the same performance as we are able to provide
with local storage. We are already seeing that it's very close with, like, literally, like, some 200 milliseconds extra latency, and we have done ton of
innovations there in terms of being able to provide this performance with the tiered storage by being able to work directly on s 3. So you are still getting sub second latency
with that. And the second 1 that we are super excited but we'll continue to invest is going from this concept of data is immutable once you put in p note 2 being mutable. So it's not only mutable in terms of batch,
but you are now able to
update every single row in p note. Right? I think that changes a lot of things that you need to join before. It doesn't need any does do not need joins anymore.
You can get accurate metrics. A lot of things open up because of this ability. So that's another thing that's super excited.
Last but not the least, there are data models that are structured. There are data models that are semi structured, and they're unstructured.
We are taking that to an extreme where we say that you can have a table in p and o and
you can decide what each column should be. Each column can be either a structured column or a semi structured. So you can have a table with
lots of different columns, and each 1 can be its own format.
So that's another thing that we are excited about. Join is a new 1, and we plan to continue to invest heavily on the join. Again, in collaboration with
Uber and LinkedIn,
build core distributed
joins, multistage.
So we already have the first version that's already
in progress in preview,
and then we plan to add the collocated joins and all those other fancy things in the join site.
In terms of the use cases, so 1 of the things that, you know, was really for the metrics use case, but we are seeing a lot more of logs and phrases, applications coming into Pinot. So people are using Pinot as, like, the strong back end defacto back end to build their
observability
systems on top of it. So a lot of companies are using Pinot as that back end. So we will continue to enhance
Pinot being your single store for your logs, metrics, and phrases. All of them can be served in a more very efficient
manner with all the features that we added so far. And last but not the least is, like, integration with all the other
players in the ecosystem.
Presto, Trino,
Kafka, Pulsar, all these spark. We want to make sure that our integration is really, really strong. As you see, the stack is be is is quite complex right now, and we want to see take a lot of burden out of the users to be able to integrate this with other systems in the world.
Unknown: Are there any other aspects of the work that you're doing at Pino or the ways that it's applied or the specifics of its implementation that we didn't discuss yet that you'd like to cover before we close out the show? 1 thing that I would like to add is
Unknown: really the way we see
the whole tech stack layout and how it's changing.
I kind of see a very strong
analogy with how the monolith
is splitting into microservices
in the stateless system.
I see a similar thing play out in the data system as well. What used to be just 1 giant, if you look at it decades ago, was, like, Oracle doing
your collection,
the storage, analysis.
We it did everything in 1 system, and that's the thing that we are seeing that change. Now there is 1 system for just collecting
and which is Kafka, and there is 1 system just for storage. There is 1 system just for compute. Each 1 is becoming so complex and, and also for a good reason in terms of evolution.
And I think it's a good thing to happen. I mean, the same thing is happening on the data warehouse side as well where everything was solved in a data warehouse, and now that is getting split apart in terms of different systems coming into picture and doing 1 thing really, really well. So what this is definitely
a little bit of an overhead for the users because now they don't have 1 tool. They have hundreds of tools. Some might argue this is a bad thing, but I think it's a part of the evolution in my view. What we really need is the tooling system to
make sure that these systems can integrate seamlessly.
And so more standardization
being evolved across all these applications will definitely help in building those type integrations.
For example, s 3 is definitely 1 of the things that standardized on the file systems. Right? And Kafka is getting there in terms of stand being this the fact of standard in terms of the messaging
API.
So the more these standardizations
evolve, there is a lot more
Unknown: ability to seamlessly integrate these different components. We are super excited about that. We continue to make sure that Pinot is super easy to integrate with other systems out there. Well, for anybody who wants to get in touch with either of you, I'll have you each add your preferred contact information to the show notes. And as a final question, I'd like to get each of your perspectives on what you see as being the biggest gap in the tooling or technology that's available for data management today.
Unknown: I kind of covered that in the previous 1. I think it's built on that. It's really that there are so many different tools out there, and it's definitely confusing for any user to know what tool should I use for what and making it really simple for them to put these pieces together. Right? I think the way I see it was previously
in in data, you would buy
a prebuilt toy. Right? And you were not really building that toy. But now requirements from within the company have changed so much that you just 1 toy is not going to serve all your purpose. So you need, like, a lot of different toys. But you go to the market and you what you get is, like, Lego pieces.
So you now need to put those things together to build the stack that you want. So it's very powerful because now you can build so many different toys from these Lego pieces, but at the same time, you need to know how to connect them. It's a different challenge from what used to be there before. The previous 1 was, like, you would have, like, 100 different systems doing 1 thing,
and you don't have very clean stack. So you have the same data sitting in, like, 100 different systems. So that was a different problem. Now you have a different problem that you can actually build an amazing stack,
but putting them together is not easy and not trivial. So I think that's the gap. Again, it's part of the evolution. I think 10 years from now, it will be a very different thing where makes it so simple and seamless, especially with the SaaS companies coming out and everything being offered as a service. You don't have to do that. It should be There will be seamless integration between all these services so that you can actually put together
completely end to end stack within within a day or 2. From my side, I think, Deepak pretty much, like,
Unknown: cover all my thoughts here. Yeah. 1 thing here is, like, about more advanced kind of integrations with all the other
surrounding areas apart from, like, the reporting, dashboarding, those traditional
BI use cases. Here, we have, like, more things like data scientist kind of integrations.
There are a lot of data scientist
people. They are
building models. They build companies to say, okay. You can roll this fancy model, saw this machine learning platform,
hop off, like, different data stores. And here, since the trend
of moving to real time data
is,
like, becoming more and more useful, and people realize how powerful those real time data are. In the old time, we don't have a way to provide the reliable,
fast
real time data. Nowadays, this is not a dream. It's
the reality. Those kind of integrations and how people are leveraging real time data and discover this kind of gold mine
is
the next thing and the missing part that you can actually leverage this real time to do more things. And people should, like, open up their mind, find out, like, what are those things actually
Unknown: being being achieved. Alright. Well, thank you both very much for taking the time today to join me and share the work that you've been doing at Pinot and now at StarTree. It's definitely a very
interesting product,
Unknown: interesting project, and an interesting problem space with a lot of complex requirements. So I appreciate all of the time and energy that you and the rest of the community have put into that, and I hope you enjoy the rest of your day. Thanks again, Tobias, for having us, and kudos to all the great work that you are doing. I've been listening to your podcast for a long time. It was very, very insightful. Get to learn a lot of good things. Keep up the good work, and thanks for doing this for the community.
Unknown: Thanks, Tobias. Thank you.
Listening. Don't forget to check out our other show, podcast.init@pythonpodcast.com
to learn about the Python language, its community, and the innovative ways it is being used.
And visit the site at dataengineeringpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com
with your story. And to help other people find the show, please leave review on iTunes and tell your friends and coworkers.