Unknown: Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
Have you ever woken up to a crisis because a number on a dashboard is broken and no 1 knows why? Or sent out frustrating Slack messages trying to find the right dataset? Or tried to understand what a column name means?
Our friends at Outland started out as a data team themselves and faced all this collaboration chaos.
They started building Outland as an internal tool for themselves.
Outland is a collaborative workspace for data driven teams, like GitHub for engineering or Figma for design teams.
By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets and code, Atlan enables teams to create single source of truth for all of their data assets
and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker, and more.
Go to dataengineeringpodcast.com/outland
today. That's a t l a n, and sign up for a free trial.
If you're a data engineering podcast listener, you get credits worth
$3, 000 on an annual subscription.
When you're ready to build your next pipeline and want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode.
With our managed Kubernetes platform, it's now even easier to deploy and scale your workflows or try out the latest Helm charts from tools like Pulsar, Pacaderm, and Dagster.
With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.
Go to data engineering podcast.com/linode
today. That's l I n o d e, and get a $100 credit to try out a Kubernetes cluster of your own. And don't forget to thank them for their continued support of this show. Your host is Tobias Macy. And today, I'm welcoming back Raj Bains to talk about how improving the user experience for data tools can make your work as a data engineer better and easier. So, Raj, can you start by introducing yourself for anybody who hasn't listened to your previous episode?
Unknown: Thanks, Tobias. I'm delighted to be here. I'm Raj Bains and,
the founder of Prophecy.
It's a local data engineering platform.
I come before this from a background in engineering,
primarily
compilers and various parts of the data stack, and now bringing that kind of power to
the visual tools for managing data.
And do you remember how you first got started working in data? So I was working first in compilers, which was in Microsoft Visual Studio, some in Microsoft Research.
And then I moved to CUDA
at NVIDIA, when we we I was kind of in the early team or you could say the founding team as we call it these days, building that. But after that, I wanted to build my own company.
Compilers is not a space where you could start start ups. So I was like, okay. What's a related space?
A lot of challenges were happening in data. Data was exploding. So I'm like, okay. Let me move into data. So I moved actually first into an operational database company that was building distributed databases
called Clustrics.
I went there as an engineer, worked on building distributed aggregates, worked on the query optimizer.
And then from there, the problem was repeatable product market fit. You know, they had a great product, but not quite a scalable go to market motion.
So from there, I moved into
product marketing, product management, trying to find
the fit and that kind of movement. But that was still in the operational
data space.
Then from there, I moved on to
Hortonworks where I manage
Hive,
and Apache Hive was the primary product in the Hadoop stack.
And through that, I got to understand the other part of data, which is, you know, where I guess, the most of the action is, which is analytical databases.
And really got to understand the data engineering, the data analytics, you know, even data going into machine learning
and the challenges that various companies had. So,
definitely, it was,
you know, pretty much
saying what my current skill set is. With that, if I have to go into a start up space, it's very opportunistic.
And then they're kind of using compiler as a thread to move from,
well, you know, c type compilers to database compilers to understanding the data space and now into power tools
for data.
Unknown: In the compiler aspect, it's definitely very interesting and,
I guess,
underappreciated
area of computer science where if you look at it the right way, most things can be broken down into the different stages of compiling,
whether it's, you know, compiling code or transpiling.
I actually had an interesting interview a little while ago with the CTO at Compiler Works to talk about their approach to using compilers for everything. And I know that you have a similar approach of being able to take some of that compiler logic to say, here's the high level goal. I'm actually now going to use that to generate the actual code that's going to run so that you don't have to be the person who's, you know, writing all of the verbose syntax to make sure that all of your Spark jobs are going to be fault tolerant and scalable.
Unknown: Yes. Definitely. Compilers
remains, you know, at the heart of our
product. And it's amazing how powerful it is and how many things we can do. I mean, going from, like because we are strong in compilers, we are able to take some of the legacy ETL products, move their codes into our product. Right? And to our and then
in a sense, like, when we developed our product, right, you have low code, which is visual. You have code. But then we have our own intermediate representation of a data pipeline.
And now, you know, you come from a format, you know, you can go to PySpark. You can go to Scala. You can go to visual. And once you are in this compiler world, all the data logic kind of, you know, can quickly move between these various things, and you'll have this much deeper understanding
of what your user is doing. You know, to a certain degree, it was just luck, but ended up being a great fit for this area.
Unknown: And so in terms of the
aspect of the
interface that is presented for the different data tooling, you mentioned that prophecy is a low code tool to make it easier for people to be able to quickly wire different pieces together.
Some people might prefer the, you know, very code heavy approach of, I want to actually write all of the logic and fine tune it and be, you know, deep in the guts of the systems.
And then there are some of the more kind of vertically integrated where you just say, you know, we're going to provide you with best practices. You just tell us which, you know, prepackaged option you want. And I'm wondering if you can just talk through what you see as being the
broad categories
of data tools and the designs that they adopt and some of the ways that that impacts
what is possible to build with them. Definitely.
Unknown: I will maybe start 1 layer lower in the stack and then build up to it. So if I'm doing data engineering, the first thing is there is I need to run it at a particular time. So you have the scheduler, orchestrator layer. Right? And why I'm mentioning that is sometimes, you know, when we went to raise our series a, you know, some of our investors were not even able to differentiate between orchestrators and ETL tools. So Tishinder, so I'm kind of going to go in the you know, kind of break it up. So you have the schedulers and orchestrators,
which is, like, using something like airflow
and then using maybe Databricks jobs if you're using Databricks.
And all the legacy ETL tools had, you know, an inbuilt scheduler, and then you had the enterprise wide
Now they are saying, okay. Run the job at this time. Run this, then that. Right? And then you have the data engines. Right? You have the spark.
You have data warehouses and then, you know, the data lake query engines.
So that is your processing.
Again, comparing to the legacy tier 2, some of them had the whole thing in them. Right? And now it's kind of broken up. And now comes more of the tooling. Now the question is, but what about metadata?
That, again, belonged in that same package, but today, there is some start ups on the operational side around, like, LinkedIn Data Hub or those. And then there is some
on the consumption side, right, where you have Calibra and those people who are more in the data warehouse analyst interface.
Then there is observability
again, which used to be part of the same ETL package. And now you have Excel data and other companies who are focusing on,
you know, end to end observability of your data. So these are all there. But all of these are still
infrastructure,
but not what you use to get your job done. Now, you have to actually get your job done. So you're going to say, okay, I'm going to use some paradigm. Maybe I'm going to write scripts. Maybe I'm going to use a visual tool. So that that kind of starts to break down things into categories. So
first is
the category of point to point data motion. Right? It's like I have a bunch of systems. I just want to ingest data.
So there, you are going to get your Fivetran,
Airbyte,
HEVO.
So these are kind of, you know, if I want to move data from Salesforce to my data warehouse, I just connect the pipe. It's going to land the data in a preset format and then going to
update it. Right? So change data capture. So this is just the ingestion piece. And they are kind of saying, I'm gonna solve the connector problem, get data into your systems, make life easier. And then, also, there is a counterpart for the
on premise operational systems and getting that data into the s 3 cloud. So there's, like, precisely stream sets. They do that.
So these are the CDC tools. Now once you've done CDC,
you have your source data from operational systems.
Now you've got to do the transforms.
So transforms
then have,
you know, 2 ways of doing it. You can write your own code. So you are writing PySpark or you're writing SQL. Or sometimes SQL doesn't always capture all the use cases, or you are using a low code approach
where you would use Matillion
or prophecy.
Right? And we are kind of different tools in that, different ways of doing it, but at least Matillion and prophecy broadly, you could say are in the low code category,
though we are very different. And then once you've done your data transformation, your data is inside the data warehouse, you want to write some SQL queries to get a regular report or something.
You can use, you know, raw SQL. You can use something like DBT, which helps you break down SQL.
And any of these can do they do some light ETL, and they do. And then they do some mostly, like, reports and then dimensional modeling and that kind of stuff. So these are kind of the set of tools that you can use on top of
your data platforms, which is like the data breaks, the data warehouses,
you know, to actually
create your business pipelines.
So I guess this completes the broad
categories
of the tools. And then the second part of your question was how does that impact what's possible with them? The point to point data motion, right, that's all you can do. Right? So that they are great. It's a specific use case, specific solution, and it solves it well. So they are great. Like, the 5 trans of the world, great products. Now the next thing is when you are building data pipeline,
you know, you are kind of deciding between the local options,
which is the coding option. And there, you know, low code options give you more reuse, more productivity.
And then if you go with the visual ETR tool, a lot less is possible than you could do in code.
If you're going with prophecy low code,
we are kind of merging the 2 approaches,
saying the old visual ETL or the material kind of ETL
with code and the best of both worlds.
And we can get later into it. But, like, we believe we can do everything
and better.
But in this stage, right, it's efficiency is important. Scale is important.
You know?
And, you know, reusability of business logic is important.
Lineage,
search,
basically being able to understand what your transforms are
doing. Because on the other side, there's a data analyst consuming the data. They've got to understand what's going on. Right? Governance is important. Right? Can you track PIA data?
So in this middle bucket of transformations,
if you kind of roll your own scripts,
which is what most of the industry is doing, you don't have lineage often. You don't have search. You kind of give up on governance. Right? Your consumers,
the people doing analytics, they don't understand how any value was computed.
Right? Your metadata architect cannot understand
where PII values are ending up. The code is severe, like, very limiting
in terms of what it offers.
Right? Apart from just the productivity of it, it's not really offering its customers, the data analyst, with basic information
or the data engineers with these deep insights.
So that's the kind of comparisons here. And then if you go back to dimensional modeling, right, yes. So DBT would do that well. On the other hand, some people use DBT for ETL also, but then you can use it at a smaller scale
because there you will not be able to do business logic reuse or stuff like that. Right? For us, our customers are like, I want reusable subgraphs. I'm ingesting data from 10 vendors.
I want to run a set of rules on them. It's like, well, if you're writing a SQL script, there is no reuse. Right? But that is for if you're doing dimensional modeling, then that's the right solution. Right? If you don't need reuse, you don't need scale. So these are kind of the,
I guess, the broad options in tooling and what's possible with each of them and how they compare
Unknown: at at a high level. To the point of the kind of productivity gains
along those different axes in terms of writing the code directly, you get a lot more flexibility to make you productive
at the
kind of level of expressivity,
but it reduces the organizational productivity. Because as you said, you have the, you know, very detailed code, but it's not in a format that's accessible to the downstream consumers of it unless you add in all of this additional engineering effort to wire all these pieces together. And so I'm wondering if you could just talk to that kind of organizational aspect of productivity by
streamlining and kind of polishing
the tool chains that the different stakeholders and data producers and consumers are working with to make it so that they don't have
to add in all this extra effort just to be able to enable
the collaboration
aspects of these, you know, data workflows.
Unknown: This is a great point, and I'd like to cover it in 2 pieces. Right? 1 is for the data engineering themselves,
And the second is organizationally.
And that also has an organizational component. And then for the larger organization.
So first, for data engineering itself. Right? There is but do these what are the challenges of the organization? The central data platform team might have a few, you know, people who are experts in Spark, and then they're oversubscribed.
So now anybody trying to build a team of more data engineers or hiring some younger people out of school, You know how hard it is to hire data engineers.
And, you know, as they are trying to build that organization, getting the set of people who could be productive with this code is hard. So just setting up that organization to be able to deliver on its mandate is hard.
Number 2
is
when they are actually working on that. Right? There is a huge amount of churn
in data engineers. That's right. They've worked for 2 years here, 2 years there. Generally, the industry, you know, you know, when they come in, can they quickly understand your code? So the you know, when they come in, can they quickly understand your code?
So the organizational challenge there is, like, you get these new people. They can't understand the code. Or when you you have 2 people who can write spark code really well, and then you get another 3 in the team who are learning, and overall, they are not productive.
Right? And then
there is basic things within data engineering. It's like, okay. I'm going to change.
Let's say I'm writing a data pipeline as a data engineer,
and I'm going to change a column. Right? What is the downstream impact of that? Am I gonna break 2 pipelines?
But how I find out is I make the change and then somebody screams at me.
Right? Or, you know, if your pipeline breaks in production, like, some column value just comes out to be wrong, now you've got to track where it came from. It might be a chain of 10 data pipelines that it came through. Right? And now how are you going who can chase it back across all, you know, multiple
piece pipelines
written and get in different writing styles by different teams. Right? And figure out where the problem is to quickly fix it. So just saying that, fundamentally,
just within the data engineering team,
code
is a suboptimal
approach. Right? It causes a whole lot of problems, and we don't think it's a good solution in general. But then the second thing that you said, right,
what's happening a lot that we are seeing in the industry is that the data analyst. Right? So if you look at it, what is the job of data engineering?
It is to provide data analysts or business intelligence and machine learning engineers
with
data
that they understand that is clean and good format, and they understand how it was computed. So there is so for example, there is a column called internal rate of return.
Right? Some engineer data engineer was feeling a little sleepy.
They Googled for the formula for internal rate of return. 5 results came up. They picked the third 1, coded it up. That might be different from what the understanding of the business is. Right? So organizationally,
what's happening is
the line of business team or the team that has to actually deliver on analytics
often
cannot understand how a value was computed even if the data engineers
provide it. Number 2
is
often
they are
waiting on the data platform team to provide them with something kind of really holding them back. And to that, right, there is a big reaction
which you could call modern data stack,
which could be called the data platform team bypass stack.
Right? It's like data analysts are like, I'm fed up with the data platform team.
They keep wrestling with the infrastructure
and not producing what we need. Let's just bypass them, build the second set of tools. And so, organizationally,
the challenge is,
you know, the analytics and machine learning teams are not getting the data.
When they get the data, they don't understand what it is. And I was talking to 1 of the companies, somebody from a data
platform team of 1 of the top retailers.
And the top retailers is, I guess, shoes retailer, you know, top 1. And they're like, you know, we enable all our data engineers to write whatever code they want.
And then I'm like, okay. Are your data analysts
enabled to understand,
you know, the data consumed? And they're like, no. They are not enabled.
But they are your customers, and they're like, yeah. We know. And then they asked us. Right? They're like, what do you think happens with us?
And we're like, we don't know. They're like, what would you do? I said, I would ask for the source data. And then they're like, yep. The data scientists
ask for the source data back. And now that this ask for the source data back, they build the same pipeline over again because they can't trust what the data engineering team is giving them. So the BI team is doing that. And then they said, you know, if you could just build lineage
and find patterns, I'm sure you could remove 60% of our ETL pipelines because they're all computing the same thing over and over.
Because, you know, what 1 team produces, the other team cannot trust. So organizationally,
that is like the consumers are not enabled. And finally comes the governance piece. Right? You are running client. Right? Personally identifying information.
You want it to come from truth. Right? So you've written some code. It's committed to get your pipelines are running. I want to know where my personally identifying information is ending up. And the data governance people have a huge challenge. Right? Adding business
definitions, knowing what formula are there
for every value, and tracking values flowing and, you know, for their governance need. So they're usually, for months, sitting with 1 team after another, trying to understand.
Right? And, you know, we work with some of the largest financial
companies and banks. You know, they are just sitting with team after team after team for months trying to figure out how PII data is flowing and if they are, you know, meeting compliance. So I I would say that the move to code by the core data engineering team,
like, has messed up absolutely every piece of the tool chain,
every data
constituency. Right? The governance people, the data analysts,
the machine learning people, and the core data engineers and such. Right? Like, everybody's life is horrible.
So that's kind of, you know, side effect of having code.
Unknown: Touching briefly on 1 of the points you made of 1 of the organizations you spoke to saying, oh, we let our data engineers write whatever code they want, and sometimes that might even mean in whatever language they want. And I'm wondering what you see as the
kind of levels of
utility and kind of ease of use that get introduced by saying, okay. We'll let them write code, but we're going to standardize the tool chains to say, you know, you're not going to write Scala for Spark over here and Java for Spark over there and PySpark over there and then, you know, use Dask over there and write it all in Python, and then we're just going to hope that it all comes together. You're actually going to say, you're going to use PySpark everywhere because we have Spark in our infrastructure. We've invested in that. We're going to use a single language, a single interface because that way the data scientists and data analysts can understand the code more easily
versus saying, okay. We're actually going to abstract out the code entirely and go to a low code solution to say, we will let you build these prepackaged components,
but you're not actually going to
write all of the code directly. You're going to write this component, and then you're going to wire it together because there are defined contracts between the outputs of that and the inputs of this. And then
then you're just going to, you know, stitch them together in this GUI builder.
Unknown: So I think 1 is that, as you see, right, the Hadoop, for example,
has the customers who were early adopters of Hadoop, when we go there, they are in the worst shape. Right? They have big scripts, and they have hive scripts, and now they have spark scripts. Some of them are, you know, the Java application people who came in and wrote spark Java.
Lesser of that, there's some Scala and a whole bunch of people who'll only write Python.
And the data scientists were, you know, also sometimes contributing,
and that's all Python. Right? So now that you have all of these,
it is a complete mess. Right? No way to run an organization.
It's kind of
the worst
way to build a data engineering team because you are not going to get
no search, no lineage,
and even across teams, you know, 1
engineer might not understand another engineer's code. And it's also challenging to hire new people and say, okay. Now you're going to write big scripts. Right? Like, I don't want to do that. Right? That is the absolute worst in the sense that it is just
that the data engineering team just gets busy wrestling
with their own code
and cannot, you know, take their head up and actually think too much about what they have to deliver.
Now let's say somebody comes in in the team and says, here is the diktat. Everybody should use spy spark.
That is great. Everybody can use spy Spark. Now they've used it. So you have this uniform code base, but the question is,
if I'm a data analyst who's using SQL,
well, what I at least used to have. Right? So let's say I was using in the last generation Ab initio. And by top of Ab initio, we have great respect for that product. They did a lot of the things really well, and it's not very well known. But if I came back as a data analyst, I could get a single formula,
you know, which would show going across pipelines
and pulling the business logic from each 1 of them and saying,
this is how it was computed. Right? And in a language that I can understand, not cold. Right? It might have some SQL function type things. It's just the basic set of transforms of how it was computed based on the absolute or hidden values. Because they are trying to understand what does this value represent.
So
in terms of that, right, PySpark is still not giving the data analyst that. Right? It's like, yes. So now every data analyst who wants to understand the code can go
get PySpark or did but then do they actually write Python?
Right? And then let's say I want to search my datasets,
you know, and understand how they came about like lineage. Right? Do you have that with PySpark? You don't, really. Most people don't. I mean well, let me say,
even if you look at the products built inside LinkedIn,
you know, the
data hub or the metadata products built inside Uber, inside Lyft, Amundsen, and this.
Not 1 has column level lineage. Right? That means that when I pick up a column, I don't understand how it was computed.
It does solve the problem within the data engineering team that, you know, new people can come up to speed quickly because the code base is a little more uniform.
You can move people from 1 place to another. If you have good coding practices, they can also understand each other's code much quicker. So the data engineering org, like leveling up, is working okay.
But then the consumers,
right, the metadata,
the governance people, the analysts
downstream, they are still not served. And then there's also some productivity issues. So they so they are not going to get data as fast as they want. And finally comes to the low code approach. At least the low code approach that we have, right, is
I think what we do uniquely is that you can use visual drag and drop, and we are generating really high quality, readable, performant code on git side by side. Right? So that means you're still producing the same assets, but you're producing them out of standardized Lego blocks.
Right? So in this approach, because it's packaged in a uniform way, we can provide search, we can provide lineage.
And it's very easy for somebody who knew to come in and, you know, build pipelines or understand somebody's pipelines.
And as well as there's nothing really you that you could
do in code that you can't do in our approach. Right? Because we let users add their own visual components. Right? So you can say, here is my spark code. This is my high performance connector that I want to write, you know, maybe to my own rest service or to something else. And I package it up. And here you go. And now everybody's using that same usable
asset. Right? And so in that sense, right,
there is nothing that you could do in code that you
cannot do, but now the same thing is available to many more users. Yeah. So ad hoc code, absolutely bad for the data engineering and the consumers.
I would say standardized code, data engineering teams, a lot of their problems solved, but their customers and less and those
still are not served. But also I would say the data
engineering team can only
hire people who know how to write Python and PySpark and scale. So there is a little bit of limit there. But at least their org is running
functionally well. And then low code is like any data user can build pipelines,
and the quality of has assets they are building is
superior than hand coding
is what we find practically.
And there's nothing that you cannot do that you could do with the code and, you know, so there's no loss of flexibility or power either.
So usually, when we look at it, you know and as we internally
with our first customers are talking. Right? It's like usually when you choose a over b, there are trade offs.
And we are at least coming to very high conviction. There are no trade offs. It's like low code is just in every way better
than coding,
and our customers are same once they adopt it. Digging more into
Unknown: prophecy,
you know, you were on the show. I believe it was about a year ago. We dug pretty deep into the product as it stood at the time, talked through some of the design elements there. I'm wondering if you can give us an overview of some of the ways that the product and the platform has evolved since the last time we spoke and some of the
learnings that you've had as more people have been onboarded onto the system, more people have been stretching it in different directions, and some of the ways that you have
increased the kind of utility
and kind of ease of use aspect to make sure that people are able to be productive
Unknown: while still having the necessary flexibility to do the kind of bespoke custom logic that they need in the places that they need it. So just, you know, as a reminder for those who have not seen the last show, so, basically, you could say there is 4 kind of what we are say are pillars. Right? 1 is there is low code development.
That means a lot more users. You could say 10 x more users are enabled, and they are 10 x more productive because you can quickly drag and drop, attach these, hit the play button, see data after every step. And side by side, of course, you're creating code. Right? The second thing is what are you creating? You're creating code standardized,
readable,
well structured on Git
using the best
practices that you want to use for your mission critical
applications or data pipelines. Right? So you are using git test CICD. So best of those practices.
And then we are giving the complete product where you are not wrestling with the infrastructure. Right? So we are saying, okay. You'll get development. You'll get metadata, lineage, scheduling, you know, everything
that you would want, you know, end to end. And finally is extensibility. I think extensibility is what we've spent most of our last year of engineering.
And it turned out to be a much, much, much harder problem than we had anticipated. So we quickly had a prototype out. We started talking a little bit about it last year, but that has been
a very hard technology to build. It has taken way longer than we've thought, and it is something that is extremely
important to all the customers.
We found so many use cases for it that we were not anticipating. Now once we talk about that, right, every customer
conversation
starts centering around extensibility,
and I can give some examples. Right? So so now within this. Right? So going
back to the extensibility piece. Right? So now what we did in the first pass was you could think of it as v 1 of our product,
where you could go from visual to code.
Right? You are doing visual drag and drop. It generates some code, and it's quite cleaned up code. Very readable. Very well structured. And then you could change the code, go back to visual. So we did the code generation. We did the parsing.
And that was working fine. But then what we said is,
you know, that we are going to add
extensibility.
That means we threw away that entire code base and said even our standard components are going to be generated from a spec.
And that spec
says, here is my standard spot. Here is a spark function.
That is my standard
operation that I want to do. These few fields come from the user.
Right? And you can't really do this in Python, for example. Right? Because 80% of the meat of that function is coming as an argument.
So, you know, all you would have is arguments and very little business logic if you did that in Python. That's why people create these XML JSON frameworks.
So now you have a function to represent your business logic. You have a function which 20, 30 lines which kind of gives you a quick layout of the UI
and how it's connected to those values that are being plugged in by the user.
Now from that,
we are generating these visual components.
But then these visual components
you know, now in this generated visual components, there is
usability has to be very, very, very high. And that's 1 of the learning. Right? There's a lot of attention to detail, but I'll go into some of the things. Now now you're generating this UI, but then when you connect to it and you open the UI for this particular visual element, you'll see the schema coming in. Right? And
now then, you know, the spec says, hey. Here is something that is going to get a spark function or an expression. You start typing. There is an autocomplete
that knows this is the spark function. And you start typing an expression in the middle. You start typing something. It says, oh, it looks like you're trying to use this column that is available at this point in time.
So that kind of autocomplete,
you know, and then that where it understands
the
kind of piece of Spark you are using. It understands the schema at this point, and it's kind of helping you along.
And then as soon as you close it, even before you run it, you know, we can tell what the output schema is going to be. Right? So there is static analysis that's quickly going to say, okay. These are your transforms, and this is your input, then this is the output schema. All of that, but all of that is generated.
Then if you save it, you get code. The code is generated.
And then if you change the code, of course, the parsing, we are going to add that we haven't gotten to it. But
going to this layer
that is generated
with
this really slick attention to detail. Right? This autocomplete,
the schema input schema, output schema evaluation,
autocomplete. And then within that, we also implemented some other things,
which kind of make the life of the user easier because
save button was a pain.
So we've moved on to autosave. Right? And then
every time users change something, they want to just hit play and see how data looks like that. So we have really, really fast but we call it interim. I I guess that's our internal term. Basically, after every transform, I want to see a sample of the data.
So you hit play and
we when we run the program, we instrument our spark logical plan. You know, we cache where we need to. But the main thing is if you hit play in 3 seconds, you're going to get the answer back. Right? So you you hit play. Usually, it's a second. Can be up to 3 seconds. And now you can see the new data after your transform. So very, very quick iteration loop. Right?
Then we had to do some things for quick productionization, like a single
click scheduling.
People are like, okay. I just wanna schedule this pipeline. And it's like, okay. Here is a button on the main
development. And I meant, you know and we've also focused a lot on UI design. Right? It's working with actually, we we found a really, really good designer who's, like, a guy from Brazil sitting out in Bali.
And my mom sometimes I'm jealous of, you know, his environment. He's in such a nice place. So, you know, working with him to get absolutely clean visual elements, like, it's single focus. Right? We have 16 inch MacBook Pros and have giant monitors. Some of our customers are using window machines with 13 inch
screens. So it's like, okay. There is a lot of clutter. Right? If you use, let's say, Microsoft development product or something. Right? These days for something like data pipelines, you've got a thing in the middle where you have, like, the visual elements, and then you have a screen on the right that's showing you some properties, a screen on the bottom with properties, a screen on the left, and, like, business logic has no space.
So we've kind of gone in and said, no. No. No. It's, like,
1 item at a time. We clean the entire screen.
Very simple visual canvas. When you click a particular, what we call visual gems,
you know, you get something that comes on screen, covers 90% of it, and it's like 1 item at a time. Very simple. Very clean. So
a lot of these things have gone a long way, and and maybe I'm
going too long on this. But I think the attention to detail in design
and really caring about
your data users and making their life easy, you know, that's something where we were treated a lot on. Stepping back to Yancea, it's like the 2 main things. Right? It's like 1 is this extensibility
that has been very challenging to build and is used a lot. And second is
the
design element of it. I think we focused on making that
significantly better, and that has made people's lives better. And maybe, if I may, I think it might be good to give some examples of extensibility.
Right? We have 1 customer who's coming and saying, okay. I want to do ingestion,
and I have to ingest these
50 tables, 100 tables from this data.
Okay. And now for many of them, I want to do SCD to merge.
And then those users tried in Spark to write the code for SCD to merge
multiple times, and their code wasn't all that. They found it challenging. Right? I don't know if your user are familiar with a CD to merge. It's like you have on the operational side, let's say, I'm in there's an ecommerce site, and I have an address. And then I change it. I change it. I change it. But on the analytical side, I want to keep a history. I had this address from this date to this date, this address from this date to this date. So when you take that operational data and actually put it in the data warehouse, you've got to look at the existing
values of those. Then based on that, modify the existing values, modify the new values. You know, the code for that is complex.
And they're like, we have to do that. Can you give us a standard transform for that? We are like, here is your s c 2 merge. Then they're like, you know what? I want to just loop over all the elements,
over all my 50 tables, and
do the ingestion. We're like, okay. Here is a table
of configuration,
and and that's something we are building. It's not released in the product, but we have a prototype working for them. It's like, okay, here is you have a loop subgraph where you can loop over all the tables from the input
and ingest each 1 of them 1 by 1,
and apply some basic cleaning rules and the right kind of merge that they want for each table. But for those, we are creating these, you know, Lego blocks. But all they have to do is connect them. Right? Then there is
data validation and quality rules. So now we have this 1 vendor who's getting data from 20 sources, and they've said, I want this set of rules to apply. You could call them data validation rules. You could call them quality rules. But then, you know, these rules have got to be reusable in any pipeline I can say. So let's say I'm an insurance person. I'm getting this from 20 different vendors. I change the schema a little bit to normalize it, and then I want to apply the same set of rules. Can I have the set of rules as a reusable thing that I can apply to any data of this particular kind?
So we've had that, right, as a reusable component or,
de identify. That's a very interesting use case. We've got some
financial companies.
They've got new data coming in, And they are like, I want to de identify it. Maybe they'll move the dates by a little bit. They'll do a certain series of transforms that they do to remove identification.
You know? And then in all of these cases, right, why I'm giving so many cases is the number of cases we
are finding is just numerous.
And in each of these cases,
they want to say this is my standard way
to de identify datasets. This is my standard way to write data quality rules. This is my standard way of doing ingestion. And they want to create
these
visual gems, visual components that are, you know, unique to them,
and then they just reuse them.
You know, they create it once. Everybody can reuse them. Deidentify.
Go to the menu. Put the de identify block there, and you're done.
Right? And if you were using SQL, you could never do that. Right?
I think
building extensibility has been challenging, and we are finding tremendous use for it. And then, of course, I talked about design.
Unknown: So now your modern data stack is set up. How is everyone going to find the data they need and understand it? Select Star is a data discovery platform that automatically analyzes and documents your data.
For every table in select star, you can find out where the data originated, which dashboards are built on top of it, who's using it in the company, and how they're using it, all the way down to the SQL queries.
Best of all, it's simple to set up and easy for both engineering and operations teams to use.
With SelectStar's data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets.
Try it out for free and double the length of your free trial at dataengineeringpodcast.com/selectstar.
You'll also get a swag package when you continue on a paid plan.
1 of the things that comes up a lot whenever the subject of low code or no code platforms
is discussed, particularly when you're talking to a group of engineers, is
you'll pry my code out of my cold dead hands. You know? I'm curious how you approach some of that
resistance to,
you know, what some engineers will see as kind of dumbing down their capabilities
or, you know, saying, oh, well, I can do all of that in code. You know, why would I use this UI builder? Because I'm not getting the expressivity that I'm used to and just sort of the some of the answer to that kind of knee jerk reaction to, but that's not the right way to do things.
Unknown: Here, honestly, it's we are finding
some challenge.
Right? And the different kind of scenarios we encounter, and they're very interesting. So 1 team says, hey.
We are a group of 5 Scala developers Spark Scholar developers.
Our high salaries depend on the fact that we know how to write Spark Scholar.
We look at your product, and we know we can do that much easily and we produce the same results, but it will reduce my salary.
So no. Thank you.
Right? So there is those teams. And, you know, there we just give up.
Right? And say, okay. You're not the right team to talk to. There's no convincing them. Right? It's it's just not in their interest. Now the other thing we are finding is there is another mix where
you have a few
programming
data engineers, and they want to enable a whole new set of people. Right? So what they're saying is, I am the central data platform team, or I have these few programming data engineers, and
they are kind of not able
to meet all the requirements being thrown at them. And they're like, I want to enable the organization. I think through them, the sellers
so, basically, instead of them building data pipelines for all the users,
we talk to them and say, you can build these standard Lego blocks. You write your spark code and said, this is how you do a. This is how you do b. This is how you do c. And now you just go into defining
standards
that people use to build their pipelines instead of building the pipelines for them. It's kind of an up scaling for them. Right? The leverage they have and the value they're adding to the organization goes up quite a lot. And I think that gets positive reception. They're like, yeah, it's all about enabling these people. And then what happens is all the
simpler pipelines get taken away by other users.
And then this team can focus on,
you know, cost and performance
and, you know, handling the difficult cases. Right? And sometimes they will go in and write scripts. Right? And write some special code for things that need to be done. Sometimes it's reusable Lego blocks. Sometimes they might have to do something unique. But now all of their time is going in
in challenging areas, in defining standards, in solving the hard problems. So I think those teams are receptive. I think sometimes the managers of those teams, their counterparts and their business teams are not happy with them, and they're trying to find a way.
So those are happy. And let me add 1 more category to that. There's sometimes, you know, the first team has brought in Spark and Databricks often,
and that they are productive. And now they're like, oh, we've set up this great platform. Now we want everybody to use it. And then everybody else is like, I don't know how to write Spark code. And so now they're stuck with this great thing that the system they've put together,
and they can't get users. Right? So we get some people, like, we have a company
that is into the housing
Internet companies. Right? And they are like that. They're like, you know, we want to enable everybody on this now. So so that category is quite
receptive in the data platform team, you know, who are thinking like that. And then there is the what you could think of, like, kind of representing the line of business group. They just want to deliver data, and they don't really care. Right? It's like their organization said they're going to use Spark.
Some user are using it right now. They need to get their work done.
Some of their source data is in Spark and in their data lake or in Databricks.
And they just have to do get a bunch of work done. They are primarily focused on delivering analytics.
So they they are the 1 who will
pull. Right? They will say, hey, we want low code.
So we kind of have these constituencies.
The people who are like, you know, we are going to code everything, there is no
debating. It's
but I don't know what the future of that is.
There is no hope for them. Right? We give up. But the other thing is
at some but the modern data stack, I feel, is kind of bypassing them. Right? It's like if they're not, you know right now, they can say, okay. You know, I am the only 1 who
knows these data pipelines. Some of them in Hive, some in Spark, some in
Big. And nobody else does, so I have job security, and I am powerful.
Right? I mean, that is
work for a while, you know, then people will run out patience and they'll get in trouble. So I think there's going to be,
you know, after maybe, you know, if we meet it, you know, talk again in 2 years, 3 years, there might be a consensus in the industry that script was a bad idea.
Unknown: There's this question of the sort of ease of utility that's offered by low code tools and by the various commercial offerings that are out there. And you made an interesting point about the modern data stack being this
kind of disaggregation
of the different individual components, but with a fairly straightforward
integration play to be able to say everything is oriented around the data warehouse or the data lake. Everything plugs in together nicely. You just need to kind of pick and choose what you want. So there's not as much kind of heavyweight integration,
custom code that's necessary to wire everything together.
I'm just wondering what you see as
the next iteration and the next generation
of data engineering where
this level of kind of ease of use and accessibility
beyond just the very hardcore engineering teams is table stakes to be able to actually say, this is a viable tool that I want to release, whether it be in the, you know, open source ecosystem or proprietary and just the level of interoperation and collaboration
that is at play to make sure that, you know, just because somebody says, okay. I'm going to use prophecy
to make it easier to build these composable units together to empower my entire organization,
you're not left out in the cold if you say, oh, actually, I now wanna go and start integrating with this other piece that's brand new and experimental, but has all these, you know, interesting and useful capabilities.
Unknown: This is a very, very good point. I think
as the new modern data stack evolves. Right? And that's 1 of the powers of code. Right? And if you say I have a visual tool only,
then you're kind of stuck in your silo.
It's hard to integrate with the ecosystem. Right now, what we are saying is you have this visual layer productivity, but then you have all the power of code. That means as new technologies come,
you can integrate with them. Right? Even if you use a different metadata system. Right? You can write a visual component that just connects to that metadata systems and registers,
say, a feature. You can say this is my feature store and just register this column as a feature or things like that. Right? So in this sense, right, the
ease of use
must be there because
the current way people are going,
where it takes just so much effort
to get basics done. That is just not sustainable.
So that part we feel is at least going to slowly just die off because you're not really getting anything for that effort. Now because a lot of data platform teams are integrating
all kinds of different tools,
And I've come from the Hadoop Hive. I've kind of, you know, had a stint there. There were 50 tools, and then, you know, they all went away. And customers are like,
Databricks and Snowflake. I want just 1 thing. Do the job for me. I don't want to spend my life integrating stuff.
So there is that. Right? That there is a move in the engine space towards a single complete solution.
And then the question is that will this come to the data tooling layer as well?
And in the data tooling layer, right, today, you know, with these things
for even the data engines and data tools. Right? And data tools are getting better. But there is a lot of integration effort.
Right? I'm going to use all these different data tools, and I'm going
to stitch them together. It's not as bad as it used to be because if you were using Hadoop, you needed to know the difference between map join and shuffle join. Right? And it's like, why do I need to know that? It's like, because that's a horrible optimizer underneath. Right? Those engineers haven't built a good system.
Now if you were going to interview
somebody for
a Snowflake user, like, would you ask them the difference between map join and shuffle join? No. It's like Snowflake's
job. They've got a query optimizer. Right?
So the table stakes
have changed in the data space. Right? Where now
if you build another data platform where somebody has to come in and say what the difference between map join and shuffle join is because you don't have a good query optimizer, you're dead in the water.
Right? That that is table stakes. Now
going back, right, if you're asking
if doing the simple things is hard,
I have to write the script. I have to create a jar. I have to do, let's say, spark submit and then write to a table, go read the table and figure out if it was good. Right? So now it's like,
for the productivity
of the overall organization,
just the simple things have to be simple. After the low code products
like prophecy
and others become widespread,
you know, it's no longer going to be sufficient to
say that, you know, that, okay,
I'd only provide this 1 thing.
Right? Because, okay, you provide me with some simpler development. Do I get lineage? Do I get search? Do my data analyst know this? Do I have governance? Right? Once you have a product that can do 10 things, just like Databricks does 10 things. Right? And then you if you have all of those, like, can I go and open a spot company today or some other processing engine? And they're going to be like, do you have the demo? Just like I need that. Right?
So the tables take
as 1 or 2 complete products take shape,
then complete tooling products take shape. I think there is going to be
the usability and completeness are going to become 2 axis
on which, you know, the table stakes are going to be that 90% of your job is very simple.
And then the question is that other remaining 10%, the hard use cases. Right? I want to use a new experimental framework. I want to be able to go down and optimize my stuff. And then that power has to be there for the last 10%.
And I think having 1 or the other is not going to be sufficient. Today, coding is giving you the power, but not the productivity.
Right? And the old detail tools gave you the productivity, but not the power. Right? And we are saying we're going to give both. But if we do
provide both in the market, we look at it and we're like, that's going to become table stakes. So it will be interesting to see how the,
data stack kind
of evolves, but it will be
challenging. Right? So for example, let's take the example of Metilina. You know? They have built a very easy to use ELT product. Right? It's visual drag and drop. It generates SQL.
Right?
That's great. If I want to do something outside of SQL, how do I do it? Now it's like, okay. So that then you run into a problem. Right? I mean, they solve their use case. Right? The simpler use case, they'll be able to solve. But then it's like, okay. You know, once visual drag and drop becomes stable stake and everybody has that,
now how do they compete? Right? And I think that's where, you know, having power and code under underneath at least for us is something, you know, that we think will become really great.
Unknown: Absolutely.
And in terms of the applications of what you're building at prophecy and what you've seen people using it for and some of the ways that it has accelerated their productivity and time to value. What are some of the most interesting or innovative or unexpected ways that you've seen it done?
Unknown: Yeah. So I think what we are seeing is some of these are business use cases. Right? So we've had some customers who were like, I don't have a data engineering team
that can write a lot of spark code. They had even outsourced
to, you know, some
spark developer
consulting firms, and they're like, they're not getting enough turnaround. Right? So 1 use case we are seeing is the insourcing. Right? We have a bunch of customers. They're like they're like, okay. I'm going to take a new college grad who's built, you know, knows, done some SQL and a few other things, 2 years of experience, doesn't know much about Spark. I'm gonna put that person on prophecy. Can they build a pipeline?
How long do they take? Right? And as they build pipeline very quickly, we've seen then them
bringing,
you know, in housing
their entire data engineering pipeline. So I think we've seen
that. And for those teams, you know, they are asking us to write. Today, I think maybe later we'll end up with the whole package that is publicly available.
Transformations that we don't have. But, you know, today, they are coming and saying, hey. I want this. I want this. I want this. All the common things they want to do, they are doing that.
And, you know, the other thing is
I don't know if it's surprising, but it's
we've built and it's also in private beta. So we we have a prophecy 2.0 release coming up in
early June. And we already have some customers on it. And then, you know, going into the Spark summit or the Data plus AI summit as it is called now.
But, you know, in that,
we have a lot of users using,
like, custom blocks for doing, I guess, all kinds of things. So that's 1 thing. The other thing is
subgraphs seems to have become really important.
People want these reusable subgraphs, and they've pushed us for it. So we built that. Then they're like,
I want to get
job metrics by default and quality metrics by default, and we've built that. But then the other thing is, like, there are really people are liking the Gitflow.
So that's another thing where and they've asked us for a visual. Like, we were not sure how many people would take to that.
Right? And they've come back and said, hey. We've got to simplify it.
But we've got visual developers
who are using branches and git
and all of that. Now they're asking us, hey, can 2 of us be developing in a branch, and can I see the cursor of the other person?
And can they come and help write
a, you know, a transform for me the way I can do it in Figma?
So we are seeing where they're moving to is, like, you know, even the simple visual developers are like, oh, I yeah. This git branch thing is great, and I want real time collaboration. And we don't have that. But yet but we are seeing that. That's interesting. The other thing that we are getting right and it's maybe not again. It might be more of an industry wide thing. But we're seeing a second paradigm, which is
some of the users who are more on the data analyst side don't want like, if you want to be focused around tables
or that is kind of, I guess, a table centric rather than a pipeline centric view. Because, like, if you go to data engineers, you know, they have 7 sources and 3
targets, and all of that is for efficiency.
But then if you look at other products, right, where you can say I have a pipeline per dataset,
which is very focused on, okay, this is how I produce this dataset based on other datasets.
That's kind of a simplified worldview that we are seeing and being asked for. We don't have that yet. But that we do see that DBT has that. Databricks' Delta Live has that model
where you have a date 1 dataset
coupled with, you know, 1 set of transform just to compute that dataset based on other datasets and kind of, you know, turtles all the way down.
And that's interesting because it's not focused so much around efficiency, but around understanding.
And if it can be efficient enough, maybe that will become important. So we are seeing some new patterns, I think. But I think some of the common cases with prophecy are primarily
I think they are just using us. I guess, the other thing I could say that I was surprised is, like, some of us are using us in
getting analytics users onto this
who are focused on like, we have a company that's like, I want to get all the trading floor people to use this because they are stuck behind data platform team. And those are very different skill set. But we are finding
interesting usages, I guess, of different kind of profiles of users that people are
trying out
of how they can be. You know, how can they do that? So interesting things. I think the organization of teams is interesting. You know, the different patterns we are seeing,
Unknown: you know, pipeline centric, model centric. Yeah. So seeing some interesting patterns. And, of course, like I said, the other pattern is, like, can you give me a visual component for, you know, this complex thing I wanna do and just kind of having that? I think those are the things that we've, you know, seen recently the most of. Yeah. The note that you just made about the sort of pipeline centric versus model centric modes of collaboration is definitely interesting and maybe something that we can explore at another time of, you know, how do people break down the boundaries and the areas of responsibility at the organizational level to be able to say, you know, we're going to collaborate,
you know, from data engineering through to data analysts, through to business users along this single pipeline to be able to say, I wanna go from, you know, this application source all the way through to this business intelligence report versus
data engineering who's responsible for all of these sort of source data collection and curation, and then data analysts do all of the modeling, and then the business users do all of the sort sort of utility of that. Definitely an interesting sort of variation of paradigms paradigms based on sort of what organizational capacity they have.
Unknown: Yeah. And just a quick note on that, it just seems like there is some amount of movement in the industry towards these unified data teams
rather than the vertical silo, you know,
rather than broken by vertical silo. And that's that's another interesting trade off that we are thinking of is, like, should we view ourselves?
Right now, they're saying, okay. Data engineering, whoever wants to do it. But then should we self thinking that do we grow toward the direction of saying, here is a unified data team. That seems to be the models people are slowly moving toward. And do we say that, okay, we're gonna cover the whole thing for them. There's some interesting design points. You know, we have to see how the market evolves, but definitely some movement in that direction. Absolutely.
Unknown: In terms of your experience of building
the prophecy platform and growing the business and now going through your series a and starting to scale the organization more, I'm wondering what are some of the most interesting or unexpected or challenging lessons that you've learned in that journey?
Unknown: Yes. So I think we started in a direction that was
hard.
I would say that looking back, we said that let's go to some of the largest enterprises
and look at some of the most complex use cases, and we started there.
So the bad of it is it took us quite a while
to show customer traction. Right? So quite a while, we were on seed round. Right? We had
seed 1, seed 2, seed 3. I mean, from the same investor. Right? It was you could just say seed broken down into 3. It took us a lot of time, but then
I think things like extensibility,
things like huge amount of business logic reuse, being able to use
business rules and
reusable pipelines. And a lot of those stuff that came into the product was because we started with the more complex use cases.
And the focus on getting 100% open source code, CICD. So we learned some of the best practices from them, and I think that learning was
was very valuable. And then, you know, we've had to scale the engineering arc
and continuing to find the top level talent without having a brand name. It's been challenging. I think we've not had good luck in Bay Area. It's just
the price to quality ratio is not as good. A lot of the people are locked into large organizations.
That is maybe more outside the technical domain. But, you know, the US has got to fix,
especially in Bay Area, like, the housing cost and immigration. Right?
It's like those have got to fig get fixed because otherwise, there's no way for
us to hire people here. The cost of living is so high and the talent is so little. Right? It's like so there is that challenge. Right? So we had to continue to grow the team, but then
we did have
some communication
cost
growth. And we were very, very
deliberate in saying we have to go fast, fast, fast
to build the product. So we kind of have gone with this model of very high quality engineers, very
lesser communication cost, and kind of built a model like that. So I I think that has worked really well for us. I think the series
a fundraising was quite challenging.
So first investors
would say, oh,
in the cloud, everybody writes code
because they're writing it today. And therefore,
that's what everybody will do to the end of time.
And then they're like, no. No. No. They're going to do drag and drop there, but nobody's doing it.
Like but the product doesn't exist that lets them do it. Right? So that was quite a challenge, and we had a whole bunch
of VCs turn us down for that. And I was actually surprised. I think it was quite because a lot of the big
brand names. Right? Because
who are supposed to understand stuff
didn't really.
And I was just surprised. Right? We got money from
Insight Venture Partners. Right? They they are, of course,
a giant form. I think they're investing out of a $19, 000, 000, 000 fund. Right? They just operate at at a different level.
But then,
opposite to what we expected, they were actually technically more competent. Like, we had George Matthews come in, Lonnie
come in. Like, they
understood the technology,
the market
way better than the Bay Area VCs who are supposed to be all this. Like, I had literally top tier firms. You know, you know the names. I won't call them. Come and say,
oh, you are building this local thing.
Oh, you are competitive with airflow.
So we can't invest in you. And I'm like, where is airflow? Where is our product? Right? I would say that the general
competence level of
you know, because seed round, you go for a lot of small investors. Right? That's what we did. And then series a is when you go to these top brand name forms. And, you know, the it was very hard to convince the top brand name forms of anything other than what's already there,
and they didn't understand the ecosystem well.
I would say that. And I was surprised, like, our investor, George Matthew, he understood everything we said, like, in first 5 to 10 minutes. And, wow, this is amazing. Right?
So in that, you know, I wonder how VC would get disrupted
further by these, you know, just more competence coming into the market. That was a very, very interesting,
learning thing for us. The go to market side, scaling the product now, we've started people using
our public SaaS version.
So we are in Databricks Connect. You can just click on it, and we just open in another tab connected. Or you can go to app.prophecy.com
and just start using the product. Right? And then that we've had to build the muscle operational muscle of being able to keep the service up at all times with high quality. And now we've also got to go for all the, you know, compliances and stuff. So that's another journey. But I think those have been interesting. I think the the other thing is right after series a. Right? We got a big series a, 25, 000, 000
upper end of the market.
And now it's, you know, the planning of how to
organize
and how fast to go.
You know, that's an interesting challenge. There's just so many decision points and so many things that you could get wrong. So it's been, like, do I hire a VP of marketing and a VP of sales? Or do we build them bottom up? Do we build them top down? Like, can I hire a VP of engineering? Will the VP of engineering be able to sell? Because we can talk to some senior
executives from,
you know, Uber and other companies for VP of engineering, which we still don't have
1. It's like,
they've always had an organization
feeding candidates to them and a big brand name. It's like, will they be able to hire good quality candidates without that? I think the interaction with venture capital was interesting. The
building and growing an organization, being able to plan is has been good. But then, you know,
I think the other thing is some of the things I think we've done well is
continue to narrow our focus.
I think right now, everybody owns 1 thing in our organization.
Something, you know, that was reaffirmed recently by Slootman's book,
you know, who the CEO of Snowflake.
Right? It's like, we've kind of taken, like, very fast speed and 1 priority for everybody
to do that. And then I think it's been very, very
interesting so far. And a lot of new muscles to build very quickly. I guess all startups go through that. Right? It's because it's, like, engineering,
but now it's, like, shipping and production. And now it is the data engineers in the field onboarding customers
And, you know, the marketing getting your word out, and it's like you have to build all this machinery very, very quickly.
And, you know, every new area
is something you don't know. Right? And some decisions are hard. Right? So it's it's been a mix of what you do. Sometimes you just go hire somebody.
Like, we got Sushant. Right? Sushant has
been in in the industry for 26 years. He's a VP of marketing. Right? He he grew up in LA
and then went to Stanford and, you know, we've been doing marketing for a while and, you know, a strong product marketer. Sometimes it's you are doing that. Right? You are saying we definitely need that. And then sometimes it's like, do I figure out how to hire a UI person and a UX person and a product manager? It's like, no. We're not gonna do that. So I just picked up Figma, worked got 1 designer, and both of us did a full UI
overhaul because that is quicker than figuring out how to hire for these 4 different roles. It's been quite an interesting and exciting journey. I think earlier, we were just kind of always running out of money and in survival mode.
And now after series a is,
you know, things have become a lot more fun.
So definitely
Unknown: a very interesting experience. You know? I recommend as many people as possible to go through this. It's been a joy. So are there any other aspects of the work that you're doing at prophecy or this overall question of developer expressiveness
and performance optimization
versus developer experience and accessibility
to the data workflows at the organizational level that you'd like to cover before we close out the show?
Unknown: I think I would like to just summarize.
And this is kind of a point of view now that we strongly believe. I mentioned it in passing earlier. But we are just looking at it and saying,
you know, that local data engineering. Right? Today, we are talking there, and I can be pretty certain that, you know, 3 years from now,
everybody will buy it.
You know? And these things change. Right? Earlier before Snowflake, everybody was using Hadoop. So I think what we believe is low code
is just 10 x better than visual ATL twos and 10 x better than coding. Right? Because it has best of both worlds.
It gives you, you know, so many more users quickly building pipelines,
reusing business logic. Right? It's standardization.
New person can come in and learn very quickly. You get
metadata and search and lineage, a complete product. Right? You get all that right. But at the same time, you have your code on get
infinite
extensibility.
Right? Get CICD best practices for robustness. And then you put that combined approach that we've developed against
either code or visual ETL. And you can just see that it is so much superior. So I think we kind of stumbled into it piece by piece solving customer problems.
But as we now look back and see, we just it's like, okay, you know, this is just a better approach,
And 1 we are very, very excited about.
I think we'll,
would love to connect back after a year or 2. And, you know, hopefully,
that will a lot more of this approach in the industry.
Unknown: Well, for anyone who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question,
I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
Unknown: I think the tooling and technology for data management today,
I would say
the management layer or the metadata layer or the information layer is just not there.
Right? There's a few startups doing it. Not quite there. If you look at the cloud, right, there's technical information. But do I have business definitions? Are they flowing? So in the data tooling,
the information layer. Right?
Let me give an example. Right? Let's say
I say, Tobias,
I want your team
to focus on data quality. I don't think we're doing very well on data quality.
Let's focus the next 2 months on data quality.
It's like,
do we even know what we're going to fix?
Right? And after you fix something, can you come back and report
what you fixed? Right? It's like what you
are missing is a place where you can
understand your data engineering,
your
performance metrics,
you know, your quality metrics, and then report on it all.
Right? I think that's a basic layer where I can say, okay. We invested
for 2 months in data quality.
We have 80% coverage on the datasets about this data quality. Within that, we found these were at this data quality, and we've improved these things to this data quality. I've got to be able to report. Right? So that business level matter to rate Hive metastore is there.
And that's just technical metadata for, say, spark or something
and for glue. But then the question is, is there
no cloud has a good metadata product that allows me to merge this technical metadata with metadata
up from various things. Right? Execution,
context, business context, cost context.
You know, put all of it together and be able to report on it and take actions on it. So I think that whole layer is just missing from the stack.
You know, we haven't gotten to it, but it just seems like a big missing piece in the stack.
Unknown: Absolutely. Well, thank you very much for taking the time today to join me and share your experiences
and insights from building the prophecy product. It's definitely a very interesting tool and a valuable approach,
and I appreciate the time and energy that you and your team are putting into that. And I hope you enjoy the rest of your day. Thank you so much, Tobias. A delight as always to talk to you.
Listening. Don't forget to check out our other show, podcast.init@pythonpodcast.com
to learn about the Python language, its community, and the innovative ways it is being used.
And visit the site of data engineering podcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com
with your story. And to help other people find the show, please leave a review on Itunes and tell your friends and coworkers.