Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
Data migrations are brutal. They drag on for months, sometimes years, burning through resources and crushing team morale.
DataFold's AI powered migration agent changes all that.
Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches.
And they're so confident in their solution, they'll actually guarantee your timeline in writing.
Ready to turn your year long migration into weeks? Visitdataengineeringpodcast.com/datafolds
today for the details.
Your host is Tobias Macy, and today, I'm joined again by Max Beauchemin to talk about the challenges of reusability in data pipelines. So, Max, for anybody who hasn't heard your numerous past experiences, can you just give a quick refresher about who you are and how you got into data? For sure. Well, first, thank you for for having me on the show again. So I think it's been, like, 3 or 4 appearances, and it's been a little while since we connected. So excited to be here again.
My quick self intro is so I've been doing data engineering
and business intelligence for for, you know, best of 2 decades now.
And, and then I got really involved in open source while I was at Airbnb. I started Apache Airflow and Apache Superset,
which became extremely popular in both cases. So for those
not familiar with with the tools or Airflow is is an orchestrator, so it's really about,
training up some some Python workloads and and running some data pipelines defined as code. And Apache Superset is an open source challenger in the the Tableau, the business intelligence space. So we're competing with Tableau and Looker and Power BI,
with the awesome power of open source. So that's what we do. If you haven't checked out Apache Superset in a while
or or you don't know about it, so I urge people to to go and check it out. The the news is that open source, this is Adeljan, is super, super competitive. Now there's no reason why you should be using proprietary software
in that area. So if you're fed up with, with Look or Tableau
or whatever it is that you use, you wanna you wanna move over to to open source. There's some really, really good set of tools out there. So check out Apache Superset. You can check it out on Presets. Preset is a commercial offering. I'm the CEO and founder of Preset, which is commercial open source on top of Apache Superset. So feel free to use the the the open source version.
If you just wanna try it, take it for a test drive. You can try superset on preset.
Io. So that's the end of the commercial portion of this,
this podcast. Let's jump into it.
And so
the conversation today is around the premise that data pipelines
are
stuck in this phase of write everything twice, where everywhere you go, you have to keep doing the same piece of work. You can't just copy and paste from one business to another.
And I'm wondering if you could just start by sharing your current thesis on the
opportunities
for code reuse and the shortcomings on either the tooling, ecosystem, or education that limit that code and component reusability
across organizations?
Right. So I've been thinking about this for, you know, 10 years, maybe maybe longer. The question, you know, since I I open sourced Airflow, I really thought people are gonna go and use
Airflow or build a higher level construct on top of Airflow
to share
code and logic and pipeline across
organization. And we really haven't seen that for
a whole set of reasons that are intricate and complicated and that we're gonna talk about today. Something that can be good for the listeners would be to check out the blog post I wrote on the topic. So the blog post is gonna be more structured probably than this conversation, but this conversation is gonna be much more interesting in in exploring, you know, the
the the intricacies
of things and getting on the next layer of talking about this challenge. But if you, if you're curious about maybe pausing the
podcast and taking a look at the scanning the blog post, it's on the the preset blogs. That's at preset.io/blog.
On there, you'll find the blog post that's titled, why data teams
keep reinventing the wheel, the struggle for code reuse in data transform in the data transformation layers. That's,
probably a good a good place to start our foundation, at least for for this conversation now.
Assuming that some of the listeners maybe paused, took a look, came back. But the the the summary at a very high level is that, you know, as a data engineer,
you'll go from organization to organization
and and kinda rewrite
essentially very similar pipelines. I think it's even more so the case
for analyst engineers. So if we're talking more about the the part of
the workload of data engineer that's about
essentially writing a bunch of SQL or doing data transformation, like creating datasets from datasets.
And typically, you're gonna go and take data from your CRM, your SaaS product,
data around engagement and growth, you know, on your product.
You're gonna be computing things like NAU, DAU, weekly active users,
and the users are new churned, resurrected cohort analysis.
Maybe you're gonna go and build a an AB testing kinda compute pipeline to look at your your AB tests and your your different treatments and and what the p values are and how,
you know, your your different experiments are are converging. And, typically,
there's, like, just 0 code reuse across organization and that layer of the data stack, which is kind of a shame, you know, and there's probably some really good reasons for for this.
And there's probably some parallels to be drawn with, you know, web frameworks. Right? People will use something like Next. Js or they'll use React, and the reality is that, you know, maybe there's not that much code reuse on on top of that,
or CRM logic and things like that. But I I think there's there's still the intuition
that
we should be able to at least, you know, reuse a little bit of code there, create some frameworks,
have some reference implementation. If if not, like, pure framework reuse,
can we have more
reference implementation in the wild so that at least you can take inspiration from from previous work?
So that's the the general kind of prompt and an idea,
you know, why
doesn't that exist today, you know. Or maybe that exists. I just don't know, you know. I was like, give me some point. If you know about some of these things, people, and, you know, share, maybe there's place for for for us to have a conversation around that. But there's
much less that seems to exist other than it than should be.
There has definitely been an overall growth in terms of the maturity and the mindset around data engineering,
data integration, and movement with tools like
Airbyte, your work on Airflow, the overall investment in orchestration,
DBT has had a significant
uptake in terms of usage.
And to some approximation,
that increases the productivity and the ability for smaller teams to get more work done.
So to some extent, maybe there is code reuse at that tooling layer.
Mhmm. But I do agree that there is a lot of rework
and individual exploration that goes into
okay. Now that I can pull all of my data in, where do I put it? How do I store it? How do I transform it? How do I figure out what type of dimensional modeling technique I want to use? How do I appropriately model the business?
And
there has been a lot of work
on the
software as a service or data as a service side,
particularly if you're thinking in terms of the work like segment where you say, oh, let's just take all of your customer data,
put it into our tool.
We will generate the dimensions that we know how to generate, and then you'll just use that and everything will be amazing. And then maybe that works for some people, but you inevitably
have to evolve it and transform it and bring it under your own control. And I'm wondering if you can talk to some of the ways that you're seeing the
overall trend as far as
individual and team productivity
versus the amount of rework that people have to do at that more detail oriented level.
Yeah. I I love that take on, like, the progress
maybe so far over the past 5 years. I think you're right on the data integration layers. The layer you know, talk about air bite. Talk about 5tran. So one is open source, the other is more proprietary or a SaaS service.
And that layer of data integration, I think, has been
solved in in some ways or is getting actively solved.
So that's very much if you if you think about it in terms of ELT, that's the EL. Right? So So you just extract
what's in your CRM, what's in your,
your HR system, what's inside, like, whatever tool you use as a SaaS service. You can get that in your warehouse synchronized very reliably
for fairly cheap, though depending on data volume, you know, depending on the I I know that, you know, Fivetran can can get pretty expensive. I think Airbyte, similarly, probably in terms of compute and overhead. But that's been
largely solved. And maybe that's a foundation block we didn't have before that is required.
Like, we need some sort of universal
data model, at least for the the different data sources as a foundation
to
reuse,
for the t of the ELT. Right? So having consistent EL
is a is a prerequisite in some ways to having, you know, a t. We could talk about other prerequisite. You mentioned
DBT, and and one thing we we haven't seen enough of, I think, is, like, the the advent of
reusable DBT projects or or just like a reference implementation of, like, hey. I built a
an AB testing framework compute engine, you know, using
using DBT, and here's the project. You can use it. Maybe it's parameterized.
So
one question around, like, why wouldn't we seen
more of these reusable dbt projects? I think there's I I try to tackle some of that in the blog post. One is, I think, the complexity in managing different SQL dialects. Right? So you have maybe I have a BigQuery implementation.
And using the long tail of
of UDFs or or just the the functions that are exposed to do some some JSON processing,
some more intricate, you know, math or stats function, and and then these are intricately different across dialects. And now,
also then as a data engineer, I have to care about other, you know, SQL dialects, and it's not super sexy or fun or interesting.
So and and it might not be even easy to test. That might, you know, wanna wanna be multi dialect code is actually gonna work on Snowflake or
on Redshift or something else. So I think that's that's a blocker
in some ways.
I I think, you know, SQL mesh is kinda interesting. I don't know how good their support is for reusable parameterized
projects too. Also, the the issue
around
if you wanna do parameterized pipeline and into in in my blog post, I I talk about
2
things that are foundational. One's the idea of, like, some sort of
universal data model that you have to dump your data into, and that might be, you know, provided by 5tran or or air by
as a foundation. So that's
one one part but to have say a unified data model for healthcare data or for CRM data or for,
you know, engagement on on various product. We call it user action and time type universal model. Some are very intricate, some might be simpler.
But I think that's a foundation we
don't necessarily have or have adopted. I think there's been some different projects in the past to try to create these unified data models. So they they may kinda exist and kinda intricate.
So that's one thing. And then we need to have some sort of, like, parameterized pipelines. I can say deploy this AB testing framework,
but here's, you know, my set of dimensions
and here's how I wanna cube the data. Or here's
some intricacies that are specific to my business. Right? So it's trying to figure out what is
kinda universal and reusable. You know, what is a constant? What is a variable? And then you would have to presumably write these these, what I call, parametric pipelines. Right? So it's, like, dynamically generated pipeline based on configuration
around your business.
Could I I could get into some of the examples. I've got some in the blog post. But you know, so that it can be it's not one size fits all. We know that much. So then for the things that are not one size fits all, you need to be able to parameterize your pipeline and instantiate it. And currently, to do that in in DBT, you have to write a bunch of Jinja.
And that's just a pile of mess. Right? So to be writing
something highly dynamic, like just writing code that writes SQL, which is messy by nature. Right? You have a declarative language,
and you're trying to bend it into being
more dynamic, and it's not what it was intended to do. So so maybe the tool set is is not really appropriate to make it easy for people to be like, oh, it's really easy for me to make this available for other people to to use it. I'm just gonna parameterize it. I'm just gonna write something more dynamic,
and share it as a as a project that people can reuse. And like, okay, there's a dialect problem. There's
the the messy parameterized pipeline issue
on top of other problems we could get into, but it's probably a good place to start. We're already, like, kinda hitting some pretty major blockers here.
Yeah. There there are definitely
any number of different
areas that you could focus in on and spend your entire lifetime trying to improve and people do. But I also think that it's interesting
before I get too far along this thought, there's also the case of
code focused versus SQL focused where right now we've been focused very much on the SQL oriented data engineering
approach of I load all of my data into my warehouse, and then everything after that is SQL. Mhmm. There is also a significant cohort of
I load all of my data into my data lake, and then I write a bunch of code to process it and do different things.
And to some extent, those are converging where there's overlap, but particularly in the data lake house architecture or the unification of batch and streaming for different use cases.
But I think, also,
some of the challenge maybe comes in from the fact that we are
typically working at least 1 or 2 layers removed from the problem that we're actually trying to solve. We're we're trying to solve a problem for the business,
but instead, we're trying to fine tune the block size of the objects that we upload to s 3 to reduce the overall latency.
So there's definitely something there that I I think
keeps us stuck and away from being able to do a lot of reuse of the the logic that is actually solving the problem we care about.
And to some extent too,
depending on who you talk to and where they're coming from, there's an argument to be made that the
way that we're modeling the data
is cumbersome
and problematic where we've been using these relational
databases
and relational theory to be able to try to model the business.
There's a whole camp of people who say, actually, everything is a graph. So model it all as a graph, and your life will be amazing. But then there's the problem of graph scalability problem. So I'm just wondering what your thoughts are on that juxtaposition
of code versus SQL focused data engineering and then also some of the ways that the specifics of how we think about the structure of the data is maybe hindering our ability to reuse more of these capabilities across
organizational and team boundaries.
Yeah. That makes a lot of sense. I've got I've got a lot to to say here and unpack. But, yeah, one one thing is on the duality
between, say, SQL and more proper
coding
or not, you know, say call it declarative to more more dynamics. And in reality, I think today that means
most like dbt versus,
maybe spark and, you know, or a data frame tape type API or say, like, data frame dot group by and that they frame that filter.
And but it does feel like
writing something like a parametric pipeline in SQL
is it feels like,
you know, like, trying to do some some odd things, like, the equivalent of, like, the lamp, like, PHP
days where, like, you're doing something that works. But
you're writing
I think in the PHP days, we were writing logic inside HTML files, You know? So you'd put, like, an if condition within an HTML block.
And then, you know, at some point, we kinda flip to to React where it's more, like, driven by JavaScript that generates HTML.
I'm not sure if the parallel is that that good or interesting, but back to
the the SQL take. I think SQL is extremely convenient and declarative
and acts at and it's not dynamic, which
makes it a lot more accessible to more people, but it's a bit of a dead end. May maybe we don't realize
at what point
at which point it it just kinda prevents us from
thinking like software engineers.
Right? So it's like,
you know, these declarative languages are good for for fairly static things. You know, you think they're
Terraform for infrastructure as code. You don't you don't want something too dynamic there.
But I think SQL in some ways
in the complexity around
dialects,
it's like those like, it's just like close enough,
like, similar enough that so that you feel like it's familiar when you switch from a dialect to another.
It's it's different enough and truly different different enough so that you can't really practically share code
across engines. It's it feels like a a a very
convenient
kinda abstraction that becomes a bit of a dead end for, you know, analyst engineers to turn more software engineer or embrace more, even more of the the software engineering practices. And there's there's been a lot of, like, narrative around how,
I think bring bringing kind of software engineering practices to
data engineering first, but even
analysis
or analytics
engineer, but it's kinda limited to Git, right, in some ways. And there's these macros. You know, in DBT, you can write these like Jinja macros, but got our day nasty to write and and, you know, test and debug.
So that's that's a bit of a limitation
there. You know? It doesn't feel like
d DBT is the best place for code reuse. The package management is just a little awkward, you know, as a as a ginger first, you know, type type thing. So there there is a parallel there with
PHP, you know, probably.
So there's that. Let's see. What else were you touching on? I think there was much more to unpack here. I think some of the other interesting
avenues to go through are the
the underlying model of the data, how we think about it. Is it all just 2 dimensional
arrays? Is that really all we care about? And then we say, well, actually, 2 dimensions isn't enough to sufficiently capture the nuance of the problem that I'm trying to solve. So let me explode this into multiple tables joined across these different attributes
that then become cumbersome to evolve and maintain, and it becomes difficult for end users to be able to figure out. So you have to pepper it with documentation and other helper functions and maybe just hand them their their spreadsheet that they actually care about. And just wondering
how you see that
data model that we have gotten locked into for the past, what is it, 40, 50 years now?
And some of the ways that alternate
storage methods, particularly in the graph space,
can either supplement or potentially at least partially replace some of the ways that we do that modeling to improve on reuse.
Yeah. I mean, let's let's take it from an angle of,
like,
dynamicity
and data models. So if we wanna do code reuse, like, clearly,
we we need part of the model to be dynamic.
And I get I get into that in the in the blog post in some ways, but your customer attribute at your company are not exactly the same as my customer attribute at my company,
and we care about different set of attributes.
And how can we model this in relational database
or traditional relational databases. You know, it's it's inconvenient to have to do dynamic modeling to be like, I'm gonna alter table, add column
dynamically.
It's not practical to do so, and it's not reasonable to do so. So one thing I'm talking about in the blog post
that's based on that premise of being in in this database world, in the traditional relational database world, is
could we identify these
these dynamic parts of the model and put them into these
these new not new fields, but these this emerging
class of, like, new
new fields,
that are more dynamic in in modern databases. Right? So you could have, you know, the extreme examples, you have a JSON blob and in in your customer
table, you would have some sort of, like, customer extra attributes. And in there, you could have as much complexity that, you know, you wanna you wanna add to enrich your customer entity with whatever is unique to your business. And maybe the framework needs
something like a customer type that's, like, really
structural
and used by the framework with some assumption around, like, this is a paid customer. This is a free customer.
Maybe there's some logic that attached to the customer type, and that's
hard and something you gotta fit into when when you wanna use this parametric pipeline.
But then if you wanna have,
store all sorts of other customer attributes, we would have a place for that. That's all in within a column. So you wouldn't have to have this, you know,
dynamic model or, you know, I I think the dynamic model is just not viable. Now let's do, like, is it grab database or document store or key value store a better place to, yeah, ultimately, everything is kind of a keyed
you know, everything is a document even in a relational database. You know? So it's all data and metadata.
So whether you say create create table,
you know, and define columns or you put that in, you know, in in a marshmallow,
for people familiar with the the the schema library in Python. So there's all sorts of different ways that you could define a schema
for your different entities, but, clearly, you're gonna have some sort of notion of entity
type and a schema for it
and an entity ID in all cases.
And what to me, whether it's modeled, you know, in in the traditional relational
way ways as a different physical table and a physical database
or whether it's stored, you know, in an object store, what metadata
living in Git instead of, you know, in the database engine itself is
is is not necessarily
I mean, it's an it's an interesting
thing to think about, but it's kind of the same problem.
And and then except for maybe the properties of of the relational the different database engine and what they can do. Right? So, of of course, in a graph database, you're gonna be able to to run certain types of queries in a
in a different way. So different workloads are, like, give me all the descendants for this node, you know, and then find all the objects
that so it's gonna be real good at doing that. It might not be as good at doing, you know, full scan or a join.
So
that as to, like, where to store it to to me is is less of
a of a challenge. But I think thinking about these parametric pipeline, what's really important is to define what is structural to the parametric pipeline and something you have to fit into and which part of the pipeline are more dynamic, or which part of the the different attributes or facts are more dynamic.
So that would be like, you know, my extra customer attribute or my extra, you know, transactional attribute or my the hierarchies that are specific to my business.
And this would have to be parameterized and stored alongside, you know, the instant the instantiation
of the parametric pipeline so that we know what to do. So say if you want for the pipeline to generate to cube the data in a certain way and
generate
some count distinct group by different combination of dimension. That would be something
that could be
unique to your pipeline. And based on on top of that metadata of the fields that
the extra fields or the extra metadata that's unique to you. So that, you know, there's quite a bit of abstraction and complexity there. Maybe the operator of the pipeline is like, what is this thing? What does it do? You know,
thinking about
the these abstractions
are pretty complex. It's kind of an issue.
Right? So maybe if you're like, okay. I'm gonna do my first AB testing
data compute pipeline. I'm gonna use this
toolkit or framework or reusable, you know, airflow DAG or what or SQL mesh, you know, abstraction.
Okay. Where do I start? Then you maybe
you PIP install this or I don't even know. Like, you you know, the DBT
installed the package,
and then you have to start looking at the the YAML configuration
maybe of that pipeline and how you're gonna do your own AB testing. And you're gonna have to load your data into a certain place and provide a bunch of parameters
and and run this thing, and and and maybe
that interface
is too too complicated of a template for people. You know, maybe they need to go through the grind of writing the pipeline to to be able to operate it down the line. I don't know. But it's it's it's quite a
it's it's quite a
complex abstraction
for
any data engineer to kinda inerrate that and try to try to work with it.
I think another interesting aspect of the state that we find ourselves in is
the fact
that data engineering as a discipline,
for 1, is relatively new. I mean, people have been doing data work for a long time, but data engineering as a specific title
has really only been used actively for about the past
decade, give or take.
And the backgrounds of people who come into data is fairly
wide. There are a lot of people who come from the systems
side where they start as a sysadmin. They just have to keep things up and running, and so then, automatically, they have to deal with the data. There are a lot of people coming from DBAs and business intelligence backgrounds. There are a lot of people coming in from data analytics backgrounds who then go further down the stack more into the pipeline management.
The many also folks. The spreadsheet people too, like, the the spreadsheet wizards, you know, that that realize they can have so much more impact if they work with a database. Those can be pretty impressive too as they become, you know, data engineers. There's, like, so many different paths and backgrounds and way to get there. And there there is also a substantial number of people with a software background, but I don't think that it's as
ubiquitous
as the web application development ecosystem has been where that is purely software, has been all the way along.
And I'm just wondering how you see that
variance in backgrounds
in the data space
impacting the ways that people think about investing in reuse and those software patterns.
Yeah. Maybe I'll I'll start with with an interesting parallel with front end engineering.
You know? So I think we've seen
front end engineering
over the past decade go from this, like,
not so respected discipline to becoming, like, a true software engineering
kind
of legit,
profession almost. You know, the perception in the past was like, you're a web a webmaster, a web engineer. You write HTML and JavaScript, and you're not you're not a true software engineer. Like, the the back end guys, you know, the the
the balding guys with a ponytail
writing, JVM code all day. They're like, okay. This this guy just writes website. Right? He's more closer. He's sitting closer to the the graphic designers than he than he is to to
than than he or she is from us. And
but we've seen this transformation
happen. It took a moment. I'm not sure exactly what are the the key ingredients,
you know, that, are the the important milestones that allowed for that discipline to to get better.
If you push and I'm deviating for from from the topic a little bit, but if you push the analogy and compare and you look at code reuse
in on the front end,
you won't see that many
templates. I think I think what you're gonna see is, like like frameworks
and toolkits
and things to abstract with these abstractions of CSS and HTML
and,
but you won't see that much. Like here's a marketing website. Right? Like I'm gonna I'm just gonna use I'm gonna I'm gonna NPM install, like marketing base and and parameterize it to fit my marketing
organization. So
so so so maybe when you get into the things that are unique to your business, like the way you treat your data or
branding, you know, on the on the front end,
you can you you know, it it becomes really hard to reuse code. And maybe the template approach, right, you might take a
marketing website
template and modify the crap out of it, but you're not gonna
install a marketing
website
package and parameterize it. Right? So maybe
maybe then that points to,
like, adding a better a better ecosystem
of templates. By that, I mean, like, reference implementations
of these different pipelines
and and for organizations
or or data engineers to just be like, oh, well, I'm I'm working on a
on on an engagement computation framework or
I'm looking to bring my HubSpot data into my warehouse and do something
with it. Well, maybe you'd you'd use some sort of template
and and fork it as opposed to, you know, take a package and parameterize
it and and extend it.
I think that's a useful analogy to build on as well where in the front end ecosystem,
you have these templates, you have all this tooling, but all of the work that you're doing is inherently visible because that was why you made it. You made it to show to your end users, and the way that the web is structured is that
when you load it I mean, it's getting less so now because of some of the protocol changes and and the frameworks. But when you load a page, you automatically have access to all of the code that went into it to to some approximation.
And I think that in the data ecosystem, we're a little bit hampered by the fact that
data
inherently has this
layer of security
and shroud of secrecy applied to it, so you don't have it as visible for other people to be able to learn from. So all you have is your own past experience at other organizations to draw from or what you're able to glean from people that you come in contact with through your own professional network and conferences, etcetera.
And so I think that that also has acted as a limiting factor in terms of our ability
to have these reusable templates, have this more broad reaching
reference implementations
to be able to draw from because it's rare that you actually have a
fully spec'd out data warehouse using,
star snowflake Skiba with highly maintained, slowly changing dimensions, etcetera, etcetera. And so it it's it's this
more
diffuse
knowledge transfer than a
easily referenceable
way of doing anything in a concrete fashion. All we have are these abstract references
of the Kimbell book or the data vault book of this is how you do it in an abstract sense. But if you wanna do it in your own business, figure it out and good luck. Pay me a bunch of money and maybe I'll help you. Yeah. I like to, you know, the one of the Kimball books was more project center. I think it was called the data warehouse life cycle toolkit is what it was called. And that one was more like, okay.
You're starting a data warehouse implementation in your organization, and how do you navigate the whole business and conduct interview and identify the key stakeholders
and get to, like, a bus matrix. So there's a whole, like, recipe that
probably does does not work anymore. It assumes, like, a very waterfall type thing. It's it's not agile by by any means, I think. It predates agile, but it's more like the cookbook was kinda interesting. I like the the parallel, the the the
the thing that's that's cool about the front end, I think you identified as it's a lot more transparent or the code is in your face, so maybe that creates some some unique dynamics to front to front end in some ways.
Yeah. On the
on the, you know, bringing more transparency
to
to what's happening or sharing more. So
I wanted to bring up,
data gravity to describe an issue where where data is not very modular. It wants to be together and mesh together so you wanna bring
everything you know about a customer inside your customer table. You know, actually I I have a blog post that talks about
entity centric data modeling and it says basically your dimension tables, you should think of them like a, you know, like not just dimensional attributes, but bring metrics. Right? So if you have a you have a customer dimension,
you have a product dimension,
don't use it only for attributes of the thing. You can totally bring in, you know, metrics like what is my number of seeds sold and number of seeds filled over 7 days, 28 days.
Some arguing, you know, can I bring
all of the properties
of the entity or, you know, as an attribute in your dimension,
take snapshots and, you know, the the there's a lot of good ideas there, but it points in the direction that
you you're gonna bring all the attributes that come from your all your different systems?
Right? So that means your your CRM, your product use usage,
you know, anything that's related to the customer,
whichever system it comes from, you wanna bring it all together. So that data
gravity, I think, makes it such that data pipelines are a big tangled mess.
You know, that that everything's connected with everything,
and that makes it harder to say, like,
I'm actually gonna share something that processes, you know, event data, and that really doesn't touch all the other dimensions of or or attributes of my my my customer or my user.
So so that's one thing. And then, you know, there's something about sharing templates or reference
implementation where I have like, I wish I could just put I could put my dbt like, we added dbt at at preset. Like, over over the years, we've grown very complex,
transformation layer with, like, maybe 300, 400
data models. And I wish I could, you know, maybe I could just
open source it not so that people would use it, but as a reference implementation.
But it's it's always a little shameful, you know, and a little bit of a mess because these things grow like like a a big,
big ball of duct tape. I don't know what what exactly is the the analogy, but it just kinda piles off on itself, becomes extremely complex.
It's it feels like it's tech debt from day 1. You know, so it's not something that you're proud of. It's really hard to make something you're proud of using, you know, template SQL
and share it in the open. Plus, there's the the intricacies of the privacy
or not not even privacy security issues would, like, show like, you know, there's probably
a thing or 10, you know, in that dbt pipeline that I wouldn't wanna share, and it would be a little dangerous.
You know? And I I don't even know what these these things are. So so there's that the shareability
of these constructs
is not good
because the dialects, because the kinda nature of it being kinda messy and tangled up and complicated.
So that that's an I don't think I identified that in in the blog post, you know. So it's like the the self perception
of the the value of the code. You're like, I don't wanna put that in the open. You know, this is dirty laundry.
Keep the curtain closed, everybody. Yeah. Absolutely. For for what it's worth, the code repository that I'm building for my day job, we actually do have open source. It's not necessarily a useful reference implementation, but people can look at it.
But I I do think that that is probably one of the biggest hindrances
in our ecosystem
of data is to your point that gravity of the data where moving at a 1000 different places to make it more componentized
and modularized
goes against the physics of the problem.
And the fact that there is so much
security layered on top of it and the issues of privacy and just the, I don't wanna risk it, so just keep it under the covers. So it hinders the
distribution of that knowledge.
So it it is much less diffuse beyond just the books that people invest in that are, by nature, abstract and
not directly
applicable without doing some of your own translation work, which is where all the complexity comes
in. Yeah. That'd be viable work to be done by someone to to kinda create templates and and share them as,
you know, incomplete
ideals, but but, like, reusable
and not necessarily as something you can
go and run with and get value instantly from, but more, maybe it's the author of a book about, you know, practical data engineering. That would be like, okay, here's an example
of a dbt pipeline that computes, you know, all the engagement, growth accounting metrics that you want for your product.
And here's how you would extend it.
That's that stuff doesn't seem to exist. Curious on the on the repo that you say you opened up with the with your own dirty laundry or maybe not so dirty laundry. But is that is that in the form of the DBT project or or a spark pipeline? Or what does that look like? So the repository
right now primarily consists of some Daxter pipelines and dbt models.
Gotcha. And and then you're processing what kind of,
is it product information?
Or
So it's primarily information
about
educational
materials and how people are engaging with it. So,
my day job, I work at MIT in the open learning department. So it's all
nondegree material
for learners outside of the bounds of MIT, so not MIT students. So
MOOCs, open course for material,
professional education, and so it's
figuring out what are the enrollments, what are the learner engagement patterns, what are the courses that we have on offer, etcetera, etcetera, and then being able to report that into the business.
That that makes sense. Yeah. So it's a that pattern,
if you think of it in terms of verticals to an opportunity for reuse. You know? So I I talk about that a little bit in the blog post too to say,
what are some of the business verticals that are highly common that data engineers do over and over
that where the the schema and the model and maybe the the the customizability
is not as important as needed? Like, what are some simple data model? Maybe that are very simple
in terms of, like, the the input to the data model, but it can be complex in terms of the output.
And I think to to me, it seemed like product,
what what I would call like user action in time and event pipeline
would be extremely useful. Right? So because everyone can model some of their data,
as, you know, time stamp,
user ID,
you know, event type,
and then extra attributes. Right? And and if you if you have something like that, you could compute all sorts of,
you know, actors and time, actions and times, action per user,
frequency of action,
click stream, like, you know, all these, you know, funnel analysis. So all from a very simple model. So maybe there's an opportunity that there in that area. And you've seen that a little bit productized
when when I was saying, like, there there's no real or usable,
you know, things that have been used and shared. If you look at tools like Amplitude or there's there's a lot of these, like, you know, tools out there that are mostly user action
event in time.
So I could see that I could, you know, there's an opportunity for a pipeline there.
I've been meaning to share one that I call it's called EGAP, the engagement growth accounting framework.
I have a dbt project. I don't think I don't think I've made it public,
but, you know, I never felt like it was good enough to share. But I I've been meaning to do this. I wanted to give a talk at the submitted a talk for the COLLS conference, like, 2 years ago and wasn't accepted for for whatever reason. So so it hurt my feelings a tiny bit. And then I was like, okay. I'll just do that some other time. Never got to it. But the premise with this, engagement growth accounting framework
was, you know, you load your data into a well defined,
you know, structure that's just
time stamp,
user ID,
action, or kinda event type,
event subtype maybe,
and x some sort of extra field with, you know, other JSON you might wanna use down the line. And then based on that, you can order you parameterize it by wanting to say, I want DAU, MAU, we collect the users on, you know, these cohort tables.
I want like, maybe there's a sub module for
experiment exposure. Like, who was part of which experiment at what point in time. So we could do some maybe testing, kind of bolt that on top of it.
And it seems like it would be immediately useful to to people, but it's kinda
hard to get to,
to sharing it because, you know, I wrote it maybe in BigQuery
in DBT and,
you know, people couldn't use it out of the box. It's it's kinda hard to put DBT to put a YAML file to parameterize that pipeline. I mean, it's not it's not that hard, but it's, you know, it's it's a little bit intricate to figure out exactly what the parameters of the pipeline should be. But maybe it's an experiment to look forward to. You know? Like like, I'm curious to try to put it out there, see if I can get, you know,
10 or a 1000 stars or 10000 stars on GitHub. It's it seems like 10000 stars material. You know?
Every company is doing that in some capacity.
Yeah. It's definitely
great to hear you suggesting to put that out there. I encourage you to do so. I'll keep an eye out for it.
And I think that this conversation won't be complete without some mention of generative AI and LLMs. And I'm wondering
what your thoughts are on the ability of this current
cycle of
innovation,
experimentation, whatever you wanna call it, how that is maybe a potential
means of doing that knowledge capture and distribution
that is currently done behind closed doors and federating that a bit more, making it easier for people to
say, this is the business problem I'm trying to solve.
You generate the 1st pass at that, and then I'll fine tune it as we go.
If we wanna, like, enable
intelligence in general to do a great thing with reusable code and templates and and frameworks,
I think it's kind of the same if you're trying to enable you like, intelligent humans or artificial intelligence. Like, you need to have good docs, good reference,
good naming, you know, things that are intelligible
humans or art AI can can work with.
One question is
would so say if you put these these templates out there, these frameworks out there, and they're publicly available for for machine to train on as opposed to rag on, you know, that's that's one thing that I really think that open source is gonna get more from
more help from AI than private source because it's just out there. Right? The the models are training against Apache percent Apache Airflow,
and they're they don't have access to the Power BI code base to to come and help out or to answer question or to write PRs or to answer issues. So we've seen the rise of these these AI bots that
are effectively trained on the superset code base, you know, and can can help us.
So that in saying that maybe the AI could work very well with these these reference implementation and temp and templates. Because you could say, okay. This egas thing I was talking about, engagement growth accounting framework,
it's out there. All the buts, you know, Cloud knows about it. GPT knows about it.
And when I asked it, I can can you help me, you know, instantiate that in my repo? Maybe I can take you, like, 80, 90% away there.
Can those questions could the would the AI help us get to that abstraction?
Probably. Would it help us, like, use,
bring this abstraction into
into useful or, like, production territory?
Probably. But the more the more that is shared, the the better, you know, for advancement of intelligence regardless if it if it's artificial or or not.
Yeah. It's it's definitely interesting to see how
generative AI has already started consuming software engineering in different capacities and how it's influencing the way people think about how they get their work done. So I'm excited to see what potential positive impact it can have on the data engineering space.
And I'm just wondering, as you continue to invest in this space,
try to build and promote this idea of code reuse, I'm just wondering what are some of your hopes and predictions around that potential future?
Yeah. So a lot of hopes. I don't know
how much prediction, but for for me, what I do is I try to do first reflex AI for just about everything I do. So new task,
new workload, new workflows,
something creative, something repetitive.
I
always try to get the AI to help me out, and I've been having, like, various level of success. I gave a talk at the Airflow Summit and at OSACON, it's the same talk. It's like
it's it's around, like, the the the hype of using AI as a practitioner over the past 2 years. As a founder, as a data practitioner,
over the past 2 years, I've had this AI first reflex
and then some reflection on as to, like, what's working, what's not working,
and what I have good hopes for in the future. So there's an entire talk on that that's that could be could be interesting.
One thing I tell the people is I encourage them to have that AI first,
reflex for just about everything.
And then, you know, and then give it more than just a shot of, like, can you do this for me and the AI is not helpful
for that, you know, without the context. If you try to provide more context, you know, give it what it needs to to help you out.
So I think I think it's a good it's a critical
thing to do, I think, to remain relevant. You know?
If if you skip
a a beat or 2 on that,
I think you can you can realize you're you're far behind pretty quickly. It's quite easy to catch up if if you're behind, but, you know, really figuring out how you can team up with this
this kinda infinite
resource, you know, and then this, you know, this thing that can really help out with a lot of things. Figure out what works, what doesn't for you. I think it's like everyone should be should be doing it. We shouldn't wait for the products
to do it for us. You know, individually, I think you gotta figure out,
how it can help you out, you know, every day.
Are there any other aspects of this problem space that we didn't discuss yet that you wanna touch on before we close out the show?
No. I think I think it's maybe a call out for for people interested in the space. So it it turns out, you know, people have been thinking about this for about
20 years too. Like, when I started in business intelligence
and, you know, data warehouse architecture,
people were already thinking about these things.
So,
just keeping,
I'd be curious if people there's a way for people maybe on Twitter,
LinkedIn, you know, to start a thread. If you've seen things
that you think are interesting
in around, like, code reuse and how to go about it, and then maybe encouraging people to put their reference implementation
out there, you know, to be like, you know, I've wrote an AB testing
framework or pipeline
as a DBT package, as a SQL mesh mesh project. And maybe I forked it for my own use, but here's how I got started, and here's the reference the reference
implementation.
Maybe there's an opportunity
to become kind of a a small community around all the people that are building that specific vertical, and they can share experiences.
So it's so it's more of a
sharing, you know, sharing what you did as opposed to, like, writing a framework that everyone should just, you know, pip install and use.
But maybe that's another aspect of open source, and it might be a required step
towards the the more reusable framework. Because, like, first, like, we gotta look at each other's work
to to understand what's common and what's different.
And maybe only then,
you know, we we we can start building a framework that works for, you know, 70, 80 percent of the of the people building the same thing.
Absolutely.
Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And usually, I ask about what you see as being the biggest gap in the tooling or technology for data management, but I think that's what we just spent the past 5 minutes or so talking about. So,
I just wanna say thank you for taking the time today and for your thoughtfulness in this space and all the contributions that you've made, and I hope you enjoy the rest of your day. Same here. Thank you for, you know, for everything you do here. I think, having this this medium of exchange is is much better than, say, just a blog post. So the blog post is more structured, but this is a lot more interesting to to really go and explore and and talk about it. So I love this
long form, you know, chat chatty, like, way
to approach,
and explore, you know, part problem space. So it's been great.
Thank you for listening, and don't forget to check out our other shows.
Podcast.net
covers the Python language, its community, and the innovative ways it is being used, and the AI Engineering podcast is your guide to the fast moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com
with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.