Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
Your host is Tobias Macey. And today, I'd like to welcome back Michel Tricot about the journey to the 1 point o launch of Airbyte and what that means for the project. So, Michel, can you start by introducing yourself for anybody who hasn't heard your previous appearance?
Yeah. Of course. Thank you for having me, Tobias. So I'm Michel Tricot, and I am the the cofounder and the CEO of Airbyte.
And do you remember how you first got started working in data and what it is about it that has kept you interested for so long?
I mean, I could go very back in the past and look at me as a teenager collecting,
DVX and,
websites because at the time Google was barely starting.
But more more professionally,
started really in 2008.
Yeah. Working on financial data. I thought I was going to be a trader, and I ended up doing
financial data, and I had a lot of fun with it.
And what happened after that is I moved in the US, joined
a very
at the time very small, became very large, AdTech company,
and everything related to AdTech and MarTech is about data and data at internet scale, and I would say that's really where I
got burned the most and where I learned, a lot of this principle around what does it mean to work with data, what is the process of data manufacturing. And at the end of the day, starting Airbyte in 2020
and having a have have been having a blast since then.
And for people who are curious about Airbyte and some of the specifics and internals, I'll refer them back to the previous interview we did, and I'll link that episode in the show notes. But for somebody who,
is just hearing about Airbyte in this episode, if you can just give a quick overview about what it is that you're building and some of the notable milestones that you have gone through on your path to this one point o release.
Yeah. So we started Airbyte in 2020,
and Airbyte is an open source data movement platform. You can almost think of it as the highway for getting data from point a to point b, and point a be being like a place where you have siloed data, and point b being a place where you can actually extract value from data.
And we started in 2020.
That was the first release, the first open source release.
And
in 2021,
we started to get more and more traction from the community
up to a point where
in the summer of 2021
the whole team was completely frozen. We couldn't do anything, we couldn't build a product, we're just spending so much time on slack helping our community be successful with Airbyte. For me I I see that as the first real milestone like yes you've released something you never know if it's gonna work but this one was the real like first milestone that I can remember because we really suffered a lot from it it was it was fun but it was very frustrating to not be able to to build more. And in 2022,
the the milestone there was
releasing
Airbyte Cloud. So it was our first time having to really operate Airbyte
before it was open source software, so we just put it there on GitHub and people would figure it out. Here, we had to be our own customers of a
this is how you need to be running Airbyte at scale, and also what are all the the pain points that go with it. And
that was yeah that was quite a journey in 2022.
We like to refer to this one as
real that this one was very much a painful year because everything
that we did in 2021 was about moving fast and 2022 was okay we need to take a step back and look at what does it actually mean.
2023
would say was
the connector builder, you know airbags is nothing without
connectors because that's what allows us to connect point a and point b together
and
I think
in some point in 2021
2022
I wrote down a document internally about like what we call nailing the maintenance
which is how do you create a very large
catalog of connectors, lots of breadth, and how do you make sure that these connectors are high quality,
they are well maintained etcetera etcetera. And 2023, we really build the first big building block
to getting to that state. And, yeah, now we're here in 2024.
We have a ton of connectors and we're preparing we're launching the the the Airbyte 1.0
version. So very, very happy about that.
On that point of having the connectors that are available be high quality,
reliable, I know that that was one of the major concerns early in your design process,
particularly
given the state of the ecosystem around the
stitch framework or protocol.
And
I'm curious if you can talk to some of the lessons that you learned in those early days of evaluating the stitch protocol, deciding that that didn't suit your needs,
building the Airbyte protocol,
the interfaces that you had around that. And I know you also went through a major revision of that protocol in that time frame as well and just some of the lessons learned early in that process that helped inform the decisions that you made about what it actually means for those connectors to be reliable,
what elements of that you wanted to be customizable versus which elements needed to be standardized,
etcetera.
Yeah. Of course. So
when we think about
building connectors
and building
a large scale of the
a large number of those,
the
the thing to think about is you're not just building a connector.
What you're building is you're building a factory
where you get raw material and input, and you get a high quality connector as an output.
And for me when I look at the past 2 3 years,
this is what we've been building. Like the byproduct of it is we get more connectors, but what we are building is that factory
to get
from someone saying hey I need to get data from x
to I have a working connector that will work for me for the for the long term. And to me that's the the big the big,
challenge in what we're doing is like what does that factory looks like
and
when we started in 2020, yes, there was the the stitch framework actually initially airbyte was running with those connectors and very very fast we realized that the quality
was not there because it was basically up
to every single human to just make sure that this connector always works
and there was no real like testing or ways to,
I don't know, to be prescriptive about what is a high quality connector.
And that's where we started to change things very fast. Like, I think in 2021, we're already we're already of the of this protocol, and we build our own. Obviously,
we we we took some inspiration because there was some there were still some good things there.
But what we did first was
when someone builds a connector,
they need to encode
the whole environment of what that connector needs.
And, you know, in 2021, some people ask us, oh, why why is that a docker image?
And I say, yeah. Because a docker image, it doesn't matter that it's an image. What matters is that by making it an image, you have to encode
what library needs to exist. Like, if you need some specific SSL
library installed on your system, it will be encoded somewhere and you will never forget about it. So it was a way of just making sure that a connector is completely self contained and it can run anywhere no matter what your environment look like. You only need to have a way to execute Docker image and that's all. And if you don't want Docker, it's fine. You can still read the Docker file and you know what you need. And you can this way, you're basically documenting what what what you need, as code.
And for us, that was really the the first step.
The the next steps was very much around
how we do the testing of those, like, how we create
whether it's sandbox,
whether it's
replaying
like API calls etcetera etcetera. And the goal here is just to create that asset of
a what what does it mean to have a high quality connector? Well, you need to be able to run test. You need to be able to run,
real tests. You need to be able to validate that the the data looks like the schema you're describing
is all these things. And we've been building every single step of that process of manufacturing a connector.
And today, we have a ton of dashboard internally that are monitoring
every single connector. We know which stream are working, which stream are low quality
we can
at that point just invest more time in those or someone from the community is gonna look at it and just help us on on fixing it.
And finally, the last one was
how do we make it more like simpler to build, disconnectors?
The the theory that we have here is that
we have a protocol and you can build anything with that protocol but it's very painful like nobody
creates a website with TCP IP. They all do they all have, like, higher level on top of that.
And that's basically what we've done since the beginning, which is first very close to the protocol. Then in 2021, we had the first version of the CDK. That's when we started to have more, like, community adoption. Then another version of it. Then we started to have low code, then no code. And that's basically what we're doing to just
minimize the the amount of effort for building, but most importantly for maintaining these connectors. And
that's I think today,
most of our connectors are using low code, no code, meaning that they can be maintained. It's just
1 or 2,
2 minutes. So and that's what we want to get to. It has to be low cost.
Another aspect of
the connectors
and the ecosystem around them is that you have done substantial investment in simplifying
that development process or the generation with the low code and no code interfaces,
specifically on the source side.
The destinations are still more complex and rightly so because of the need to be able to manage how updates are inserted into the platforms, the different types of destinations, whether it's a data lake or a database or a data warehouse. And I'm wondering if you can talk to some of the struggles or some of the challenges in
simplifying
the interface and simplifying the development effort for
being able to generate those destination connectors and beyond just the source connectors that are a
little bit more straightforward, at least in the at least in some cases. Obviously, there are very complex source connectors as well. But Yeah. So, today, a lot of the
destinations that
we support,
they have a lot of specificities
so
you could load any data into like a SQL based store by just saying like
insert
this is going to be very inefficient.
So then you go and you start looking at like what is the most efficient way to load data into Redshift? What is the most efficient way to load data into Snowflake or BigQuery or like ClickHouse etcetera etcetera.
And that's why like having a CDK for those like more
heavy infrastructure
is building an abstraction you can do it that's actually we have one that is coming out with a CDK we built the oracle destination on that abstraction and most of the destination are going to sorry not oracle
Databricks
and we're going to be able to use that abstraction for more and more destination so but it's probably not going to be to the point where you can do low code no code because these systems you want to take advantages of their specificities
now a place where I think destinations are going to become easier to build
is gonna be when we start addressing more, like, API types of destination
because those
already have,
some kind of,
like,
generalized
interface, which is it's an API, and you're pushing data there. There might be some specificities, but they will be easier to encode in a low code, no code. So this is a place where, yes, we will be investing in a in a framework there.
Another
interesting aspect of both the timing of when you first launched Airbyte as well as the
location in the data stack that you're targeting
is that there have been numerous
shifts in the past 4 years,
in particular, starting in 2020, the rise and then subsequent
decline or dissolution of the modern data stack, which is a term that nobody wants to really throw around anymore.
And then, also, maybe even more impactful is the rise and adoption of generative AI
and the need for vectorization
of data for the semantic retrieval
capabilities.
And I'm wondering if you can talk
to some of the ways that those industry trends have impacted the way that you think about the areas of focus and the capabilities that you're building into Airbyte and the requirements
around what Airbyte can deliver to the end consumers?
Yeah.
So,
you know, I I think I can still remember the first readme we had about air bite which was
open source
ELT
tool or something like that.
I don't want to just pull the the cover to myself but
I do believe that in 2021 we were the one who talked about data movement for the first time.
Maybe that's wrong, I don't know but I like to think that that is true
and the reason we quickly moved away from just ETL and ELT and just going for data movement is
we wanted to position
Airbyte and to develop Airbyte in a way that is very
that is at the infrastructure layer
so that
it's not about just pushing data into a data warehouse, although today it is one of our main use cases,
but we've seen people doing other things with airbytes that were also super interesting and that's why we wanted to just go a level lower and just look at it as data movement which is moving data from point a to point b provide intelligence on top of these pipes and just make sure that data can be sent to a place of value.
And for me that's why like when I hear about the modern data stack yes I mean we use
the word a little bit we stopped using it as you said
but at the end of the day
what
the modern data stack is
really what it's just like how do you create an architecture
and infrastructure that helps you build
like a long term system
and today with
people are going to continue to adopt warehouses
like there is no question there like people need to provide and to compute analytics to do dashboard etc.
Maybe we don't call it modern data stack but at the end of the day once they do something like that we want to guide them
on how do you future proof that. So, yes, you have your warehouse, think about your storage, think about your compute, think about your network, like data movement, like how you bring data into it, and then what comes after that. Like, do you need to do reverse CTO? Do you need to do dashboarding? So those are real problem that people will have. We we just stop calling them modem data stack, but they will still be there. Now on the gen AI side, that's actually where we got a lot of pull from the community.
The same way we got some pull from from reverse CTL, never really got to it, but we got a lot of pull from the committees.
We provide pipes.
How do we make sure that our pipes can connect to
different systems than warehouses? How can they make sure they can connect to data lake? How can we make sure they can connect to vector databases
or other type of destination? And for us, that's just how do we evolve, how we build this nation because sources are not going to change that much. Like, we still get need to pull data from Salesforce, from HubSpot, from Postgres, from
Oracle, from SAP. It's just where does it go and what kind of intermediate process do you need to put in place. So if you need to do embedding of the data, well, how do you configure airbag to do so? And that's things that we,
we need continue to develop because it's part of building pipes.
As far as the
ecosystem of data movement,
that is
maybe the core essence of what data engineering is. It's just moving data from one place to another so that you can use it for some process.
Even just the and I say just,
maybe that's not the right the right word to use. But
the constraining the scope to
extract and load, which is a big area of focus
and something that was the,
you know, initial step of the modern data stack is having that easy extract and load
capability. There are a lot of complexities,
both explicit as well as incidental complexity that comes out of it, a lot of edge cases, performance challenges.
I'm wondering if you can talk to some of the hard won lessons that you learned in the process of building and scaling Airbyte about the hard realities of data movement and data integration.
Yeah.
The the the hard thing about it
is
the moment
you depend on
a system that you don't control
but that you're supposed to link these systems together,
well, in a way you're responsible when they go down and that's really the problem. And that's why like people accept
like except people like Airbyte nobody wants to do something like that it's just it's just crazy.
And
this to me is
the this is the real complexity, it's like sometimes even if you want to create the best possible platform
sometimes you cannot
create a pipe that is as good as you would want it to be. You know we have people who say oh pulling data from this API is slow, I say yes it's slow but we don't have a choice it's because it's rate limited on the other side so you cannot go fast.
But in a way it always reflect on on the tool
and now the question for us as like operator and and builder of a platform is how do
we
train and educate people to understand that yes pulling data from this API is actually really hard because you have all these rate limiting things maybe there are some pieces of data that you cannot get out maybe the
like this particular
API endpoint does not support like incremental updates.
Although the platform supports it, the source does not support it or the destination does not support it. And I think this is the real challenge when you're moving data is like you can build the most
perfect
platform
you're always dependent on the source. So now the only thing you can do is just educate,
make
the most of it like you know when you have like rate limitings you can say yes if we don't rate limit they're gonna close down your account and that's like explaining like why why we have to do these things. But I think here it's more like
educating the the humans that depend on that data and just providing them like yes like this is actually something we did for 1.0 is this concept of checkpointing
is
sometimes it takes forever to pull data from an API.
The last thing you want is something goes wrong in the middle and boom you have to start from the beginning. Like this is the worst possible expense. And here, this is us having to walk around the fact that it does not support incremental and just create a system of checkpointing to make sure that although the expense is not the best it's not the worst.
And this is the kind of thing we need to do as an infrastructure product is yes all these weird edge cases
and setting the right expectations.
Over the past 4 years,
as you said, you started
with a very straightforward here's the open source project. Here are a bunch of connectors. Let's see what happens to now you're committing to the 1 point o release.
Obviously, there have been a lot of evolutionary steps architecturally.
What are the core decisions
from an architectural perspective that you made early on that have proven to be
most effective and most stable?
And what are some of the aspects that have required the most constant change and exploration
to settle on what you're now committing to as a stable one point o version?
Yeah.
The
one
that
I'm so happy we had made that decision in the past is running connectors at dock as docker images,
absolute game changer
and I know it adds a little bit of overhead
but it means that in terms of how we develop
how we think about maintenance
and that's the thing that is the most important like disconnectors needs to always work etcetera. So for me that was probably the best decision we've made in the past.
The worst decision that we've made in the past now
was,
I don't know if it's the worst one, I think it was a good decision at the time but we had to pay the price later, was we did not optimize for we did not optimize for scale
at the time. And it's just for me, for like as a founder, is you should not optimize for something like scale until you have to hit some scale.
And what we're optimizing at the beginning was very much like time to value making sure that people can just download the software get up and running experience the value and boom it's done.
But
it had some implications it means that there were some developments or some like architecture choice that were suboptimal for operating air by that scale.
And, yeah, we
I think when we started to run cloud
and to operate cloud, that's really when we started to hit this kind of,
of problems.
In
2023,
we onboarded our largest customer on cloud.
Thousands
of jobs run-in parallel.
It broke our cloud,
So nobody saw it but we saw it and we felt it to a point where we are having that meeting and someone was saying hey I think we made a bad deal here it's going to cost us more than they are paying us,
it creates fire everywhere on cloud
and I say yeah actually no we want to be able to take on this kind of volume for the price that we that we provide and it actually forced us
to work on
like a completely new way of
running
connectors.
We
it something we released I think the first version was in December it was still like behind feature flag was how we can very fast very quickly like spin up new data planes so that if we have a customer that needs a lot more like if we run into limitation on Google's API or like AWS APIs for Kubernetes
boom we can spin up a new cluster and everything works magically
and that's really but that took a lot of time because the the the fundamentals were not there at the time in the platform so we had to really build through okay we need scale we need a lot of scale
the other one that was and this one
like got us into
yeah was
we started to use DBT
in
our connectors directly. So when we put data into warehouses,
we would just transform it using like a generated DBT script on on a warehouse.
And although it was very good at the beginning when the volumes of data were small,
the moment we started to have more
data,
we started to have people complaining about, like,
cost of running,
like, the normalization of Airbyte. And,
like, the last thing we want is to incur more cost. So
that was also a big change in like how we architect not just the platform but really some of the connectors
was around like how we manage data warehouses. But initially it was a great decision because it allowed us to move very fast but at some point we had to really focus on okay
what does it mean to run air bite at scale? Like, we cannot do that. We have to be more efficient. We have to be smarter.
Some other interesting
evolutions
of the air bite platform are around
the operability of it, where in the very early versions, it was you deploy it. Here's a web app. You point and click. Here are your connections
off to the races.
And then
over time, you developed more API interfaces for being able to do
connections as code,
triggering syncs as code.
And
now you also have the PyAirbyte interface where you can issue the web server component and do it entirely in a Python script. And I'm wondering if you can talk to some of
the internal conversations and decisions that precipitated
each of those
decisions
and the those capabilities in the platform.
Yeah.
I think it's it's it was very much a a maturity of the audience we were talking to. I think when we started,
a lot of people were adopt I mean, that was what happened in 2020 and 2021
is a lot of companies were just revamping
their data stacks I think covid gave them a lot of breathing room to invest in this kind of project
and a lot of the people we are talking to at the time they wanted to have like a very easy solution just UI.
Now you know when you have an open source project like that you don't control how people use your software.
And what we started to see happening very fast was people
not using Airbyte for their own personal use case but just to create an application on top of it And they started to act around, like, our
terrible API that we had at the time and just trying to operate air by programmatically.
And even ourselves internally, we we have, like, we started to have more, like, needs for, like, like, workflow orchestration
and we could a ui does not does not is not sufficient
and very fast like API became something we had to do.
The piece around pyar byte is actually
it really shows
the power of how we designed
connectors as
very thin
pieces of code
because
you can't transpose them to any systems.
And Pyabat is an example of it of let's get rid of the platform all the heavy features and let's just use the connector itself within code and just pull the data from code. But for me it was really like a progression in terms of the maturity of the user which was first very early
BI
people
bringing or data engineers bringing,
warehouses wanting something very simple and then you start having people who develop application on top of airbag to or who have like more advanced use case with orchestrator and things like that and then it was oh now we have application developers that are building
with airbyte
and the platform
is good but maybe it's
too prescriptive for them like they don't want a UI they don't want an API they just want to have access to the data. So how do we serve
this audience? And payabite I think for me is just now we're in the software development land of people building application or people in notebook doing experimentation
and they don't have to take on like the the whole platform. But when they go to production, they probably need to have the platform because it has all the bells and whistles for reliability.
Absolutely. Yeah. That that's definitely a big piece
of my
understanding about the utility of PyAirbyte is that a big part of the value of the Airbyte platform is all of the state management
of being able to see when the syncs ran, making sure to manage the checkpointing,
the incremental state so that I don't have to rerun everything all the time.
But that point of being able to quickly test out the connectors in a development environment, as you said, brings a lot of value because you don't have to set up the entire platform just to test something out.
Circling back a little bit to the
LLM and generative AI application
ecosystem
of generating the embeddings,
it's very
straightforward to do that in a naive manner of just give me the text. I'll chunk it at arbitrary delimiters and just feed it through into the vector database.
Now that that ecosystem
has developed a bit more maturity and a bit more understanding about what are the appropriate chunking strategies,
how do I
associate
different attributes and metadata with those vectors so that I can do things like filtering. And now that we have things like graph rags starting to gain some attention,
being able to associate those vectors with graph nodes. I'm wondering if you could talk to
some of the challenges that you have of being able to understand
how best to surface those concepts
in Airbyte. And
when it's just a matter of it doesn't make sense to do that in air bite because there's too much customization
and application specific logic that needs to be embedded and so you need to move that into a different layer.
Yeah.
I actually think that right now
we know we still know almost nothing
that's so
and you see that you know we we talk to a lot of our like whether they are startups or
more mature companies
getting
the
most of the time they don't use platforms right now. They work and build everything very vertically,
and the reason they are doing it is
the pace of innovation and development cannot be blocked by
what
feature a specific platform supports.
And that's why you're seeing a lot of it built very vertically.
I think platform will come but we're probably 1 to 2 or 3 years out,
like,
ahead of of the of that release.
So
at that point,
air bite,
if you look at it from like UI perspective,
we can do the first POC.
We can start,
like, getting you to to get to that
like first
moment of like understanding
that you can get all that data with airbyte into a vector store. Now the thing that we really need to figure out is
okay
how is the world going to evolve because chunking maybe today we have something for chunking or for, like, embedding or, like, appending metadata
that will work for some of the use cases, but it will probably never work for all, like, most use cases. And that's where that's why we have
PayA Byte actually because
today
we just want to be there when people
look and say, oh, I don't want to build
a connector to bring data from, I don't know, Gong calls or whatnot.
We just want that that that we just want to be there and just provide Ihabite and then
be with them as they experiment because they they don't know, we don't know what they need today. So for us it's just about can we create
can we be there at the entry point and learn from how they're using it so that then we can bring this learning into the platform and make it a platform and at some point have that as a as a data movement platform for this type of use cases.
And that's really the the path we've taken where, yeah, we we try to address to be there once they develop.
Another element of
what you're building and how it exists in the ecosystem
is that the competitive landscape for data data movement has changed
fairly substantially over that same time period where
when you started out, it was the Stitch data ecosystem
was
kind of the option if you wanted open source. There was a high degree of variability in the quality. There was no
cohesive platform for operating those. Meltano came up to try and address that
situation and help to bootstrap that community.
On the commercial side, there was Fivetran,
who is still one of the more notable fully commercial options.
But then there have also been a number of other entrants,
most notably, I'm thinking of DLT as a very composable
data movement
capability. And I'm wondering how that shifting landscape
helps you
understand
what are the needs in the ecosystem so that you know how to build and address them as well as how to position yourself
for people who are coming to this problem fresh and trying to understand what are my options,
what are the differentiators, and how do I think about which tool to select to solve my problem?
Yeah.
I would say that,
primarily,
if you don't have connectors,
it doesn't matter what you do. Like, that's the value you need to get. You you know, when when I think about how we built Airbyte, there was like a ping pong ping on like connectors platform connectors
where we started with a ton of connectors
but like mediocre platform
then we invested a lot in the platform did not invest as much in connector and now that we have a platform that is running
at high speed high reliability etcetera now we're focusing
again on connectors
because
at the end of the day people
take what the platform gives you as a given they don't think that there is effort to be done in the platform when this is actually a lot of the things that we provide is hey you don't have to worry about data schema changes etcetera etcetera.
So
there will always be other players,
at the end of the days
people will pick another platform
if the connectors are not
the right quality.
That's that's gonna be the thing because
and and now that we have dominant player on like fully commercial
and and airbyte on open source, yes other player can come but at the end of the day people will make the choice based on the connectors that are available.
Now there there is there is a question around use cases
and I think the the thing we're looking to, to address with spyabyte is
how do we address use cases where it's more sophisticated
use cases
than pushing data. Like, it's a sophisticated use case, but more like downstream on on warehouses.
Here, the sophistication
comes from
what kind of preprocessing you need to do on your data. The fact that you're not using a warehouse to process your data, like, and Piabyte is here to just do this first discovery. And
I I think I think we have a strong community, a lot of a lot of connectors and and a lot of use cases that are addressed today. So
Absolutely. And also, as you look across the entire data landscape, it it's definitely not a zero sum game.
There are still so many people who are just writing bespoke code, spending all of their time just on maintaining those different point to point connections that
if they choose any of the tools, then it's a win for the whole ecosystem.
Yes. Exactly.
And, yeah. Honestly, like, the that's the
that's also I would say that's the power of open source is
preventing build
by providing I mean, I'm a lazy engineer.
Most engineers are a little bit lazy, not in the bad sense, but they just want to not have to build the thing that,
yeah, they don't they don't want to build, and they're gonna find open source. So the question here is,
do they start adopting Airbyte or do they start adopting something else? And I think today,
a lot of it is coming for Airbyte.
But yes we have to be very
ready and and provide like very strong like I don't want platform I just want the connectors because I'm building something that no platform can do today and that's why like having our connectors as code
in a code use case is very important.
Now
we've talked a lot about the history, the journey that you've been on. We've touched a few times on the fact that you're gearing up for this 1.0 release.
When you do commit to a 1.0 version, there is a signaling of a certain degree of stability and commitment to the way things are right now and the fact that you're not going to drastically change anything.
And I'm wondering if you could talk to
the decision process, the discussions,
the
confidence building that you've done both internally and the community that makes you feel that you're ready for that 1.0 version and what that 1.0 version is intended to signal.
Yeah.
We had a lot of conversation internally about when should we do 1.0. At some point, we're even talking about that in 2021.
That would have been a huge mistake.
The the thing about 1.0 is
it's really about we have hit a milestone
in terms of the technology.
Now technology
is not a milestone by itself, it has to be proven. It has to be proven by the type of workload that we are running,
the type of,
feedback that we are getting.
And,
you know, like, today, we believe that the platform we have
is
the
the foundation or, like, the the ramp for, like, even more data movement use cases. And that's what we wanted 1.0 to be.
And the proof points that we wanted to have was,
do we have
more than a certain number of, like, enterprise users? Whether they are paid users or whether they are open source users.
And and we got to this one. We actually hit that number,
earlier this year, probably, I think it was in April or something like that.
But that was something we wanted to see, like, critical
data
analytics pipeline being handled
with airbyte with a certain amount of scale.
The the other piece was around the connector quality because as I told you, we have this ping pong between connectors versus platform versus connector.
And for us, it was, do we feel good
about
the process,
the manufacturing
process that we have in place for connector? And today, we do. Like, the our ability to bring in more contribution from the community,
our ability
to react when a connector has an issue, the the our ability to just maintain that connector over time. I think today, we've hit the we've hit the number we wanted. And, yeah, and after that, it's just
diversity of workload.
We wanted to have, yes, the click
click the UI. We wanted to have the programmatic with API.
We wanted to have more of the CICD with Terraform, and we wanted to have Piabyte for people who are building on top of data connectors.
But that was really what we wanted to get with, with 1.0.
And as you have been gearing up to that 1.0 launch,
what has that meant in terms of
the
requirements that you have around new connectors being adopted,
the requirements around existing connectors
and their ongoing maintenance, the commitment to
which connectors
you are going to continue to support, which ones may still phase out and be deprecated,
and the commitments to
the
community engagement
and what they can expect from Airbyte going forward.
Yeah.
So
I think here we need to make the distinction
with regard to connector between sources and destinations.
Destinations,
our goal here is,
like, the moment we start building a destination ourselves like as airbyte,
this is something that we'll continue to maintain because we've also made the choice of
we see a lot of use like, a lot of users
and
enterprises asking for that particular destination.
At that point, it also becomes something that we, would charge for. So
the moments they are, like, air bite maintained,
they will be there for, like, probably ever. I'm sure there will be some exception, but,
that that that's the thing.
Now in term of, like, community or, like, marketplace
connectors,
we
at the source level, what we've done is we've deprecated a few of those
in the past and it was more about us
reinvesting into high quality connectors
and making sure that
as we build that engine,
we also simplify the work for ourselves so people can still take it
but right now in order to make that connector fit with the systems we are building it's too much of a lift and it would require a full rewrite to be compatible with the the level of quality we want. So at that point, we just deprecated them. Some people are actually taking them and just reshaping
and putting them back,
but
for us, that was a one time thing that we did this thing of deprecation. It was more about, hey, this is all the learning we've had. These connectors that were built in 2021,
we don't want to maintain them. And nobody in the community is doing it so we're just gonna remove them because they reflect badly on the platform and people who are using air bite. So we just we just remove them. But now it's gonna be very different. We're also going to we're pushing more of sort of the the metric that we look at internally
around, like, connector
quality and connector,
reliability.
So these are now all available on our documentation.
That's also the thing we wanted to do with 1.0 is
providing the right expectation for how is that connector behaving, how is it working
and making it easier much much easier
to modify and to maintain. And that's to me the
the the big thing about about 1.0 is like
we're not gonna remove connectors,
we're just gonna provide the right tools
to make them better and we are also we're also
providing the good visibility into the quality.
I'm not sure if it fully it fully answers your question though.
No, I think that you did a good job addressing it. I was mainly just trying to get at what are the
changes that people can expect
around Airbyte from the community side as well as the commitments that you're making with this 1 point o release and Yeah. Any substantial changes that need to be made ahead of that release to put you in a good position going forward?
Yeah. The the other thing we've done is we've we
part of the system that we're building, there is also, like, how do we manage community contribution?
You know I have a graph
and at some point it was showing how fast do we reply to PRs
and there was a moment where it was not good.
Now we got it down to maybe like a day or 2
but that's part of like making the making the process much much
stronger and our goal is to just
especially for connectors
like how can we get that faster into the into the mainline.
So and that's part of building that process, building that factory for connectors.
Another interesting decision that you made early on that it seems like has paid dividends
is that
you
are keeping all of the connector
definitions
from the source code perspective in a mono repo. So all of the connectors, all of the platform
is all in well, actually, now it's in 2 repositories, but it it all of the connectors at least are in one repository
versus being spread around GitHub and having to be recollected
via some,
website, etcetera.
As you move towards this one point o release, is that something that you intend to continue with? And what are your thoughts around the
possibility or viability of having some sort of,
kind of community
discovery platform for people who are building other connectors out of band of the AirPipe repository
or any support that you want to have about maybe third party connector libraries?
Yeah. Actually, we we do have some,
there is a company that has been building connector
by themselves, and they host them on their own repo.
So the thing that we're releasing also is one point one point o is this concept of, connector marketplace.
And connector marketplace
can be both for community members, but also for, like, vendors who want to build their own connector.
The thing that we would like to
but this is a place where also we're going to listen to what people are telling us.
Ideally, most of the low code connectors
should live
on the air by triple. And the reason we we we think it's a it's a pretty low lift is that one
now we can very quickly review this type of PR,
but also there is no security
concerns on disconnectors because they are low code. So we control the execution framework of these connectors. It's just a yammer file
that gets,
that gets executed.
Putting,
connectors
on
other repo,
It's a place where
either it's I mean,
would we put them on the platform right away? I don't know. Because at some point, like, we're dealing with data,
and we want to make sure that it's not just the quality, it's also the security of these connectors.
And And we want to make sure that everything that we run behind the scene to to validate
that
the connector is not doing something crazy,
like, we want that to run before it runs on onto onto the app by platform. But people can already do it. They can already build connectors on repos. It's just we want to vet the connectors
because
and
I've worked in data since, yeah, 2008.
Like, the amount of
like, data is a key asset
that every companies have very privacy centric sometimes.
We have to make sure that this data is always safe and, and that's what we guarantee basically with,
I mean, that's what we yeah. We guarantee with having that on on Airbyte. It goes for our security processes and reduce.
As you have been
building Airbyte,
growing the company, growing the platform,
and the ecosystem, what What are some of the most interesting or innovative or unexpected ways that you've seen the technology applied?
Yeah.
The the first one is one we discovered in, in 2022
was people just built a a Redis
destinations
and they were just using airbag to do a cash warm up. I was just, yeah that's a cool use case, I love it.
It's just you know it's this kind of thing where you don't have control you're just giving like a a knife and a and scissor to people and a piece like
like hammer and wood, and they just build something that you never thought about. The other thing, and this one became a product, was this powered by airbyte thing. Initially, we're just thinking of airbyte as hey. You're just using it to
push data into your warehouse, and people started to build application on top of it. Just, yeah. That's a cool use case. Never thought of that. And it became
we kept seeing people asking for, hey. How can I build on top of Airbyte? How can I build on top of Airbyte? And that's when we released
the first version of the API so that people don't have to click,
on on buttons and and fill form.
But that that's, I would say, were 2 very interesting ones because one led to a product, the other one just
nice,
very, very smart.
And we have a few others, but I don't know if you we want to cover all of them.
And in your own experience
of going through this journey,
building this platform, building this business, what are the most interesting or unexpected or challenging lessons that you've learned?
Yeah. First one was
when we started our batch, someone gave us an advice. 2 advice actually. 1 was
don't optimize for scale, optimize for time to value. The second one was you will never know who is using your software.
Well, we we followed the first advice which was around scale. We did not invest in scale initially.
That was very cool at the beginning, became a big,
a big boulder that we had to to push forward at the end, but now we're there.
The one around not knowing our our users,
the fact that we had Slack very early on,
the fact that, you know, sometime, like, very early on in the in the UI, we're asking for, hey.
What what do you do? Where are you from? Etcetera. Like, having
this ability to know the customer
or the or the user, whether it's on cloud or on open source,
that was just
it was possible. Like, people were willing to to help us by giving us, like, information like that, but it just gave us like a direct line to people and
especially
early on, like now we still do it but maybe like not at such a large scale but very early on every time we were thinking about oh should we build x or y
We just go
on on on on Slack or we would go and send a newsletter, like put that in newsletter
and within 30 minutes or one hour, boom, everyone was just telling us we prefer x or we prefer y.
So in terms of, like, product development and where you focus your team,
invaluable.
Now I would say that the thing that are hard is
building open source is hard
because you don't like, especially for infrastructure is you don't control who is using your
software, meaning that
you will have people
who face issues,
and they
will not be successful using Airbyte. And as a product builder, it pains me to be in that situation where I know that someone cannot be successful with Airbyte.
And, yeah, you have to live with that which is some people will
will not
be successful with it. But it's also because you cannot control with downloading,
and you cannot say, oh, yeah. Your use case is probably not a good fit for for Airbyte. And I I think that's the the the the place where especially as we're building open source, like,
how
how do we want to widen the
the net or the the spectrum of user that we want to be able to,
to make successful?
That's really the challenge.
And on that note, what are the cases where Airbyte is the wrong choice and somebody should either build something custom or use some other technology that's off the shelf? You know what? I would have had an answer for you
a few months ago, but now we released Pyervice.
So I would say
there is probably
oh, okay. Maybe for pre one point o, yes, there is one around, like, streaming type of use cases. Like, we don't do those.
Now it's planned for post 1.0.
But, otherwise, with Pyabyte now, you can do whatever you want. So you have access to all the catalog of connectors on code, and you can add all the logics you want on top of it. So,
yeah. Probably that would probably be the
the former gap and the and the no gap today.
And looking forward to that post one point o
world,
you hinted at streaming. What are some of the other capabilities that you have planned for the future of Airbyte once you have moved past this 1.0 milestone?
You've committed to stability
in the platform and the interfaces,
and so you have a more,
stable target to build off of and move fast on.
Yeah.
For us, it's gonna be very much on, like, operational use cases. I think we've been very focused on analytics, data warehouse,
use cases.
All the AI use cases are putting us into the direction of operational use cases, and reverse ETL is an operational use cases. Streaming,
in my head, only makes sense for operational use cases where it's not a human making decision but a machine making decision.
So that's gonna be a big big focus for us post one point o is like how do we address things that are not analytics but are also, like, operation,
driven.
The other one is we have a, like, a a large
road map around, like, enterprise connectors.
We're starting to release,
some of those, but we have a long road map and we're working with our enterprise
customers to build them. So we want to release them in the in the next few months.
Speed is gonna be a big deal. I think we have some low hanging fruit that we want to to make faster
you will see
we've made a few changes in the past few months like speed has gone up quite significantly
but I think we can still do a 10 times so like a time stamp somewhere.
So that's I'm very excited about. Like, data needs to move as fast as possible. We can never be a bottleneck.
We didn't really touch on it
throughout this conversation, but I'll also
say thank you for all of the work that you and your company have done on the outreach perspective of providing a lot of high quality blog posts,
conducting the state of data engineering surveys year over year to help collect some of that information and just all of the investment that you're putting back out into the community as well.
With that being said, are there any other aspects of the work that you've been doing on Airbyte, the 1 point o launch, the future that you're building towards that we didn't discuss yet that you'd like to cover before we close out the show?
No. But I think we're gonna have some new,
unstructured data is gonna be unleashed
and I'm very curious to see like how that's gonna pan out. We have some theory
but
it's basically you're saying that now 80% of the data that was very very expensive
to leverage is gonna be unleashed.
I'm looking forward to that world and, yeah, want to make sure that we're positioning Erbide also in that in that field because with data geeks, we want to push and get more data through through the pipes.
Alright. Well, for anybody who wants to keep in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes.
And as the final question, I'd like to get your on what you see as being the biggest gap in the tooling or technology that's available for data management today.
Yeah. I think the biggest gap is what do we do with unstructured data?
Like, the
it's gonna be a lot of things around, like, the security, the privacy of that data because,
well, it's not as easy as doing a regex on something.
So
to me that's a big that's a big deal today.
Absolutely.
Well, thank you very much for taking the time today to join me and share the work that you and your team have been doing on Airbyte. Congratulations
on the 1 point o milestone. I've been using Airbyte for several years now, so appreciate all of the,
value that that has provided for my own work. So thank you again for all the time and energy you folks are putting into that, and I hope you enjoy the rest of your day.
Thank you, Tobias.
Thank you for listening, and don't forget to check out our other shows. Podcast.netcoversthepythonlanguage,
its community, and the innovative ways it is being used, and the AI Engineering Podcast is your guide to the fast moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email host at data engineering podcast.com
with your story.
Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and
members.