Unknown: Hello, and welcome to The Machine Learning Podcast. The podcast about going from idea idea to delivery with machine learning.
Unknown: Your host is Tobias Massey, and today, I'm interviewing Kevin Stumpf about the challenges and promises of real time ML applications and the work that he's doing at Tekton to help support them. So, Kevin, can you start by introducing yourself?
Unknown: Yeah. For sure. Happy to, and thanks for having me.
I am the cofounder and CTO here at Tekton. And with Tekton, we help organizations
get real time machine learning applications into production with our feature platform.
Before starting Tekton, I was over at
Uber where I helped build the central machine learning platform team called Michelangelo.
Unknown: And do you remember how you first got involved working in ML?
Unknown: Yeah. The first time where I got really deep into it after, like, taking
grad school classes and whatnot on the topic was,
roughly 10 years ago now when I,
cofounded
a trucking startup. We were actually
matching long haul truck drivers with
shipments that shippers or brokers,
needed a truck driver for.
And we used machine learning to
predict prices, because oftentimes we didn't have accurate pricing information. And so we 1 of the most
curious things that truckers are interested in is, well, how much am I gonna get paid for? It slowed. And so we tried to help them out by, having an ML algorithm that predicts the prices.
And then through that startup, I eventually ended up at Uber,
where I, as I just mentioned, was part of the central machine learning platform team, Michelangelo,
which helps pretty much all the different
ML teams at Uber get machine learning into production.
And that has everything from predicting
ETAs
when you open up the Uber app to recommend
excuse me,
recommending which restaurants
you may wanna purchase from an Uber Eats,
pricing
predictions,
etcetera. Like, there's a whole long,
class of different ML problems that are solved with Michelangelo.
Unknown: And in terms of the focus for the conversation today on real time ML, I'm curious if you can describe a bit about what that is and how you might differentiate it from other approaches to machine learning that folks are using.
Unknown: Yeah. Real time machine learning should really be contrasted to batch machine learning or also sometimes called offline machine learning. Real time machine learning,
I'd classify
as any
application that uses machine learning
where the machine learning prediction
affects the end user
immediately,
right, in the moment where that prediction is being made and not hours later,
days later, or whatever it is. And
common examples here are, like, real time transaction fraud detection where somebody's swapping a credit card somewhere,
and it's either immediately denied or accepted,
or it's real time recommendations
where
the
items that you've just clicked on in Amazon
affect the next recommendations that you get on the page that you're gonna browse a couple seconds later,
or dynamic pricing systems, which
react to the movements of supply and demand
literally in the moment. Like, if you look at, like, the the ride sharing applications out there, real time bot detection, all that type of stuff. So all of these are applications where the prediction is made and the end user is immediately affected by it. And when you look at
the offline machine learning, that's typically where,
like, a a common example would be maybe you run a
a model
in big spikes in batches, like once a day, once a week or something where you look at all of your customers
and you predict their likelihood of churning,
And then you may send them an email with a coupon code or something like that.
Unknown: And
in terms of the kind of utility of real time ML, I'm curious if there are any common categories of problem that it lends itself well to or particular industries that have a higher predominance of using real time ML and just some of the ways
that people should be thinking about whether or not it is a suitable fit for the problem that they're trying to solve?
Unknown: The best thing to think about is actually, like, where
and you can use your human intuition honestly for this, which
is where do you have most of the information
that will help you make a prediction
about something that you wanna predict, like, say, a recommendation or whether you wanna detect fraud in a transaction. Is that information
very new, very fresh? Is that information that's basically just come into existence a couple seconds ago, a couple minutes ago, a couple hours ago? Is that the type of information that you, if you personally had to decide and personally with infinite time, had to make a recommendation without a computer?
Is that the type of information you would use in order to make that human prediction, so to say? If that type of information
has only come into existence in the last couple of minutes, hours, up to milliseconds ago or up until right now,
then that's typically a good use case and a good case for,
real time machine learning.
If on the other hand, you'd say, hey. The like, 99% of the wealth of information
is just buried in
what the user,
what I know about the user since the big bang up until a week ago,
then,
and I
only need to react to that information in batches once a day, once a week, or something like that, then that's a good case that, hey, that's probably that is is best served by batch machine learning.
Unknown: And as far as the
technical and organizational
requirements for being able to actually
take on this real time ML workflow, I'm curious what are some of the additional complexities or maybe even in some case, simplicities that get,
incorporated into the
supporting infrastructure and architecture
to be able to actually start thinking about these real time predictions?
Unknown: Yeah. When we look at the operational and organizational
requirements
before we we we look at the the the scope and the architecture,
of these real time ML applications, at the end of the day, real time machine learning really covers,
4 different parts.
1 is it it touches the operational system. That's the system
in which your microservices are running or the other production applications like your back end system, your postgres database,
all the stuff that cannot go down that's mission critical is the operational stack. That's 1. Then there's the data pipelines, right? Like somebody needs to actually calculate machine learning features, run training pipelines, make predictions,
and that typically sits next to your organization,
or close to your organization's data engineering efforts and other data pipelines, whether they're batch or stream. Then it also touches the observability
parts of your organization because you need to know whether your model is going off the rails or not.
And then finally, of course, data science is involved because somebody needs to come up with the ideas to how to train a model, what features to implement, etcetera, and then finally train the model. And so you've got these 4 things, operational system, data pipelines, observability,
data science, and your organization
needs to be set up to cater to all of those for real time machine learning. And so typically what you'd find is that you either have, like trios of people like a data scientist and, data engineer and a software engineer work together
or some lucky organization have them all baked into 1 in the form of a machine learning
engineer.
Some platforms, like, obviously, Tekton,
is trying to make that significantly easier so you don't need as many
folks with different backgrounds and whatnot to,
to actually power real time machine learning. But good indicators that your organization is actually ready for real time ML is that you can already spin out APIs and services with ease, and you follow DevOps best practices already, and all that stuff is is not very,
very foreign to you.
Unknown: And real time as a term in the technology
industry is always interesting because it's always so fraught and overloaded with concepts and kind of,
different applications of use. And so far, what we've been talking about with real time ML is focused on being able to take data as input that was generated recently and then generate a prediction from it. There are also elements of ML or applications of ML that might be considered real time as far as things like reinforcement learning where you're actually actively
updating the model itself with data as it is generated
or kind of streaming ML where, again, you're constantly kind of retraining the model, and you never actually go back and do a big bang, retrain the model, deploy it, and and so you're you're constantly evolving the model as it sees new data. I'm wondering if you can talk through maybe some of the nuance of these kind of 3 different categories of how you might think about real time in this context of an ML application.
Unknown: Yeah. And and at the end of the day, it really just comes down to terminology and what you include and what you exclude.
And personally, I include in real time machine learning anything where you have a model that's making a prediction
online,
even if it's just acting on precomputed
batch features. It doesn't have to act on, like, real time data or streaming data. It's totally fine if it just acts on pre computed batch features, for instance, to make a prediction in the operational online system,
and that has been most helpful in having conversations with our customers,
to to differentiate it from batch machine learning. Reinforcement learning are, like,
somewhat related is like online
learning where you're continuously
updating the model as it's running in production. I would generally classify that also as an online system. It's certainly acting online. That's in the operational world,
where it's continuously
acting on new information and updating itself. To be frank, it's something that I
basically never see in production used by anyone. It's super heavily
touched upon in the research field, but I think there's a scenario where research is, many, many years ahead of actually industry.
And there are some companies who of course use it. Google, I know Twitter has some, online learning systems and whatnot, but by and large, the the vast, vast majority of real time machine learning applications don't leverage these,
Unknown: these mechanisms yet. And as far as the actual model architecture of what you're deploying to be able to do these kind of low latency predictions, I'm curious how those constraints and requirements influence the, I guess, styles of machine learning algorithms that people are able to employ. So thinking in terms of, like, stochastic models versus deep learning, etcetera, as well as the size of the model that is,
kind of viable to be used in this context? So, again, thinking in terms of, you know, a random forest versus chat GPT or something.
Unknown: Yeah. Actually, the the the whether you look at the decision trees like your XGBoost
or
the linear models or the deep learning models, At the end of the day, that can all be used and they all are heavily used in real time systems. Yes. There is,
certainly
a
very big difference in the inference cost
and whether you need a GPU cluster under the hood,
or whether a a small CPU is enough to quickly enough, make the inference, but they can all be used for real time applications.
The size of them is, of course, different, and your JET GVT model is gonna be significantly larger than your good old, linear regression. But they're yeah. They they can all be used, but the operational complexity is, of course, a little bit more significantly higher the more complex the model is. And the other interesting element of using
Unknown: machine learning in this online mode versus a batch workflow is
the potential for things to go wrong where, you know, obviously, there are some notable examples of, like the Tay chatbot where it's continually learning and and trying out new things,
but also just in the case of, you know, maybe I have a bad batch of data that made it into my feature store, and so now I'm giving erroneous predictions or or recommendations.
I'm just curious how the kind of failure modes
and guardrails around machine learning models need to be thought about in this online versus batch context.
Unknown: Yeah. 1 of the biggest failure modes here is
train serve SKU, and train serve SKU for those who are not familiar with it basically means that you're training your model
on
data
that looks different in its shape or form
from the data that you make predictions with in the production system. If there's a skew, then your bottle model basically is not set up for success
because it sees completely different shapes and forms of data on the production system. So it doesn't know what to do with it. So the prediction is gonna be a a flip of coin basically. And it's very hard to notice that because you're not gonna get a exception, say from the model running the production system. It's not gonna know that it's never seen anything like it. It's just gonna do its best and make a prediction, and it's very hard to identify these types of issues. And so trained service queues is very easy to introduce with real time machine learning and significantly easier than with
your common batch or,
offline machine learning application. And, 1 funny example that I have here is we talked to 1 company
where 1 of the large consulting firms came in and it was a big bank, and they went in and they trained a model for them. I think it was a fraud detection model. The model was trained, put in production,
and then it just ran in production for
literally years, and it was never retrained and updated. And nobody actually looked at whether this model
is,
going off the cliff
or
whether you're
leaving a ton of money on the table by not retraining it, which is almost guaranteed to be the case.
And the reason why I'm bringing this anecdote up is that with real time machine learning, it's extremely easy to just not continuously retrain your model. Just train it once, you throw it behind an endpoint, and then you let it run-in production, rather than retraining it frequently.
Versus with batch ML applications. What I most commonly see is that a lot of companies actually, they retrain the entire model before they make batch predictions.
And it's just all the same Jupyter notebook or the same Python file that's running on an airflow. Just retrain the whole thing and make a prediction. And as long as it's not cost prohibitive to do to do the retraining step, it's actually not a bad idea to do that and to include the information that you have up until the point in time in which you're making the predictions. Besides retraining, another thing that's important to keep in mind is that with real time ML applications,
1 of the biggest challenges is that you need to calculate the features
both for your training pipeline, which runs offline,
and for your inference pipeline, which runs online. And so you need to be able to provide the feature values,
to your
real time machine learning model that's running in the production
system at super low latency. Right? Like, you need to be able to provide this feature in a handful of milliseconds with high availability
while offline, you just crunch through a ton of data
in
huge spikes as you generate the training data. So to make this a bit more concrete, what you would typically do if you have a real time feature or a streaming feature is in the
production system, you may run a Flink streaming
job or a Spark structure streaming job, or maybe just run some Rust code that's executing your transformation in real time.
While offline, you reimplement everything using, say, Snowflake SQL because all of your data is in your data warehouse. And now you have to make sure that the 2 implementations are identical because if they're not, you introduce train search skew. That's very hard to debug, and your model performance may be awful or not as good as it possibly could be. So those are 2 very common failure modes that are very easy to introduce with real time machine learning and best avoided, of course, with something like a a feature platform. And as far as that
Unknown: kind of train, serve, skew challenge, it also feeds back into the question of data quality, particularly in an ML context where the quality of the data is a huge predictor of the quality of the output model. And I'm curious how teams are addressing some of the question of how to,
you know, identify the critical elements of what quality data looks like for a given model and some of the kind of monitoring and ongoing validation that needs to be performed as new data as is fed into the platform so that you can, for instance, maybe, you know, shunt bad data into a dev letter queue to be reconsidered or, you know,
factored into updated experimentation
and to some of the aspects of kind of ongoing
validation and maintenance of the model as it continues to serve
without having
Unknown: to kind of take it down for maintenance or have it be spitting bad, you know, bad predictions for x period of time without being aware of it. When it comes to observability and monitoring with machine learning, I'd broadly look at 3 different categories. 1 is just data quality monitoring, the next 1 is prediction monitoring, and the next 1 is operational
health monitoring. Operational
health monitoring is what we already know from how we run microservices in production. Like, what are the what are the serving latencies look like? What does the uptime look like? In the case of a feature store or feature platform, what's the what does the feature freshness look like that I'm continuously serving? All of that, I would bucket under operational health where if things go off, oftentimes you have a, say, a provisioning issue or you're you're running on an unhealthy node or something like that. So that's super important to monitor because if that's going off the rails, then your predictions are not gonna you're not gonna get good predictions. Then there is model
prediction monitoring
where you wanna make sure that the prediction that you're making is still having a positive impact on the business situation
where where you wanna create value? Like, am I
are my recommendation predictions, are they still serving the business? Like, am I still
selling as many items as I expect to? Like, what's my click through rate on the recommended items? And if those are suddenly
if the click through rate is suddenly falling off the cliff, then you have a problem which may very well be related to your model. And then the third is data quality monitoring associated to it, which is like the what does the quality of the features look like that is being fed into the ML model? Then here, you can again bucketize, and you can either say, well, do I have full on outages
where my maybe there's no data coming in or the distribution
of my ML features is now,
or the really, the the statistics now look completely off. Maybe I'm suddenly multiplying
the user's age by a 100, and that's not anymore within the range the and the schema that the model expects.
So those will be full on outages. In the separate category under these data quality issues is Drift. Common problem in machine learning where the the data may still be real and representative of reality, but it's shifted over time
where,
maybe
your,
users' preferences have just shifted over time, and you really ought to now
retrain your ML model because your model just hasn't seen these types of distributions,
these types of means and standard deviations or categories of your ML features before, so it needs to be retrained. And so sophisticated
solution should monitor
all of these elements. Of course, that's quite a lot, and that's why platforms and tooling helps you. But those are the different areas where you wanna have visibility,
in order to make sure that your model isn't going off the rails. And
Unknown: for teams who are trying to build these types of systems, I'm curious what are some of the gaps in knowledge or understanding that you see them commonly encounter
and maybe some of the useful patterns that they have built up to be able to account for some of these error cases and failure modes
so that there is kind of a a general team understanding of how to address these systems and build them robustly versus the kind of the cowboy case of, hey. I built this model and put it in production. Isn't it swell? And then everything just breaks.
Unknown: Yeah. I think the worst things here are the silent errors.
The loud errors are,
you know, even if you don't think about them, you'll realize that when they're in production, and the loud errors had classified as suddenly your latencies
are going through the roof, and so you're not able to make
the the item recommendation
within the 200 milliseconds that the rest of your team needs it.
Or maybe the prediction isn't returning anything useful, and you're just clearly, you're recommending garbage. And those will be the loud failure modes, which are easy to detect, the silent ones,
where you're silently introducing train serving
skew because you're not monitoring properly
the quality of your data. Those are
really, really tricky to detect, so you better have a good data quality
data quality monitoring solution in place. And again, here, it's it's not enough to just have data quality monitoring solution because that may just show you that the your batch data that you use for training is of good quality and doesn't change, but you also need to make sure that you are, for, at prediction time, calculating your features the exact same way.
And that's why, of course, I I would recommend to anybody to use a feature store, a feature platform
to minimize the risk of introducing train serve, SKU simply by
incorrectly,
calculating the feature values or or having differentiated
implementations
between serving and and training. And so, takeaway here should be, like, data quality monitoring, super important. And then second,
have a a feature platform or a system where it's guaranteed that the implementation
of your serving features and your training features is identical because if you don't have these systems,
silent errors will eventually smuggle themselves into your system that are really, really, really hard to detect and very hard to debug.
Unknown: And in terms of the kind of featurization
strategy,
what are some of the useful tactics in terms of being able to actually
build the features in such a way that they are debuggable and composable so that you don't have some giant complex feature that you can do kind of a a surface level validation of, hey, This gave me a feature, and it's within a range that I think I expect. But then as time goes on and you're you undergo the issue of kind of inevitable data drift because the world moves on, You kind of waste a lot of time with debugging things until you get to the point of, hey. It's this feature, and I'm putting too many different conflated
requirements into it.
Unknown: Yeah. 2 things I'd say here. 1 is,
it's important to not only monitor
the quality of your predictions
and the output of your model to see, hey. Are they suddenly looking super different? Are the distributions of my prediction suddenly shifting entirely?
But it's extremely important to look at the inputs to the model because typically for every single 1 prediction, you have on the order of
tens, 100, 1,000, maybe even higher than that or of magnitude features, and you wanna know which 1 of those is
changing,
which may, at the end of the day, affect
the quality of your prediction. And so may even have some leading indicators here in your feature where you don't
model's performance. And so that's where modularity in a way or really more granular monitoring comes in. You just wanna break it up and look at the feature quality and not just the at the prediction quality. And then apart from that, you touched on modularity
and feature engineering. And here, of course, it's important to
try
to use a system where
you reuse individual building blocks
off your,
in your feature engineering system
where everybody doesn't reimplement
customer lifetime value or 30 minute transaction count feature on their own, but they all have the same common understanding of what the definition should look like and share it and reuse it. And then similarly, if you have derivatives of features, they shouldn't
they should depend on the literal feature implementation
rather than have a redundant and copy implementation of the upstream computation.
And so you should manage those features
in code, and feature engineering transformation should be managed in code backed by Git, etcetera,
and use common
software engineering principles
to keep it dry, to not repeat yourself,
and to reuse whatever you can.
Unknown: And I guess, in terms of the experiences that you've had of working with customers who are starting to go down this path of building machine learning systems, wanting to be able to react,
you know, promptly to input data,
what are some of the
kind of pitfalls that they have encountered and some of the useful mistakes that, you have seen people make that you and others have been able to learn from?
Unknown: Yeah. The most common mistake is just
always forgetting the train service queue issue
or to even
to start with a system where
you say, ah, it's gonna be good enough. I don't need to think too much about my foundation. I'm just gonna hack something up, put it in production,
and then you've got the stickiness effect of your application that's running in production. It's not implemented
using good practices, and then it's gonna be pretty hard to replace later on with good practices where you can avoid trained serve SKU. Another thing that's quite interesting is when you look at real time machine learning,
the
the real time data processing
industry as a whole is actually still
far behind the batch processing industry.
Right? Like, we
when you look at the sparks of the world, the snowflakes of the world, we're really good now at conveniently
processing large amounts of data. Like, over the last 10 years, 20 years, we've come a long way,
from the map reduces of the world,
to just being able to write a simple SQL query
that can easily process,
terabytes, petabytes of data, and we kind of forget what that all looked like
10, 15 years ago when you had to implement individual Java applications to do any map reduce type stuff. Streaming today is still pretty hard, and it is a significantly harder challenge because
you're dealing with fundamentally
unbounded datasets. Like, a batch dataset is bound. It's limited. You know exactly how many rows there are in there. Of course, it can be growing, but you can arbitrarily slice it while streams are,
unbounded. And so when you think about things like stream aggregation, stream joins, etcetera, those are hard problems.
And there are a lot of different players who are out there and who are trying to make stream processing easier,
but those aren't really deeply proliferated
yet. And,
that makes it harder to introduce,
specifically streaming features
in machine learning applications. Of course, TechTown, we we have our own answers to make that simpler for customers to express common streaming,
machine learning features. But as a whole, that has just been a pretty interesting realization
after having come out of Uber where we had a pretty sophisticated streaming stack on how immature the industry generally still is.
Unknown: And in your experience of building Tekton and helping customers through this journey of being able to build and deploy these real time ML systems,
Unknown: is
is just like a a really good cause
that can actually save lives, which I really like. And apart from that, I'm also always excited to see and sometimes surprised to see when I
see safety applications of machine learning. So it's not just about fraud detection, but it's literal
end user safety, whether that's the safety
of a rider of a of a of a Uber trip or the safety of someone who's playing a game where you can literally protect whether somebody is could be potentially under
in a in a in a potentially harmful
situation that you may wanna do something about. And those use cases aren't all too commonly talked about, but I love seeing those. And then maybe finally, 1 thing that I also find always interesting is when I look at customers who use, feature stores is
oftentimes what comes before you have a machine learning model that's running in production is that you actually have a
human implemented heuristic, a rule that's making a decision based on input data, which can actually get you quite far as well.
And these rule based systems, they also need to act on data,
that's streaming data or batch data or,
real time data. And we see customers actually use Tecton as well to power some of their
rule based systems before they're ready to
upgrade those to, machine learning driven systems.
Unknown: In your experience of working in this space, I'm curious
what are some of the
notable evolutions in terms of kind of capability and functionality
and some of the,
future potential that you envision once we do do get to a point where these kind of streaming data systems are more on par with batch capabilities?
Unknown: Yeah. The the future that I have in mind is 1 where
you can
look at an organization's
data systems,
and you basically see this giant web of,
with nodes
and edges that connect to different nodes. And
every every,
node is a data consumer or a data producer, and the edges are the actual data processing.
And it doesn't matter whether it's streaming or batch, but you've got the full lineage graph. You see exactly where the data is coming from and where it's going to, what's failing, what's succeeding.
It doesn't matter whether it's,
analytics
feeding into a dashboard
or whether it's a machine learning feature that's feeding into your fraud application.
You see exactly how things are shared. That's, the vision that I and the others have in mind that we're all working towards where you just have this fully integrated view of all your different data producers, consumers,
transformers.
And we're, of course, far from that where even today within the same team, you oftentimes don't have this visibility,
let alone across teams in the same organization. And maybe at some point down the road, we can even have this type of visibility across organizations
and not just within the same organization.
Unknown: And for people who are considering
whether and how to apply ML to their problems, what are the cases where a real time approach is the wrong choice?
Unknown: Real time ML is
not the right choice if you
don't care about the information
that happened in the seconds or minutes or hours leading up to the prediction. Like, if it's 1 of those use cases where it's more about, hey, everything from big bang until yesterday
and that'll always stay this way. You don't think that you'll get much more juice out of the prediction by bringing in more recent real time information.
If that's the case, coupled
with you making these predictions and acting on them in in batch, like once a day or once a week, then that is a batch machine learning use case, and there's no real need
to force,
real time machine learning complexity on top of that. If on the other hand, you need information up until the moment where you're making a prediction
or if you are
Unknown: acting immediately on the prediction as you as you make it and it affects the end user, then real time machine learning is a good choice. Are there any other aspects of the kind of real time ML ecosystem and applications and the ways that you're working to help support those capabilities that we didn't discuss yet that you would like to cover before we close out the show? I think we went pretty broad and comprehensive
here. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team at Tekton are doing and the kind of future direction of real time ML applications. I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest barrier to adoption for machine learning today. The biggest barrier
Unknown: for adopting real time machine learning is typically that
companies don't yet have the right tooling and the right platforms in place
to actually make it easy to
develop these real time machine learning models and features
that can be consumed
by ML models that are running in production.
That's, of course, why self servingly, we're we're going after this market, and we're trying to make it significantly easier for enterprises to
have a system that
Unknown: is easy for them to develop real time machine learning applications with. Alright. Well, thank you very much for taking the time today to join me and share your experiences of working in the space of real time machine learning and some of the ways that it can be applied and how teams should be thinking about it, potential for their problem domains. Appreciate all the time and energy that you and your, team are putting into making this a more tractable and approachable problem. So thank you again for that, and I hope you enjoy the rest of your day. Thank you. Thanks for having me.
Unknown: Thank you for listening, and don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest in modern data management,
and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at themachinelearningpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at themachinelarningpodcast.com
with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.