Unknown: Hello, and welcome to the Machine Learning podcast. The podcast about going from idea to delivery with machine learning.
Unknown: Your host is Tobias Massey. And today I'm interviewing Erez Kaminsky about using machine learning and safety critical and highly regulated medical applications. So, Erez, can you start by introducing yourself?
Unknown: Yeah. Thanks for having me, Tobias. It's a real pleasure.
So my name is Erez.
From Boston,
I, I'm the CEO of a company called KETRIX that helps
regulated software
teams perform CICD
and just develop fast and stay compliant
and make safe and reliable applications.
I spent,
basically most of my adult life working on high reliability applications and
the nuclear engineering domain
and medical devices and pharmaceuticals. And kind of most recently before doing this, I I was leading AI for
Amgen, which is the largest biotech company in the world,
as a medical device division. So I've been thinking for a long time on how to do this and then kind of classical developer.
I worked at a big company developing this very complicated thing and said,
you know, what if I had better tools to do this with? Are there other people who feel the same way and left to to build the tooling company, really? And Ketrix is a software development tooling company
Unknown: for highly regulated industries, including kind of regulated AI and so on. Before we get too much into the bulk of what we wanna talk about today, I'm also interested in how you first got started working in the machine learning space.
Unknown: Yeah. So really my background is I'm I'm a I'm a physicist, and and I worked in in physics and
helped to build numerical simulation
applications. That's kind of where I always say that I'm not a software engineer. I'm just a person who knows how to build fancy calculators,
in low level languages or high level languages.
And so that started an interest, right, because numerical simulation, you get really into computing, into
creation. The time is, like, 2014,
2015, 2016.
So AlexNet
made a bit a big impact. You hear a lot of it. And even in the research community I was in, there was starting to be conversations of can we accelerate certain numerical simulations with, neural networks.
But the real jump for me is,
I went to work for a company called Wolfram Research that develops Mathematica and Wolfram Alpha,
So a lot of both kind of more classical expert system work, but also more kind of modern machine learning work. And there, I got exposed really to everything that was going on there as part of their machine learning group and and helping to support them developing products and,
you know, we do a lot of different things even even back in the day, which is not too long ago, but in machine learning time, you know, even 5, 6 years ago is is eons ago. So that's how I first got involved, and then I've been around the Boston machine learning community for a long time and helping different
government private
establishments
do machine learning, get trained on machine learning,
and then did a lot of work at MIT around machine learning specifically for health care and biology.
Unknown: And
the
idea of safety critical, highly regulated systems is
a massive area of active research and development in the software community,
and I can only imagine at the additional levels of complexity that are involved when bringing ML into the equation. But before we get too much into the specifics,
I'm interested in getting a description of what some of the regulatory
burdens are that are that exist for ML teams who are building solutions for medical applications.
Unknown: Yeah. And and I think, you know, I'll start more in general than that even, but we'll get into medical. I think that
there's basically 2 type of things in the world. Right? There's things that you could just build and no 1,
focuses on, no 1 cares about, there's no public pressure about them, they're not society critical, and there are things that are.
And generally speaking, society critical or or just critical things are regulated. There's a a wide range of what it means to be critical. Are you mission critical? Are you safety critical?
And with that, kind of,
the burden of regulation and the burden of evidence goes up.
But, generally, everybody's asking for the same type of thing, which is
if you are claiming that this system or application you built can help people
in a moment of need with something they need to rely on
to
make sure their finances are okay or make sure that
they are safe from harm,
there's usually a regulator in the world that regulates that for every jurisdiction,
and we see that's just growing, both kind of the amount of regulation but also the scope of what is being regulated, and a lot of the discussions over, I think, regulated AI touch that.
But fundamentally,
all regulations that deal with product, there's regulations that deal with your company, and that's more,
you know, are you compliant with SOC 2? Do you have proper
privacy controls overall?
But then there's regulations that deal with your product,
and and that's our focus is basically how do you know that a system
is doing what you said it did or what you said it was going to do.
And at the end of the day, all the regulator is asking people to do is have a long document that describes for
a less educated
observer, right, someone who's just coming to audit you or inspect your facility,
that explains what your system does,
in a very robust way, and the robustness really depends on how dangerous your system is, like very dangerous systems more, less dangerous systems less.
And then
have evidence,
that those things are being are being met, Those requirements are met by the system for every version of the system.
And while that sounds very easy, just kind of share with us what you do and then show us that you do those things. If you've ever
built any web system or any complicated software system, you realize that's really, really challenging to do and very, very difficult to do and takes an immense amount of time.
Now machine learning introduces into that problem the question
of what does it mean for something to do what I said it does.
Right? Like, it's a statistical model.
It will always have certain
parameters in which it operates correctly in and certain parameters in which it doesn't operate correctly, and then you need to try and assess. And and
in the same way that you validate your software system to meet certain requirements,
validate your machine learning system
or subsystem,
as the FDA calls it, to meet certain performance parameters
and be consistently meeting them. And I think that's a very challenging environment for people who grew up building mobile apps, ad tech,
even Fintech,
to try and do. Like, it's so hard to do that. It's so hard to make something reliable
and meet certain requirements, and machine learning just adds some complexity. And I think a lot of that complexity
is about the subject matter expertise more than the regulation, frankly.
It's just like, what does that even mean to show that a neural network does what you said it does? What is what do you say it does? What do you think it does? You know? Absolutely. Yeah. Model validation is a hard enough problem on its own without having to do it in a provably correct manner.
Unknown: And before we get too much even into that model development, model validation, model design question,
in terms of
security
and safety in the software realm. There's also the whole question of the software supply chain and being able to manage and validate the dependencies that go into the the system. And then with ML, there's also the questions of the
accuracy and reliability and the awareness of and correction for bias within that source dataset.
And I'm wondering what are some of the ways that the combination of software supply chain along with data supply chain come together to compound the complexity that you're trying to address?
Unknown: That's such a good question, Tobias, and and it's actually the the first place where we started to work on on our company and and the product is how do we help people integrate open source software and, you know, kind of third party software into the regulated applications because it's it's complicated and
people didn't use to develop
with that much open source in the healthcare community,
medical device community until kind of recently,
and that's caused a lot of issues, right, because if you developed a medical device like 20 years ago, even a very advanced medical device, it would mostly have embedded software
and a a very small amount of dependencies.
And now, you know, you wanna spin up, some image classifier
with TensorFlow. There's already a lot of dependencies going on in there. Could be, you know, in the many thousands easily.
And I think there was it's it's it's not an impossible task. It's a task of of
understanding what your system does and how these different dependencies affect the safety of the system and the connection to harm,
and then being able in a sophisticated
manner to triage that based on both the security aspect aspect
of that dependency,
but also how the dependency works inside your validated application. And I think, you know, that's very challenging for the men mentality of software teams where, you know,
you just kinda put whatever dependency you want in, whatever makes it work, and let's move forward.
But here, we really need to understand what we're doing, so that's why we we sometimes need to stop and think about it and make sure that, you know, we're not introducing a dependency
that,
would hit our performance or would get our end users exposed to unnecessary harm.
And we have a big offering that helps do that, manage that softer bill of material and that softer supply chain because it's a common attack vector
into all kinds of products. Right? And then there's recent study that says that kind of the average
cost of a cybersecurity hack in health care is about $11, 000, 000 for a company,
and that's just gonna go up and up and up.
So it's something that I think people don't think about enough, what I'm introducing into my application,
and they need to do it more. And we've developed a lot of both kind of best practices and software to think of how to do that and how to
connect back from the things your system can do poorly to harm another person
to what you're using to develop it.
And most of it, like you can imagine, is just saying, you know, have we thought about this? Do we know where we're using this? And is the thing we're using it in high risk or low risk? If it's low risk, maybe we need less attention to it. If it's high risk, we need more, and just to have some kind of procedure and thought process. And and I think the new
guidance from both FDA and Congress, which is part of the patch act that passed in December of last year,
has a huge focus on that. Right? If you build a medical application, particular medical device in the United States by October 1st this year, that device has to have a very robust software supply chain methodology.
Unknown: Another aspect of the security question before even touching on the complexity of ML is also the operating environment that you're dealing with, which the software bill of materials is 1 aspect, but then there are also questions around system hardening.
Does it need to run-in an air gapped environment?
Who has access to these systems? And I'm curious, again, how that factors in with these regulated and safety critical systems.
Unknown: It's a huge factor. Right? Because, again, it's just another vector that introduces risk. Like,
let me give you an example. It's a pretty interesting
recall by a large medical device company recently.
They
developed insulin pumps. That insulin pump, had a vulnerability
exposed to it that would allow a user
to take control of certain aspects of the pump. As far as I recall, it wasn't actually something super dangerous, but, you know, if I if if, you know, my child had an insulin pump, the very idea that someone can connect to it
makes me extremely uncomfortable and and makes me not wanna do it at all or change your brand. But because they had good risk controls, and this is where you do see big companies doing a good job with these processes,
the risk was actually quite small because they mitigated it enough from an architecture perspective that you couldn't really hack into the device from the Internet.
You could maybe hack into the device maybe if you were can connect to it via Bluetooth and you were close enough.
But I think, you know, as kind of a white hat type person, that's a totally
different paradigm. Right? Like, can anyone in the world access it now or just people that are in my vicinity?
And they announced it and they got back to all the users. And To me, that was an example of, look, this stuff works, right? Planning everything
works, but yeah, you need to have the context of where your device is used,
what other devices
it's connected to,
and then have an understanding
of what could go wrong. And
if it's a pacemaker, I think it needs to have a very different architecture than if it's an image classifier for radiology images,
right, just because of how immediate the risk is.
And I think being sophisticated about that is just part of developing
serious systems. And if you wanna build serious software that's deployed in serious application that people depend on, there's just an amount of work that goes into that.
We often hear it just to spin this around in in the side of like, should I run this highly safety critical process on the cloud and the cloud is out of my control, right? And AWS or Google
or IBM, they do a bunch of different things in the cloud while you're working in this kind of server farm,
and maybe some of those things can cause a latency issue or it can cause a performance degradation and you know, our response is, well, you know, maybe you shouldn't put something that can kill someone on a cloud. Maybe that should only be on the device and the way it's sequenced with commands,
should be much more hardened, as they say in the DOD space.
Unknown: In terms of that risk evaluation
framework, you mentioned that the level of
control
and the level of validation
is dependent on whether it is seen as as a high risk or medium or low risk. I'm wondering what are some of the factors that come to play to determine
what that level of risk happens to be
and how much,
education and awareness you have had to do with developers to understand what are those heuristics to identify.
This is high risk. I really need to be sure of x, y, and z, or this is low risk, so I can ignore b, c, and d.
Unknown: Yeah. And isn't that the question of the century, I think, is what does it even mean to be high risk and low risk? And I think we're gonna see a lot of regulation roll out in the AI space around that that, yeah, you could still build little widgets and apps that don't do anything, but if you're making decisions about
loans that are gonna substantially impact people's lives,
well, it better be
not biased as many models are today, and we already know this. And I think that's gonna be a good question. How do you triage? What does it mean to be high risk? I think it's gonna change away from a paradigm around harm to a paradigm around
harm that's not physical, harm to people in the abstract and I think we're already seeing that with social media companies, right? Like I know a lot of people who have kids who are like, my kids aren't getting on Facebook and Instagram until, you know, they beg me to be on Facebook and Instagram, and I can't deny them anymore because we don't we don't feel it's a safe place for people
because the companies are not worried about this. But fundamentally, it's all about risk triage, and the way you do that is by saying there's an element that's high risk and what does that mean to have risk? Risk is a probability, is a calculation
that is the result of the severity of
something happening, usually it's called a harm,
not getting drug infused to you or
falling off a staircase, that's a harm, right, and then people put risk controls to prevent that harm like a rail or something like that,
or empty slip kind of little patches on the floor.
And then the risk is really a result of the severity,
how, you know, could it kill me? Would it cause a severe injury that
I would need to get hospitalized for? Would it cause a light injury
or would it cause no harm? That's usually kind of, the different aspects of the adverse event that can happen.
And then there for physical devices, this is where it gets a little complicated, there's also a probability,
the probability of harm and the probability of that harm actually happening, right? So there's like the probability that I will get in front of the stairs and there'll be water on them and then I'll slip,
So the probability that there's water, let's say, you know, the stairs are outside, it rains so many days a year, so 1, you know, 30% of all days in Boston, there's probability of water.
Then if there is water, what's the probability of me actually slipping? Let's say it's 1 in 5. And if the risk is quite low, maybe overall, we say now that the first probability, p 1 times the second probability times the severity,
we can do a little matrix and calculate it and say in our risk management, it's either too much or or fine.
Right? The issue is in software, you don't have that aspect.
Because you don't really know when harm will appear in software, people assume that the probability is usually 1. And so you're only trying to mitigate the severity
with most software things because,
who knows will my child be exposed to bad content
on Instagram? I don't think they can produce a probability of that. And so it's mostly about the severity and then what risk controls could you put in place. And I think the future is really about
teams understanding what the product does,
the possible risks it would have, and then ways to mitigate that risk, which is, by the way, a little bit different than how a medical device thinks about it because a medical device says you don't just need to mitigate
risk. You also need to have evidence
called validation that the system performs what you wanted it to perform. But I think in some cases, it's gonna be less about proving it works. It's mostly about proving it doesn't do bad things for most common applications. But does that make sense of how you would think about it? Starts from the end. Absolutely.
Unknown: And another
layer in the case of ML systems also is how much authority does it have to take an action, where
is the model able to take direct action on some other system that will have an impact? Or
is this purely a an informative
output where a human is the 1 that's going to be interacting with and interpreting it? But even in that situation, there's also the question of ensuring that you're surfacing the level of confidence
and, addressing any potential for bias where in the radiology
example that you gave, what is my level of confidence that the,
scan of this X-ray actually includes a tumor
versus just some other obstruction
or abnormality
in the physiology of the patient. And
if the,
if the level of confidence isn't surfaced and the practitioner
just assumes that it is going to be accurate,
then maybe they're taking an action that is not properly informed, and they don't do enough of their own due diligence or rely on their own expertise to actually critically evaluate what the model is telling them.
Unknown: Yeah. And I think that's the most common risk control for machine learning systems is that there's a human in the loop that can prevent any action, and that's why most ML systems today in medicine
are really not taking any actions.
They're the vast majority of them are suggesting certain things to focus on for a physician. Vast majority of applications in medicine, no surprise, are image processing, image classification applications. That's happened kind of post AlexNet. Now there's gonna be a big boom in in language models for sure.
But the biggest way they, you know, mitigate the risk is by the severity. Just if there's you know, you have to it's just suggesting to a person and just helping them focus, but the person makes the decision,
it's a lot less risky.
But it it'll be impossible for us to keep doing that forever. Like, eventually, the whole power of AI and machine learning and software, because because I think it all part of that is to automate,
things in our lives, and and we will,
have to do that.
And we just need to figure out how we do it in a safe but innovative way. Right? Because in most things, innovation wins, but in medicine, you need to have innovation, but needs to be safe.
Unknown: And in that ML space, bringing it back around to the design, development, and validation of the models,
I'm wondering
what are the ways that the regulatory
requirements
impact the approach that teams take to
developing and testing their ML models or even their willingness to invest in machine learning given the additional burden versus
other ways that they might use their skills in the ML space?
Unknown: So I think a lot of what's needed in the market is folks who really understand
the aspect of validation of the machine learning models. And
I think there that's gonna be a big need in a lot of time spent by folks, trying to find people like that. They can help the regulatory folks in their organization, the quality folks
understand
how do you even test if this model works. Right? You need a subject matter expert, and I think that's where there's gonna be a big gap.
Not that many people have approved the medical device that's based on AI, and so the amount of people who know how to do that is not very high,
but that's gonna grow and grow. So I think that's, like, a major focus for us is, like, how do we help companies
reduce the the need for so many SMEs
that can't be found
in order to accelerate their innovation?
Unknown: For the
development process, another aspect that you mentioned is the question of auditability.
There's record keeping requirements.
I'm wondering if you can talk to the ways that the
development teams can structure their work to
reduce the
number of procedural
tasks that they need to add on to their existing development work. Just being able to make that something that happens in tandem in some of the ways that,
your product at Ketrix is working to, further reduce that,
kind of managerial burden of the work.
Unknown: Yeah. I'm trying to to think you even have to answer that because it's a very complicated question. Right? The whole issue is that the regulator
in general, both FDA and others, they regulate the total product life cycle. They wanna regulate everything from the early design, the early risk management,
the production,
the testing,
deployment, and then, you know, there's a huge area called post market surveillance, which is what happens after something's in the market,
and all of that is a big cross functional team,
and we believe that that team wants to do that work through commonly used development tools and we just enable them to do that instead of using some system, some legacy system that, you know, you can't even find a developer that knows.
And so
the way we see it is is actually a collaboration between people and computers. We have a lot of AI in our system
and
that AI
basically guides people onto how to develop things
based on their existing processes.
It could be configured to them and can suggest stuff and can help them author certain documents,
and then through that,
we just reduce the amount of of communication
and amount of lack of knowing what needs to happen. A big challenge in developing something that has so much process control
is that what if you don't know the process. Right? What if there's not that many people in your organization that know
anything about the standard operating procedure to develop this type of thing and you need to wait on that SME to to keep working? So we're trying to have that SME partially in the computer
because I think it's an iterative thing because if you wanna develop
fast moving, large, complicated web systems that are learning,
you probably need to have something similar to that helping you build them.
And that's what Ketrix is really, it's this mind that sits behind all the different tools and and helps guide the overall process to meet
some existing
standard.
Unknown: Given the need to integrate with that suite of tools for the,
overall life cycle of the model, I'm curious how you're approaching
that from the design and development of your platform, given the fact that the ML space is in a state of high volatility because of all of the activity that's happening, particularly around LLMs, but just in the overall advancement of machine learning as a discipline
and the fact that there are still a lot of,
as yet to be decided best practices?
Unknown: So I think we you know, our approach has been to start from the things that we know matter. Like before you even wanna work on AI, first you need to know how to build software,
right, and we use Kedriks to develop Kedriks. It's recursively developed in the same system, which allowed us to do that. So we started by first, can we help people build large complicated software systems
And then we use that ability to build our own system and to integrate it with a lot of different systems. And now,
we kind of
go to the tried and true things. In some ways,
being in a regulated industry helps you with that because
it's not like everybody's gonna switch on doing the latest cutting edge thing in medical devices.
There was some kind of
time till that echo
of the innovation reaches the regulated industries. And in that time, we
we can kind of see how things are developing and be that gate
that, ensures that the tools
that are kind of well suited and the best in class tools and also the best in class processes
can be used for that.
And so basically, we integrate with what our customers want and what people want the most and also what makes sense for us, and then we're slowly building it up as we connect to more and more things. But I think, you know, there's a lot of needs before we even talk about large language models that are not met yet. Like, the vast majority of companies can't even deploy in a monthly or weekly fashion, which is
very outdated compared to how most software is deployed today.
And and we're very much focused on enabling that core technology that would let everything else happen and also build the tools that would allow us to do this. And for the
Unknown: validation
aspect, there are
a number of different approaches that are being developed and explored,
things like Shapley values for being able to understand what is actually happening at each of the phases of the model, the idea of causal modeling
using Bayesian statistics as opposed to stochastic methods. I'm wondering
how you have seen teams try to
tackle that question of how do I either understand what my model is doing or explain why a model has made a certain decision.
Unknown: And I think that's the the, in some ways, the hardest part of this entire question.
Because even if I solve it for you to some specific case, it would always depend on your system and not just
on the technology you're using. Right? That's kind of the developer's perspective. It really depends on the intended use.
I think there's
every 1 of those aspects of the error generation
and forecasting of what, the way the failure mode could happen and if it emerges, right, like if the loss goes down, maybe it's starting to get intelligence and all that discussion.
I think it would always be kicked back to the manufacturer and that's actually very clear in the regulation that the manufacturers are just gonna take more and more responsibility and liability about that.
And,
basically,
whatever is right for
your use case and your technology,
the regulators would expect you to have an SME in house that is able to understand what that is and what's the appropriate controls to put in place and that you've put those controls in place. And I think it could be that as those controls, I think, even, it's gonna be a combination of things mostly tracked over time
because I think that any any person who's built a lot of validated or high reliability systems will say, in some ways, it's not about what I choose to measure.
It's about
if I accepted that measurement in the first go around, how does that measurement change over time?
Right? I'm kind of, in some ways, more interested in that because I know that it's iterative, and I know that things change in in medicine,
let alone the software can be wrong. What about the clinical model? What if the way I'm treating the disease is totally wrong? I don't understand the biology. So I have to kind of always monitor it, and I think it's to me, it's less of a question of the particular error. Like in forecasting, I did a lot of work in supply chain forecasting forecasting of all kinds of things.
There's a lot of different arrows you could use. I think the most important thing is just be consistent to make sure it's appropriate for your intended use. Like, if your intended use, it's not a good idea to have
an absolute value because you need the negatives, maybe it's a bad idea to have an absolute value in the way you do it, maybe it's fine.
But as long as you pick something that's appropriate for your intended use of
Facebook profit or whatever you're using for forecasting,
then it's
basically more of a of a question of has my model become worse or better over time.
And I think that's that's where the best practices
will will emerge
of, like, how do you technically do that? But I I
in my view, is I just less focus on a particular methodology and more of a way to develop products,
in general. And then every organization, every product
Unknown: is so unique, and they need to do whatever fits for them. Yeah. Also, that question of change management is interesting because up till this point, the we've largely been talking as if we're doing an initial builds and validation and deployment, and that's the end of the story. But for anybody who's ever worked on software systems over any period of time knows
software is constantly evolving. Machine learning models are constantly evolving both from the software perspective, but also from the data being fed to it.
Your example of a change in the clinical methodology
for treating a particular symptom is also very interesting because that introduces
another potential avenue of drift. It's not just drift in the source data that I have, but also drift in our understanding of the world from a perspective that is completely outside of the software and the data that I'm working on. And I'm wondering how that question of change management and model evolution factors into the regulatory requirements around that life cycle management.
Unknown: I could I could answer this question for an entire lifetime.
But I I that's the core of it. Right? And that's why that's like my thought is when I got into a fairly regulated company coming as a developer,
I felt that as I went to industry conferences and saw the industry works, I was like, well, this doesn't really align with how anyone does it today. Like, there you can't just not change the software for such long periods of time. Like, it's not safe even if you can't change it very fast, which is why the FDA came out or the congress with a patch act to say you have to have a way to have hotfixes. We it's not legal in the US
right now to have a medical device that you don't have a robust hotfix patch process for. It's not just frowned upon. It is no longer legal to do that. There's an, you know, a law about the matter called the patch act. We meet a lot of teams, especially innovative young teams that are they think that, like, creating the first validated version is their challenge and how do you create the submission,
but that's never the problem. The true problem is how do you create
consistent,
validated versions that are referenceable, that are well recorded.
And we built an entire system to deal with change because you make it once, but you change it forever, and it grows and you add stuff for it. And I'll also say that our developers over time, the people who work for us, kind of feel that this is an approach that's generally good for all software. Like, just to have a little more structure and think about what you're gonna change instead of just changing it. But, basically, the the requirement
is
you need to know what you're changing. You need to know how that change affects your system. You need to make sure that it's not adding risk. And if it is, you understand why it's worth it. Right? You can always have a benefit risk analysis that maybe the the benefit is greater than the risk. But I also think regulation is changing to allow that more,
but, again, not in the sense of allowing you to do changes without any record keeping. That's
very unlikely to me, that that's gonna be any future state in any high risk industry, that you could just change stuff all the time
without any reference. The the main thing is that,
changes are gonna be allowed more frequently by the manufacturer, pushing more liability to the people building the software.
And then,
basically,
there's now this new regulation
that came out of the FDA. It's in draft right now. It's the AIML draft guidance.
And in there, it says it was just released a few months ago. You know, basically, if you can segment your system well enough,
for example, as a microservice and have the machine learning model as a separate piece, maybe you can preapprove with us the way you're gonna change it as long as there's a plan, as long as you can explain to us the decision points in general, it doesn't even need to be that specific that you're gonna make to change things. And the fact that you have a way to change
that's gonna regenerate proper documentation and evidence over time, I think that's how
the FDA sees it that you can change as long as you know what you're doing. And I think that's really, really hard for folks because, you know, you usually think of change in in, like, the Git perspective. You only need to change the code.
But frankly, in validated systems, code is just 1 of many, many, many different design artifacts
that need to be produced to show that the system works as you said it does, and
we kind of have this innate trust that health care and medical device and pharmaceutical companies are doing this because we would never trust them with their children's lives otherwise.
And so
change is kind of a core part and
just maybe a finishing note, why is is change such a big part? It's because in the nineties, the FDA did research
and showed that 20% of all software recalls in the United States are software related. And out of that 80 percent 20%,
80% of them. So
it's like 16% of all recalled in the United States are a result of changes to the original system
that caused harm or caused something that
required a recall.
And so change is kind of inherent to this, change is inherent to software,
and I
think the kind of
best in class assumption right now and especially the highly regulated industries, the change is not gonna happen very often, and it should be avoided. And I I I I just don't think that's true. I think change is a blessing, and we just need to do it in a controlled
manner.
Unknown: Yeah. And that is an interesting
parallel as well to the whole advent of the DevOps movement where software as an industry was
largely stuck in this mode of we are going to release
large blocks of change infrequently
because that gives us a lot of time to verify the things that we're changing.
But it was shown that that's actually
the least effective way of making sure that you're doing things safely because you're never going to be able to,
cover the entire service area of what you're doing
and that it's actually
much better and much safer and easier to manage to have more frequent small change sets because every time something breaks, you have a much smaller delta of what to evaluate versus, oh, I'm only gonna release once every 6 months. So now here's, you know, 4 months worth of software development, 2 months worth of testing, and then your software engineers are sitting on their hands for 2 months. And so there's a lot of wasted cycles and still a lack of any trust in the system.
Unknown: Right. And you wanna shift left everything
in general. Right? That's, like, the approach we're talking about. And it's shift left because you're shifting it left on the v model,
which all this regulation, you know, is all about, the v model. How do you validate
something and you validate it based on the v model? Yeah, I
think it's that's true, but with that said, I also think the changes you have to, in these type of systems, have a lot of rigorous checks
before you even change something because you're never gonna catch everything. But just the core things you know it should do and the core things that you're mitigating need to be tested. That's a legal obligation. All risk controls for any
product that is released, as a medical device in the United States need to be mitigated for every single release. I'm just thinking that, like, as we were talking about it, right, it's like the the challenge is that that's
it's just complicated. It's not black or white. It's not like release fast and and
see what happens or don't do that at all. I think it's just
it requires some knowledge and understanding and sophistication in how we deploy software. Absolutely. It's very hard to make things that work well. Yeah. Definitely.
As somebody who works in the software and technology industry, I'm constantly amazed that anything works. Yeah. And that's the the, you know, especially the more you do it, the more you're like, how
when I started this journey, I was a developer
and I was very frustrated.
Like, how can
anyone do what what these regulations are asking to do? Like, no 1 will make anything if that's the case. And frankly, over time, you realize that the things that bother you the most in this regulation, which is mainly design verification, the act of saying that
this thing,
you know, my original requirements are met by the thing I built eventually and any change, I need to redo that. That aspect,
which is the most burdensome and most difficult and most annoying,
is also the most obviously a must have. Like, if I'm building something that people rely on, yeah, I have to sit
and say, yeah, I think we did everything we said we did and we didn't forget anything.
It just people want it to be black and white, right? You know how people are and
how developers are, especially. Developers, they're very flexible, but they also
want it to be very clear and, you know, develop faster, die.
Unknown: And and that also
goes to the airline regulations of having to have checklists for every procedure in flight because if you don't have the checklists, then you start to lean on your, you know, memory or intuition. You say, oh, yeah. I already did that. I don't need to worry about it, and you invariably
slip up and miss something. And so this is just another
manifestation of that of we need to have these checklists to make sure that we're actually looking at all the things we're supposed to look at and not just assuming that it's already been done.
Unknown: Yeah. And you can't assume when someone's life's on the on the line. Right? You can't just say, we'll see if in production, if we get a bug, we'll fix the bug. And a lot of developers tell me 2 things. 1 is if you do this this way, you'll make less features, and
the response is, yeah, of course, that makes sense. Maybe you need less features. You don't need all the features in the world in certain systems. And I think there's also, like, a fear of it slowing down things and adding more documentation, but
there is a point to that and I think I'm very much against having documentation as a sec of documentation. I think you need to have it in a way that makes sense for your organization, for your product, for the risk level, but I also think that without that, without
the ability for someone to step in and see what you're doing and audit you, there are just bad incentives. And and everything here, we're just, you know, in 2023,
we forget how it used to be before this regulation, but people are selling literal snake oil,
telling people that their kid will be cured of of polio if they just drink olive oil. We've gone a long way from there, but it's not easy to do that. Right? There is
I heard someone say it very well.
Said that the best companies in tech
might have better quality than the best companies in medical devices,
but the average company in tech has way worse quality.
So and that's what the regular is concerned about. Overall, am I improving the quality of things?
Unknown: From your experience of entering this space, building the product around KetchX, what are some of the ideas or assumptions that you had about the potential space for solutions that have been
changed or updated as you worked through with customers
and came to understand more about the overall,
problem space?
Unknown: I think 1 is just how hard it really is to build,
regulated web based applications. Machine learning just adds more complexity to it that people often run kind of against the wall and say no one's ever gonna regulate machine learning models and high risk applications because of the statistical nature.
Fundamentally, I disagree with that assumption because we regulate,
biopharmaceuticals
and biopharmaceuticals
are produced in a statistical manner. They're grown. They're not made.
And if we can regulate that, we could really regulate everything. And this goes into cancer patients, you know, direct IV. And so I think
that the 1 assumption I had is
it can't be that complicated. Well, it is. Some assumptions I had were where can we reduce it, and there are places to reduce it, but there's also places that you just totally understand why they're asking for it.
And the third is that there would be some limit to regulating
AI systems, but I don't think that's true.
I think there just needs to be a lot of subject matter expertise and knowledge that there's not enough people who know.
And that's what software tooling is made for. Right? It's made to automate subject matter expertise
and make it easier for people for less people to do more.
Unknown: And in your work of helping customers
gain control of their overall development cycles,
reduce some of the operational burden of developing these systems and help with some of the automation aspects and just understanding what their requirements are? What are some of the most interesting or innovative or unexpected ways that you've seen ML used in the medical space?
Unknown: There's so much of it going on right now. I think,
most innovative. Most innovative is hard. I think some of what people are doing with images is amazing.
I think
the way it's been used, even like linear models or multi linear models for a long time is pretty
pretty cool and useful. Right? People think about regulated AI. There have been approved machine learning applications in medicine since the late eighties, early nineties,
just not what we do now.
I think what it's gonna do for home care, for people that live in places that have
less access to health care is gonna be transformational. And I think that's to me is why I'm I'm so excited about machine learning is we're gonna take a lot of things that people know and no 1 else knows, and we're gonna
make it more accessible to people to increase the
level of care. Because at the end of the day, the growth of population,
is outpacing the growth in providers, physician, nurses, and so on.
And we need to find a way to reduce the cognitive load it takes to perform certain medical interventions and then the amount of time
physicians
spend on it. And I think there's a lot of amazing stuff going on in surgical robotics as well regarding this. How do you make a robot that's just easier to do surgeries with
To just make it less burdensome to learn how to make how to do surgeries and and I'm pretty excited about that. And I think it all ties together to, we want systems and software to automate tasks in our life and reduce the challenge of doing that, and machine learning
is just kind of the 1 of
the most advanced stages of doing that.
Unknown: And as you have been working in this space and building the KETRIX product and business, what are some of the most interesting or unexpected or challenging lessons that you've learned?
Unknown: I've learned that there's amazing people out there in the world that are spending their life building
really important medical applications,
and
they're yearning for a way to do it better and faster
and right. And I think that was really exciting to me is to meet a lot of dev teams that are not just trying to do this, but are trying to do this really, really, really well and are thinking of patients all the time
and how
to make good companies to do that with and and are excited about what we're building. Right? It's always exciting where you have this idea. You're sitting in some
company
while other people care. And and so things that seeing that they do care and also the overall acceleration
and the way the regulator
in the US and the EU has taken on software over the last 24 months, I think, is very exciting. There have been more movement in the last 24 months or 12 months in software
than there has probably in the last 30 years.
The real big set of regulations came out in the early 2000,
but even they were just less robust than what's going on now. And we're seeing it
all over. We're seeing the EU
start to really enforce a lot of things around software, whether that's GDPR or or many other things or the EU AI Act or MDR, medical price regulations, or IVDR,
as well as a lot of clinical trial regulations for data. And we're also seeing the same thing in the US, and and
it excites me to see that everybody's seeing the impact software can have on everybody's lives and want to make sure that that's gonna happen in a reasonable,
efficient,
and safe way. And it really excited me to just see how many people are thinking of that
desire
to help other people,
save lives. And
I just think it's limitless what's gonna happen in medicine. We're gonna see a transformation over the next 20 years,
that's gonna reduce the the cost of things and also give access to treatment that's extremely advanced
to all kinds of places in the world.
Unknown: And for people who are building or thinking about building products in this space of medical devices, medical applications,
trying to address all of the regulatory aspects of it, what are the cases where Ketrix is the wrong choice?
Unknown: If you're building software in medicine and you're not using Ketrix, you've already made the wrong choice. And I think every person that works for us will say,
this is the only way to do it. I think if you're building certain embedded systems that take a long while to develop,
it could be in some ways
less attractive or more attractive. But I think if you're building complex systems and and we even have people designing hardware on top of KETRIX now just because the system is so intuitive and the AI in it is so helpful.
But I I I really
you know, that's my belief in we're building a company to just help people that build software and medicine do it faster,
you know, at 10% of the cost and 10% of the time.
And I think that in 5 years from now, we're gonna see that it's way over that. I think that the amount of the amount of money people are spending to build medical software
is outrageous,
is outrageous
and it is a shame for us as a society
And it is a shame for the millions of patients out there in the world that need treatment
and need funding for devices that are not getting it. And we're gonna reduce that by so much that
every entrepreneur and every innovator that wants to start sees something in the clinic and wants to start a company never thinks again about it, but, I don't really wanna do medical devices because it's hard. And I think that's part of a broader story of automating society that we're gonna we're gonna be part of on how do you do it in other spaces. But I think if I was for my first time trying to develop this, I would go and read
FDA's good principles of software validation and understand what's even been asked here and
then figure out how to go from there because most people just wanna build and they don't think about the regulatory aspect of things and regulatory strategy and you can save a lot of time doing it right early on and not fixing it later.
Yeah. And I think our best users are leaders because it's, just over time, what happens is if you don't do it well,
if your company doesn't go anywhere, it doesn't matter. But if it goes, the company is really successful, it'll matter a lot. This might be the only thing that matters ever in your company at the end of the day.
Unknown: And as you continue to build and iterate on the Ketrix product, what are some of the things you have planned for the near to medium term or any particular problem areas or projects you're excited to dig into?
Unknown: More platforms,
more ecosystems,
more
community around it, a bigger way to collaborate. I think that in coming months years, you're gonna see a lot of collaboration features that help in a wide variety of ways people,
work together to make products, in a totally distributed fashion
and
across companies and across the supply chain. That's something I'm very excited about because there's a very robust world of people that help you make biologics or pharmaceutical agents or hardware medical devices, but it doesn't exist in software and it's kind of sad because software is the easiest way to do that. Right? I can just ship someone code.
And so we're gonna have a big focus on that and how people work together to make stuff.
And,
Unknown: yeah, it excites me. It's all about people collaborating together and building ideas together and then making sure those ideas are are working really well. Right? That's all the FDA really asks for. Are there any other aspects of the work that you're doing at Ketrix, the product and platform that you're building, or the overall space of building software and machine learning applications for the medical space that we didn't discuss yet that you'd like to cover before we close out the show?
Unknown: Not really, except maybe the fact that people need to focus on the overall product, and there's a tendency to focus on machine learning as a product.
I don't think the vast majority of models are not products.
They're building blocks for other products,
and people need to start thinking about it that way.
Machine learning as a model generally would be a feature.
LLMs are more complicated than that, but they're generally a feature unless you're chat GPT and there's not gonna be a lot of other chat GPTs. You're a feature within a broader context, and you need to understand how that broader context looks and works.
Unknown: Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest barrier to adoption for machine learning today.
Unknown: Fundamentally, it's subject matter expertise
as it has been for any new technology ever.
Some of it is resources, but I think there's also a lot of great resources,
technical resources, like compute and the United States has decided now to create a central
governing resource for AI research and and share that across communities and university.
I think it's very exciting, but I think it's all about
knowledge,
and software is also about knowledge, but things are progressing very fast. And if it's hard to find someone to build a model, how about someone who understands those models well enough
to explain that to a regulator?
Right? And then Albert Einstein always said, if you can't explain it to someone who doesn't know what you're talking about, you probably don't know well enough. That's all the FDA is asking for, show us that you know it well enough that you can explain it to us.
And I think that's going to be the challenge in machine learning overall and the challenge in machine learning in medicine because it just kind of over
the machine learning complexity.
You get another layer of regulatory complexity, another layer of software development and DevOps and all those together.
You know, how many people know how to develop
web based machine learning products that are validated
to medical regulations?
Not
Unknown: many. But how many of those do we need? A lot of people. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing to
simplify and, reduce the overhead of working in this space.
Definitely appreciate the time and energy you're putting into that, and I hope you enjoy the rest of your day. Yeah. Thank you, Tobias. It's been a real pleasure. I enjoyed it.
Unknown: Thank you for listening, and don't forget to check out our other shows, the Data Engineering podcast, which covers the latest in modern data management,
and podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used. You can visit the site at the machine learning podcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts at the machine learning podcast.com
with your story. To help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.