Unknown: Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
Introducing RudderStack Profiles.
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
You specify the customer traits, then profiles runs the joints and computations for you to create complete customer profiles.
Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack.
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
Whether it's real time dashboarding and analytics, personalization and segmentation, or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface.
Go to dataengineeringpodcast.com/materialize
today to get 2 weeks free.
Your host is Tobias Macy, and today, I'm interviewing Elad El Dor about operating Kafka in production and how to keep your cluster stable and performant. So, Elad, can you start by introducing yourself?
Unknown: Yeah. Sure. So, nice to meet you, first of all,
and
and thank you for letting me be on the podcast.
So my name is Elad,
and I'm a DataOps
group manager
at,
IronSource,
which is an Israeli company
that was merged with
Unity.
And
on my job, I
primarily
focus
on stability,
deployment, and cost reduction
of
big data analytics
clusters on AWS,
such as
Druid,
Trino,
Spark,
etcetera. And I also,
am involved with
Kafka
production issues
and cost reduction issues, but mainly
production and stability
issues on, on Kafka.
Prior to that, I was a SRE
for
a company called Cognite,
where
I where I was
in charge of stability,
of big data clusters
on prem, for Linux on prem,
mainly for Spark Streaming, Spark Batch,
and Presto on HDFS
at various,
customer sites.
Prior to that, I was
a Java back end engineer
for
for about 10 years.
And,
I recently published a book called the Kafka troubleshooting in production,
which will be the
I
think, the main topics of what we will discuss today.
And it talks about how to handle
production issues in Kafka clusters,
both on prem and on the cloud on AWS.
Unknown: And do you remember how you first got started working in data and what it is about that space that keeps you interested?
Unknown: So
I I started working on big data,
about 7 years ago when I developed the
my first
Spark streaming application,
consuming from Kafka
and persisting to HDFS on Linux on prem clusters.
Before that, I just wrote, you know, back end applications
that were
reading from some database,
writing from some into some databases, but it wasn't,
it wasn't formally
big data. And
5 years ago,
I understood
that,
on big data clusters,
there are production issues that I didn't see before.
And,
instead of so I reverted from being a developer,
I reverted to being an SRE and solving
stability issues
in these clusters, which meant knowing better not my code,
but,
the
the infrastructure,
the the cluster that my code or some code, some big data code, runs on, whether these are Presto clusters,
Spark
clusters, Spark batches,
Spark streaming,
Kafka,
HDFS,
and this it was a whole a whole new world for me.
And that that's what brought me specifically,
to handle
Kafka issues, which I which I do for the last,
like, in production. I do it since 2018,
but I started doing it a year before. So maybe
the question is more what brought me to be not to be a developer anymore,
but to become an SRE slash data ops
of, of big data clusters,
and specifically
focusing on cost reduction, whether on prem or on the cloud.
And
and and focusing
on on Kafka.
So
I I I I will elaborate on on the Kafka issue. So I understood that Kafka is, like, stuck in the middle of everything,
everything. Like, if there is a problem with Kafka,
your whole data pipeline is just
stuck.
Producers
can't write, consumers
can't read.
And and that's what
it was a combination of 2 things. 1, the understanding that Kafka,
the best ROI for my time will be to focus on Kafka.
And the second thing was that
I
stumbled into a book called the System Performance by Brenton Gregg,
who was a lead performance engineer at Netflix,
and now he works under the,
Intel's CTO
on distributed
clusters.
And this really
opened me a world of
that I wasn't aware of before,
of monitoring and detecting bottlenecks
in Linux,
clusters.
And not only understanding
what what's your bottleneck what's the cluster bottleneck,
but also it opened a way to reduce, costs.
And what I saw
when I stumbled into production issues in Kafka,
Spark, Presto,
Druid,
For this matter, any service, any applicative service on running on Linux cluster
is that,
understanding how to diagnose issues in a Linux cluster
can really help you detect
what what's the problem in in many cases, and also allows you
to reduce costs on the on the cloud.
Unknown: In terms of the Kafka focus, you mentioned that you were working as a back end developer. You were
interested in the use cases for Kafka, the production requirements around it, how to make sure that it was stable and performant.
And I'm wondering if you can talk to, at least at a high level, some of the different environments in which you've had experience working with Kafka and some of the
categories of operational challenge that you've had to deal with in that process?
Unknown: Sure. So I started with a on prem cluster,
a Kafka on prem,
and it was
on a Kafka distribution of Hortonworks,
HDP,
before they were acquired by Cloudera.
So it was
it was a free
distribution
of Kafka,
and
and now I work on with the Kafka Open Source
on AWS Cloud.
So in either cases,
it wasn't a managed service,
so
we had to to diagnose the issues and solve them ourselves.
Now
the because both
are practically Kafka open source, although it was
when I was on prem, it was part of the HDP, but I don't see a lot a lot of difference here.
The main
issue here was on prem is on prem versus cloud.
And this is why my book focuses mostly on cloud deployment,
but also on on prem deployment. So I started
from
from on prem,
and there's a big difference,
in in the
in some parts, there are big differences on on prem because,
you are in charge of the hardware failures, for example.
Now the the most common failure for hardware
on prem,
no matter which cluster it is, are disks.
And in Kafka on prem, usually,
you you you need a lot of storage.
And so in order not to pay much, you use HDD
disks and not SSD.
So they fail for a lot of
their ratio failure is pretty high, and you need to take care of it because it is not cloud.
Another issue is that on on prem environments, you usually the SRE is not on-site.
So you have a tier 1, tier 2 support,
which are usually unaware
of
of signals
that can say that something is going to get wrong,
something is wrong in the in the in the cluster.
So as an SRE of on prem, you reach the problems once the cluster holds
sometime.
But there are also
but on the cloud,
the the cloud provider will tell you, okay, you don't, you don't need to
to handle,
disk failures.
However,
when
even on the cloud, a broker can halt
because of disk,
deterioration,
and you wouldn't know of it until you get into legs.
But you don't need to replace the disk. You just can replace the broker,
which is much easier
in the cloud versus on prem, where you need to,
get into the drawer of disks and replace the disk.
Another big difference is the scaling.
If you have more traffic,
you can scale out. On the cloud, you can just scale out or scale up, and
by spinning in your or spin a new cluster
pretty easily,
while on on prem,
it's very, very tough,
because think of it like if you want to scale up, you need to have disks
on-site, and sometimes you don't have these disks on-site. What happens if you ship the cluster to another when your customer is in another country?
Other,
another issue is,
like, what happens when you need more RAM? On the cloud, you just,
you just
add more spin a cluster with instances with more RAM.
But on prem, you need to check that you have enough slots in order to insert DIMM,
DIMM sticks, memory sticks,
into the
into the class into each machine.
And you need to do it, like, manually.
So I think the
the scaling,
the 2 main difference of the scaling
option, which is much easier on the cloud,
and handling failures,
hardware failures,
which is
also
when you are the when you are an SAW of on prem, it's it's on you or on on on the first years
that manage the issue once, but but while in the cloud,
you just
detect the issue. You need to detect the issue and then replace the the machine.
The main so these were the 2 like, I came from on prem,
not only from Kafka. Like,
only for Spark application. It's it's pretty much the same issue,
between on prem and on the cloud, these these 2 issues.
However, the the benefit of, deploying on prem
is the fact that you don't pay
every for every hour.
You just, pay once,
there is I think today, there is a growing discussion,
whether
a cloud based
companies,
maybe they need to go on prem for some of their clusters.
So
it's the math the math
behind
this calculation,
sometimes, I think,
favors on prem even though
you need to
to handle this. The scaling is very tough,
and it needs on you to to detect,
hardware issues, but, maybe the cost sometimes
justifies this.
Unknown: Yeah. The on premise versus in cloud debate is definitely
ongoing and always very nuanced.
And to your point, yeah, where
on prem,
the
cost over time is much lower because you own the hardware, so you don't have to pay continuous upkeep for it. But
there's the opportunity cost of having to move slower and more deliberately and,
perhaps reducing the number of chances or experiments that you take because of the fact that it is so much lead time to bring in that hardware and scale up the cluster.
And, also, on the point of Kafka, that's an interesting aspect as well where
the my understanding of the way that Kafka itself is designed and
some of the aspects of
having to define up front the number of topics in order to accommodate a certain number of clients seems as though it lends itself more readily to that
fixed installment
on prem environment versus the cloud environment where you are incentivized
to elastically scale up and down. And I'm wondering
what you see as some of the
challenges of bringing Kafka into the cloud because of that potential for elasticity in the clusters. I
Unknown: money like, use the cloud. Okay?
The money is the the the money is the only reason why going on prem because it's indeed it's it's tough. But it's interesting that you mentioned,
that that Kafka might sound
a reasonable a good candidate
to to be deployed on prem. And by the way, there are companies that most
of their
clusters,
are
that that host the open source analytics tool or messaging tools are deployed on the cloud, but Kafka
is deployed
on prem.
I I know of
2 not small Israeli companies that have their Kafka
deployed on prem.
And
what like, from from my experience, because I was deploy I I was SRE for
various customers when I was working for on prem. So I had I saw
several examples.
And the main thing that I saw in the main difficulty that I saw when provisioning a Kafka
cluster by the way, most of the clusters were over provisioned. This is 1 problem in on prem, that you
over provision
because it takes time
to purchase more disks or dings or brokers. So it's so it's hard to to say what what's going to be the the size of the cluster.
But if I if I will take 1 1 factor that makes it hard
to to provision
upfront the Kafka cluster is the storage, is the retention.
Because on on prem, you need to remember that most of, like,
people on the cloud that were born to the cloud, let's say,
that started working only and they work only on cloud environments,
they tend to forget,
that there are many on prem clusters.
And
much of these on prem clusters are deployed by, enterprises,
or by companies that are not really high-tech, and they have
big on prem clusters.
And these customers, the the man the the owners of
the customer, let's say, they are also the owners of the data center, and you provide them,
the data center.
And
they have retention
requirements
that they serve not only how much time will it take you to recover from lag,
but they have which is usually in in cloud and Internet companies.
That's the retention, several hours, they want days or even a week.
So
this changes the whole picture because then
when I started
when I started as a necessary, I saw cluster that were just a a bunch
of machines that cost
sometimes tens of 1, 000 of dollars.
And the CPU utilization is is, like, ridiculous. It's 10% user time.
And
the RAM usage, well, you never know what's your RAM usage, because
we might later talk about the page cache, but it's very hard to understand what's the RAM usage in Kafka.
But the disk storage was the utilization
usage was really high because of this retention.
And then you come to to a point, and I dedicated a chapter in the book only for this,
to how how you
whether you use RAID,
RAID 1 plus RAID 0 or use JBOD.
Because
I can give a real example of a customer
that had
17,
brokers.
And
just in order to satisfy the retention
requirements of the customer,
and then
the customer decided to double the amount of retention
because that's the order that he got. It was some guy government
law enforcement.
So you don't mess with them. They tell you, okay, I need to double my retention, so you need to satisfy this.
And
the the the first reaction was, okay, let's double the amount of brokers,
but then I I,
I convinced that the
my managers that,
you you we just need to
switch from RAID 10,
to JBOD.
Now RAID 10 gives you double the amount of replication. So if you have replication factor of 3, you will get 6 copies per segment if you use rate
10. But if you use JBOD,
then you save
then you save half of the disks, and then you don't need to add even 1 broker,
just
use
the same number of disks. And if you want even more storage, you can just add more disks. But then you run into the issue of, okay, I don't have enough disks in my,
in my drawer.
So you need to add another drawer to each broker.
These are these are things that on the cloud, you don't even think about them. People on the cloud even don't know about them. But
if I will take 1 thing that that is really, like,
that makes provisioning Kafka on prem to be tough is is the
retention requirements.
And managers
for on prem clusters tend to be very sensitive for retention. They they want
they want
10 times the amount of retention or 20 times the amount of retention,
that Internet companies have, for example.
Unknown: And bringing us around to the book that you wrote, which you mentioned,
what was your motivation for bringing this all together in the written form, and what are the overall goals that you have for the book and the people who are reading it?
Unknown: So
back in I remember myself as
a as a back in engineering trying to
deploy my application,
my my streaming application on Kafka,
and going to something didn't work in the Kafka on dev. So I remember going to the
to the DevOps room,
and, you know, everyone are afraid when you go to the DevOps room, because they're the most important the critical part
of the of the organization.
And
I I asked them, okay. I don't know what's wrong with my Kafka.
And I had no clue, and they didn't get an answer because they had no time. And then I went out to the room. I decided that I will know Kafka
from all the angles. Like, that was my, like, really initial motivation. But
if to be, like,
more serious,
when I started handling production issues in Kafka,
and I went over to
sites customer sites, I saw that people just didn't were clueless
about what's wrong with Kafka. They blamed
everyone blamed
in the other part,
the
the devos blame consumers, consumer blame producer, producers blame Kafka.
And
so I I wanted to to
and it it is in the middle of the data pipeline. So
that was my motivation. Like,
the best ROI for my time was definitely
Kafka and Linux operating system, like understanding the metrics
and how to
diagnose
using Linux metrics, diagnose problems in Kafka.
After now this is was why I wanted to
learn Kafka
from real production issues.
My motivation for the book was that these,
support engineers and DevOps and developers that encounter issues in Kafka,
which is not a managed service, so they manage it themselves,
will have a a cookbook
for understanding,
like, like recipes,
understanding how to how to handle production issues in Kafka. And this is why the book is split it into
3, sections,
3 logical sections. It's the the data section,
the Linux OS section, and the Kafka metrics section section.
And, also,
for, most of it cloud based based, but but also 2 chapters dedicated
specifically to to on prem.
Because,
the
duplicated
when I was, starting to work on the cloud.
On the cloud, it's just easier because you have, you know, you have monitoring in front of your eyes. You don't need to get logs from from, support engineers.
But,
so I I saw that that
that there there was no book. There was nothing that even resembled
a book about real production issues in Kafka, and there is a real need,
for for this because
there are so many Kafka deployments out there, whether on prem or the cloud.
And
and people are just you know, I I talked to 1 CTO of a start, Israeli startup who told me, I must use Kafka because that's the message of us today, but it's hell.
Okay? Managing it is so tough.
And I heard it after he said it from different people,
different manager ops managers,
or
tech tech tech leaders at some small companies,
and
understood that, it's just
a common common problem.
And then
then the idea for the book,
I I mean, it didn't come to me. I got an offer to to write the book, and then,
it took me some months because it's a lot of work.
Then I then I decided to compile to to gather all the stuff that I compiled for
during the 6 years of me working in Kafka.
And that was my my motivation. Like,
I'm not saying that people shouldn't use Amazon MSK, yeah, or, yes, or Confluent or Ivan,
but I think that
for for for those who already manage Kafka,
they should have a better
guide
than than no guide, first of all.
And, also, for those who
consider they they manage their own Kafka and consent, say, okay. It's tough. It's too it's too much. It's too tough.
We need to pay
license and move to some managed service.
I think that if they have
the core the right guide,
then it can save them money.
And,
I work in the open source with open source for 20 years, and, like, I see this as my small contribution
to the open source community.
Unknown: To the point of saving money and whether to run your own Kafka cluster or use a managed service,
there are a lot of considerations
that go into all of that
as well as the
use cases
of what you're going to be building on top of Kafka
and what are the
sizing and scaling requirements. I'm wondering
for people who are
in that position of deciding, do I want to use Kafka? How do I want to use Kafka
as far as
self managed or managed service? And
what are the parameters along which I need to project what my cost is going to be? What are the
different elements that they need to be considering and planning for as they start to evaluate and do those initial deployments?
Unknown: So so first of all, you need to understand your traffic.
Okay? So,
let's assume you are not a small
company and that you you decide you you you are the decision point whether
you need to use Kafka, first of all. And then if you use Kafka, whether you're going to use cloud manage or you manage it yourself. So,
regard regarding alternatives to Kafka, I I'm not aware of any alternatives. However, I never search for alternative because Kafka is just everywhere.
So let's say you chose Kafka and you have enough traffic
so that
you can build
even a minimal cluster of,
33 brokers. Yeah?
But let's assume you have more, and let's assume you have multiple Kafka clusters per each team or service or a group or whatever. So in that case, first of all, you need to know your traffic,
the number of topics,
the per to per cluster, the number of partitions,
and
who who are your consumers,
per topic, who are your producers.
So understanding the producing rate,
the consuming rate,
and
then you need to provision
a cluster. Let's say, even before you know whether it's cloud based or not, you need to understand how much CPU you're going to require,
how much RAM in order to support not the Kafka process itself, but the page cache because you want data to be read from the page cache and not from the disk,
and allocate enough disks,
cheap disks in order to support
as cheaper as possible,
to handle the the retention requirement.
And
now
after you have this estimation
and you know
which how many brokers you have and what's the size of the broker,
now it depends
the decision whether it's
cloud based or not. If you are cloud based, you you will
usually
pick
cloud based.
But again, some companies decide, okay, I will put my Kafka on prem because I know it's it cannot be spot instances.
But most of the companies which are cloud based will will deploy it on the cloud, and
they will deploy it
probably on on demand instances if they're on AWS.
Some
if if they have a strong DevOps team, they will, deploy it on a on a Kubernetes,
maybe. But,
but
I I don't have an experience with Kafka and Kubernetes, so I can't say anything about it. I also didn't write about it, of course.
But,
so so and if you're on prem, you will deploy it on prem, of course. Now the question may of whether to deploy it on like, purchase a a license
for, for Confluent or for Amazon MSK,
or for Ivan.
These are some of the many services. Or use it or or manage it yourself.
It really depends on on the
on how your ops team, whether it's DevOps or DataOps,
know how to
handle
production issues.
And let's say, you know, ops teams are
are rare. Good ops teams, I mean, are are rare.
And and I think that if you
if, like,
you need good mon an excellent monitoring
of Kafka. And when I say an excellent time, I don't mean
hundreds of metrics.
Okay? I mean, I mean, a minimal subs a minimal set of metrics
that will show you
where production issues can occur.
And in my book, I have 2 chapters, 1 on producer metrics, 1 on consumer metrics,
and
is 3 chapters
on
CPU RAM and Disks for the broker themselves. So I think, like, half of the book talks about
how to diagnose issues,
and
and from this,
the the reader can understand
what to monitor. So you need an excellent
monitoring, but not too large. You need,
like, specific
monitoring,
in order to deploy it yourself. Because if you are blind in Kafka,
you will pay a big price. You will have downtime,
and you will just lose data.
So you must have an excellent,
excellent monitoring
and a team who knows how to how to manage the Kafka cluster. And,
otherwise,
to choose,
choose cloud based or there are again, because it's the knowledge of handling Kafka clusters is is pretty rare.
What happens is that some
places, some customers,
and they saw it in on prem,
They just
sometimes Kafka holds, and they lose data because
there's not enough knowledge.
That's not, like, that's not, of course, the situation at my current company. We have an excellent,
DevOps team, and its team leader was also the technical editor for my book. And he affected
a lot
on the content and also on how to,
like, what
some parts of what to focus. His name is Oronon,
and I
I am very grateful
for him to to
to that he
invested the time
to to read and to edit the book.
But, again, it's at some companies, you have an excellent DevOps team, and some customers,
you don't
have. And if you're willing to lose data at some time, then
use the open source. And if not, then
go to
then pay the license for Managed Kafka.
Unknown: Data lakes are notoriously
complex.
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte scale SQL analytics fast at a fraction of the cost of traditional methods so that you can meet all of your data needs ranging from AI to data applications to complete analytics.
Trusted by teams of all sizes, including Comcast and DoorDash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.
And Starburst does all of this on an open architecture
with first class support for Apache Iceberg, Delta Lake, and Hoody, so you always maintain ownership of your data.
Want to see Starburst in action? Go to data engineering podcast.com/starburst
and get $500
in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
And on that note of
data loss
and the
cluster uptime and stability,
what are some of the failure conditions
that cluster operators need to be thinking about that might lead to data loss and some of the ways to
mitigate that and
plan for it in order to be able to reduce the time to resolution?
Unknown: In terms of preventing,
data loss, let's start with prevention
and then go to to
to what can cause,
data loss. First of all, you have a replication.
Now replication goes side by side with the size of the cluster
because the higher the replication factor,
then each segment will have more copies on different brokers.
However, you will need more storage, and more storage
means,
like,
more storage means sometimes,
more brokers.
And even if not more brokers, it means more more,
more disks.
So it will cost you more,
higher replication factor. Also, you can add another level of,
assurance
and
deploy RAID. Let's say RAID 10, RAID 1 plus RAID 0, and this will double the amount of, of storage that you that you need.
So,
replication is 1 thing.
Making sure that your your producer,
they have a
they receive a axe acknowledgments
from the broker
is another thing, although it may affect
latency.
Retention policy.
Retention policy, it's pretty simple policy,
and you wouldn't believe the number of times that that consumers lose data because of it. Because
you can define it by time or by size or by both, and then,
the threshold is the minimal between both. And
and and, if you define it by size
and suddenly the topic size the the traffic increases
even by a small amount of percentage and you your consumer lags, then you will lose data, or some consumers will lose data.
So
if,
like,
if the audience will take 1 thing from regarding retention is that
highly consider
configuring retention by, by time
and and not by by size, because
you you really don't know the traffic at any point of time.
And and some producer can increase the traffic by
by multiply of 10,
because,
there was some filter and the developer just removed that filter.
And and suddenly you get 10 times the traffic.
However,
if again,
nothing is simple in Kafka. If you configure retention,
by
time, then
you might get into storage, a 100%
storage. So, again, this equilibrium
between
retention size
retention time and and storage. Again, we reach storage.
How do failures?
Monitor hardware failures. Now on on prem,
you can just check the status
of the disks. I repeat again. Disk, disk, disk. This is the the most critical
the cheapest part of Kafka,
but the most critical part of Kafka, and
and the cause for many, many problems. So
like, on on prem, you can you there are tools that you can monitor the state of the of the disk,
and just
a very, like,
recommendation that can save
several clusterizers, I assume.
If a disk becomes read only,
so you can just read and not write,
usually, it means you need to replace the disk,
because it, even running FSEK on on on the disk will help maybe for several minutes, hours, or days. But after that, it will become again read only,
and producers cannot write to it, and it will create a whole
mess. So monitor the disks.
On prem, you have the the
tool to to check
its,
itself.
And on the cloud, you can check,
you can check the the disk utilization.
So the IOSTAT
tool, the Linux tool called IOSTAT,
IOSTAT minus x,
print every 1 second
can show you that if if the disk utilization is 100%
all the time,
then something is wrong with your with your disk.
Another metric is, what there there are a lot of metrics, but the the another 1 that happens to be around every
production issue is the system time, the CPU system time. There are 4 main CPU metrics. You have the user time, system time,
I wait time, which is either disk time, wait time for disk or wait time for network and context switches. If the system time goes, let's say, about
10%, then
you should suspect something this is not good in your cluster. If your contact switches
time reaches, let's say, more than 3, 4%,
then
check
check what happens there.
A common
cause, by the way, to a high
context switch
time percentage
could be,
that you just have too many disk kind of threads compared to the number of disks that you have.
And this is if you want like, if to if you want to be, like,
like, very cautious, you can you can regular backup regularly your data,
either by using the feature of a tiered storage of Kafka, which I admit I I didn't use until now,
or you can just
read add another consumer
and just
take the data from the topic and persist it to some
cheaper storage, like HDFS,
or s
3, where the storage is separate for the from the compute.
So this is regarding
some ways to prevent data loss.
But in order to but these these are the these are the, like,
what happens after
after you already have a problem.
But the question is how you prevent the problems.
And from my experience with Kafka in production in the last,
6 years is that Kafka cluster talk before
they hint you before they they creep you down.
They tell you stuff,
and they tell you this through the disk utilization,
the IOSTA tool, which you can see using the IOSTA tool. They tell you this using top command.
Check the system time,
your CPU system time. Check the CPU context switch, CPU wait time.
If if you have GC issues in your Kafka
process,
then it will tell you this through
spikes in the user time during a full GC. You can use the JSTAD
tool, which is part of the JDK tools in order to check for
full GC the frequency of full GC.
Again, it's it's like, this is what half the book talks about half the chapters almost talk about
how to, like,
various cases that they stumbled upon
regarding
what can lead you to
to to data loss.
I had, I had a colleague asking me,
I I gave him a copy of the book and he has he asked me, okay, but you didn't talk about the the issue of un replicated
partitions.
And I replied to him, well,
every
problem in Kafka can result in under replicated partitions
or in in partitions that are not. They don't appear in the list of in sync replicas in the ISR.
So
and there are tens of problems, which might not even be on the Kafka itself. Like, it could be in the operating system because something
is,
like
sometimes it's even not in the Kafka. Again, like,
if you deploy another service, if you deploy
a antivirus
antivirus,
okay, or firewall,
and they scan
the the segments all the time, and you will have high disk utilization, this is not the fault of Kafka,
but it will cripple down your Kafka.
So
and there are some I saw on prem clusters where you shut down the firewall, and then it comes up again because of some policy.
So
so so you need to remember that not always you run alone on the on the broker, but asking,
like, how like, what can cause data loss
is,
like, is like asking what should I monitor in Kafka and and and, how to deal with it. And that's
that's a topic of the book in general.
Unknown: And given your experience
both working as a back end engineer and
operating Kafka clusters at scale.
I'm wondering if you were to
be in the room today
redesigning Kafka from the ground up, what are some of the aspects of the system design that you might choose
to revisit
or revitalize?
Unknown: I think the design is is,
makes a lot of sense, the log based
approach.
I I don't have many
changes.
I don't think of any change other than 1 change which is,
for me, it's a bug. I don't know if they declare it as a feat
as a feature
that, they spread.
If you have more than 1 disk, then the partitions
are being spread,
among the disks,
given the number of partition per disk
and not given the numb the the amount of storage per disk,
which I I don't think that the Kafka community understands
how it affects,
some Kafka cluster owners
to decide whether they go raid
or, or j bot.
It's it's like something that if it will be fixed, the the amount of cost that will be saved
for disk and maybe for cluster will be pretty big,
because
there are
there are ops teams that say, okay. I don't want to go JBOD because because of this. I don't want to handle the spread of the data
among the disk.
But other than this feature, feature slash bug, of spreading the data evenly
based according to storage and not
not number of partitions.
I do I I must I I must admit that I'm not in the I'm not in the applicative side
anymore for a long time because
to to be honest, it it's it interests me much less
than the upside.
So
my focus is on the upside, and
and then it comes down to, like,
this this question comes down to, like
it could be asked regarding every
big data cluster, but in Kafka,
it's really hard,
to detect to understand the root cause of production issues.
To be honest, I don't know why,
fix a problem in Druid or Spark or Spark Streaming or Treyno
takes much less time than understanding
why why, why data was,
it was lost on Kafka
or why the disk utilization is so high.
I don't fully understand it,
but, like, if I take the amount of time that I invested
in every Kafka production issue compared to other class, it will be I'm not getting, like, 10 times
between 5 times and 10 times more. So I think that
and this brings me back to the motivation of writing the book
that Kafka uses the operating system in a way that
no
other open source that I know of uses it,
especially
not the CPU, by the way. It's really like, on the CPU, it's, like, very simple. User time, and that's it if if you do it correct.
But the usage of the page cache, the trashing of the page cache in in cases that you have legs,
and the the
the amount of stress
that comes down on the disks
is something
that I I don't think the the the
the
those who created Kafka
or those who
who develop it or contribute to it,
I think there's some
split between those who develop and those who who maintain,
and and there's not enough
connection between the 2 2 sides.
And maybe this this is another when I think of it,
I'm thinking out loud now. This is another reason why the book is important both for developers and for
ops team in order to not not only understand Kafka, but also understand
for developers to understand the ops team,
because there are so much problems that are being caused by
1 disk that goes rogue.
1 disk.
You can you can have a cluster with tens of disk and 1 disk go bad,
and
the this can cause the whole cluster to halt.
And
a healthy cluster should not suffer from such an issue. Remember,
getting a call about 5 years ago
from a customer side that had 3 brokers,
6 disks, HDD disks,
configured in RAID 10,
and 1 disk went bad. So you see
but but in RAID 10 in RAID, you see only 1 logical disk.
So and the utilization in Iosat shows the utilization
of the,
the highest utilization among all the disks in the RAID, and then you see 100
percent. So
I had to I guess that that, it was 1 machine the 1 disk that was,
that got screwed.
And I told the support engineer, go to the room
and check the if the light,
waves
on 1 of the disc, and he told me, yes. There's the light was on 1 of them.
But
imagine that you need to do
I needed to get
to get
this because because of the lack of monitoring. So
when I think of it,
then the work on something needs to change
on
like,
not on the work on the disk because this is what Kafka does. It writes
massive amounts of data to the disk, and it reads
it tries not to read from the disk, but at many customer sites or cloud
or cluster, it reads from disk.
And first of all, ops team dot node, it reads from disk.
Secondly,
like,
many cluster have deployment of RAID. So
how can you assist them in understanding
the that 1 disk
got screwed?
They don't have a and in the cloud also, by the way. Why when 1 disk gets bad,
there is
the cloud provider won't tell you that because they they can't really tell you, okay. I'm on the disk is the disk is
on 80% utilization
for 30 minutes.
We don't know if it's good or not, so we will not tell you ever anything.
And then you get into high, high weight, and you replace the broker.
So the mitigation
for these issues in Kafka
is,
is is
is brute force.
And
and figuring out that you have a disk issue,
you need to be a magician in order to to to know this.
And and developers of Kafka just assume that,
okay. Someone will handle it. It's not us. We just develop.
And I think
for me coming, I did a shift, a career shift, which is not common, like going from developer to
ops. So
I I understand the frustration from both sides,
and I think that there should be more people
that know,
both paradigms.
And
because a lot of production issues
originate
from the lack of understanding
or
the lack of
cooperation or knowledge sharing between the between the 2. If if the Kafka community had better communication between the developers and the ops team,
then
I think it would be much easier to detect,
disk issues,
which cause,
I bet it causes, like,
at least third of the problems
in Kafka.
Unknown: Yeah. It's, definitely always a challenge balancing
the developer of, I just wanna get something shipped and do something cool with some fancy new feature in the operations team of, I just want you to stop crushing my machines so that I can sleep at night.
Unknown: Yeah.
Yes. But but I I it's
like in the Kafka community,
I think there is lack of ops team that will tell that will check
these these developments.
And but it's it's not even new development.
Just for I am telling I'm asking the Kafka community, like, go to deployments
and of Kafka
and check ask customers,
what's the percentage
of production issues which is caused by disks.
And then
try to I don't know. Maybe maybe every Kafka tool needs a better monitoring of of of disks.
Like
and, also, if there is monitoring,
how do you read
like, Brandon Gregg has an excellent explanation of how to read
a IOSTAT, the output of IOSTAT. So you have the utilization.
Like, the service time is obsolete. No 1 needs to look at it. So every utilization,
you have throughput, which is the read megabyte per sec and write megabyte per sec, And you have IOPS, which is the read per second, write per sec.
And
and this is a fact that not many
people know. The saturation
of disks is 60%,
Only 60%, which means that every increase
in the disc utilization
going is bringing the situation,
the the level of IO 8 will become worse and worse and worse.
And it's not like in CPU. In CPU, it could be,
like, the recommendation on cloud is, like, 75%
CPU
CPU saturation.
Above that, your load average will increases
in a nonlinear way.
So
how do you read the how Kafka,
use ops team should read the output of iostat? Well, it's simple. Look at the disk utilization.
The disk utilization of 100 I saw developer ops team telling me, okay. We reached 100%. This is not good. No. This is okay
because Kafka works in bursts. So it writes a lot of data for a small amount of time, so we'll have high disk utilization
caused by
read meg write megabyte per second. Write write write per second. This this is good. And then you should see 0.
And then again, you have a burst of writes.
And
but if you have burst of,
if you have 100% utilization
because of reads,
then it means that you read from the disk, which means that you have a problem somehow. Maybe you have a a consumer lag.
Maybe you replicate some of
a a broker that replicates the data
is is is, lagging
behind,
which means that this partition is not in the it's
this broker is not in your ISR list of this partition.
So just look at the output of the iostat, minus 6.
It prints every 1 second. And if you have several seconds of 100% utilization
from
writes, then this is okay. But if you have the same
100% utilization
from grids, then this is not okay.
Now if the community will take this and make it
an alert or some monitoring
tool, then
it will reduce, and ops team will understand know how to read this,
then it will reduce the
it will reduce the frequency of production issues.
It it is more than 10%, for sure. Just this
feature, detect leg detect
legs,
by looking at the diastat.
And
Unknown: in your experience
of working with Kafka clusters
and helping customers
and end users manage
and ensure their uptime? What are some of the most interesting or unexpected or challenging
production problems that you've had the, opportunity to diagnose?
Unknown: Okay. I will tell
the most
this is
the okay. The most bizarre. I will tell the most bizarre production issues, the most the most interesting ones. So the most bizarre, I I I already
mentioned it before,
was a cluster of, 3 brokers, sick 18 disks, 6 per
broker
configured in RAID 10,
that,
1 1 disk, 1 40 disks
1 40 disk crippled down the cluster because,
because,
it was on ray 10. So, in fact, only 2 disks were up, and the disk utilization
were already was already pretty high, so it just added to the party this problem.
And then I had to guess that 1 disk was faulty,
and this was the support engineer really saw that 1 disk was faulty after he went to the server room. And, of course, the the other it's it's,
the it's twin disk because of straight m
also,
didn't function.
And
so on that and the disc utilization was already very high. So it brought the disk utilization to 100%
on the brokers. So 3rd of the
of the leaders, partition leaders were on that broker, and then
producers
stopped, couldn't cry to them. Consumers didn't read from them.
And once consumers
don't read even from 1 partition, then the day then they just cannot function.
It depends, of course, on the nature of your consumers, but this was
a streaming application that had to read from all consumers.
So it got stuck. Even if it wasn't the application that needs to read all consumers, then you have partial data. So
that was but that was a very bizarre
problem.
That combine on prem and high disk utilization and guessing that 1 disk is 40.
The most interesting
problem
that I ran into
was,
it took,
it took several weeks, I think. And,
it involved
brokers
that,
from a certain point in time, every broker that was added to the cluster due to some
failure of other broker,
every broker was,
got its, at some point, to 100%
disk utilization,
and
and nobody managed to write and read for me. Of course, no manage nobody managed to write, so there was nothing to consume.
And
every time we replaced
this broker with another broker, and again, the same phenomena happened.
And,
like,
and then again looking from the IOSTAT after a long, long, long time
of trying to understand what is going on here, it was like voodoo,
we noticed that
that at some brokers,
the disks reach 100%
utilization,
with
because
for for all the time, they were in 100% utilization
because they
but but when we looked at the throughput,
we saw that they are
half
the throughput
of the throughput that causes other discs on other brokers to be an 1 or 2 percent utilization.
So not only did the healthy brokers, reach 100% utilization for small amounts or few seconds and then
the utilization went down,
These 40 brokers these brokers with 40 disks reached 100% disk utilization,
and it kept
100% utilization
and while the throughput was half.
So just dividing
just correlating the disk utilization with the throughput
and the amount of time the disk utilization was another was 100%
led us to the understanding that these are just faulty disks.
But this took a lot of excels and the and just
checking trying to correlate every
OS mon OS metric
until we we found it out, and that was by far the most interesting
production issue
that I stumbled into
in Kafka.
Unknown: And in your work of writing the book and
consolidating
all of the information and experience that you've had working with Kafka. I'm wondering if there are
any insights that that helped you gain or any new knowledge that you were able to obtain in the process.
Unknown: Of course. Yeah. Because,
mainly,
like, from there there were 2 issues, like,
like,
it helped me formulate
the the
the 3 legs that Kafka stands on. Kafka
needs to understand, let's say, which is the data part, the OS part, and the Kafka part.
I was surprised to see that the Kafka part is
is only third of the book,
which shows how much the data part is is important, how
how
the the
how the way that
the data is spread
among the partitions
is so important for the, for the health of the cluster.
And,
also,
I was surprised of how many issues production issues
originate from
a problem with with storage.
And,
and,
also,
I found out several producer and consumer metrics
that that were new to me,
because I thought that
I that
many issues can be fixed with, with tuning the linger
and the batch size on the producer and that found out
several very important metrics in the consumer and producer.
So this was also
new to me. I ran them I ran across them during the time,
like, from production issues that I dealt with
during the time that I I wrote the book.
And
I must say that I must say that a lot of things written in the book were not things
that only I discovered.
So I worked with several
with several people
that,
that that I work we worked together and found the issues together. So
in part, it was just documenting
what team of ops people,
found out,
including me, but also
other people. I I I I have, like,
in order to to make this more more specific, I have an for example, the the issue of the storage.
Just to emphasize,
when I wrote the the part of the storage usage,
turns out
that it's much,
that it's more vast
than what I thought of. So if just we if we just take,
this issue, the storage usage issue.
So,
for example,
running out running out of disk space
due to retention configuration.
What happens when you configure
both time based
and size based?
The way the the the option there is an option to lose data, and I was surprised to to see how 2 simple configurations
like this can cause
data loss.
And
so retention policy and its effect on the data loss,
the
explaining, like, how to add storage to a cluster, how it differs between on prem and on the cloud.
The the fact that
that when you're on prem, this is not only technical decision.
It's also managerial
or financial decision because you can't tell
the owner of the data center, okay. I I I I had a mistake. I don't need the 2 terabyte disk. I need 4 terabyte disks. So I need to throw away all the 2 terabyte and buy 4 terabyte disks
so it will not go smoothly.
So understanding
these aspects,
like, this chapter became partially technical and partially
how is a is a provider
for an on prem customer,
how you manage this this issue?
Of
again, in DIMMs, it's the same issue. Okay. I I I it was a mistake. I bought 16 megabyte
this 16 gigabyte and it 32 or 64.
How do you pass this decision? How do you mix DIMMs
in on prem?
And, also,
the effect of of the retention on data replay. Something you need to replay the data because you did some wrong transformations.
So what's the effect of, like,
customer
ops team need to understand
that they need storage also for
a for a for replay.
The data skew, how data skew can affect data loss even you if you have a lot of storage.
If you don't partition the data correctly,
then you will get data loss at some point, even 1 partition.
And it's for certain consumers, this is like data loss in all partition. You need to replay the data
again. So the data aspects were also something that I learned
along the way.
So this is only an example of
1 chapter. The issue is discussed in the chapter of storage usage.
But I not only that I learned during the writing of the book, I think that if I wouldn't write the book, I would forget almost
everything.
So for me,
the for for the my the personal benefit for me is that
I know I remember
stuff,
really, like, Kafka related stuff that I knew,
and and I didn't forget them, but also that I learned along the way.
So I think the investing
10 months,
during the weekends,
every weekend for the for this amount of period to write the book was
beneficial
for my technical knowledge. Absolutely.
Unknown: Are there any other aspects of the work that you've done with Kafka, your work on the book, the overall
Kafka
operations ecosystem that we didn't discuss yet that you'd like to cover before we close out the show?
Unknown: I think we covered a lot of
technical
technical stuff. We we can discuss the
the cost reduction,
but but in very short, like, I would like to mention that,
there is a chapter on cost reduction in in Kafka,
but it relates it relates to like, I
I brought 6 examples, 6 real world examples
of cluster that I stumbled upon,
which
per each example, I I I write
how much,
I specify how much CPU,
RAM, and disk each cluster has,
and also the usage of each of these resources.
And then I ask whether the cluster can be scaled down
or scaled in.
And then I discuss
other metrics,
monitoring metrics.
And by correlating this monitoring, Kafka monitoring metrics, and
the operating system
metrics usage,
I I give recommendation
regarding whether
you can scale in or scale down the cluster.
And I think,
like,
this is, like,
the the cost of the Kafka cluster,
it's not big, I think, compared to other clusters in an organization.
But,
because for for cloud based,
I I assume that most of the deployments are on demand,
So even if you have reservation, again, it's on demand, it's not spot, so it's important,
especially
in today's market,
to squeeze in every penny that, that you can save.
So the the cost reduction part is
something that
that can help
to reduce cost,
on Kafka.
But there is a part that I didn't talk about, which is might be a bigger part even than the machines themselves,
which is the data transfer
between consumers
and brokers.
Because if you don't configure
if there is no rack awareness in the cluster,
then consumers will read data,
only from leaders,
and these leaders can be
can be will will most of the leaders statistically
won't be in the same AZ.
And for some,
for some companies, this can save
100 of 1, 000 of dollar per year configuring reco reco awareness.
But since I don't have an experience with reco awareness,
I I didn't,
discuss it thoroughly.
But
for for those who listen,
checking for echo checking if you can configure echo awareness,
in your consumers and producers,
and consumers and brokers, sorry,
this can be beneficial. So you need to check your data transfer cost,
and maybe it's worthwhile for you to invest in a
in in deploy in test in deploying and testing,
and validating the track awareness,
works. And then you will read not from the leaders, but from the closest replica
and save,
data transfer costs.
Unknown: Yeah. That can definitely be a substantial cost when running in the cloud, and that's always 1 of those,
surprise gotchas when you're first getting up and running in a cloud environment.
Unknown: Not not only when you're getting started, but also when you're,
after years,
then you you see the the data transfer. Yeah. This brings us back to what we started with regarding cloud versus on prem. Thus, it is the the amount of reasons why
it becomes, like,
I don't know, like, for some clusters,
it might be a good idea to to check the,
like,
to
I'm saying this because I came from on prem,
so it's not Chinese for me.
And,
okay, so you don't have managed services,
but it might be from
from the
financial perspective,
the valid choice for some clusters to to be deployed
on prem.
Unknown: Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd just like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today?
Unknown: Oh, also,
I I usually
work with
analytics
clusters.
And
I think that
if there was a tool
that wouldn't
would would show
at any given point of time a correlation between the traffic,
whether it's the incoming traffic
or a query or query load.
So correlating between
the the load on the cluster
and the
and the usage of the cluster
in terms of,
like, CPU,
RAM, disk,
or even internal usage. For example, let's say, a droid cluster that uses
sometimes the bottleneck is the number of workers,
or for Trello clusters, the number of split queries. So if there was something that some tool that would show correlation
between the load and the cluster and the real,
usage
and cost,
it would allow
a ops team to teams to
to better understand whether they can
save cost on the cluster, whether they can scale it down,
or, maybe replace on demands with spots, or maybe replace, on demand reservation with on demands without reservation, and then auto scaling them. So something that we show some tool that will show correlation
between usage
applicative usage
and resource usage,
and will enable to
to save costs because especially today
in today's
economic
economics, it's
it's becomes pretty important to save costs.
Unknown: Alright. Well, thank you very much for taking the time today to join me and share your experiences
of running and operating Kafka clusters and the work that you've done on the book to make that easier for everybody else to do as well. It's definitely a very
challenging and necessary
task. And as you said, Kafka is very widely deployed, so I appreciate the time and energy that you put into
sharing your hard 1 knowledge with everyone else, and I hope you enjoy the rest of your day.
Unknown: Cool. Thank you very much. Again, thank you for hosting me.
And
I hope that the audience will, will gain something from,
from this podcast.
Unknown: Thank you for listening. Don't forget to check out our other shows, podcast.init,
which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
which helps you go from idea to production with machine learning. Visit the site at dataengineeringpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts atdataengineeringpodcast.com
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.