Unknown: Hello. Welcome to the Data Engineering podcast, the show about modern data management.
When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy them. So check out Linode. With 200 gigabit private networking,
scalable shared block storage, and a 40 gigabit public network, you've got everything you need to run a fast, reliable, and bulletproof data platform.
If you need global distribution, they've got that covered too with worldwide data centers, including new ones in Toronto and Mumbai.
Go to data engineering podcast dotcom/linode
today to get a $20 credit and launch a new server in under a minute. And go to data engineering podcast.com
to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. And don't forget to go to data engineering podcast.com/chat
Unknown: to join the community and keep the conversation going. Your host is Tobias Macy. And today, I'm interviewing Bin Phan about Alexio, a distributed virtual file system for unified access to disparate data sources. So, Bin, could you start by introducing yourself?
Unknown: Yeah.
Hi, Tobias. My name is Bin.
I'm glad to be here. So I'm right now the founding member of Alexa Complaint and also,
the PMC member for Alexa Open Source Project.
I was working on the project for about
almost 4 years. And before I joined the team, I was working in Google on a large scale
and distributed systems
storage systems, very similar to,
in the same space as Alexio. And before I joined Google, I was working I was actually a PhD student in Carnegie Mellon working on
distributed systems and focusing on storage systems. So I've been working this space for a while.
Unknown: And do you remember how you first got involved in the area of data management?
Unknown: It's from my early PhD years. So actually, maybe it's on the 3rd year I start to look at key value systems, key value
study system as my 1 of my part of my PhD thesis. So I was involved in,
like, we we hacked actually
a lot into memcached
at that time to emphasize on memory efficiency and throughput.
So we had 1 paper out of that.
And then,
I actually because my work on that, I visited UC Berkeley M Plab where I get to know the founder of Alok Steel, Haoying Li, who was a PhD student in UC Berkeley MBLAP at that time,
because he is also working on similar, like, in memory storage. Yeah. So in the same space.
So we know each other. And and then after that,
he got a because I graduated and just go to Google to work. And Haoying Li, he continue
studying in UC Berkeley.
And finally,
his research project was founded by VC and get created this company. And so I, he called me and actually, I pick up the call, and I feel like this is very interesting,
and very
exciting opportunity. So I just quit Google and joined the Aluxio.
So, yeah, that's my,
Unknown: post story. And as you mentioned, Aluxio started its life in the AMP Lab at Berkeley. And I know that it was originally
released under the name of Tachyon. So can you give a bit of an explanation about what the Alexio project is, and anything
organization that you've built up to support it? Yeah. So this is a great question.
Unknown: So in the early days, it was a pro research project called Takyon at that time. Actually, it has even
less known name before even Takyon. But in the Takyon days, it was open sourced from the very beginning. The motivation for having this project is actually
heavily related to Apache Spark. I think at that time,
Spark is already
taking off from UC Berkeley. And a lot of people are looking at Spark,
and they see the great potential for this computation new computation framework aggressively using memory. So 1 complaint people always have is at even at at that time or even now, memory is still
considered as a maybe more expensive resource storage resource than the other storage capacities.
And Haoying Li, the founder of the project at that time was taking a look at this project, and he found
it can be more efficient if you have a system external to
Spark and let different Spark jobs or different Spark contacts to share data more efficiently. So that's how,
in the beginning, he, start this project. And and actually, he open sourced it. And gradually,
this becomes
more like a general purpose file system, distribute file system, rather than just,
1,
sharing or caching layer for Spark, but also for other computation frameworks,
like
Presto,
like,
MapReduce,
or even TensorFlow in a in a new days. So, that's how basically the project is evolving from various it starts as a very specific purpose, just help Spark to manage the data more efficiently,
but gradually becomes a,
in memory file system because
this in memory file system needs to handle
data persistency. Because the first thing once you talk about something in memory, the first thing people are asking
is, what if the memory if you restart the machines and you lost data from memory, how do you deal with this? So from the first day, Taqian or Alexio at that time
is designed to handle,
this kind of basically data persistency issue with even volatile
storage media like memory. So we have all so this is very
fundamental to the design for this system and becomes fundamental in architecture
if you compare Alexa with other,
distributed storage systems like HDFS or like Google file system from the paper. So this is I I would say this is like something fundamental
to the,
entire ecosystem to our entire design. And it turns out very
interesting in the days, especially nowadays, people are talking about separating
storage and compute.
Storage is getting
far far can be far far away from the compute because the storage can be some cloud storage. And in this case, having a Luxio
works perfectly in this architecture because we can be in the middle between compute
and the real persistency story since we are not targeting to solve this data persistence issue. Long story short, I think we we start
from a
caching layer for Spark using memory
to a general purpose file system. It's a file system. It's not a caching system. A distributed file system with the awareness
of some other we call ender stores to provide data persistency. So, the architecture becomes different from
10 years ago when people are talking about, oh, yeah. We should move compute to storage. Let's have cup deeply coupled storage
and compute. And we build open source around this idea. We have more than 900 contributors from all over the world joining this project because they want to contribute, for example, adapter
to read their own data source, their own
persistent data store. And we have a lot of interfaces for people to
customize. So they have different cache policies.
This open source project is getting more and more popular, and we also provide a enterprise offering. So the core features in terms all the things I mentioned so far are in the open source edition, what call community edition. We also provide
as a company, we also need to survive. We need to have a business model around this open source. So we provide enterprise offering,
which
adds more
enhancements in security
or in, like, large scale or high availability.
This kind of enterprise readiness,
features, like owning
a lot of these large corporations they are looking for when you run services in their environment.
Unknown: Yeah. So this is our business model. And a couple of the things that you mentioned in there preempted some
of the questions I have, particularly in terms of the persistence of the data in memory as far as what happens when you have a memory failure in a machine or the instance gets rebooted and how that data distribution gets managed, at least in terms of the long term persistence of the data where you rely on the underlying storage systems to provide those guarantees.
I'll probably have a few more questions later on about how you manage
distribution among the Eluxio layer once the data has been retrieved. But before we get into that, I wanna talk a bit more about some of the use cases that Eluxio enables because, as you mentioned, there was a big push of moving compute to the data because of the inherent gravity that these large datasets have. But as we move more toward a cloud oriented environment where you have these object stores that don't have any
guaranteed physical location, so you have varying levels of latency as you're accessing these object stores. Or if you're trying to run-in a hybrid cloud environment where you have your own local data centers, and then you're also trying to interface with public cloud for bursting capacity or for taking advantage of some of the specialized features that they offer. So, yeah, going back again, just wondering if you can talk about some of the use cases that Alexia enables that would be impractical or sort of too difficult to want to deal with without it. Yeah. Good question.
Unknown: The first case,
I would recommend to,
explore is really if you have data remote
from this from compute, and we see cases like people are having
a different data centers,
and they have cross data data center traffic
to load data because you wanna do a join and 1 table is in 1 data center and
another parts of the computation, other tables are in the local data center. So in this case, having a Luxio in place will help greatly
to reduce the computation time because you can either preload the data the table from remote data center to the Aluxio,
caching layer, or Aluxio have the intelligence built in to bring the data on demand. And next
time, after the code reads, next time if you read that same data again, this will be cached locally. So we do see a great performance gain in cases like
this. So we have a published use cases with Baidu,
the search giants in China. They see
performance benefits by 30 x in this cases. See, the other case, interestingly,
I see more is really the cloud,
as you mentioned. So actually, this is also what I observed. The companies of the new generation,
they typically
born on cloud. They start from AWS or Amazon
or Azure on the 1st day. And even the older generation complain, they have aggressive
plan nowadays to move their either data or compute or both
to cloud.
And once you are moving to cloud, the natural
storage choice for you is for AWS, it's s 3. Right? But s 3 is s 3 is awesome. I like s 3 a lot. It's cheaper compared to other storage capacity, and it's scalable.
It's very easy. The semantics is very simple. It's very easy to manage. A single global namespace.
However,
there are also costs come with all the convenience here. For example,
once you have all your data in s 3, your compute, if it's, let's say, on EC 2, will naturally always
drag the data from s 3 to a local compute engine. So this will go across network. And s 3 sometimes has different semantics compared to the traditional file system. For example, the rename can be expensive. And also listing a huge bucket,
a bucket with, say, 1, 000 or 1, 000, 000,
objects inside can be very
slow. So in these cases, people do want to have a
more familiar or more traditional performance implication
for the storage per storage service.
And also,
the new trending machine learning
applications, they
emphasize a lot on iterations. They take data from last iteration
and, do some computation
and output to next iteration. And from that on, the previous iteration the data from previous iteration may not be important at all. So this kind of like a you need a temporary data store rather than, some persistent data store. So we see cases on running
computation
on cloud. They won't have a
caching or another tiering
on top of the
cloud storage provided by the vendors like s 3 or Microsoft, and allows you to can fit perfectly in this case. So So I named the 2 cases. I think there is a third case, which is very interesting.
I see mostly from,
users in China. The Internet companies in China. They like to do, a topology that I have a centralized
main
HDFS
data source. But because my HDFS is HDFS service is heavily, heavily loaded because we have giant
number of different applications
depending on this data. So what we see is to
guaranteed SLA,
they also
set up we call satellite
clusters,
and each cluster is maybe a zone just for 1 specific computation service, for example, Presto.
And in this case, the data and compute becomes is still in the same data center, but it's still cross different machines, go cross network.
And because they have seen issues with, like, high pressure on their HDFS service, so they see, okay, let's put another,
storage
here collocated with my satellite compute cluster
and mirroring the data
in the
main storage.
And we don't need the whole set, just need working set. Right? So Alexa work perfectly in this case too. So we see a lot of use cases around this in this,
Unknown: especially for Internet companies in China. And so can you talk a bit about how the Alexia project is implemented to allow for these data reflections
and high performance access of data, particularly in the face of varying levels of latency, whether it's the case of interfacing with cloud object storage or the situation that you just described where you're using it as a way to keep a working set accessible to a compute layer that's remote from a centralized Hadoop or other sort of data lake that it would be too large to keep in memory of the compute instances, but
Unknown: data lake? Yeah. So as I mentioned in previous question, Alexia was born actually in the in beginning to to solve challenges like this. So there are few things we are doing to help reduce the latency for,
and reduce the
performance variation, the retrieval latency variation.
1 thing is we aggressively use memory as as the main storage media. So although we provide optional
choices, you can use Alexa to manage memory or plus
or optionally SSD plus optionally,
hard disk. But a memory a lot of we still see a lot of users use Aluxio to manage memory to, which has a higher bandwidth
when you, have a very hot data insight. And also,
we have done a lot
to provide what we call
short circuit,
data read and data write. So essentially, we aggressively
use
leverage data locality. For example, if there is a distributed application, for example, it's a spark. And a spark, if it's using HDFS
interface to access data, it will
try to allocate the tasks
closer
to the storage. And this is by having some APIs to
the,
nodes serving this data. So Alexio, if collocated
if it's deployed collocated with a spark or other computation framework, we also provide a sim compatible interface.
And in this way, we will try hard to match,
the task to our, like, workers serving the data.
In this way, they don't need to go cross network to fetch data. So, yeah, memory and data locality.
And also
yeah. So the third part actually is related to,
metadata.
So, Luxo is not just a data caching layer. We also, as I mentioned, this is a file system. So we have our own
dedicated metadata service. It's basically it's a file system. So we maintain
the INO trees and all these journal,
all these basic
elements for required for
a file system.
That means if you have a slow list operation on s 3,
that will happen once. But after that,
Alux understands how the metadata is on s 3, and we will serve this instead of having s 3 to serve this metadata
queries. So by doing this, we can reduce,
even the operation
variations on the metadata side.
Unknown: So to your point about the tiering of storage where your primary
operations are being done in memory, but you have the capability
of also using the local disk on the Alexio nodes. I'm wondering if you can talk a bit about the life cycle possibilities that you have as far as managing
when the data gets moved from 1 layer to the other in the Alexia nodes, but also as far
as aging out data that you are accessing where it gets initially pulled into the working set by
a request from the compute engines. And then at some point, it is no longer relevant, or if you need to page out data because more data is being requested than can be held within the Alexio cluster, just how that overall
tiering and life cycle management happens within the Alexa layer
Unknown: itself. This is also a core feature provided by Alexa.
Essentially,
from very high level speaking,
you can think a Luxeo is a cache.
It's a distributed cache for file like file data. So each worker, a Luxeo worker a Luxeo worker is the component
handling all the data capacity, like, really provide data capacity in it. You,
can add more or n different workers into the cluster. Each Elixir worker implements
implements a cache. And by default, we use LRU as the caching policy.
We also make this pluggable. So, we in addition to LRU, we have some other,
policies,
built in and use we do see users
provide their own
caching policies,
replacement policies too because they understand their workloads better. So that's on the caching side.
On the other side, we also provide
functionalities
for users to
actively control
the data life cycles.
For example, we provide commands. You can set a TTL
for a file or a directory.
Say, I set the TTL to be 1 day, then after a day, the data in this the data in Alexeyo can be, freed. It's up to the workloads, but we do not guarantee it's in memory anymore if you set a TTL.
And there are also policies like I want to really ping this data in memory layer. So that's the
another command you can use to really
put a data in the top tier
and closer or faster
to the computation.
So,
yeah. So that's basically, what we do for the life cycles. And we are actually looking at more, complicated
different policies
in the future,
because we do see,
especially for enterprise users, they have a more sophisticated
life cycle management requirements.
But right now, these are all available
Unknown: in open source, community edition. And as far as being able to manage the life cycle of the data in the underlying systems where, for instance, you might have initial data getting loaded into s 3 from an inbound data pipeline, and then you do some analysis or processing s 3 or some form of cold
storage.
You wanna store it long term in s 3 or some form of cold storage. Is there any capacity in Eloxio
for being able to manage that life cycle? Or would it just be based on the
mount points for those underlying storage systems and then having the computational framework be responsible for reading from 1 location, and then writing back to a different mount point for that, more persistent long term storage and archival?
Unknown: This is a great, great question.
And we, this is basically what I mentioned. Like, we do see, enterprise large enterprise. They have requests like this. They really want to have a very intelligent data management system
to migrate data from
hot storage to warm storage and from warm storage to cold storage to,
improve data efficiency and also reduce the their data cost.
And right now, with the current Eloxio
open source
edition, what you can do is I just mentioned, like, you can write ETL jobs to move data from 1
location to another, like, if they are backed by different mount points and representing different end of storage.
So that's possible doing that. And we don't have automation
on here yet, but I I believe this is on the road map in the future.
Unknown: And another question I have, particularly as it relates to metadata
is
in a computational layer, if you're trying to search for either a particular set of records
or just do a,
discovery process of seeing what data is available in which systems,
does Alexeyo
proactively
retrieve the metadata from those storage systems so that those initial, for instance, LS of the file structure
return more quickly? Or does it do a fetch at the initial retime and then just cache it for a particular period?
And then also, I'm wondering if it has any
native support for some of the more,
high level
data storage formats, such as parquet or avro, where you can potentially
query from the computational layer into the underlying files to determine if a particular set of records has the information that you're looking for before you necessarily
retrieve it into the Alexia layer for further processing?
Unknown: This is not a great question.
Go back to the metadata question part first. The boast mode, like,
loading the data on the first
operation and,
loading metadata on the first operation and put it into our metadata store.
And also,
you can also the second mode is you can also preactively
load the metadata. Like, if you know,
my computation will touch this bucket
in the next
hour or so, Let me just try to run something first to pre to to preload that metadata. So both are supported.
And in terms so this is how the metadata enters a Luxeo and being remembered by a Luxeo. And remember and and regarding how we manage the life cycle for the metadata, we also provide different policies. For example,
you can set
up cache expiration
time. The metadata I remember for this I n o tree for this part of the file system
will be true for maybe
an hour. And later on, we will just do another
load once some application is accessing data there. So this is 1 policy. The other policy is, oh, okay. So there might be out of bounds modification to my end of store
frequently. Let me just don't trust all the cache metadata. And on every single operation, I need you to check the updates on the vendor's Underscore site. So there's another policy to be very aggressively.
But we do see this will,
you can see that this will make
the metadata operations
slower because it's just adding more overhead. So it really depends on the application.
And we also adding new features so that for certain end of stores like HDFS,
there are hooks
you can put.
Once there are some modification to their metadata, you can trigger, you can register some action
on their site. So we can just preactively
update the metadata once there's modification there. So, I think this is,
yeah, this is also something
we'll do in the new feature in the new release. So the second part is regarding how we can optimize, maybe if we understand more about data structure
in the file, like parquet or do we do some push down? So right now, Alexa is a file system. So we understand
the data format as a just a very plain bit of stream here. So we do not go into the file file
structure
to,
put more, to provide more optimization.
But on the other hand, I'm working with actually some
researchers from universities
and to see,
if this is I know this is a parquet file. And, can I do something smarter?
For example, can I do some clustering, or can I do some more intelligent,
replication?
Yeah. So that's something I think it's a very interesting direction,
Unknown: and we will see how the readout goes. Yeah. And another possible approach to that would be to rely on other metadata storage systems where particularly for Hadoop, where you have something like the Hive meta store or the
iceberg table format that's being worked at,
in Netflix and with other people where you can potentially interface with those metadata storage systems that are already doing the work of parsing the records within those more high level storage formats, and then be able to use that to determine which subset of files have the information you need to then retrieve into Alexio?
Unknown: Yeah. Yeah. That's definitely another way to go. And we are
very interesting
to to see the, some like, if there's some experimental results or some benchmark. So we are very interested to see that and and see if we can just, influence their roadmap for Alexio in the in the in the in the future. And, I think this is something very
interesting. And I hope this can like, because this open source project, university researchers or some other people, if they feel this is something worth doing for their workloads,
we are very happy to provide some help or some collaboration
Unknown: and on this end. And in terms of the actual code itself,
I'm wondering if you can discuss some of the particular challenges that you faced in terms of building and maintaining and growing the project and discuss some of the evolution that it has undergone in terms of the overall architecture and design of the system? Yes. I have a lot. So,
Unknown: I should just share 1 interesting story. My first,
big project in Alexio after joining a team is to make Alexio modularized.
So if you know a little bit more about Java JVM applications so so Alexio is building Java, like 90, maybe 95 percent of source code is within Java. So the way it works is you just build Alexa into jars, different jar, binary jars, and JVM will just execute the jars.
So in early days,
because
it's a research project at that time and
developers or maintainers for the project were
really
looking for fast growing. Right? Just like move fast. So everything, all the components
were in the same
single Java module, which means
you compile a single
jar. Even it's on the client side, you put in Hive or you put it into MapReduce
or it's, it's also the JAR you you execute the service. You provide RPC service, provide the the the daemon long running daemons. And this is a really bad design, but it gives a lot of convenience because like, the Java
in Java land, because it's in a single module, you can reference all the other classes without worrying about managing dependency.
And my first project on Luxo is to break this a single,
monolithic JAR into
multiple JARs with a clean dependency.
So we can distribute a much thinner jar to applications
like Hive or MapReduce,
or Spark. And and it takes me
quite a few, I would say, months to figure out the relationship between all these dependencies,
the callers,
and then figure out the way to break this into multiple different frameworks, different JARs, different modules. And, so 1 lesson I learned from there is once you
start,
once the project starts to grow
to a certain degree, we should start thinking about architecture much earlier.
So we can have a much better management
in terms of the like, all the dependencies.
And and this will simply simplify the life down the road much easier. And after that, actually,
we are pretty happy with the structure. And but still,
after adding
new features,
more modules come in, and how to make this plugging
pluggable, and how to make different components
very easy to
upgrade.
It's also, it's forever going process. So that's something regarding the source code management and also as a Java. If you work in the Java land, you will understand the Jaha issue, which is the
pollution
of the binary libraries
on other
applications,
class pass. And it's particularly a problem, a challenge for us because
we are supposed to talk all different kinds of, data stores. So
originally, we put everything, like, we just put everything together on the class pass, and then you will see all the dependency, how issues. We spend a lot of time to dealing with that, and you have to do a lot of dirty work in, like,
workarounds or hacky
ways to solve this JARHEL issues. So later on, we decided to do this in a very clean way and use the class loader,
to isolate the
the libraries. So, in this way, we see much less issues in in terms of Jaha.
But it just it also makes the code a little bit more complicated. But I think this is totally worth it. Yeah. Code wise, this is something also we learned,
very very important.
Other, the other part, I would just wanted 1 last thing is really,
the resource management.
People know Java. Like, Java is a good language because you don't need to handle the you don't need to delete something you allocated from the heap. Like, the garbage collection
will help you to be lazy. This is only 1 part
of the resource. There's a lot of other resources. For example, the sockets you open, the locks you acquire,
especially this can be a distributed lock. And also, for example, the, the all the handle file handlers, all these different pieces, and even sometimes for efficiency,
we also use the low lower level Java APIs and there can be memory leak related. So how to handle this well to make sure you don't leak the resource? So
in the beginning, we all just because it's a distributed file system. Right? So it once you have this kind of issues, it's very hard to debug to diagnose. And, once you know that, it might be relatively easy to fix. But even to to diagnose the issue is very hard. So
we, what we later on decide in a source code we do is to really have a very strict pattern. Like, everyone should just follow this pattern when you acquire some resource,
and we leverage some good things from Java. They some good functionalities from Java they provide
to make sure, once you once you quit, like, normally, this what resource will be returned. But also,
for example, if you encounter some exception, so you don't really expect, you go the code path you you go through the code path you don't really expect. We make sure in that way, we also return the resource,
correspondingly.
So, you know, this is this also helps a lot,
to make the source code much more robust.
Unknown: In terms of being able to
write it in such a way that you can easily add new back end capabilities
for interfacing with different storage systems and also making it easy for different computational frameworks to interface with
Eloxio. What are some of the particular challenges that have come up on that front? And what are some of the trade offs that you've made in order to be able to provide a single API
to the computational layer that works effectively across all of those different
underlying storage systems that might have varying capabilities
or more advanced functionality that's not necessarily exposed?
Unknown: Yeah. So So 1 challenge, as I just mentioned, like, a different
if you want to talk to different enders stores,
usually, they require different meta
different libraries and how to make all the libraries to work together
happily. So as I mentioned, like, we use the class loading to isolate all the dependencies
to make all of them happy. And also the other part is how to
how to find a common API.
So that,
like it's
it's useful enough and also it's common enough and
to cover all the different storage,
types. And we do have multiple iterations on top of that. Right? So in the beginning, we have 1 set of API. So we call the we call this understore API.
And later on, we realized this is not sufficient, especially
when we see more use cases with object store. And the object store has very different semantics, not just the semantics, but also like a security
all the security requirements with the traditional file systems.
So, we do,
put a lot of emphasize on that to make it work. So, I think
after multiple iterations, we
gradually converge to a version that works for most of the cases. This is something we call us to a southbound, basically, how Aluxio talks to the enderscore. We also have the API. We call the northbound,
northbound API that's,
that that's what we provide to different applications.
And we on that side, most of the applications we see so far use the our Hadoop compatible API.
So in that way, applications can just assume this is just the Hadoop file system. This is, just the Hadoop file different implementation for Hadoop file system. So they can continue use whatever the, they assume,
but just chain if replace Hadoop
HDFS with, Aluxio.
On top of that, we also have our own
API, file system API, which is more similar to the Java file system API.
And adding more features like setting TTLs, doing the mounts,
and doing a lot of Elixir specific,
functionalities.
And 1 interesting trend we see nowadays is as in in the beginning, it's because people have their legacy.
They want to have run their legacy applications on the distributed file system. So they want the POSIX API.
So we do, we do have a contributor
from, I think it's from IBM contributed the POSIX API
for us.
But later on, then we just realized actually a new generation of the machine learning applications, they
leverage
aggressively on the
heavily
on POSIX API. So they typically assume the data is on the local
directory.
So this becomes
now popular again
among this, machine learning
tools. So yeah. So but, essentially, the way we handle the northbound is we provide different set for like HDFS compatible for
distributed application like Hadoop or Spark, and the more traditional
applications or even a new wave of machine learning applications we provide the POSIX API. And from the operational perspective,
Unknown: I'd like to get an understanding
of what the different scaling factors are in Alexio
and some of the edge cases that come up as you try to add and remove nodes, particularly as it relates to data distribution within the cluster?
Unknown: Great question. We see a few different depends on the workloads and the environment. We see different
different scale, scalability factors here. The major 1 is really the bottleneck on the masternode.
Our masternode, you can think it's equivalent
to Hadoop file systems,
name nodes. And because the master node
is using unkeyed Java memory to store
all the,
the mapping from file to maps, from,
inotrees
to different detailed
information.
So this can be very memory hungry.
And also this can be GC intensive. There's a lot if there's a lot of small modifications,
intensive modifications
to the file system. So this becomes 1
scaling bottleneck once you hit once you store,
say, 100 of millions of files into
a LogCIO
namespace
and,
and becomes a unstable part for the for the manager service. So in the new release in the next coming months, we're having the new 2.0 whole new 2.0 release. We are moving this part,
the on heap memory
for file system information,
files basic file system, I n o tree
to a off heap implementation
on that. And by doing this, we can scale the file system to be much larger, like contain billions of files and directories. So this is 1 thing. The other part is really
how many workers you can handle because there are heartbeats
between workers and,
master. And there are,
different communication,
the control messages sent over. So we do if you if you scale to thousands of nodes,
or
beyond, you will see the master node becomes a bottleneck
or at least
very
pressured because of this kind of workloads. So 1 way we handle this is we
testing this aggressively
on release to make sure we can scale to thousands of worker nodes. But also we,
optimize a lot on RPC sites, threading pool, all these different things. So, we really want to make it work. I think in the new release, we can handle at least
a a few 1, 000 workers
in 1 deployment. So these are the
efforts. And the the the bottlenecks we have seen in terms of scalability and our efforts to improve
on top of it. So,
Unknown: what's the other part of question? Just wondering how scaling the node count up or down impacts the data distribution in the cluster. So I'm wondering if you use things like consistent hashing to ensure that you're always able to retrieve a record from a given instance or if you use more of the name node style approach where you have metadata in the master about where all the files are located, and then also any sort of redundancy
so so that if 1 node does go away, you don't then have to retrieve the data all over again from the source system and can instead just replicate it from within the Alexio cluster? 2 parts. 1 is on
Unknown: the metadata
service part. We do provide
high availability mode. You can run
multiple master nodes, master service inside Alexa deployments. And we internally,
we use Zookeeper and index release. In the coming release, we're using raft to decide who is the primary
masternode
serving the manager service. And also on the data side,
we do have replication scheme built. So, this is actually something very interesting I want to highlight. If you think about HDFS,
the default replication factor is a 3. Meaning,
we want to have this data 3 copy
in avoid to avoid, we lost data in some time window. So that's the replication factor. And in us, because we are not aiming to provide persistence,
so, we do not have a target by default. We do not have a target replication factor. It really depends on the workload,
the temperature for the data. For example, by default, if the data is hot, then
we may the application
may just trigger more access to retrieve to data and turns out this will make more replications in Alexa space,
cached into different workers. So in the coming release, we also provide replications.
So we can set, okay, I want the minimal set of replication
to be 3 for this data copy or a maximally this can be 10 copies. And beyond that, I don't really need more. So more fine grained control on the replications
against the,
fault tolerant, like failures.
And also regarding the consistent hashing, as you as you said, like, we
use actually, we use both. We use the masternode
to,
remember where the data is located across different workers. But on the other hand, we have a policy that when we load data into Alexio
to avoid some skewness data data skewness, We can also specify a consistent hashing of similar some something similar to consistent hashing way to place the data. So we can get a more uniform distribution
across in Alexa space when we drag in data into,
from the data source. And you can set said, okay. I want replication to be 3 or 4. And then even you have 10 different workers reading the same data, and they will not compete too much for the end of store
bandwidth. And also this will guarantee
the 4 or 3 copies you specified are uniformly distributed. Basically, they will coordinate. And when I was looking through your documentation
Unknown: and blog posts to prepare for the show, and I was impressed that you consistently
you and then rely on users or beta testers to ensure that new releases
are bug free and, not subject to regressions.
So I'm wondering if you can talk a bit about your overall approach for being able
to run these verifications at scale
and also
any operational
considerations that you've built into Alexio to make it easy to be able to build and deploy these clusters in an automated fashion?
Unknown: Yeah. We put a lot of emphasis on that in terms of testing and also the easy of deployment the ease of deployment.
So regarding the testing,
there are multiple different levels of testing happening. So first, we, because I'm from Google,
I used to used to work with Google. And in my team in Google,
we emphasize a lot,
on unit tests and integration tests. So I I try to when I join the team, I try to bring the same,
mindset.
And we do have a lot of we do have requirements for the unit test. Anyone who contribute the code will ask, okay. Can you also write the unit test? And also, if a possible, get a integration test too. So these are building with a source code coming with the source codes. Once you try to submit something to the our GitHub repository,
this test will be automatically triggered so we can have some sanity check. But later on, we realized, okay, this is not sufficient, especially
after a few releases. We noticed
if some issues will only be,
identified if you're running
this service for for a while, like 4 hours or 4 days or even 4 weeks. So what we do is we do,
we build more tasks. We called nightly integration,
nightly task, which we write
the whole
info entire infrastructure
on AWS
to, like, launch this test nightly so we can run different workloads including high bench, including TPC DS, or more micro
benchmark like a just pressure the master node or just pressure the worker node and see, what's the result in terms of performance in in terms of correctness. So we do have 90 build 2. And before release, we have more
strict
tests, like, in more complete test suite for the release.
And also, we will ask,
partners, friends,
or, users heavy users
in, in a community to help us test the beta in their environment. Even that, you know, bugs are
they will always be bugs. Right? So we,
we will also
try hard to help users to get the bugs
fixed in the first time, in the in the in, like, immediately. So we have the we just recently launched the Slack channel so we can talk to the users more in a more real time fashion.
But also we talk to these users like periodically
to check out whether they see any issues. If they set any issues, we will just try to solve them, really solve them, help them to solve these issues earlier. I I I think all these are actually common practices in from different,
industries and companies, and we just try to
implement it in our own, suiting our own team and our environment,
and it works pretty well so far. And the regarding the skill scalability task you mentioned,
that's something we,
actually, we were very proud of. Like, we are doing this test with thousands of, like, worker instances
to, really pressure test our service. And we do spend a lot of efforts on this to make it affordable
calls because we always use AWS or the cloud infrastructure to build our, tests. So, as I just mentioned, we have a blog on that, and we actually have a white paper,
summarizing all the caveats. And, we see we're running this scalability test on AWS. And I think it's a really good read.
Unknown: And going back to the
deployment topology,
as you mentioned,
the primary use cases for being able to provide data reflections
so that you can get faster access to some of these underlying storage layers and possibly at a remote location in the case that you described with the Chinese Internet providers.
So can you discuss a bit about what the typical topology is in terms of how Alexia is deployed in relation to the storage systems and particularly with the compute? I'm wondering if you have the compute operations running directly on the same nodes that Alexia was using, or if they're just collocated
from a network perspective.
Unknown: So we have our recommendation
to co locate Alexio with the set of nodes running Alexio
to be the same set of nodes running the,
computation, for example, Presto or Spark. So in this way, you can trigger what we call the short circuits
data IO, read or write. And so they don't need to go through because we try to aggressively leverage data locality.
So if they happen to be on the same nodes, you don't have to go through network. So that's the recommendation
for
topology to deploy Alexio. We also see cases that Alexio
is not really
located collocated in the same set of nodes,
due to a variety of different reasons. And in that case, it's really depends,
what's the bandwidth and and also what's the,
latency
from your compute to your data source versus your compute to Eloxio
deployment.
If it is still a large ratio, meaning even it's not on the same,
set of machines, but it's within the same rack or a same data center versus the remote data source is really remote. It's across the ocean.
Or in that case, I think that's still considered as close enough words. And and that's basically what we,
see. Essentially,
the recommendation I will say is make sure Alexa is close enough
Unknown: to the compute compared to how the data source is to compute. And for somebody who's planning a deployment of Alexa, what should they be thinking of in terms of system capacity as far
as RAM that's available
and disk that's on the system, whether it's, spinning disk drives or SSDs,
and any sorts
of optimizations that they can make either at the node or network layer? The system requirements
Unknown: is not really
I I I would say for the master it depends on the master or worker nodes. For our manager service running on the master node, you probably want some
relatively beefy nodes
with enough memory. We'll see, like, on the other 10 to 100 gigabytes. So you can have enough space to,
store the file system in, file system information and making sure you don't really see a lot of g c's. For the worker nodes, it really depends on your target SRA. If you're really talking to a very
strict SLA,
returning data in a very short latency
compared to reading from disk, then use Alexa to manage and to allocate and manage
memory for you. But if you are really dealing with the SLA to reduce the the target if your target is really reduce the latency of reading data from a remote data center. In that case, I will say, like, having a LogCIO to manage SSD or hard disk is
already good enough
for really good good enough for the workloads. Because in that case, having data in memory
versus having data on as local SSD or a disk do not really provide, like, significant
gain. On top of that, You already reduce the network latency
by,
moving removing parts, dragging data from a remote data center. So that that yeah. That's my philosophy to set up Alexio deployments.
Unknown: And what are some of the cases
Luxio for a given use case?
And I'm curious if there are any other projects or products that are working in a similar or possibly adjacent space that might be a better fit for those situations.
Unknown: There are cases we think Alexio is not really the best fit,
including if you have your
database
is your source of choose.
And Alexa right now is integrating
with file system or objects 2 or more.
And, database is not really, 1 available or ready to use data source for Alexa.
And in that case,
having Alexeyo in between,
is not really I don't think that's the target use case. And another use case we think is not really a good fit for Alexa is if you have already colocated computation and the storage
and your network bandwidth is awesome
and your hard disk bandwidth is also
has no pressure at all,
like where your memory is a lot. Then from time to time, we see there is also marginal benefits running
on this case. But we do also see some other benefits if you run-in the colocated case. Like, I mean, when I say colocated, I mean, the data source is already colocated with the computation.
So we do see use cases
by putting Alexa again in here
for SLA because different applications are competing for the disk spindles
and they do see large variance, but serving Alexa serving data from Alexa memory will just increase the bandwidth data bandwidth. But if that is not a problem, then in the collocated environment is not our target use case. Let me see. The similar or adjacent space,
we do see, for example, there is a product called s 3 guard. That is really more like a cache for s 3, but that is only for the I think, that's only for the metadata part. So if you want to cache data,
that does not provide you. So Aluxio does both data caching and metadata
management. So s 3 guard is 1 way to translate s 3 APIs, the of your store into a
file system API
for, different applications to consume. But that that's only on the metadata side. And there are also
some projects we see
doing similar, but only on the data pass. They do not handle the or they do very
straightforward
translation
on the metadata side. For example, Hadoop itself, you can use Hadoop to access s 3 directly. That's no problem. But it's just doing a client side translation
and use some different client to access s 3. And there will be no data caching and no metadata caching either. Yeah. So and also we see cases,
they do only data caching, but not the metadata management.
So they, I think we see a few
GitHub repositories
providing functionalities like this. Essentially, they just use a local
node once you read from s 3, and you just put this data. You download it from s 3 to your local node. And once the same node request data, you can just serve it from local storage. But this is not a really distributed.
There's no coordination between different local cache. So this is very
I would say this is very
single nodes, and there is no coordination
to help
applications to access data from a different,
cache on a different nodes. So yeah. So my in general, my takeaway is our
project is pretty unique. I guess 1 of the reason is in the early days is it was a research project.
So, we we just do whatever
this is, this is, we think is interesting to do. And later on, it turns out, has
real use cases. And, it turns out, solve real prop important problems for users. And it's open source, and I think a lot of users just take it,
instead of having similar launch a new similar project. And what do you have in store for the future of the Alexio project and company? So in the near future, we are releasing Alexio 2.0.
As I mentioned,
there are major upgrades to Aluxio,
including we're moving, I know, trees to our off heap storage. So this is something we are doing. And also we are changing the entire RPC system to be more scalable, more extensible, and a lot of new feature added features like async propagation,
from Alexa space to the data source. So that's really in the near future, in a few months. And in the long run,
Alexa is really
we just want to be the unified
data access layer for any applications. And now we start from big data world. And in the future,
we hope this can cover, like, different different other areas for different workloads.
For example, we start to also
see more and serve more machine learning workloads. And so we just go across more areas. And also,
we used to see only
people use big data or mostly use big data for unstructured data. But nowadays, in the last few years, definitely, people build more also build more systems to derive insight from struck more structured data. Like, for example, Presto or Spark SQL,
all different
SQL query big data SQL query engines. That's also becoming a
trend we see, and we hope we can have more integration,
deeper integration with this kind of systems. As for the company, we are
still early stage,
I will say. So,
also we in the early days, we focus a lot on engineering,
And
now we think open source is something we really
want to focus in the community. I mean, the user community is something we really want to focus
in the in the next few years to help really understand how users are using Alexeyo and understand how we can help users to improve that, improve the value. And actually, I'm part of the efforts. And if you need any help or see any interesting you have any interesting ideas, feel free to talk to me and we'll be very happy to have more discussion.
Unknown: And for anybody who does want to give you that feedback or follow the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today? I think 1 actually very
Unknown: important change in the last few years is really the trend moving data to the cloud.
And I will say this is something you see every maybe 10 year such as, such a huge change.
And cloud providers,
cloud storage vendors, they just provide their own mostly objects to our service
with some, maybe with some file system wrapper on top of that. But still we are still in the early days to learn how to build a right architecture when you really move your computation to cloud or is a public cloud or on premise cloud. When this happens, how do we really architect
the system well?
And based on this new architecture,
which means
the computation and the storage are more and more
disaggregated.
Then what are the different
possibilities,
new possibilities, and what are these toolings associated? Like 1 thing you mentioned,
how
to automatically
migrate data from hot to warm to from warm to cold storage.
And
also, for example, in the future, can we have even a smarter way to decide which cloud storage vendor I should pick based on the price or based on the topology.
So all this data management, I I just see we are entering a new
era that we are able to do a lot of more things.
And, I think this will take a few years for us to really understand,
Unknown: what are the new things we can do in this new era. Well, thank you very much for taking the time today to describe the Alexio project. It's definitely a very interesting platform and 1 that seems to fit a very big need in the big data community. So I'm happy to see the work that you're doing on that. So I want to thank you again for your time, and I hope you enjoy the rest of your day. Yeah. Thank you so much, Tobias. I'm very happy to share my experience.