Unknown: Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
Atlan is the metadata hub for your data ecosystem.
Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities.
Push information about data freshness and quality to your business intelligence,
automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans could focus on delivering real value.
Go to data engineering podcast.com
/ atlan today, that's a t l a n, to learn more about how Atlan's active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork, and Unilever achieve extraordinary things with metadata.
When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends at Linode.
With their new managed database service, you can launch a production ready MySQL,
Postgres, or MongoDB cluster in minutes with automated backups, 40 gigabit connections from your application hosts, and high throughput SSDs.
Go to data engineering pod cast.com/linode
today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don't forget to thank them for their continued support of this show. Your host is Tobias Macy. And today, I'm interviewing Isaac Brodsky about Foursquare's unfolded platform for working with spatial data. So, Isaac, can you start by introducing yourself? Thank you for having me on the podcast today. I'm Isaac Brodsky. I'm a software engineer
Unknown: with Foursquare. I've been working on Unfolded for a couple of years now. Unfolded started as an independent startup end of 2019,
and about a year ago
joined into Firstware.
Unknown: Do you remember how you first got started working in data? Absolutely.
Unknown: For me, that started
a little bit out
of college.
I remember
before that, I think that most of data that I had worked with was, like, some small MySQL
databases or something like that, so not big data.
Out of college, though, I joined Uber
and was working on their marketplace data teams.
And this was, you know, quite a big shift for me because I went from working on, like,
small data problems, smaller applications
to
data applications that had, of course, you know, very, very large volumes because it's all of the
supply and demand indications in the Uber marketplace.
So I started there. I started working on kind of 1 aspect of that system, 1 of the query systems.
And over the next, I wanna say, roughly 4, 4 and a half years at Uber,
it kind of expanded what I was working on to other parts of the data
ecosystem. So I started from query systems and then moving into
storage and database systems
and moving into kind of a a little bit of a niche area
around this system Uber was built in called h 3, which is a spatial indexing system, this hexagonal
grid system.
That was something that we were subsequently able
to open source
a few years after that.
Unknown: And so you mentioned a little bit about what the unfolded
platform is. I'm wondering if you can just give a bit more of the story behind
how it came to be and why it is that this is a problem area that you're interested and motivated
to continue working in?
Unknown: Absolutely.
So Unfolded kind of came out of this experience
of me and my
unfolded cofounders
at Uber. We were there working on
geospatial data in kind of all different
forms, me on the back end and indexing side, couple of my cofounders on the visualization
and front end side. I'm 1 of them also on the data science
side of things.
We spent, you know, several years just thinking,
breathing, living, geospatial data, and we had a lot of ideas for where we can take geospatial geospatial data next
and how we can continue developing some of the open source software that we had all been involved with at Uber, you know, in the the open source ecosystem.
That led us to start in unfolded
at the end of 2019
with this
idea of we wanna take
how we work with geospatial data, how we evolve that open source ecosystem to the next level.
That was really kind of the founding premise, I think, of unfolded.
Over the next roughly,
year and a half, we built up this platform, which we call unfolded studio or, you know, unfolded, you know, platform,
you know, released that publicly
and built up a geospatial
data tooling. So something where you can bring your data,
work with it in our Fold Studio, view it, view it very performantly directly in your browser,
and hopefully get some useful insight out of it and make some business decision based off of that data.
And we we built up that product over that that year and a half.
Like I said, roughly a year ago, we joined it to Foursquare.
And
to some extent, that's that's changed things a little bit now that we have kind of this broader
geospatial organization around us. But to some extent, it's also kept things a little bit the same because there's still this focus on,
obviously, very great focus on geospatial data, and there's been a great opportunity to continue developing with the Unfolded Studio platform.
Unknown: In terms of being able to work with geospatial data and build analytical use cases or kind of mapping or visualization use cases around it, what are some of the core challenges that are often presented to people who are trying to engage in that type of information and use it for various purposes?
Unknown: Absolutely. I think there's a few really main challenges that I see come up again and again.
1 is just that the scale of geospatial geospatial data can be really significant, and that can be a really
big impediment to getting started with it. If you think about, like, you know, I'll take 1 example, which is, like, satellite data, like, Landsat or raster data. It's kind of remote sense, and that's what we call it in geospatial parlance.
These datasets can be absolutely massive.
They can be on the scale of, you know, terabytes, hundreds of terabytes, something like that.
And working with these data,
even with big data tools, can be quite a challenge.
And you can think that even kind of basic things that you want to do with this data, like visualizing it, can be challenging.
That's kind of an extreme example of it when you're working with, like, satellite data.
But even for
a lot of companies that have geospatial data in some form,
this data can often be a challenge to just visualize and just see in a very basic level what you actually have in your data. It can be a challenge because it's large, or it could be a challenge just because it's not in a form which is really readily
viewable on a map or plottable on a map. So an example of that would be a case where a company has,
let's say, a customer list with ZIP codes or addresses or something like this. There's some kind of geospatial aspect to it, but, obviously,
ZIP codes
somehow correspond to a place in the real world.
But at the same time, unless you're really familiar with how to use ZIP codes,
it can be a little difficult to actually plot this on a map and actually see, does this data set does it make sense? You know, are are these ZIP codes all in the same state or metro region or something like this? What are the outliers?
This is something that unfolded studio does pretty well since we allow you to just click a button, shapeify, and get the geospatial
aspect of your data in quite, quite easily.
Another area I think that is
kind of important to
working with geospatial data is understanding where that data came from and what the
geospatial implications of it are.
I would say a good example of this is whether that data came from like satellite imagery
or a GPS log or from
a state, you know, boundary database or something like this. And all these behave a little bit differently. And I think you have to be kind of cognizant,
at least a little bit, of of where that data came from and
what that means for it. And when I say what that means for it, an example I have is, like, for GPS log data, which we usually think of GPS as just being the location that you're at and, you know, providing
is exactly where you are on the earth. And the reality, of course, is that there is always some kind of error that gets attached to that data. Right? If you're in, like, a dense downtown area,
there are radio signals bouncing around,
And this means that GPS just might not know where you are very well. And I think that's something that you have to be cognizant of working with geospatial data that, you know, for however much you want to understand the data and you understand
what's happening
in the real world, there's always some limits to that. As far
Unknown: as the other interesting element of the geospatial
sources that you're working with, a lot of times there's also some time element attached to it that you need to factor into how you're building your analysis or how you're using that data.
Because if you have, for instance, satellite imagery,
if it's from 30 years ago, it's probably not gonna do you much good if you're trying to figure out where you want to, you know, figure out a building site for right now because there's a good chance that somebody has already taken that building site because your satellite imagery is out of date. But I'm wondering how that kind of chronological and time element factors into some of those relevant geospatial spatial data challenges.
Unknown: Absolutely. It's actually a really good point that the the time dimension and the time domain can be a really challenging
aspect of it. Like you said, there's always this push for having the most
recent and up to date data. I think the example that you gave is
really spot on that all of these use cases really did not have an up to date data.
There's another aspect of it as well, which is
that the time dimension is very challenging
when you have things that are varying over time.
So if you wanna look at, let's say, demographics
or patterns of business transactions or things like this, they're varying over time.
You know, for non geospatial data, I think we have kind of okay or, you know, a little bit more tractable tools for this because you can plot it on a chart and you can kind of see what is that trend over time.
For geospatial data, this becomes
quite challenging
because not only do you have this time dimension, which introduces, you know, a much larger dataset, but you have it in geospatial dimensions. So now you're multiplying it by, you know, perhaps, 2 more dimensions or, in the worst case, even 3 more dimensions.
And this just leads to an explosion of the amount of data that you need to work with and that you need to if you're in a a full studio, for example, that we need to load into the browser.
For some cases,
this leads to a situation where people have geospatial data. They have geotemporal data. They have this combination of spatial and temporal data,
but they can't,
like, even visualize it or work with it because they don't have the tools to even bring it into
a tool
for for viewing that or for analyzing that. And they look at, like, very constrained slices, which I think is a very limiting way of looking at the data and working with the data.
Unknown: As far as the
existing
tooling for working with these types of information,
I know that there are other products such as
the OpenStreetMap
project. There are things like Mapbox and Carto that are very focused on the kind of geospatial analytics use case. I'm curious what the
state of
the geospatial ecosystem
looks like and what was missing that made the unfolded platform a useful and valuable entry to that ecosystem. You know, looking back kind of a few, products, what geospatial
Unknown: kind of meant to a lot of people, products, what Geospatial
kind of meant to a lot of people
was more of this cartographic
and, I would say, GIS use case. And this is something where we're
we're doing things more like plotting, you know, boundaries, plotting locations on maps, and it's a very large industry, of course. It's a very important 1. Obviously, a lot of utility companies and and governments need to know exactly where things are in the real world. And I think that's kind of what GIS meant to to a lot of people when they heard these terms like geospatial.
More recently, though, there's been this kind of bringing this
to big data systems.
And Advent Technologies like H3 and and S2 before that have made it possible to have big data systems that contain
analytical
geospatial and geotemporal data.
At the same time, we kind of only had
a lot of the big potato tools around that. So we had
notebooks, we had Spark, we had kind of all these tools that we were using to process this data,
but these are very
low level tools, I would say, for working with this. As somebody who wants to get insights out of a geospatial dataset, you would need to go into
these low level tools
and
figure out exactly what your query is, figure out exactly how to get your data out,
all these different things, and you're not working with it in a kind of spatial first, visually first
way of working with the data.
More recently, there's been the shift, like I said, to the this analytics data.
And some of the things that we noticed at the start unfolded
was kind of a need to be able to bring in
these large analytics datasets into
the browser for analysis.
We, of course, wanted to do large analytics datasets because these are kind of the datasets that we feel have a lot of tremendous amount of value,
but they were really difficult to work with. So this led to
the creation of some technologies like hex tiles, which allow us to bring in the appropriate
set of analytics data into the browser
at the right time and in a format that we can use for analytics.
And it led to the
focus
of the product as being this browser based application of oldest studio
in order to allow for quick iteration on the data, in order to allow for actually visualizing
what geospatial data is it that you're working with. As far as the actual H3
Unknown: library and the hex tiles format, I'm wondering what are some of the types of
metadata and
the data modeling elements that go into
the information that are contained within 1 of those tiles or cells or however you want to think about this individual unit of information that you're pulling into the browser and exposing for being able to analyze and manipulate?
Unknown: So when we think about modeling
hexagon data and
textile data, it is a little bit of a different way of modeling data than I think we normally think about.
We think about,
let's say, points or polygons or things like this. We kind of have a little bit of a
more immediate sense of what, might represent.
When we think about a hexagon,
it's a little bit more abstracted way of working with the data.
When we talk about having a hexagon worth of data, we're talking about this hexagonal shaped grid area on the earth and associating some attributes with that.
So if you look at, for example, the unfolded
data catalog that we have of just, you know, freely available datasets that people can bring into their
analytics.
Some of them include things like census demographics.
It includes,
you know, land land use and land cover status and improvement status you know, human improvement status. And it also includes some things like
whether there are roads there, and you can also imagine a lot of other datasets that you can bring into this that are things like political boundaries and admin boundaries, which is, you know, which state this is, which country this is, which city this is.
It's a little bit of an abstract way of working with the data, though, like I said, because you're associating
this property, this attribute.
For example, what is the population
of this area with a hexagonal cell?
And, when you're doing this, you're saying that in this
area, essentially, this is the amount of population
that I'm going to attribute to that particular
cell. So we're going from
this geometry as the census defines it, which are these, you know, roughly block areas
into this hexagonal grid. In particular, there's kind of a natural question of, okay, why do we want to bring it into this hexagonal grid? And I think there's kind of a couple reasons that that way of looking at the data, that way of modeling the data, are advantageous.
The first is by having it in a common grid system.
It makes
it tremendously easier
to bring together different data sources.
So for example, we can bring together census data,
GPS log data, remote sensing information,
things like this,
all into the same grid system and join based off of that. And that allows us for to have a
clearly defined
join between these different layers of data
and bring together data that comes from very diverse sources,
for example, census data and remote sensing data, bring it together in the same
analysis.
And it's, of course, possible to do that with other methods, but this 1 has this unique advantage of having a common join key and
being able to then add on additional layers of information as you see fit. I think another advantage of this approach of projecting things into a common grid system
is that you do have this common grid system then.
My background is more in, like, computer science and computer software engineering.
And so to me, certainly,
it seems really comforting to have this this grid system because I can
define how I want to work with my data
in terms of this grid system and in terms of this nicely abstracted
mathematical
construct over the data Rather than worrying about exactly where these points are and does this point relate to this other point,
how exactly that happens,
I can instead think about it as the grid and how I work with
data in a grid form. And so this allows me to
step back from some of the
really
low level details of the data and get more of a high level sense of what's going on in my dataset
and what are some of the interesting,
you know, analytics and conclusions that I can draw from it.
Unknown: Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.
Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks.
By the time errors have made their way into production, it's often too late and the damage is done. DataFold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests.
DataFold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production.
No more shipping and praying. You can now know exactly what will change in your database.
DataFold integrates with all major data warehouses as well as frameworks such as airflow and DBT and seamlessly plugs into CI workflows.
Visitdataengineeringpodcast.com/datafold
today to book a demo with DataFold.
And so now that you have the information in this common grid system, you have a way to easily abstract across multiple different sources of data and put them into a common working space to be able to join across them.
What are some of the additional capabilities
that you built on top of that with the unfolded platform and some of the primary use cases that you were driving towards with the kind of initial build out and now as you continue to progress the product, some of the additional use cases that you're looking to add into it?
Unknown: Absolutely. So this is kind of low level
infrastructure. It's in kind of foundations that we need to do
these really interesting geospatial analytics. Just this capability of unifying your geospatial data together is that foundation.
Like you said,
now that we have that foundation,
we're building out these
analytics capabilities
on top of it. The basic form of this is you can bring your datasets together and the immediate next step is, okay, now I want to do a good example of this, I think, is something like site selection, which is, tremendously important to the retail marketplace
as somebody who's, you know, operating
or wants to operate or is, you know, in any way thinking about retail, of course, the location of where that retail establishment is is of really
importance.
And
working with geospatial data gives you this opportunity to see, you know, is this a good location to open a new store or to invest in 1 of your 1 of your stores?
Having this strong foundation of being able to bring different datasets together
makes that a stronger
analysis because you can bring in, let's say, demographic information
along with transaction information or
maybe footfall patterns, patterns of traffic in that area.
Then on top of that, you can do kind of simple analysis of, okay, maybe I want to weight these different variables in different ways to come up with my score of,
you know, whether I wanna select a site or not,
then even more advanced
reporting and analytics capabilities that you might wanna build on top of that
to communicate your results and to, you know, communicate this to other stakeholders in your business.
And, like I said, even think of more advanced analytics use cases that you wanna build on top of this that might be using more more involved or more advanced
analytics approaches involving either machine learning or, you know, other advanced, stats methods.
Unknown: And as far as the machine learning element, you mentioned that the unfolded platform has a browser focused capability as 1 of the use cases, but I also know that it has a Python SDK available to it. And I'm wondering if you can talk to some of the ways that you can factor those geospatial elements into an existing machine learning workflow that an organization might already have set up.
Unknown: So I think there's kind of 2 aspects to that. 1 is, you know, as you mentioned, the browser based workflow, and I think that as of a few years ago,
everybody would have heard
browser based and would have said, okay, it's kind of less capable,
and there's not as many APIs available, and you can't access some of the, you know, cool advancements in hardware acceleration.
And that's just completely inverted as of, you know, today or maybe as of a couple years ago because browsers are now able to
actually
access
graphics hardware and graphics acceleration.
So there's been this
tremendous proliferation
of
tooling, which works with browsers and which takes advantage
of,
modern capabilities. So that's 1 area that, you know, I I don't have something where I can say, like, okay. We we can do this, this ML pin in the browser today, but there's,
I think, a very clear path
of getting there now. And you mentioned this other aspect of UnfoldEdge
as a platform for managing your data, which is, of course, the SDKs and APIs that we build around the unfolded platform.
And this is a, you know, important part of
unfolded
as well because it's 1 thing to just be able to fit your data into a map, but it's another thing to be able to, as you said, manage it and bring it into other workflows in your in your organization
or in your use case.
For that, we've built up this set of
SDKs and APIs
that allow you to
connect the unfolded platform to your current data sources so that you can just simply run a query and get your data into a map
as well as bringing your data in and out of the unfolded cloud.
So if you want to do that via a browser, that's great. But if you want to do that same operation via an API
or via an SDK, for example, in, in Python,
that's also
readily available to you. And this means that the same tool that you're using for visualizing your data and for doing your human
driven analytics,
you know, the part where you're actually at the mouse and keyboard and looking at the data,
you can bring that same workflow
into data pipelines or into automated analysis where you're plugging in instead that Python SDK
and you're accessing those datasets.
I mentioned accessing, like, text file data
of census, for example, and you can also bring that in through through SDK. So you're not just looking at it in a browser, but you can also be looking at that
in, for example, in in online
ML model where you wanna use that as 1 of your variables. Use that as kind of a
for your machine learning use case.
Unknown: Now digging more into
the unfolded platform, can you talk through some of the overall implementation approach and the architecture that it is built around to be able to power these geospatial use cases and also being able to be performant when you are working in that browser environment?
Unknown: Absolutely. So there's a couple of aspects to it. I think that the
driving
idea that we had when we started and 1 of the things that it really allowed us
to scale globally here is around a sort of hybrid front end, back end
system.
Yeah. And this was really important to recognize the
both the strengths as well as the weaknesses of of both of these
aspects of working with geospatial data. You know, browsers are tremendously
fast and
performant
of working with data that they can hold in memory, but they do have these weaknesses of not being able to
load an extremely large dataset directly into the browser.
This was kind of the
the guiding principle, I think, that we started with. We built up from there.
So we took the the strengths from browser implementations where we can bring in data and we can work with it very quickly and we can give very quick feedback to a user. We use technologies like WebGL and WebGPU or using the graphics hardware
on somebody's laptop or desktop
computer, and now even on phones, of course, to accelerate
how we visualize this data.
But we also pair that with
a cloud back end where we're able to work with larger datasets.
So an example of that could be somebody who could upload in a dataset to the unfolded platform, but maybe that's too large to comfortably load into
into a browser system.
They can upload that to our cloud platform, which is using we have standard cloud services.
And we can put it through a process with hex tiling, which is a, essentially data
preparation job that runs on our cloud environment
and preprocess it into a form
where it can
be served
piece by piece, tile by tile into the browser for analytics.
Having this combination of a front end and architecture, I think, was really important to our ability to work with a variety of customer datasets.
Unknown: Now that you've merged with the Foursquare team, I'm wondering, what are some of the
additional
opportunities
and new constraints that are provided by being part of that larger organization and some of the ways that that influences
the ways that you think about the goals and the specific areas of focus for the platform as it continues to grow and evolve?
Unknown: So joining into Core Square has really expanded our ability to work with, I think, some of
the large and interesting datasets that we want to use to power these analytics. So we've been able to release more datasets as part of our data catalog,
and we're always working
on more data,
both open data as well as
proprietary
Foursquare datasets
that we can deliver to customers through through that platform.
And I think that that's been a great advantage of joining into
to Foursquare is because the company has this overall focus on
geospatial and geospatial data that
goes together just tremendously well with having a geospatial data tool because,
of course, you do need to have geospatial data available to you. You asked about constraints, and, you know, obviously, joining into a company always brings its own, you know, new challenges and rewards and just ways of working with people. But at the same time, I really felt over the last year or so that we've been with Foursquare that we've been able to
continue developing the unfolded studio product
and, you know, continue developing this in a direction that
that makes sense to us as
as people are excited about geospatial technologies. So I'm really tremendously excited about what we've been developing and releasing over the last year, and to me, I see that as kind of extension of what we were building with unfolded before.
So if you look at, like, our release notes on the unfolded
website, you'll see that continuing
cadence of release and features and release and updates to our platform.
So from that, since I don't see a huge set of new constraints that get added, you know, Foursquare being
independent and being geospatial focused already,
I think there's a very good match for what unfolded
has been trying to build.
Unknown: And so for somebody who is interested in being able to build analytics around geospatial information, either assets that they already have or assets that they want to be able to use from your existing data catalog. Can you talk through what the overall workflow would be from,
I have an idea. This is something that I want to either figure out, or I want to build a project around this geospatial information
through to using the unfolded platform and its various capabilities
to actually delivering that in a production context.
Unknown: So I would bring in unfolded really on it early on in that journey
of understanding
your data sets and what you can do with it and productionizing
that. The reason for that is that I think it's
of
really, really high importance
to, as early on as possible, see that data on the map and see what is the distribution of the data.
As I'm sure, you know, your listener knows,
the quality of data
and what the actual underlying data is
matters tremendously
when we're building
data products or we're building things around data.
And you really,
really have to know what is happening in that data,
where did that data come from, what does the data represent
in order to do anything useful with it.
And
I think that getting a good understanding of that
really starts with
visualizing it and exploring it and understanding
what is in that dataset.
So I would say that workflow really starts with bringing the data in, just, you know, dragging and dropping it into the Unfold Studio product,
and seeing that auto map and seeing what is the distribution of data. And you might learn something surprising at that point even of, like, oh, maybe I don't have data from a certain state or I don't have data from a certain country or something when you thought that you did.
Once you've been able to do this kind of exploration phase and understand what's in your dataset,
then I think that there's kind of an iteration phase of, you know, trying hypotheses,
trying analytics on top of this data set,
you know, to the extent possible, seeing what you can
learn from it and what are the actual conclusions from this data set. This is kind of this loop of, you know, visualizing,
analyzing,
trying to derive different statistics from it, see what's
what conclusions you actually read from the data. And this can all happen in the Unflow Studio
product and by doing things like looking at different aspects of the dataset,
projecting different metrics, or projecting different analytics from that dataset.
The next kind of step in this workflow, I think, looks something like
determining how to share these results and how to
communicate results to stakeholders and to others that
you need to present this to or you need to convince.
So this could look like reporting or charting or things like this that you add on top of the geospatial form of the data
in order to make it clear what is going on in that dataset.
And the next stage of that is really sharing it. So that could be if you're working with, like, an open data set or with public data that could be publishing your map or publishing your map to your stakeholders.
It could
be embedding that map into
an application or dashboard, something where you're taking, like, the unfolded studio map or you're working with 1 of the SDKs we mentioned earlier
in order to actually put that into your own website
or into your own dashboard or something like this so you can see this map
in the appropriate storytelling context or in the appropriate reporting context so you can see it alongside
kind of the other data points and numbers
that help to bring that map into context.
Unknown: Once somebody has the information
in the unfolded studio, they're analyzing it, they're working with it, what are some of the, maybe, additional
libraries or tools that they might want to pull in to bring alongside it to be able to, you know, maybe juxtapose
the map that they're building up in the studio with, for instance, you know, a
leaflet map or an open street map to be able to say, okay. These are all of the maybe
political features that I'm building up from the census information that I have, and I would then want to overlay that on top of the kind kind of real world geographical map that I'm used to looking at or just be some of the other, you know, ways that they might think about working with that data and being able to
bring understandable
context for themselves and for the people that they're presenting it to.
Unknown: What you mentioned, I think you can do in Unfold Studio
as it is without going out to other mapping providers, or you can print those data providers into
Unfold Studio. So I often talk about, okay. I'm gonna drag and drop a file into
unfold studio, and I'm gonna view that on a map, paper him over a bunch of other capabilities in the Studio product to bring datasets in.
So I talked about bringing files in. I talked about bringing this data in as this hex tile format, which which we've been working
on. But a full studio also has a number of other capabilities for bringing datasets in that could range from data connectors or, you know, datas from your
your enterprise data warehouse, let's say, in Snowflake.
It could range from external files that are just being brought into the browser as, you know, the URL.
And it could also include data
which is coming from sources like vector maps or raster maps that bring in this kind of, you know, natural mapping context to what we think of more as, like, the mapping context
around the data. So that could be, for example, the boundaries of
states or cities,
or it could be from Raster. I'm thinking of something like where where is the land or where is farmland or or things like this to be brought into
Unfolded Studio.
So a lot of that could be brought into Unfolded Studio.
I I think you have a lot of good
examples
of different types of data that you'd want to use
to put your map into context.
I would say there's kind of 1 other aspect to this, which is other tooling that you wanna have available
alongside this. And this kind of gets us again, the the SDKs that I mentioned before,
for example, the Python STPA,
or maybe you want to work with this in a notebook environment
so you have both your Studio and visual tools for working with the
data, alongside
some of the more traditional data science tools that you might also be familiar with, that you'll be working with in more of a notebook environment and more of Python and Pandas or Spark or something like this
and making the data available alongside it. If you're familiar with those tools, that can be a really powerful combination.
Unknown: Unstruck is the DataOps platform for your unstructured data.
The options for ingesting, organizing, and curating unstructured files are complex, expensive, and bespoke.
Unstruct Data is changing that equation with their platform approach to manage your unstructured assets.
Built to handle all of your real world data from videos and images to 3 d point clouds and to geospatial records to industry specific file formats, unstruc streamlines your workflow by converting human hours into machine minutes
and automatically alerting you to insights found in your dark data. Unstruck handles data versioning, lineage tracking, duplicate detection, consistency validation, as well as enrichment through sources including machine learning models, third party data, and web APIs.
Go to dataengineeringpodcast.com/unstruck
today. That's
unstruck,
and transform your messy collection of unstructured data files into actionable assets that power your business.
Now going back into to the data modeling question, but from the other side, as you are actually preparing your data for being able to load into unfolded
or preparing it for being able to be
analyzed in that context and in that environment. What are some of the considerations that go into thinking about
what are the attributes that the data needs? What is the structure that it needs to be in? How do I think about, you know, partitioning or segmenting the data to make it performant or making it easy to be able to join across different data sources and ensuring that maybe the chunking is the right size and just some all of those kind of considerations that go into being able to work effectively with that spatial information?
Unknown: My
view of that is that hopefully you have to make very few of those modeling
choices.
That's really
the thing that I would love to see come out of our development on this product.
An example of that is if you have, let's say, customer data,
which has associated
ZIP codes.
And
in that case, you have this kind of sort of geospatial aspect, which is your ZIP codes. But unless you have gone and figured out exactly how you map ZIP codes into geospatial
areas, which itself is a little shaky, that's kind of its own ramp that I could get into, You're not gonna have those coordinates.
And what I would love to see, and we've made some progress in this in Unfold Studio, I'm happy to say, is something where you can just bring that data in, and
the product of the system walks you through how to get that from whatever suite you have it in into a state that's visualizable on a map. So for ZIP code, that's something where we can provide
the coordinates or boundaries or, you know, geometry to you
and, okay, it's just a click of a button, and you've solved that problem. How do I go from my
identifier
to the actual shape behind it?
So that's, you know, that's kind of my overall
goal for all of these things is just that
these issues of partitioning
or
of having the appropriate identifiers
go away and are solved by the tool because I think that for a lot of
data professionals that are working with this data, that isn't their primary concern. Their primary concern is usually more of a
business 1 of what
decision can I make from this data and not something of how do I go into these low level details?
And, you know, that being said, I do have a lot of opinions of of how that data
should be modeled and what are some effective ways of doing that.
1 methodology that I'm really
quite optimistic about is
1, using h 3. Obviously, that's a project that I've been involved with for a number of years now, and so it kind of makes sense that I'm gonna
kinda say, like, I think this is a good way of working with the data. But I do think that it's a really strong
abstraction, and it's a really strong
way of working with data to
project us into a grid system and do analytics on top of that.
So that's kind of
the
1 of the guiding principles that we use for
how we guide our users along in terms of
in terms of getting their data ready to work with on a map and ready to work with that geospatial way.
Unknown: This might be a bit of an aside, but in terms of the h 3 library, I know that there are different kind of hierarchies that you can get into as far as what scale of the region you're working with. And I'm curious if there are
any considerations or challenges that users need to think about as they're maybe trying to
project into different scales of region and then being able to join across those scales where they might need to kind of, you know, resegment the data as they go from, you know, 10 mile scale down to, you know, 1 mile scale, for instance.
Unknown: So as we work with this H3 printed data, like you mentioned, it is a hierarchical grid system. So a cell or, you know, hexagonal area at a finer resolution
has this
logical and some kind of spatial relationship
to a cell
that is larger, that's at a coarser resolution?
Yeah. This is a question that I think comes up quite often because we have this choice of what resolution to put the data at.
There's kind of a natural question of, okay, which resolution should I pick? Yeah. I don't think that there is a
really hard and fast rule that you can deploy
for
which resolution
is appropriate for your your dataset. I think that you can come up with some good kind of stats ways of doing this and some good data science approaches to how to choose that.
I would say that the best way that I have to explain
how to to choose that resolution
really depends on the use case of the data that you're working with.
So
at Uber, we're working with
data which is essentially cars moving around cities. That's kind of
the general primitive that that Uber has to deal with. And in that case, I think it makes sense
that Uber chose to use a resolution
which is geared roughly to, like, a city block. This is roughly around resolution 9 in the the h 3 hierarchy.
And this allows for
a grid which maps really well onto the the business problem that
Uber is dealing with in their marketplace.
As I look more at data which is at finer resolutions, which might be more like footfall data or
traffic to individual
stores or retail locations.
Sometimes what I'm looking at is at finer resolutions.
So so I'm thinking more like 12, 13, 14, something like that, because this is happening on an even smaller spiel
or even finer spiel, I should say, with smaller cells
than that Uber use case. This involves something more like somebody walking along the street or crossing the street where which side of the street you're on has a tremendous impact.
You think, of course, for a car, which side of the street you're on doesn't have quite so much impact, So you maybe don't need that
level
of precision or level of resolution
in your analysis.
You know, if I have something which on the in the other direction
is not about cars or people moving around cities, but it's maybe around movement between cities or between regions,
Maybe in that case, even a coarser resolution
is gonna make the most sense for your use case. So I think that this kind of use case informed way of looking at which resolution to choose is a,
you know, probably the best approach that we have right now to to choosing things.
I would get 1 other
thing to that, which is that
it is okay for that resolution
to
be a little bit more coarse than you might think. I think that people kind of have a natural tendency to wanna go for something which is really, really high resolution.
So you could, of course, just choose to use what's the highest resolution that's available to you in the library.
But as we know with GPS data or whatever data source you're getting it from, there is going to be some kind
of error that's introduced into that data. And there's going to be some kind of uncertainty that's introduced into that data. And I think that we do ourselves a disservice
if we view this as
extremely
high precision data when that's not actually the case. Just in terms of data collection, it might not be that high precision. And in terms of data analytics, it might not make sense to
look at it on that scale. It might make sense to
Unknown: assume that there's a little bit more error and account for that in your analytic rather than trying to pretend that the data is more specific than it really is. Yeah. And for 1 more point in this direction, 1 of the other things that I'm thinking about as far as, like, how to pick the appropriate resolution for your information is when you're dealing with maybe more statistical measures, and you're talking about maybe population density or, you know, number of houses within a given spatial area, and
how if you choose a resolution that's too coarse and you, you know, apply all of the information to that region and then you try to then zoom in further,
then being able to figure out how do I redistribute that calculation
appropriately based on this finer resolution, you're gonna end up losing a lot of precision in that case. And but then there's also still the issue of, like, okay. Well, if I go too fine, then I'm going to spend all of my time just doing data entry and figuring out how to allocate these statistics to these, you know, fine detailed areas and just, you know, figuring out what is the appropriate balance for
the kind of level of statistical uncertainty that I'm willing to accept at finer resolutions.
Unknown: And in the hex tile system that we're working with, it's not only a
h 3 based system and hex system for working with the data, but it's also a hierarchical 1.
And I think that that really helps with the use case that you're mentioning where maybe you have
demographic information or census information that you wanna join into your analysis,
and you might need that at different resolutions.
Data, I think, is a really good example of this because there are methods that we can use either for disaggregating the data. So projecting the data at a finer resolution and saying, okay. This is
the areas within a census block that have
population and that we want to attribute population to,
as well as for aggregating that data to coarser
cells. So if you want to look at that on a really fine level, we have that data.
And getting access to the the coarser data. It's just a matter of zooming out and having the application load in that coarser
form of the data,
which is just aggregated up from that raw data or finer data, I should say, and
having the choice of which resolution that you want to bring into your analysis.
Unknown: As you're talking to people who are first encountering the unfolded studio and unfolded platform. I'm curious if there are
any
sort of conceptual challenges that they run into as they're trying to figure out how do they think about the problem that they're trying to solve for,
how do they understand
which features and which capabilities
are most useful for their specific
application. I'm wondering what are some of the elements of customer education and kind of messaging around the system that you've built that you run into as far as how to help them understand
what the capabilities are, how they compose together, how they integrate with the broader ecosystem of their data platform and the the information that they're trying to work with. I think customer education is a really important
Unknown: aspect of this, and that's something that I've definitely learned over my experience with with Unfaulted and and even before that at Uber.
We are presenting, I think, a new
newer way of working with geospatial data,
and that takes some getting used to. So we we do have some capabilities which are not, you know, fully oriented around this h 3 grid system, but that's where a lot of our
unique capability and really strongest capability
comes from. And like you said, that does require
education
and training resources
so that people can understand what that data looks like and understand what they can do with it and understand why that might be a useful way of working with their data. I think that some of the main
educational questions we've got, 1 is around just this hexagonal grid system and this way of working with analytics data. And that's something that
I usually
recommend people play with it a little bit, project some of their data into it, understand
what this relationship between
the underlying data is and what the grid system is,
and
understand some of the things that you can do with your data in the grid system
in order to get an appreciation for why you wanna go through this kind of modeling
problem that we were talking about
of understanding how to get your data into the grid. So once you're you have that data in the grid,
understanding, okay, I can do this analytic on it. I can do this joint on it. I understand kind of what's on the other side of that
that modeling mountain, so to speak, you know, helps give you the
impetus for getting through that.
I think the other thing that is
customer education
matter with Unfold Studio is that
there are a lot of companies and there are a lot of use cases that have geospatial data in some form,
but they don't necessarily
realize that it's either geospatial or geotemporal
or that they can work with it in this way. They can do something but they can get useful analytics out of it. So they might already have
customer lists or another geospatial dataset list of retail locations,
but they don't necessarily see, how can I actually plot that on a map in a So maybe they've done something that's kind of like a vanity project of just what does this look like on a map, and does that look cool enough to to share with some people?
But they haven't seen their data presented in,
really a geospatial first way,
in a way that
makes them aware that they can actually do decision making based off that data. Maybe they don't have, kind of, the associated datasets
that are needed in terms of transactions
or demographics
in order to enrich that analysis,
or maybe they just haven't taken a look at that data in a geospatial or geotemporal
first way.
Unknown: As you have been building the unfolded platform and working with end users, what are some of the most interesting or innovative or unexpected ways that you've seen it used? So I think some of the ways that I kind of expected. And, of course, coming from the mobility sector and Uber,
Unknown: I always expected mobility and transport and things around people moving around or vehicles moving around.
And
coming from kind of this business background, you also think about, okay, retail and restaurants
and things like that where there's, I think, a very clear business connection between the location and the use case in the business.
The things that I think were more surprising to me were less on those
business use cases and were more on, like, the humanities
uses of this.
So 1 that I can think of off the top of my head was with a partner Contour.
This was their global health fires map and this was, an early project using our our Hex Tile technology.
And this was both technically challenging because we need to make this map over the entire earth, and it needs to be temporal as well. But it was also really educational because
being in California, I'm aware that, okay, wildfires are, you know, are a serious issue,
but you don't kind of see that in the context of the entire Earth. You don't see that in the context of what is the the pattern of wildfires in, let's say, in the United States versus
somewhere in Asia or Africa or, you know, another part of the world.
So this was something that was
exciting and kind of different to me because it served as an educational tool. And it served as something where I can just kind of flip that map back and forth and see how are wildfires developing in Africa
over the last 12, 13 months.
Another use case that I was kind of
found extremely unexpected
was around the the COVID pandemic.
So it unfolded starting in, like, the the tail end of 2019
was kind of in the right place to
see all of the the COVID pandemic and response to it unfold
over the next you know, especially over the next year of kind of that initial response.
And at the time, you know, I recall thinking very pessimistically of how are retail establishments going to do, how are businesses going to respond to this, and how are people going to respond to it, how are people going to change their habits or their patterns,
and which areas are being affected, things like this.
And over the next,
you know, year, year and a half, I recall seeing a number of really interesting
maps and
visualizations
come out of that
in terms of things like, okay. What are the effects of
the COVID pandemic, especially in the United States?
What are some of the things that correlate or do not correlate with
either effects of COVID or on various measures of things like social isolation and things like this and kind of understanding
the impact as well as
what things
go along with the response to the COVID pandemic. I I thought it was a really unexpected,
of course, use case. Again, also a really educational 1 and and 1 that I felt that I learned, you know, some really interesting things from. And
Unknown: in your experience of building the platform and growing it and going through acquisition and now continuing to evolve the product, I'm wondering what are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
Unknown: So going through that process over the last few years has been a really
interesting and educational
experience.
I feel tremendously grateful that I was able to start a company and that I was able to just go through that process.
That was my thought process going into it was
I've seen part of the industry, part of the engineering
career,
things like that, over my my prior few years at Uber.
But starting a company and
building up a product from that really provides a wonderful
educational experience
in other aspects of
professional development and
of building a business.
And that was
the idea that I went into that with. I was very lucky to be in a situation where I can say, I'm gonna give this a shot. I'm gonna learn whatever I've done from it. I'm going in with the understanding, of course, that, you know, whenever you start a company, there's a significant risk that that company is not gonna work out.
And going into it with that
mindset and that approach, I think, was really
personally enriching. I had the opportunity
with Unfulton not only to do
engineering, backend engineering, data engineering, these kind of things that I previously been doing, but also to work more on the business operations side of things and just the kind of day to day
tests of what's needed to actually keep a business running and just have benefits programs and things like this. A technical audience that can very easily be, like, kind of boring sounding thing, but at the same time, it's really important to
understand that how the business actually operates and how the business gets things done.
And coming in from that perspective, I think, to a lot of technical audiences,
that's something that should be
really exciting and really interesting to understand,
you know, because this is understanding how the system works in a different part of the system in addition to, you know, the the part that we're looking at as technologists in terms of a system that we're building.
How does the rest of the the organization
get things done? What are some of the concerns that they need to deal with in order to have
financial reports or benefits or things like this? So this was, I think, 1 of the,
to me, just most enriching aspects of it was not technological
but was more of
kind of the other aspects of running the business.
Unknown: And so for anybody who is interested in working with geospatial and geotemporal
data, what are the cases where unfolded is the wrong choice and they might be better suited with some open source libraries or a different commercial platform or, you know, maybe going a different direction and deciding that maybe the geospatial aspect of their project isn't actually necessary?
Unknown: So a few things that have come to mind. 1 is
coming back to
what I said about the the history of some of the geospatial
industry,
I think there are cases where
it's not about geospatial analytics. It's not about business analytics,
but it's more about
what I would call this GIS or cartographic
use case. And this is things where it's the apt position of
coordinates
is of supreme importance.
So examples of that could be things like plotting parcels or plotting,
you know, utility locations, things like this. These are, of course, really important use cases,
but they have a different set of concerns than what we've been talking about today. What What we've been talking about has been much more around business analytics
and trying to derive
approximate locations that you might wanna open a store.
And whether that store is in, you know, 1 corner or a storefront down from it might have secondary importance to, you know, what is the overall traffic in that in that neighborhood.
In these use cases, which are much more cartographic,
I think that users are gonna be served better by other tools because it's not
a focus of the unfolded platform. It's not kind of the background that we're coming from, and it's not the way that we're approaching the product.
Another area that I think, you know, we're we're relatively early on on
is when companies have
really significant
data infrastructure built up, they have a way of working with that data,
which is not going to integrate well with a third party tool.
So companies that are
extremely invested into
their own way of processing the data or their own set of queries really heavily built up, they're gonna have a harder time integrating with a third party
vendor and a third party solution.
That's not to say that it's not possible to do so. You can still bring unfolded in as an SDK and use it for,
let's say, for a visualization
or for, you know, part of your analytics workflow.
But if that's not
the
analytics workflow that your data scientists or your data analysts are gonna be comfortable with, then I think you need to be cognizant of
which part of that workflow you're bringing it folded into. And if that's something like visualization or kind of reporting, that might work well. If that's something where you're gonna try to move them from a set of tools that they're really familiar with and they're really comfortable with to a new set of tools,
that's something that you have to be a little bit more careful with and provide. You know, you have to get a lot more certainty that's going to work out.
Unknown: So as you continue to build and grow the unfolded platform, what are some of the things you have planned for the near to medium term or any particular problem areas that you're excited to work with?
Unknown: So in the the near to medium term, I think we wanna continue on
a few of the
themes that we've been developing over the last year or so. 1 is bringing in
all the data that somebody needs in order to do their analytics.
So this looks like developing additional data connectors and database connectors
so that this does slot into the workflows that they have today and so that it can be brought into those workflows. And the relevant data that they have can be brought into the product
and so that data can be brought into the product in a way that can actually be used for analytics. So today, what we wanna bring in, like, let's say, a hex tile dataset,
we have some catalog ones, and we have some capability for bringing customer datasets into that format. And I'd like to see that develop into more of a feasibility for
a lot of the datasets that we work with to be transparently
brought in in that format and transparently joined into your analytic.
That access to data is an important 1, and and making the product
accessible to
more users that don't have that geospatial
background, I think, is also a really important 1. So something where you can run-in your data
and have it displayed on a map without having that really in-depth understanding
of how geospatial data and how geotemporal data should work or, you know, does work, but something something that we handle for you,
I think, is an important important
line of development that we're working on as well.
This is something where I think it's really important for us to continue developing
the capabilities for users to bring their data in. We handle a lot of these technical details under the under the hood, essentially, as a product.
So being able to bring in an even wider variety of data,
which, again, is maybe not gonna have latitude and longitude points associated to it but might have
other kind of implicit geospatial or geotextemporal aspects.
Bringing that into the product, I think, is kind of another
secondary focus. And then the 3rd focus that I'd
mentioned is building out these analytics capabilities even more.
So I mentioned some things that we wanted to do a little bit earlier on in terms of more advanced
analytics or advanced ML use cases.
And I think that there's a tremendous opportunity for us to
continue developing
specialized analytics for different use cases. So, for example, in in retail or QSR or something like this, where they have, you know, site selection and anomaly detection and things like this,
developing these as
just packaged
analytics that they can run on their their datasets as I think are really important focus area for us going forward.
Unknown: Are there any other aspects
of the unfolded platform
or the use cases for geospatial analytics or the overall ecosystem around that that we didn't discuss yet that you'd like to cover before we close out the show? I think we covered quite a lot of the really interesting
Unknown: aspects of that ecosystem
that we've been working on. You know, these workflow aspects of how do I get my data in,
some of the analytics aspects of, okay. I I have geospatial data.
How should I model it? How should I
make decisions using that? You know, what kind of methodology? That's h 3 or another methodology
for working with that data. And then how do I communicate that out are the systems that we've touched on today? And I think this is kind of the the core
loop of
of working with geospatial data. So I I think we've we've touched on some of the main points.
Unknown: Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as a final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
Unknown: I think there's a couple of
gaps that I would
identify right now. 1 is
kind of around data cataloging
tooling and
platforms for managing datasets that the companies have. This is something we've been working on at Unfolded in terms of our our dashboard product and
how we manage datasets that customers are uploading to our system or datasets that customers are working with.
This is something that I think is, you know, relatively early on that is seeing continued investment in. But as an industry, we don't have as many
kind of standard tools or conical tools that we would slot in as this data catalog and data management
applications.
The other area that I think
I'm very
optimistic to see developments in, right, and really love to see developments is maybe another way of putting it, is around the management of data quality
and understanding
what is going into a datasets.
Is data flowing through our pipelines the way that we expect it to?
You know, what are the
statistical attributes
of datasets. I've seen some interesting developments of this in the last
year or so in terms of
how people approach exploring datasets and how people
look at the the quality of datasets.
I really wanna come back to this point that
a lot of the the data tool that we're working with as data professionals is really only as good as the quality of that data
and
the strength of the data that we're putting in it. If we're making data driven decisions, but we're putting in low quality data or data that we don't have confidence in,
I find it very difficult to find that we're gonna have confidence in the results.
And so I think a focus on
how do we evaluate
the quality and how do we track the quality of the datasets that we're working with is
of very great importance to us as data professionals.
Unknown: Alright. Well, thank you very much for taking the time today to join me and share the work that you've been doing with the unfolded platform. It's definitely very interesting problem space. It's always great getting to catch up on what's happening in the world of geospatial analytics. So I appreciate all of the time and energy that you and your team are putting into that, And I hope you enjoy the rest of your day. Thank you so much for having me on and really appreciate it and talk with you today.
Unknown: Thank you for listening. Don't forget to check out our other shows, the Data Engineering Podcast, which covers the latest on modern data management, and the Machine Learning Podcast, which helps you go from idea to production with machine learning. Visit the site at pythonpodcast.com
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email hostspythonpodcast.com
with your story.
And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.