One gigabyte of data for a grocery bag. This is what you get when you make a robotic delivery. That’s a lot of data, especially if you repeat it over a million times like we did.
But the rabbit hole goes deeper. The data is also incredibly diverse: sensor and robot image data, user interactions with our applications, order transaction data, and more. And there are equally diverse use cases, ranging from deep neural network training to creating polished visualizations for our business partners and everything in between.
So far, we have been able to handle all this complexity with our centralized data team. By now, continued exponential growth has led us to look for new ways to work to keep pace.
We have found that the data mesh paradigm is the best way to go. I’ll describe Starship’s view of the data mesh below, but first, we’ll give a brief overview of the approach and why we decided to follow it.
What is a data mesh?
The data mesh framework was first described by Zhamak Dehghani. The paradigm is based on the following basics: data products, data domains, data platform, and data governance.
The key intent of the data mesh framework has been to help large organizations eliminate data engineering bottlenecks and deal with complexity. Therefore, it addresses many details that are relevant in a business environment, ranging from data quality, architecture and security to government and organizational structure. As it stands, only a couple of companies have publicly announced that they adhere to the data mesh paradigm, all big billionaire companies. However, we believe that it can also be successfully applied to smaller businesses.
Starship data mesh
Make the data work close to the people who produce or consume the information
To run hyperlocal robotic delivery markets around the world, we need to turn a wide variety of data into valuable products. The data comes from robots (e.g., telemetry, routing decisions, ETA), merchants and customers (with their applications, orders, offers, etc.) and all operational aspects of the business (from brief operator tasks remote to global spare logistics). parts and robots).
The diversity of use cases is the key reason that has attracted us to the data mesh approach: we want to carry out data work very close to the people who produce or consume the information. Following the principles of data mesh, we look forward to meeting the diverse data needs of our computers while maintaining reasonably light central oversight.
Because Starship is not yet enterprise-wide, it is not practical for us to implement all aspects of a data mesh. Instead, we have decided on a simplified approach that makes sense to us now and puts us on the right path for the future.
Define what your data products are, each with an owner, interface, and users
Applying product thinking to our data is the basis of the whole approach. We think of anything that exposes data to other users or processes as a data product. You can display your data in any way, such as a BI dashboard, a Kafka theme, a data store view, a predictive microservice response, and more.
A simple example of a data product on Starship might be a BI dashboard for potential site customers to track your site’s turnover. A more elaborate example would be a self-service pipeline for robot software engineers to send any kind of driving information from robots to our data lake.
In any case, we do not treat our data warehouse (actually a Databricks Lakehouse) as a single product, but as a platform that supports a number of interconnected products. These granular products are usually owned by the scientists / data engineers who build and maintain them, not the dedicated product managers.
The product owner is expected to know who their users are and what their needs are with the product and, based on that, define and meet product quality expectations. Perhaps as a result we have begun to pay more attention to interfaces, components that are crucial for usability but laborious to modify.
Most importantly, understanding users and the value each product creates for them makes it much easier to prioritize between ideas. This is essential in a start-up context where you need to move quickly and you don’t have time to make everything perfect.
Group your data products into domains that reflect the organizational structure of your business
Before we got to know the data mesh model, we had been successfully using the slightly integrated data scientists for a while at Starship. Indeed, some key teams had a part-time data team member working with them, whatever that meant for a particular team.
We proceeded to define the data domains aligned with our organizational structure, this time taking care to cover all parts of the company. After mapping the data products to the domains, we assigned a member of the data team to cure each domain. This person is responsible for taking care of all domain data products, some of which are owned by the same person, others by other domain team engineers, or even other members of the domain. data team (e.g., for resource reasons). .
There are a number of things we like about setting up our domain. First, now every area of the business has a person in charge of its data architecture. Given the intricacies inherent in each domain, this is only possible because we have divided the work.
Creating structure in our products and data interfaces has also helped us better understand our data world. For example, in a situation with more domains than members of the data team (currently 19 vs. 7), we are now doing a better job of making sure that each of us is working on a set of interrelated issues. And now we understand that to alleviate the problems of growth, we should minimize the number of interfaces that are used across the boundaries of the domain.
Finally, a more subtle advantage of using data domains: we now believe we have a recipe for dealing with all sorts of new situations. Whenever a new initiative emerges, everyone is much clearer about where it belongs and who needs to do it.
There are also some open questions. While some domains are naturally inclined to expose source data and others to consume and transform it, some have a good amount of both. Should We Divide Them When They Grow Too Much? Or should we have subdomains within larger ones? We will have to make these decisions later.
Empower people who create your data products by standardizing them without centralizing them
The goal of the data platform in Starship is simple: to make it possible for a single data person (usually a data scientist) to handle an end-to-end domain, that is, to maintain the data team. the central data platform out of the day. work today. This requires providing domain engineers and data scientists with good tools and standard building blocks for their data products.
Does that mean you need a complete data platform equipment for your data mesh approach? Not really. Our data platform team consists of a single data platform engineer, who in parallel spends half of his time embedded in a domain. The main reason we can be so skinny in data platform engineering is the choice of Spark + Databricks as the core of our data platform. Our previous and more traditional data warehouse architecture put a significant strain on our data engineering due to the diversity of our data domains.
We found it useful to make a clear distinction in the data stack between the components that are part of the platform and everything else. Some examples of what we offer domain teams as part of our data platform:
- Databricks + Spark as a work environment and versatile computer platform;
- line functions for ingesting data, for example, from Mongo collections or Kafka themes;
- an Airflow instance for scheduling data channels;
- templates for building and deploying predictive models as microservices;
- cost tracking of data products;
- BI and visualization tools.
As a general approach, our goal is to standardize as much as it makes sense in our current context, even fragments that we know will not remain standardized forever. As long as it helps productivity right now and doesn’t centralize any part of the process, we’re happy. And of course, some elements are completely missing from the platform right now. For example, tools to ensure data quality, data discovery, and data lineage are things we have left for the future.
Strong personal property backed by feedback loops
Having fewer people and teams is really an asset in some aspects of governance, for example, it is much easier to make decisions. On the other hand, our key governance issue is also a direct consequence of our size. If there is only one data person per domain, you cannot be expected to be an expert in all potential technical aspects. However, they are the only person with a detailed understanding of their domain. How do we maximize the chances of them making good decisions within their domain?
Our response: through a culture of ownership, discussion and feedback within the team. We have liberally borrowed from Netflix’s management philosophy and cultivated the following:
- personal responsibility for the result (of the products and domains themselves);
- seek different opinions before making decisions, especially those that affect other areas;
- request code reviews and revisions as a mechanism for quality and opportunity for personal growth.
We’ve also made a couple of specific agreements about how we approach quality, we’ve written our best practices (including naming conventions), and so on. But we believe that good feedback loops are the key ingredient in turning guidelines into reality.
These principles also apply outside of the “construction” work of our data team, which is what has been the focus of this blog post. Obviously, there is so much more to providing data products on how our data scientists are creating value in business.
One last thought on governance: we will continue to iterate our ways of working. There will never be a “better” way of doing things and we know we have to adapt over time.
That’s all! These were the 4 basic concepts of data mesh applied to Starship. As you can see, we’ve found a data mesh approach that suits us as a fast-growing company. If you find it appealing in your context, I hope reading about our experience has been helpful.
If you would like to participate in our work, please see our Careers page for a list of open positions. Or check out our Youtube channel for more information on our world’s leading robotic delivery service.
Contact me if you have any questions or thoughts and let’s learn from each other!