Anthony Cecchini is the President and CTO of Information Technology Partners (ITP), an ERP technology consulting company headquartered now in Virginia, with offices in Herndon. ITP offers comprehensive planning, resource allocation, implementation, upgrade, and training assistance to companies. Anthony has over 20 years of experience in SAP business process analysis and SAP systems integration. ITP is a Silver Partner with SAP, as well as an Appian, Pegasystems, and UIPath Low-code and RPA Value Added Service Partner. You can reach him at [email protected].
Organizations worldwide handle large data volumes to complete various transactions. The challenge, however, comes when analyzing enormous data, especially if you have multiple sources. In addition, if you have many business streams, it may be tedious to control how to access and share data with different users.
A data mesh comes in handy to address the complexities of data security and control. It helps you manage, share, and control all your data streams from a central domain, regardless of the size.
Discover the key functionalities of a data mesh as part of your Data Architecture solution and how it works. For a quick recap of what a Data Architecture and Data Fabrics, see our blog Data Architecture – Understanding The Benefits and Data Fabrics Explained
What is a Data Mesh? 
Data mesh is a decentralized framework that organizes and distributes data using a specific business domain. It can combine various data links into a unique domain from where you can manage, use, or share them on a distributed framework.
Here’s an example; assuming your organization consists of many departments, say marketing, sales, finance, and customer service. You receive multiple requests from external stakeholders daily to deliver products, services, or business information. Instead of handling client requests in batches, you can organize them in sub-domains and assign domain ownership to designated staff.
As a result, each department will serve its clients’ requests from a unique domain. Moreover, the department will determine, control, and access their domain since they better understand their products and client needs. (see below)
Core Principles of a Data Mesh
Zhamak Dehghani, a ThoughtWorks consultant, invented the data mesh concept in 2019. He emphasized that an organization that seeks to organize and deliver a reliable data enterprise should capitalize on the concepts for a successful implementation. Dehghani’s core principles are as follows:
Data as a Product
A successful data mesh implementation requires each domain team to organize its data assets into unique products. Each data product should be clean, complete, and available to all users anytime, depending on the regulations. Moreover, to achieve the best user experience, data products should be;
Identifiable – the data product is registered under a central database for easy identification.
Addressed – the data has a unique address to help users access it.
Trustworthy – the product defines clear objectives and reflects the actual state of events.
Self-explanatory – the data product labels semantics based on the organization’s naming codes.
Domain-oriented Ownership
The data mesh concept requires organizations to build frameworks around business domains to reduce over-dependence on the primary data teams. For this reason, you should have different domain teams that collect, transform and distribute data based on their business functions.
For example, if you’re into fashion design, you can have a domain containing fashion products and a separate website for your visitors’ behavior analytics. That way, the fashion and data analytics teams will handle their data sets separately instead of piling all data into a central domain.
Self-serve Infrastructure
A self-serve infrastructure enables instant access to data by all users. It also reduces duplication of efforts since each domain will have a clean, filtered set of data products each time. For efficient management of the self-serve structure, the data engineers can handle the technology while the domain teams upload the relevant data products.
Federated Governance
A good data infrastructure allows shared responsibility for better control and implementation. Although it is okay for a domain team to handle its data products, federal data governance should also be available to monitor compliance with standards and policy implementations.
What is a Data Product?
A data product is a reusable asset with a corresponding business identity, usually designed for a specific use. It may exist for internal or external use at various organizational levels. Consider a data product as an enclosed container with solutions to many business challenges.
Examples of data products:
- A decentralized table uniting employee data from different departments.
- A customer dashboard uniting data from sales, marketing, and finance departments.
- A computer application (app) with command lines and user interfaces.
What are The Key Attributes of a Data Mesh?
A data mesh is a logical approach to better data handling in an organization. Its design and implementation follow laid-down procedures unique to an organization. Due to its immense application, your data mesh should have the following attributes:
Decentralized Architecture
Organizations usually decentralize their cloud-based applications to enhance efficiency; data mesh is no exception. Since you’ll retrieve and transmit data across many platforms, it also pays off to decentralize your data architecture.
Decentralized Security
Data security is a vital tool in cloud data management. In a decentralized network such as a data mesh, security should also be decentralized. The framework should have delegated authentication to allow only authorized access at various entry points. Some decentralized security protocols you can implement include;
- Encryption
- Privacy management.
- User and data identity.
- Compliance with security regulations.
Product Mindset
Ensure your domain teams have a data product mindset to implement the framework effectively. A product mindset enables the team to regard data assets as products while considering other business teams as customers.
What are the Main Functionalities of Data Mesh?
The data mesh architecture supports various built-in functionalities for effective data handling. Additionally, it has essential non-built-in capabilities that you’ll find helpful to the organization. Here are the six main functional capabilities you should expect from your architecture:
Data Preparation – You can retrieve and prepare essential data from relevant external sources for dissemination to targeted users.
Data Governance – Enables distribution of policies and compliance protocols to data domains.
Catalogues – A data catalogue sources data assets and supply chain information.
Data Engineering – Facilitates the creation of data pipelines for analytics and reuse by business domains.
Data Integration – The features allow you to retrieve and channel data to the audience or target market.
Persistence Layers – They store specific data sets within the domain to enhance better performance and data access.
Top Benefits of Data Mesh
A data mesh empowers business domain teams, granting them ownership, control, and autonomy of their products. Ultimately, it is the best solution to everyday data management challenges. If you’re skeptical about implementing a data framework for your organization, consider the following seven benefits:
Manage Data from a Business Domain
Multiple data sources are prone to failure, especially if you embark on massive data integration and upscaling. However, if you have a robust, domain-based resource, you can always have clean, fresh data. And using a data governance compliance policy, you can say goodbye to frequent data dumps that often arise from bulky data.
Deliver Data at Super-fast Speeds
A reliable data mesh ensures instant access to data by authorized users. As such, the self-serving system provides super-fast access to the resources, eliminating complexities and delays.
Learn from Domain Experts
Implementing data architecture requires hiring highly-trained experts to handle the data sets. As a result, your domain teams will tap immense knowledge from the domain experts.
Improve Domain Agility
The data framework is the key to decentralizing all your data operations. It reduces data backlogs, enabling your domain teams to focus on the relevant data products. It also improves the speed of accessing the business domains by clients and staff.
Reduced Processing Cost
Data architecture decentralization takes your business from the old batch processing to real-time streaming. As a result, you can monitor resource allocations and bundle width costs in real-time. You can also budget for data processing, hence reducing unforeseen expenses.
Improve Data Safety
A data mesh platform ensures data safety and compliance with security protocols within different domains. It also provides a means to monitor and audit all data-sharing processes within the organization. For instance, you can track data log requirements on each domain from a central location.
Increase Operation Flexibility
Installing a data mesh platform is one of the ways to increase operational flexibility. Unlike the conventional data architecture, a data mesh helps you rearrange all the technical implementations to business domains. As a result, you’ll eliminate operational bottlenecks that slow down the data processing.
Common Applications of Data Mesh
A data mesh is best known for helping teams scale up their data processes. It is, therefore, frequently applied in daily transactions involving bulky data handling. Common use cases of data mesh in operations and analytics include the following:
All-round Customer Support
A data mesh system enables support teams to obtain information and reduce handling time, boosting customer experience. It also helps marketing teams channel their campaigns appropriately, thanks to accurate demographics.
Data Privacy Operations
The platform helps organizations enhance data privacy in compliance with regional data laws. Domain teams can therefore withhold classified customer information without sharing it with other users on the platform.
Business Intelligence
The system is widely used for gathering business intelligence in many organizations. For instance, data mesh architectures can provide customized information to data users when conducting a marketing initiative. As a result, marketing teams can deliver marketing information to the right people using relevant channels.
Machine Learning
You can standardize all domain data from various sources to reduce time spent on data processing. In turn, you can increase the number of models going into production and achieve automation goals faster.
Virtual Assistance
Data mesh has practical applications in virtual assistance. For instance, your customer care team can use chatbots to respond to client inquiries when off duty. You can also link your “Frequently Asked Questions” section to selected data sets so your clients have adequate information based on their queries.
Summary
We have learned that a data mesh is a decentralized data architecture that organizes data by a specific business domain—for example, marketing, sales, customer service, and more—providing more ownership to the producers of a given dataset.
The data producer’s understanding of the domain and data positions them to set data governance policies focused on documentation, quality, and access. This, in turn, enables self-service use across an organization.
While this federated approach eliminates many operational bottlenecks associated with centralized, monolithic systems, it doesn’t necessarily mean that you can’t use traditional storage systems, like data lakes or data warehouses. It just means that their use has shifted from a single, centralized data platform to multiple decentralized data repositories.
