What is a data mart, and how does it differ from a data warehouse?
Over the past decade, the dizzying increase in the amount of data produced has accelerated the development of Big Data. The worlds of application development and data management have begun to converge.
In this context, knowing how to centralize, structure, process and analyze a mass of data for a specific problem is essential: this is the whole purpose of a data mart. What exactly does this concept cover? And how is it different from a data warehouse? We explain.
What is a data mart?
Data mart: definition
A data mart, also known as a data store or data counter, is a specific database intended for a given group of users.
Used in business intelligence, it is extracted from source systems, cleansed and made available to users in a specific area of the company, or to a restricted group of users.
Data mart example - © Talend
👉 The data mart must serve the end need, and therefore transcribe the data initially stored in the data warehouse as intelligibly as possible and as closely as possible to the business language.
Example of a data mart
We can thus imagine that, within a company's HR department, a first datamart compiles all the indicators relating to the use of the main ERP, and that other "bricks" of the HR requirement are datamarts directly associated with secondary and very specific applications, for example the monitoring of employees' e-Learning.
Advantages of data marts
- Provides users with a full range of indicators for the data they need on a daily basis.
- The same group of users can have access to a single data mart or to several data marts, each corresponding to a specific need, depending on the IT architectures in place and the confidentiality of the data.
Data mart vs. data warehouse: what are the differences?
Depending on how it is conceived, the data warehouse can be seen as a collection of data marts and their gateways, or, more commonly, as the centralization of all data for use by the data marts, in a single system ensuring security, availability and technical consistency.
It therefore takes on a more technical coloring, and will probably not have a single "Sales" field, but perhaps several components of the company's income and expenses, which each domain will arrange according to its own conception of sales.
The data warehouse will also ensure the traceability of information throughout the company, whereas the data mart is limited to satisfying the specific needs of one business line.
How do you build a data mart? 3 options
Data mart integrated into the source application
If you prefer data marts dedicated to an application, it may be because the application itself offers integrated analysis tools. This seems like the ideal solution.
Advantage: to meet the application's needs as closely as possible, and to ensure consistency between the data and its output.
Disadvantages:
- costs in the medium and long term, since you have no control over indicator output;
- you may not be able to enrich it with the rest of your company's data, and vice versa;
- you may be overlooking options for feeding this data back into the datawarehouse.
👉 So you lose in potential what you gain in speed of implementation.
The datawarehouse-independent datamart
This is a more advanced version of the previous one, since it may have been set up internally, but still from a very specific source on which it is highly dependent.
Advantage: you have more latitude when it comes to rendering elements.
Disadvantage: the fact that it is not integrated with the rest of your data warehouse always reduces your potential to respond to user needs in the medium term.
The data mart as a building block of the data warehouse
To maximize their potential, data marts should be built around a data warehouse. Their integration can be :
- ️ upwards: a set of data marts enabling the constitution of a datawarehouse,
- ↘️ top-down: the centralization of data in the data warehouse enables the creation of all the necessary building blocks.
Advantages :
- connection with other areas of the company, enabling key performance indicators to be refined and explained precisely. For example, you can
- highlight a correlation between declining results on a particular circuit of your e-learning platform and an increase in incidents on a production line.
- optimize your production rate based on pipe analysis of your CRM tool.
- the arrangement of these bricks within or around a datawarehouse increases your chances of perpetuating the correct interpretation of your indicators in cross-functional use.
Disadvantage: loss of independence
Which tools for my data marts?
Of course, there's no shortage of ETL tools for processing mass quantities of data and rapidly analyzing them.
But there are also dedicated storage tools, open source or proprietary, available as turnkey solutions for your data mart.
As with any choice that pits open source against proprietary solutions, it's support and in-house capacity to develop or adapt components that will be the criteria to take into account.
From data mart to DataOps
Integrating your data marts with a data warehouse should be a major objective of your architecture. And the proper evolution of this data warehouse is its corollary.
As technical teams are exposed to ever-increasing demands and a growing need for responsiveness, we have had to adapt our development and deployment methods using the continuous integration techniques that have proved their worth in the application world. Data engineering must now submit to a new paradigm: DataOps, derived from DevOps.
In short, adapting the principles of DevOps to the world of Data offers a new response to the challenges of setting up data marts in a context of strong growth.
Article translated from French