Many companies are on the verge of becoming data-driven, a business model that bases its decisions on data, or rather, on the insights derived from the analysis of the data at its disposal. The objectives of this journey are obvious: by improving decision outcomes, companies aim to optimize their processes, reduce costs, increase competitiveness and, last but not least, consolidate their market position.
Every modern organization has large amounts of data that can be analysed in depth. Yet despite the promise of increased competitiveness, only a small percentage of organizations is genuinely data-driven; specifically, 23.9%, according to NewVantage Partners. One reason is the lack of skills required for data science projects, i.e., the transformation of vast volumes of raw (and heterogeneous, structured and unstructured) data into information, then into knowledge, and thus into tangible value.
The path from raw data to information is a data pipeline.
Data Pipeline as a pillar of data science projects
Formally, a data pipeline can be described as a process (or method) adopted by the company that begins with the acquisition of raw data from various sources (SQL and NoSQL databases, files, IoT, etc.) and performs various operations on it to make it available for analysis within a repository such as a data lake or data warehouse. In this regard, the data pipeline enables the transformation of data into usable information for end users and is therefore essential on the path to the data-driven enterprise.
The data pipeline is necessary (above all) because of the extreme heterogeneity of business data, which is not ready for immediate use and requires processing for standardization and integration. Given the business requirements, i.e., the information that the company expects to obtain from its data, it is up to the data science professionals (scientists, analysts, engineers) to design and implement a scalable pipeline that is as automated as possible and that delivers results within a reasonable timeframe, which today can even be in real-time.
Data Pipeline: architecture and differences with ETL
There are typically three elements or main stages to the architecture of a given pipeline:
- the ingestion or data collection;
- the data transformation processes;
- the storage
The architecture described above explains why the terms data pipeline and ETL pipeline (Extract, Transform, Load) are often used interchangeably, although they are not the same concept.
Formally, the ETL pipeline is a type or sub-category of data pipeline based on a rigid sequence of operations and batch processing with the aim of loading data into a data warehouse for business analysis. On the other hand, the data pipeline is a broader and more flexible concept: it (also) incorporates the principle of real-time processing, can include additional operations such as data validation and error handling, and can be used for multiple purposes, such as feeding data lakes.
How a typical data pipeline operates
Typically, a data pipeline consists of a series of steps that, as mentioned, begin with the raw data and make it usable for analysis and decision support activities. The organization defines pipelines according to the data to be processed and, most importantly, the project objectives. They have four core elements.
Data Mining
The first step involves extracting raw data from various sources: database tables, spreadsheets, and unstructured documents such as images, code, IoT data, and much more.
Data governance
To ensure data quality (data quality is a key issue in any data science project), integrity and security, governance rules must be applied to the data.
Data transformation
Data transformation is the key area of data pipelines. The aim is to modify them so that they have the correct format and can be used for any kind of analysis. The process may involve multiple steps, such as standardization, de-duplication, masking, verification, filtering and aggregation.
Data Sharing
Once transformed, data can be stored in a local repository or in the cloud for data sharing, analysis, and visualization, enabling summary reports and dashboards to support critical decisions and accelerate the journey to a data-driven enterprise.
