Data Lineage and Observability in Large-Scale Pipelines

Authors: Santosh Vinnakota

DOI: https://doi.org/10.5281/zenodo.15086741

Short DOI: https://doi.org/g89tkw

Country: United States

Full-text Research PDF File:   View   |   Download

Abstract: Data lineage and observability play a critical role in modern data engineering, especially in large-scale pipelines where data undergoes multiple transformations before reaching its final destination. Ensuring traceability, accuracy, and performance monitoring in data workflows is crucial for compliance, debugging, and optimization. This paper explores key techniques for implementing data lineage and observability in large-scale data pipelines. It also discusses industry best practices, tools, and methodologies for tracking data transformations while maintaining system reliability and performance.

Keywords: Data Lineage, Observability, Data Pipelines, Monitoring, Data Engineering, Big Data, Data Transformation

Paper Id: 232283

Published On: 2021-11-17

Published In: Volume 9, Issue 6, November-December 2021

Share this