Data lineage is the record of where a piece of data came from, what happened to it on the way, and where it went — the path from a source system through every transformation, aggregation and hand-off to the report, model or decision that consumed it. It is the metadata that answers the questions regulators, auditors and analysts ask most and organisations answer worst: which systems feed this number, what changed it, and who would be affected if the source were wrong. Three types of lineage exist and are frequently confused: business lineage, which describes the flow in business terms — the revenue figure comes from billing, is adjusted for returns in finance and reported in the board pack; technical lineage, which traces it at the level of tables, columns, jobs and code; and operational lineage, which records what actually ran — which job, when, on which data, with which result. In the DAMA-DMBOK all three live in the Metadata knowledge area and serve Data Integration and Interoperability, Data Quality and Data Governance; regulations from BCBS 239 to the GDPR and SOX make them evidence. This guide sets out the three types with what each is for and who reads it, how lineage is captured, the granularities and directions that matter, the regulatory drivers, and the errors that make a lineage programme collapse under its own scope.

What data lineage is for
| Question | Direction | Who asks | Type of lineage that answers it |
|---|---|---|---|
| Where does this figure in the report come from? | Backward (provenance) | Auditors, regulators, executives | Business, then technical |
| If we change this source field, what breaks? | Forward (impact analysis) | Engineers, change managers | Technical |
| Why is this number different from yesterday’s? | Backward, at run level | Data quality, operations | Operational |
| Which personal data flows to this third party? | Forward | Privacy, security | Business and technical |
| Is this dataset derived from an authoritative source? | Backward | Analysts, stewards | Business, with catalogue metadata |
Our guide to the data governance framework places lineage in layer 4, the data assets and metadata layer; the data catalog vs data dictionary guide covers the catalogue that stores it.
The three types of data lineage
| Type | What it records | Granularity | Produced by | Read by |
|---|---|---|---|---|
| Business lineage | The flow of a business concept — revenue, customer count, exposure — from originating process to consuming report, in business terms, with the transformations described as rules (“adjusted for returns”) | Process and dataset | Stewards and owners, from the glossary and the catalogue | Owners, auditors, regulators, executives |
| Technical lineage | The actual path: source tables and columns, ETL and ELT jobs, SQL, code, APIs, intermediate tables, target columns | Column and job | Automated scanners parsing code, query logs and pipeline metadata; engineers where scanning fails | Engineers, architects, change and impact analysis |
| Operational lineage | What executed: job instances, run times, input and output datasets and their versions, row counts, failures, the user or service that ran it | Run and record | Orchestration and pipeline platforms, logs | Operations, data quality, incident response, forensic investigation |
The three are layers of the same path. Business lineage is technical lineage abstracted to the glossary terms; operational lineage is technical lineage instantiated at run time. A programme that builds only one has a map without a territory, a territory without a map, or a log without either.
Granularity and direction
| Granularity | What it shows | Effort | When it is enough |
|---|---|---|---|
| System level | Which systems feed which | Low; a diagram | Architecture overview; first regulatory response |
| Dataset or table level | Which tables and files feed which datasets and reports | Moderate; catalogue-scale | Most governance and privacy needs |
| Column level | Which source columns, through which transformations, produce which target column | High; needs automated parsing | Regulatory reporting (BCBS 239), critical data elements, impact analysis |
Direction is the other choice. Backward lineage from a report to its sources is provenance — the audit question. Forward lineage from a source to everything downstream is impact analysis — the change question. A tool that captures the graph gives both; a documented lineage built by hand usually gives only the direction it was built for.
How data lineage is captured
| Method | How it works | Strength | Limitation |
|---|---|---|---|
| Automated parsing | Scanners read SQL, pipeline definitions, transformation code and BI semantic layers and infer column-level flows | Scale and accuracy for supported technologies | Blind to hand-coded, undocumented or unsupported steps; spreadsheets and manual transfers vanish |
| Runtime capture | Orchestrators and pipelines emit lineage events (e.g. OpenLineage) as jobs run | Operational lineage for free; always current | Only what runs through instrumented platforms |
| Query-log mining | Database and warehouse logs are analysed for read-write patterns | Finds flows nobody documented | Infers, does not prove; noisy |
| Manual documentation | Stewards and engineers record flows in the catalogue | The only way to capture business lineage and the steps tools cannot see | Decays unless owned and reviewed |
In practice the graph is assembled from all four: automated technical lineage for the platforms, runtime capture where available, manual business lineage over the top for the critical data elements, and stewards closing the gaps the scanners leave — the spreadsheet in the middle of the finance process, the file transfer to the regulator.
The regulatory drivers for data lineage
| Requirement | What it asks for | Lineage type and granularity |
|---|---|---|
| BCBS 239 (risk data aggregation, banks) | Accuracy, completeness and adaptability of risk data, with data taxonomy and architecture; the ability to trace risk figures to source | Business and technical, column level for critical risk data elements |
| SOX 404 / ICFR | Evidence that financial reporting figures are produced by controlled processes; IT general controls over the systems in the flow | Business lineage for significant accounts; technical lineage for in-scope systems |
| GDPR Articles 30 and 15; PDPL and DPDP equivalents | Records of processing; the ability to tell a data subject where their data went; transfers to third countries | Business and technical at dataset level for personal data flows |
| EU AI Act, Article 10 | Data governance for training, validation and testing data, including provenance | Business and operational lineage for training datasets |
| Model risk management (SR 11-7 and equivalents) | Data inputs to models understood and controlled | Technical and operational lineage into and out of models |
Building a data lineage programme that survives
- Start from the critical data elements, backward. Take the regulated or board-level figures and trace them to source; that scope is finite and the value is immediate. Our guide to the six data quality dimensions covers how those elements are selected.
- Capture technical lineage automatically where the platforms allow. Warehouse, pipeline and BI scanning first; treat the coverage gaps as a list of undocumented flows to fix, not to draw.
- Layer business lineage over it. Stewards map the technical path to glossary terms and describe the transformations as business rules; owners approve.
- Keep operational lineage in the platform. Run-level detail belongs in the orchestrator’s logs and lineage events; the catalogue links to it rather than copying it.
- Assign ownership to the flow, not only the datasets. A lineage path crossing three domains needs an owner for the path — usually the consuming report’s owner.
- Use it. Impact analysis in change control, provenance in audit responses, run history in incident reviews; a lineage graph nobody queries decays as fast as a manual diagram.
Errors that make data lineage programmes collapse
- Everything, column level, from day one. The scope defeats the team; start from the critical elements.
- Technical lineage presented to the board. A column-level graph is unreadable to an owner; business lineage is the layer they need.
- Trusting the scanner’s coverage. The steps the scanner cannot see — spreadsheets, manual loads, undocumented scripts — are where the misstatements live.
- No owner for the path. Datasets have owners; flows between them usually do not, and the gap is where changes go unreviewed.
- Documentation without a use. Lineage that is not consulted in change control and audit is a museum.
- Confusing lineage with provenance metadata. Lineage is the path; provenance is the record of origin and custody. Both matter; the catalogue should hold both distinctly.
Frequently asked questions
What is data lineage?
The record of where data came from, what transformed it and where it went — the path from source systems through every processing step to the reports, models and decisions that consume it — captured as business lineage (in business terms), technical lineage (tables, columns, jobs, code) and operational lineage (what actually ran, when, on what).
What are the three types of data lineage?
Business lineage, which describes the flow of a business concept in glossary terms for owners, auditors and regulators; technical lineage, which traces the actual path at column and job level for engineers and impact analysis; and operational lineage, which records executed runs, inputs, outputs and results for operations and incident investigation.
What level of detail do we need?
It depends on the use: system level for architecture, dataset level for most governance and privacy needs, column level for regulatory reporting such as BCBS 239 and for critical data elements. Start from the critical elements and go as deep as the question requires.
Can lineage be fully automated?
Technical and operational lineage largely can, for platforms the tools support; business lineage and the steps tools cannot see — spreadsheets, manual transfers, undocumented scripts — need stewards. The graph is assembled from both.
Which regulations require data lineage?
BCBS 239 for banks’ risk data, SOX and ICFR for financial reporting flows, the GDPR and equivalents for personal data processing records and transfers, the EU AI Act for training data governance, and model risk management standards for model inputs — none uses the word ‘lineage’ and all require what it provides.
Where this leaves you
Build data lineage as three layers of one path — business for the owners, technical for the engineers, operational for the operators — scoped from the critical data elements backward, captured automatically where the platforms allow and by stewards where they do not, owned per flow and used in change control, audit and incidents, because a lineage graph is evidence only while someone maintains it and only where someone reads it.
References
- DAMA International — DAMA-DMBOK Data Management Body of Knowledge — The Metadata and Data Integration and Interoperability knowledge areas.
- Basel Committee on Banking Supervision — BCBS 239, Principles for effective risk data aggregation and risk reporting — The regulatory driver for column-level lineage in banks.
- OpenLineage — open standard for lineage collection — Runtime lineage event standard.
More on data governance
- Data lineage — you are here
- Data governance and the DMBOK
- The data governance framework: five layers
- Data catalog vs data dictionary
- The six data quality dimensions
- The data governance maturity model
The data lineage standard, the business lineage template for critical data elements, the flow ownership register, the impact analysis procedure for change control and the lineage coverage report are in the Data Governance Toolkit, or start with the free templates.