Data lineage standards: what large enterprises actually need

When a DG Manager at a European bank asked their data engineering team whether the company’s lineage was standardized, three teams gave three different answers – all technically correct, all using Collibra, all citing OpenLineage compliance. The bank had a tool. It had connectors. It even had a dedicated lineage team. What it didn’t have was a shared understanding of what “standardized” actually meant.

We see this more often than we’d like to admit. The term “data lineage standards” sounds like it refers to one thing. It doesn’t. It refers to at least three distinct layers of requirements – and most organizations are only addressing one of them.

Key takeaways

• Data lineage standards operate at three distinct levels: regulatory requirements (BCBS 239, DORA, EU AI Act), open technical specifications (OpenLineage, W3C PROV), and internal governance protocols. Conflating them is the single most common reason lineage programs stall.
• OpenLineage solves interoperability across data pipelines. It does not define ownership, naming conventions, or how technical lineage maps to business context. Adopting it is necessary but not sufficient.
• BCBS 239, DORA, and the EU AI Act each impose specific, auditable lineage requirements for regulated enterprises in the EU. These are not aspirational guidelines – they are compliance baselines.
• The root cause of most failed lineage programs is not the tool. It is the absence of defined internal standards: who owns what, what gets named how, and which data elements are critical enough to track at all.
• Before running any automated lineage scan in Collibra, five internal governance decisions must be made. Skipping them produces technically complete maps that nobody uses or trusts.
• Defining Critical Data Elements (CDEs) and their ownership model is the single highest-leverage action for any organization starting or restarting a lineage program.

What “data lineage standards” actually means – and why the term misleads everyone

Data lineage standards are the formal rules governing how an organization captures, models, and maintains records of data movement across systems. They operate at three distinct layers: regulatory requirements (such as BCBS 239 and DORA), open technical specifications (such as OpenLineage and W3C PROV), and internal governance protocols covering naming conventions, ownership, and metadata schema.

Most conversations about lineage standards collapse all three into one category – usually the technical one. That’s where the confusion starts. A data engineering team adopts OpenLineage and declares the lineage standardized. A compliance team files against BCBS 239 requirements. A DG Manager tries to get business users to trust the lineage maps. These three conversations are happening in parallel, and they rarely connect.

The practical consequence: organizations invest heavily in technical interoperability (layer 2) while the regulatory requirements (layer 1) are only partially addressed and the internal governance layer (layer 3) is barely started. The result is a lineage system that passes a technical audit but fails a regulatory one – and that business stakeholders find impossible to interpret.

Understanding which layer you are trying to address at any given moment changes what you need to do next.

Layer	Examples	Who defines it	What happens without it
Regulatory	BCBS 239, DORA, EU AI Act	External regulators	Audit failures, fines, regulatory action
Technical/Open	OpenLineage, W3C PROV, DAMA DMBOK	Standards bodies	Fragmented pipeline lineage, no interoperability
Internal governance	Naming conventions, ownership model, CDE definitions	Your organization	Technically valid lineage that nobody uses

Regulatory standards: what BCBS 239, DORA, and the EU AI Act require

Regulatory frameworks are the only lineage standards with external enforcement. For enterprises operating in regulated EU industries, these are non-negotiable – and the requirements are more specific than most governance programs acknowledge.

BCBS 239

BCBS 239 is the Basel Committee standard for risk data aggregation and reporting, applicable to systemically important banks. Its lineage requirements are explicit: institutions must demonstrate end-to-end traceability of risk data from raw source systems through all transformations to the final regulatory report. This means column-level lineage for Critical Data Elements used in risk reporting, bi-temporal audit trails (capturing both when a change was recorded and what the data state was at a specific point in time), and documented data ownership at each step in the chain.

What this means in practice: if a regulator asks where the “Net Exposure” figure in a risk report comes from, your lineage must trace it back through every aggregation, every join, every transformation – back to the authoritative source system. A lineage map that stops at the data warehouse boundary does not satisfy BCBS 239. We helped a Swiss bank achieve exactly this level of traceability for sensitive CDEs – the case study covers the architecture in detail.

DORA

The Digital Operational Resilience Act took effect in January 2025 and applies to financial entities operating in or serving the EU. DORA’s lineage requirements focus on operational resilience rather than risk reporting: organizations must maintain documented, auditable maps of critical ICT data flows and dependencies, specifically to support resilience testing and incident recovery.

What this means in practice: when a critical system goes down, your lineage must be current enough to identify which upstream sources feed which downstream systems – and in what order they need to be restored. Static lineage documentation that was last updated 18 months ago during a project phase does not satisfy DORA. The lineage must be actively maintained, which makes automation and event-driven update protocols a compliance requirement, not a preference.

EU AI Act

Article 11 of the EU AI Act imposes documentation requirements for high-risk AI systems that map directly to data lineage. Specifically, it requires technical documentation including “a description of the training methodologies and training data sets used.” For organizations deploying AI systems in regulated contexts (credit scoring, fraud detection, medical decision support), column-level lineage from source systems through preprocessing pipelines to training datasets is the mechanism that satisfies this requirement.

What this means in practice: AI governance is becoming a lineage problem. Organizations that have not extended their lineage infrastructure to cover ML pipelines and training data provenance are creating a compliance gap that will only grow as the AI Act’s enforcement provisions mature. Our Collibra AI governance article covers this intersection in more depth.

Organizations in banking, insurance, or pharma that haven’t formalized lineage standards against these frameworks are not just behind on governance – they are exposed to regulatory action.

Technical open standards: OpenLineage, W3C PROV, and DAMA DMBOK

Open technical standards address interoperability – the question of how lineage metadata can be captured consistently across different tools, pipelines, and platforms.

OpenLineage is an open standard for metadata and lineage collection at the pipeline level. Hosted by the LF AI & Data Foundation, it defines a common model for run, job, and dataset events, with integrations across Apache Airflow, Apache Spark, dbt, Snowflake, and BigQuery. Emerging since 2020 and now widely adopted, it has become the de facto standard for pipeline-level lineage capture in modern data stacks. What it solves: if you have multiple pipeline tools in your stack, OpenLineage gives them a common language for emitting lineage events into a central platform like Collibra.

What it does not solve: OpenLineage captures the “what” of data movement at the pipeline level. It does not define data ownership. It does not determine which assets are critical. It does not establish how technical lineage nodes connect to business terms in a glossary. Those decisions belong to your internal governance layer.

W3C PROV is a domain-agnostic data model for provenance information, published as a W3C Recommendation. Built around three core concepts – entities, activities, and agents – it provides a formal ontology for representing relationships between data, the processes that created or transformed it, and the responsible parties. It is primarily relevant in scientific, research, and cross-institutional data sharing contexts. For most enterprise data governance programs, W3C PROV is the theoretical foundation that OpenLineage operationalizes at runtime.

DAMA DMBOK is not a technical specification but a data management framework. Its relevance to lineage standards is the explicit distinction between three metadata layers: technical metadata (schemas, lineage graphs), business metadata (ownership, glossary definitions, classifications), and operational metadata (job execution status, data freshness, quality metrics). A lineage program that only captures technical metadata is incomplete by DAMA’s own definition. Collibra’s architecture is designed to integrate all three layers – but only if the governance layer underneath has defined what goes in each.

Standard	Scope	What it solves	What it doesn’t solve
OpenLineage	Pipeline-level metadata capture	Cross-tool interoperability, runtime lineage	Ownership, business context, CDE prioritization
W3C PROV	Provenance data model	Formal ontology for entities, activities, agents	Operational implementation, enterprise scale
DAMA DMBOK	Data management framework	Metadata layer definition (technical/business/operational)	Tool configuration, naming conventions

The critical practical conclusion: adopting OpenLineage is a necessary condition for enterprise lineage standardization. It is not a sufficient one.

Internal governance standards: what you must define before Collibra runs lineage

This is the layer that most implementations skip – and the primary reason most lineage programs underdeliver. No automated tool, however well-integrated, can compensate for the absence of these five governance decisions.

1. Critical Data Element (CDE) identification and prioritization

Before scanning anything, define which data elements are critical. Not everything needs end-to-end lineage – and attempting to map everything is one of the most reliable ways to produce a lineage graph that is too large to use. CDEs are typically the data elements that drive 80% of business value: the metrics that appear in regulatory reports, board-level dashboards, and risk calculations. Start here. The goal at the foundational stage is 100% documented lineage for these elements.

2. Naming conventions for assets and lineage nodes

If your data engineering team names a dataset “customer_v2_final_CLEAN” and your business glossary calls it “Verified Customer Record,” your technical lineage and your business lineage will never connect. Internal naming standards must be defined before automation begins – not retrofitted afterward. This includes conventions for data assets, lineage nodes, transformation steps, and the relationship between technical names and business terms in Collibra’s Business Glossary.

3. Ownership model

Define who is the Data Owner for each CDE, who is the Data Steward responsible for validating lineage exceptions, and what the escalation path looks like when automated scanning finds something it cannot resolve. Without this, lineage exceptions – dynamic SQL, undocumented ABAP logic, manually maintained reference tables – accumulate with no one responsible for resolving them. The data governance roles framework gives a practical starting point for this structure.

4. Business context mapping protocol

Technical lineage maps tables and columns. Business users need to understand reports and metrics. The gap between these two representations is bridged by connecting technical lineage nodes to entries in Collibra’s Business Glossary – but this connection does not happen automatically. Someone must define which technical assets correspond to which business terms, and maintain that mapping when either side changes. This protocol needs to be owned, documented, and reviewed on a regular cadence.

5. Update and validation protocol

Decide how lineage stays current. Event-driven updates triggered by successful ETL job completions are the most reliable approach at enterprise scale – polling everything in real time is expensive and fragile. Define who reviews flagged exceptions, how often full re-scans run, and what the acceptance criteria are for a lineage path to be considered “validated.” This is not a tool configuration question. It is a governance decision that happens to be implemented in tool configuration.

We’ve built this governance layer from scratch for clients across banking, pharma, and retail – including a custom SAP-Collibra lineage implementation where the technical work was straightforward and the governance definition work took three times longer than anyone expected. That ratio is typical.

“When a client comes to us saying their lineage is a mess, the first thing we do is not touch the tool. We sit down and ask: which five data elements, if their lineage was perfectly documented tomorrow, would change something for your business? The answer to that question defines the whole scope of what we actually need to build.” [Paulina, Solution Architect, Murdio] 

How Collibra implements these standards in practice

Collibra is designed to support all three layers of lineage standards – but the architecture only delivers if the governance layer underneath has been defined. Here is how the three layers map to Collibra’s capabilities.

• Regulatory layer: Collibra’s lineage graph supports bi-temporal audit trails required by BCBS 239. The Integration Service handles technical lineage ingestion from external scanners. For DORA, Collibra’s dependency mapping and impact analysis capabilities allow ICT data flow documentation to be maintained actively rather than statically. For EU AI Act compliance, extending lineage coverage to ML pipelines requires custom integration work – Collibra provides the data model and governance layer; the pipeline-level connections typically require implementation expertise.
• Technical standards layer: Collibra supports OpenLineage integration natively for modern pipeline environments (Spark, Airflow, dbt). For legacy environments – SAP ABAP, custom ETL, proprietary BI tools – standard OpenLineage connectors are insufficient. This is where custom technical lineage development is required, using specialized scanners (IBM Manta Data Lineage, or Solidatus) integrated into Collibra’s architecture. We’ve done this for SAP S/4HANA environments where custom ABAP code created lineage blind spots that standard connectors could not penetrate, and for Snowflake technical lineage where the goal was complete end-to-end coverage from source tables through the consumption layer.
• Internal governance layer: This is where Collibra’s Business Glossary, Data Stewardship workflows, and community structure do the actual governance work. CDEs are registered as assets with documented ownership. Business terms are linked to technical lineage nodes. Steward workflows handle exception resolution. The tool can enforce the governance standards you have defined – but it cannot define them for you. That is the governance layer’s job.

For enterprises in regulated industries, a functioning data governance framework is the prerequisite, not the output.

Signs your lineage standardization isn’t working

These symptoms appear reliably when the governance layer is absent or inconsistent. We call the underlying condition “The Standards Illusion” – a state where an organization has adopted technical standards and deployed a tool, but has not built the governance layer that makes those standards operational.This is related to but distinct from Decorative Lineage – the pattern we describe in our data lineage testing article – where the gap is between captured lineage and current system reality, rather than between technical adoption and governance activation. 

• Different teams interpret “data owner” differently. In practice, ownership questions route to no one or to whoever is least able to say no. Lineage exceptions accumulate unresolved.
• Lineage coverage stops at the data warehouse. Technical lineage maps pipelines and tables. Nothing connects to BI reports, dashboards, or the metrics that business stakeholders actually look at. Trust in the system is zero among non-technical users.
• Technical lineage exists but is not connected to the Business Glossary. Engineers can trace a data flow. A CDO cannot find which source system feeds the “Adjusted Net Revenue” figure in the quarterly board report.
• Audit preparation still requires manual investigation. Collibra is deployed and lineage is “in there somewhere” – but when a regulatory inquiry arrives, the response still involves spreadsheets and three days of engineering time. The lineage is not organized around the questions regulators ask.
• CDEs are not formally identified. The lineage system scans everything or it scans nothing. There is no prioritization layer, which means the most important data elements receive the same coverage (or lack of it) as irrelevant ones.
• Lineage maps are outdated within weeks of each scan. No update protocol exists, no one owns staleness as a metric, and the maps are trusted by no one because everyone knows they reflect a past state.

If more than two of these apply, the problem is not the tool. It is the absence of the governance layer that tells the tool what matters and keeps it current.

For a structured approach to measuring lineage program health, the data lineage metrics framework covers the KPIs that signal whether your standardization effort is actually working.

When to standardize in-house vs. bring in a specialist

Not every lineage standardization effort requires external help. Here is an honest breakdown of where the boundary typically falls.

Handle it internally if:

• Your scope is limited to one primary system and one team. The governance decisions are straightforward when the stakeholder map is small.
• You have no external compliance deadline. The governance layer can be built incrementally over 12-18 months with internal resources.
• You already have functioning data governance adoption and your data stewards understand their roles. Lineage standardization is an extension of existing governance, not a rebuild.
• You have certified Collibra expertise in-house and the technical integration is with modern, well-documented systems.

Bring in a specialist if:

• You have a BCBS 239 or DORA audit window. Regulatory deadlines compress timelines in ways that make trial-and-error governance design expensive.
• Your environment includes SAP custom ABAP, legacy ETL, or hybrid on-premise/cloud architectures. Standard connectors will not give you the coverage regulators expect. The data lineage challenges that emerge in these environments require architectural expertise, not just configuration.
• A previous implementation failed or stalled. If the program has already lost internal credibility, rebuilding it from within is difficult. An external assessment of what went wrong carries more weight than an internal one.
• You are not sure whether the problem is tool configuration, governance structure, or both. The answer changes what needs to happen next – and guessing wrong is expensive.

We offer architectural review sessions specifically for this diagnosis. Not a generic audit – a focused conversation about your specific environment, your regulatory exposure, and what the realistic path to defensible lineage looks like from where you are today.

If you are at the point of defining or rebuilding your lineage standards and want a realistic assessment of what your specific environment requires, we offer data governance implementation support tailored to regulated enterprises. Our starting point is always the governance layer, not the tool – because the tool can only standardize what the governance layer has already defined.

Schedule an architectural review with a Murdio expert 

Frequently Asked Questions