Data lineage requirements: a practical framework for large enterprises

A team at a European financial institution spent six months configuring Collibra, mapping hundreds of tables, and building what looked like a complete lineage graph. When the compliance audit came, three questions stopped them cold: which Critical Data Elements have documented end-to-end traceability? Who owns each CDE at the source? And what was the lineage state at the time of the last regulatory report?

None of it had been defined before the project started.

We see this pattern more often than we’d like to admit. Organizations jump to tool configuration before agreeing on what the lineage needs to deliver – and end up with technically impressive maps that fail regulatory scrutiny and that business stakeholders can’t interpret. The fix isn’t more configuration. It’s requirements definition that should have happened first.

Key takeaways

• Data lineage requirements fall into five categories: regulatory, functional, integration, organizational, and success metrics. Missing any one of them creates a failure mode that no tool can fix.
• Regulatory requirements (BCBS 239, DORA, EU AI Act) are the only non-negotiable categories for EU enterprises in regulated industries. They define the minimum floor – everything else builds on top.
• The single highest-leverage action before any lineage project is defining your Critical Data Elements (CDEs): which data assets are critical enough to track, and who owns them at the source.
• Lineage that stops at the data warehouse boundary does not satisfy most regulatory requirements, and does not build data trust with business users. End-to-end coverage to the BI layer is a functional requirement, not a nice-to-have.
• The most common failure mode is the Configuration-First Trap: starting with tool setup before agreeing on what “complete” lineage looks like. The result is a technically valid lineage that nobody uses and that fails compliance tests.
• Organizational requirements – naming conventions, freshness SLAs, ownership models – are the hardest category to define and the most common reason implementations stall after go-live.

What are data lineage requirements?

Data lineage requirements are the documented specifications that define what a lineage graph must capture, how it must be maintained, and what compliance and business objectives it must satisfy – before any tool is configured or any metadata is collected.

They are not the same as data lineage standards. Standards (such as BCBS 239, OpenLineage, or W3C PROV) define the external rules and technical specifications that lineage must conform to. Requirements define what your specific organization needs its lineage to do – translating those external rules into actionable, auditable decisions about scope, ownership, depth, and success criteria.

The distinction matters because organizations routinely adopt OpenLineage, deploy Collibra, and still end up with lineage programs that fail audits and get ignored by business users. They addressed the standards layer. They skipped the requirements layer.

The Configuration-First Trap-and why it’s expensive

The typical lineage project in a large enterprise starts with tool selection, then license procurement, then configuration. Requirements – if they get defined at all – are treated as an output of the configuration process, not an input to it.

This creates a predictable failure mode. The engineering team produces technically complete lineage maps. The compliance team reviews them and finds gaps: CDEs without documented ownership, transformation logic that stops at the warehouse boundary, audit trails without bi-temporal metadata. The business teams find the maps uninterpretable because there’s no connection to business glossary terms or report-level KPIs. The project is declared complete, the maps decay, and two years later someone starts the cycle again.

The cost isn’t just the wasted configuration effort. It’s the audit exposure during the gap. For BCBS 239 banks and DORA-regulated financial entities in the EU, an incomplete lineage program isn’t a governance gap – it’s a regulatory risk. For pharma companies deploying AI systems under the EU AI Act, undocumented training data provenance isn’t a technical oversight – it’s a documentation requirement with legal consequence.

Understanding the difference between data governance and data lineage is a good starting point – but requirements definition is where that understanding gets operationalized.

The five categories of data lineage requirements

1. Regulatory requirements

Regulatory requirements are the only category defined externally. For large enterprises in the EU, three frameworks impose specific, auditable lineage obligations.

BCBS 239 applies to systemically important banks and requires end-to-end traceability of risk data from raw source systems through all transformations to the final regulatory report. Column-level lineage for CDEs used in risk reporting is not optional – it’s the mechanism that answers regulator questions like “where does this Net Exposure figure come from?” Lineage that stops at the warehouse does not satisfy BCBS 239.

DORA, which has applied since 17 January 2025, requires documented, actively maintained maps of critical ICT data flows and dependencies, specifically to support resilience testing and incident recovery. Static lineage documentation updated 18 months ago during a project phase does not satisfy DORA. Active maintenance – meaning event-driven updates tied to pipeline execution – is a compliance requirement, not a technical preference.

The EU AI Act requires technical documentation including a description of training datasets and methodologies for high-risk AI systems. For organizations in banking, insurance, or pharma using AI for credit scoring, fraud detection, or medical decision support, column-level lineage from source systems through preprocessing pipelines to training datasets is the practical mechanism organisations use to satisfy Article 11’s technical documentation requirements. .

These frameworks are covered in detail in our article on data lineage standards. For requirements purposes, the key action is translating each applicable framework into a concrete scope statement: which data elements, which systems, which reports, and which audit trail depth.

2. Functional requirements

Functional requirements define what the lineage must show and to whom. These are the specifications that determine whether the lineage is actually usable once built.

Minimum functional requirements for enterprise lineage programs:

• End-to-end coverage – from authoritative source system through all transformations to the final BI consumption layer (PowerBI, Tableau, or equivalent). Lineage that stops at the data warehouse leaves the most visible layer – the one business users interact with – outside the governance perimeter. This is where data trust breaks down. See how this manifests in practice in our data lineage challenges article.
• Column-level granularity for CDEs in regulatory scope. Table-level lineage is not sufficient for BCBS 239 or DORA compliance.
• Business context mapping – technical lineage nodes connected to business glossary terms. Without this, the lineage is readable only by engineers. Business stakeholders cannot confirm that “Customer Revenue” in the dashboard derives from the correct source, and the governance program loses adoption.
• Impact analysis capability – the ability to assess downstream consequences of a schema change or data quality issue before it propagates. This is the primary operational use case for lineage outside of compliance.
• Bi-temporal audit trail for regulated CDEs – capturing both when a change occurred and what the data state was at a specific point in time.

3. Integration requirements

Integration requirements define which source systems must feed into the lineage program and in what priority sequence. This is where scope decisions get made – and where most projects either over-promise or under-specify.

Key questions to answer before configuration starts:

• Which ERP systems (SAP, Oracle) must be connected, and do any rely on custom code (SAP ABAP Z-tables, Z-programs) that won’t surface with standard crawlers?
• Which cloud data warehouses (Snowflake, BigQuery, Redshift) are in scope?
• Which BI tools constitute the consumption layer?
• Which ETL/ELT pipelines carry CDE data, and do they use event-driven metadata (OpenLineage-compatible) or require batch scanning?

The SAP ABAP question matters more than it’s usually given credit for. Custom ABAP code operates as a black box for standard lineage scanners – the transformation logic exists but can’t be parsed without specialized tooling. Organizations that don’t flag this in requirements discover it during implementation, after scope and timeline are already committed. Our custom SAP lineage implementation case study covers exactly this architecture challenge.

Integration requirements should also specify update frequency: event-driven triggers tied to ETL completion, scheduled batch scans, or hybrid. This isn’t just a technical decision – it directly affects whether the lineage satisfies DORA’s active maintenance requirement and whether it’s fresh enough to support incident response.

4. Organizational requirements

Organizational requirements are the hardest category to define and the most frequently skipped. They govern how the lineage program operates as a sustained process, not just a one-time configuration effort.

Five decisions that must be made before any automated lineage scan runs:

1. CDE identification – which data elements are critical enough to warrant end-to-end lineage? Not everything. A useful heuristic is to focus on the roughly 20% of data assets that drive the majority of business value  (key financial metrics, regulatory reporting inputs, AI training datasets). A lineage program that tries to cover everything achieves coverage of nothing useful.
2. CDE ownership model – who is the accountable data owner for each CDE at the source system level? Lineage without ownership has no mechanism for remediation when something breaks. This connects directly to the data governance roles structure already in place.
3. Naming conventions – how will lineage assets be named consistently across systems? Inconsistent naming produces technically accurate lineage that can’t be cross-referenced or queried reliably.
4. Freshness SLAs – what is the acceptable staleness threshold for lineage metadata? Regulatory frameworks like DORA imply near-real-time freshness for critical flows. Operational use cases may tolerate daily batch updates. The SLA determines the update architecture.
5. Governance process for changes – when a source schema changes, who is responsible for updating lineage metadata? Without a defined process, lineage decays from day one of go-live.

These decisions are the internal governance layer that most lineage programs never formalize. They don’t require a tool. They require organizational alignment – and that alignment is significantly harder than any technical configuration. For a practical foundation, the data governance framework article covers the broader governance structure these decisions need to sit within.

5. Success metrics requirements

Success metrics requirements define what “working lineage” looks like as a measurable state. Without them, there’s no basis for declaring the project complete, no mechanism for ongoing health monitoring, and no ROI case for the investment.

Core metrics every lineage program needs to define upfront:

• CDE coverage percentage – ratio of in-scope CDEs with verified end-to-end lineage vs. total identified CDEs. Target 100% for regulatory-critical in-scope CDEs; prioritize these before expanding scope.
• Lineage accuracy – percentage of mapped data flows that match the actual physical flow. Track directional improvement; full accuracy on day one isn’t realistic in complex enterprise environments.
• Lineage staleness – time elapsed since the last confirmed metadata sync for each CDE. A current SLA breach is a compliance signal, not just a technical debt item.
• Audit response time – how long it takes to answer a regulatory data inquiry with documented lineage evidence. Reducing this from days or weeks to hours is the most concrete ROI indicator for the compliance case. Our article on data lineage metrics covers the full metrics framework per stakeholder group.

Define these metrics before the project starts. They serve as acceptance criteria during implementation and as ongoing health indicators after go-live.

How to document requirements before implementation

A requirements document for a lineage program doesn’t need to be elaborate. It needs to answer five questions clearly enough that the implementation team, the compliance team, and the business stakeholders can all read it and agree it reflects their actual needs.

1. Step 1: Identify applicable regulatory frameworks. List which frameworks apply (BCBS 239, DORA, EU AI Act, GDPR) and translate each into specific lineage obligations – which data, which depth, which audit trail, which update frequency.
2. Step 2: Define CDEs. Work with data owners to identify which data elements are in regulatory scope and which drive primary business decisions. Document names, source systems, current ownership, and known quality issues.
3. Step 3: Map integration scope. List every source system that feeds CDE data, flag any with custom code or non-standard metadata, and sequence them by regulatory priority.
4. Step 4: Agree on organizational decisions. Run a structured workshop to define naming conventions, ownership model, freshness SLAs, and the change governance process. Document the outputs formally. These decisions cannot be delegated to the implementation team.
5. Step 5: Set success metrics and review cadence. Define the specific KPIs from category 5 above, set baseline measurements, and agree on review frequency. Quarterly lineage health reviews prevent the decay pattern seen in most mature implementations.

The entire process takes two to four weeks for a tightly scoped pilot with committed stakeholders and pre-identified CDEs, longer where ownership is contested or systems are complex. It’s the fastest way to prevent a six-month implementation from producing results that satisfy nobody.

Signs your lineage requirements were never properly defined

The symptoms are recognizable:

• The compliance team rejects the lineage maps because they can’t answer “where does this figure come from at the column level?” – functional and regulatory requirements weren’t defined.
• Business users don’t use the lineage tool because it shows tables and joins but not what “Revenue” means in business terms – functional requirements missed the business context layer.
• The lineage was accurate 18 months ago and hasn’t been maintained since – organizational requirements (freshness SLA, change governance) were never defined.
• SAP transformations are missing from the lineage graph – integration requirements didn’t account for custom ABAP code.
• The project expanded indefinitely because nobody agreed on what was in scope – CDE identification was skipped, so every new system became a candidate for inclusion.
• Nobody knows which CDE owner to call when a data quality issue appears downstream – ownership model was never defined.

If more than two of these describe your current situation, the lineage infrastructure is there but the requirements foundation is missing. This is fixable – but it requires going back to requirements, not adding more configuration.

When to bring in external expertise

Requirements definition for a lineage program sits at the intersection of regulatory compliance, data architecture, and organizational change – three domains that rarely live in the same internal team.

A reasonable dividing line: if your organization has a clear data governance framework already in place, active data stewardship, and an implementation team with Collibra experience, you can likely work through requirements with structured internal workshops and an experienced facilitator.

If you’re starting from an incomplete governance foundation, or if BCBS 239 / DORA compliance is a hard constraint with an audit deadline, external support in the requirements phase pays for itself in avoiding rework during implementation. The end-to-end lineage case study for SAP S/4HANA Cloud shows what a fully specified requirements baseline enables when implementation starts clean.

The data governance consulting approach Murdio uses starts at requirements – not tool configuration – for exactly this reason.

Getting lineage requirements right is one of the most upstream interventions available to a data governance program – and one of the most underinvested. If you’re starting a lineage initiative or restarting one that stalled, that’s exactly the conversation we have most often with new clients.

Talk to a Murdio expert about your lineage requirements