The data lineage process: from first CDE to operational governance

The most dangerous moment in a data lineage project is not when the implementation fails. It’s when engineering declares it a success, hands over a technically complete graph, and considers their part done. That’s when the governance process should begin. Usually, it doesn’t.

This article is about that process – not what data lineage is, but how you run the governance layer that makes it operational, trusted, and maintained.

Key takeaways

• The data lineage process operates at two levels: technical capture and governance operating model. Most implementations complete the first and skip the second.
• The correct sequence starts with scope and Critical Data Element (CDE) definition – before any tool configuration begins.
• Business lineage mapping – connecting technical flows to your Business Glossary in Collibra – is consistently the first phase to get cut when projects run over time or budget. It is also the phase that determines whether business users ever trust the lineage at all.
• A lineage map without a defined ownership model becomes stale within three to six months of go-live.
• For EU enterprises in regulated industries, BCBS 239, DORA, and the EU AI Act define the minimum scope of your lineage process. They are not optional contexts.
• When engineering delivers lineage and governance has no process to maintain it, you do not have a lineage program. You have a lineage artifact.

What the data lineage process actually involves

The data lineage process is the structured sequence of governance and technical steps an organization uses to capture, maintain, and operationalize the full record of how data moves from source systems through transformations to BI consumption. It includes both automated technical lineage capture and the governance layer that defines ownership, freshness standards, and business context mapping.

Most explanations of this process stop at the technical layer: data flows from source to transformation to consumption, metadata is collected at each step, and a lineage diagram is produced. That description is accurate and incomplete. The technical capture is the mechanism. The governance layer is the process that gives it purpose.

The distinction matters because the failure modes are different. Technical lineage fails when systems are too complex to scan, when legacy environments have no automated connectors, or when push and pull architectures are not reconciled. Governance process failures look different: CDEs without documented owners, maps that stop at the warehouse boundary, lineage that is technically valid but that nobody uses or trusts. Both failure types exist in parallel in most enterprise programs – and they need separate attention.

The six phases of an enterprise data lineage process

The sequence below reflects what we have seen work across banking, pharma, retail, and manufacturing environments in the EU. The order is not incidental. Skipping or resequencing phases is one of the most reliable predictors of a program that will need to be rebuilt.

Phase 1: Scope and CDE definition

Before any lineage tool is configured, the governance team must agree on three things: which data domains are in scope, which assets within those domains are Critical Data Elements, and who owns each CDE at the source.

Scope without CDE prioritization produces noise. An enterprise with thousands of tables cannot meaningfully track all of them at equal depth. The correct starting point is the data elements that drive the highest-impact business decisions or carry the greatest regulatory risk – typically the KPIs in executive reporting, the data elements cited in BCBS 239 or DORA disclosures, and the personal data flows relevant to GDPR.

This phase produces a scoped CDE registry that every subsequent phase references. If it does not exist before Phase 3 begins, automated scanners will collect everything indiscriminately and the resulting lineage graph will be technically complete and practically unnavigable.

For a detailed framework on how to document scope and requirements before configuration, see our article on data lineage requirements.

Phase 2: Standards and regulatory alignment

The second phase establishes the rules the lineage program must satisfy – external regulatory requirements, technical open standards, and internal governance conventions. These three categories operate independently and all three must be addressed before lineage capture begins.

Regulatory requirements (BCBS 239, DORA, EU AI Act) define the minimum floor for what end-to-end lineage must cover and how it must be documented. Technical standards (OpenLineage, W3C PROV) define interoperability across the data stack. Internal governance standards define naming conventions, freshness SLAs, and ownership models specific to your organization.

Conflating these categories is the single most common reason lineage programs stall at the standards layer. A full breakdown of the three-layer model and what each requires is in our article on data lineage standards.

Phase 3: Technical lineage capture

With scope defined and standards agreed, the engineering team can begin automated lineage capture. For most large enterprises, this means a hybrid approach: event-driven push methods (OpenLineage and similar) for modern cloud stacks, and pull-based scanners for legacy environments including SAP ABAP.

Each architecture type introduces specific challenges. SAP environments with custom ABAP code, Z-tables, and Z-programs require specialized scanners to translate legacy logic into metadata Collibra can visualize. Modern cloud pipelines using dbt or Spark can emit lineage events directly via OpenLineage. The two streams must be reconciled into a single graph.

The technical complexity of this phase – and the specific failure patterns it introduces – is covered in depth in our article on data lineage challenges. What matters here is that Phase 3 is not complete when the scanner runs successfully. It is complete when every CDE identified in Phase 1 has a traceable path from source to BI consumption layer. Stopping at the data warehouse boundary is one of the most common reasons lineage fails a regulatory audit even after technical implementation.

We have documented this pattern directly in our custom SAP lineage case study, where the challenge was not just running a scanner across a multi-layered SAP architecture but reconciling MDG, BW, the data lake, and Tableau into a single coherent lineage graph.

Phase 4: Business lineage mapping

Technical lineage tells an engineer how data moves. Business lineage tells a Data Steward, a DG Manager, or a CDO what it means. The two layers serve different users, and bridging them requires explicit governance decisions that no automated scanner produces on its own.

In Collibra, this phase involves linking the technical assets captured in Phase 3 – tables, columns, pipeline steps – to the Business Glossary terms that represent business concepts. A table called MARA becomes “Material Master Data.” A KPI called NET_REV_Q3 becomes “Global Net Revenue Q3.” This mapping is not automatic. It requires data stewards to review technical assets, governance managers to agree on business term definitions, and a workflow process to manage the review and approval cycle. This process is straightforward for self-explanatory field names such as client_first_name, but requires more governance effort for cryptic identifiers such as clnt_n_2 where business meaning is not evident from the technical name.

One factor that significantly reduces the manual effort in this phase is the semantic layer – a modelling layer that sits between the raw technical lineage graph and the Business Glossary. Rather than connecting individual columns directly to business terms one by one, the semantic layer groups related technical assets into logical business entities first, which makes the mapping process more structured and the resulting connections more maintainable. Collibra’s Semantic Assistant and automated stitching capabilities provide partial automation of this linking, suggesting business term, measure, and column matches for stewards to accept, reject, or correct rather than building every connection from scratch – reducing the volume of manual stewardship required. Governance decisions about which business terms apply to which entities still require human review regardless of how the suggestion was generated. If your organization has an existing semantic model, integrating it into the Collibra workflow early in Phase 4 reduces both implementation effort and ongoing maintenance overhead considerably.

“The business lineage layer is almost always the first thing cut when a project runs over time or over budget,” says [Paulina, Solution Architect, Murdio]. “But it is also the only layer that makes lineage useful to anyone outside of engineering. Without it, you have a map that the compliance team cannot read and that business users will never open.”

Phase 5: Validation and trust-building

Before lineage is considered operational, it must pass a validation cycle that includes both technical accuracy review and business user acceptance. Technical validation confirms that every CDE’s path is complete, that transformation logic is correctly represented, and that the lineage reflects the current state of the data architecture rather than a snapshot from six months ago.

Business validation is where trust is actually built. For Collibra lineage programs to have lasting adoption, the business stakeholders who rely on CDE-linked reports must have a mechanism to verify that the lineage makes sense to them – that the business terms connected to technical assets actually match what those assets represent. This does not require extensive technical knowledge. It requires structured review sessions, clear ownership, and a process for flagging discrepancies.

Running an audit dry-run as part of this phase is strongly recommended for regulated enterprises. The dry-run simulates an incoming BCBS 239 or DORA inquiry: can the governance team produce documented proof of data origin, transformation logic, and ownership for each CDE cited in a regulatory report within an acceptable response window? We recommend a 48-hour benchmark as a completion criterion for this phase.

Phase 6: Ongoing governance and maintenance

This is the phase most programs do not have.

Engineering delivers the lineage graph. Governance receives it. Six months pass. Pipelines change, new source systems appear, CDEs are reassigned to different owners. Nobody updates the lineage because nobody’s job description includes updating the lineage. The maps become stale. Business users notice that the lineage no longer matches the data they see in reports. Trust collapses faster than it was built.

“What we see most often is a technically solid implementation followed by months of silence,” says [Stan, Senior Consultant, Murdio]. “Then the lineage team realizes half the CDEs have changed ownership or new pipelines were added without any notification process. You get a perfect map of how things worked at go-live, not how things work now.”

Phase 6 requires four concrete governance decisions: who owns each CDE (not just as a name, but as a responsibility with a defined process for updates), what the freshness SLA is for each domain, who is responsible for reviewing and approving lineage changes when underlying systems change, and how stale metadata is identified, flagged, and remediated. For context on data governance roles and how ownership structures support lineage maintenance, that article covers the organizational layer in depth.

Where the data lineage process breaks down

Technical failures in lineage implementation are well documented. The governance process failures are less visible but more consequential. These are the patterns we see most often.

The Configuration-First Trap

Organizations begin with tool selection and license procurement. Requirements – what the lineage needs to actually deliver, which CDEs must be covered, what the regulatory baseline is – are defined as an output of the configuration process rather than an input to it. The result is technically impressive maps that fail the next compliance review because the scope was never tied to regulatory obligations. Full detail on this pattern, which we call the Configuration-First Trap, is in our data lineage requirements article linked in Phase 1 above.

Stopping at the warehouse boundary

Lineage that ends at the data warehouse does not satisfy most regulatory requirements and does not give business users visibility into the reports they actually consume. End-to-end coverage means tracing data from source systems through the warehouse and into the BI layer – PowerBI, Tableau, or whatever reporting tooling the organization uses. Everything short of that is partial lineage, which is technically interesting but governance-incomplete.

The Handover Gap

Engineering delivers a technically complete lineage graph and considers the project done. The governance team receives it without a process for how to maintain it, how to connect it to the Business Glossary, or how to handle the first pipeline change that affects a covered CDE. The map exists. The process does not. Within months, the map is an artifact rather than a living governance tool.

Missing ownership model

A CDE with no assigned owner is a CDE whose lineage will not be updated when the underlying system changes. Ownership must be defined at the CDE level, documented in Collibra, and tied to a notification process that is triggered by technical changes to the systems feeding that CDE. Without it, Phase 6 is aspirational rather than operational.

Business lineage never started

Phase 4 is outscoped mid-project. Technical maps exist. Business terms exist. Nobody connected them. The compliance team can see data moving through pipelines. They cannot answer the question “which pipeline produces the ‘Regulatory Capital Ratio’ reported to the board?” because that connection was never made.

How Collibra structures the data lineage process

Collibra data lineage separates into two components that require different governance inputs.

The technical lineage layer is built via Collibra Edge, combining pull-based scanning with direct OpenLineage event ingestion from modern data pipelines. Collibra’s legacy CLI Lineage Harvester end-of-life (July 31, 2026) means that new implementations should be built on Edge rather than the Harvester. Collibra automates the collection and visualization of metadata flows, dependency mapping, and impact analysis once connectors are configured. For hybrid SAP/cloud environments, this typically involves a combination of pull-based scanning (JDBC, proprietary scanners for ABAP environments, run through Edge) and push-based OpenLineage event streams from cloud platforms.

What Collibra does not automate is the governance layer: which assets should be covered (Phase 1), what naming conventions apply (Phase 2), and who owns each CDE (Phase 6). For Phase 4, Collibra’s semantic mapping tools can suggest connections between technical assets and business terms, but accepting, rejecting, or overriding those suggestions is still a governance decision, not a system default. These are governance decisions that must be made before configuration and enforced through workflows after go-live.

The distinction is worth internalizing before any Collibra lineage project begins. The tool handles capture and visualization at scale. It does not substitute for the governance process that defines what to capture, who owns it, and what it means.

How to measure whether your data lineage process is working

Measurement is covered in depth in our article on data lineage metrics. For the process assessment specifically, four indicators are most predictive:

1. CDE coverage rate – the percentage of CDEs in scope that have complete, end-to-end documented lineage to the BI layer. Anything below 80% signals that Phase 3 has gaps.
2. Lineage freshness – the average age of the most recent lineage update per CDE domain. Staleness above 30 days in actively changing environments indicates Phase 6 is not operational.
3. Audit response time – how long it takes the governance team to respond to an incoming regulatory data inquiry with documented proof of CDE origin and transformation history. This is the most direct measure of whether the process is working as a compliance capability, not just a technical one.
4. Business lineage coverage – the percentage of CDEs with confirmed Business Glossary connections. A score of zero on this metric means Phase 4 never happened.

Signs your data lineage process isn’t working

The most common early warning signs are visible long before a compliance audit surfaces the gap.

1. Your compliance team still asks engineering for manual reports during audits. If the lineage program is operational, the governance team should be able to pull documented CDE provenance without requesting custom queries. If engineering is still involved in every audit response, the process layer is missing.
2. Business users do not trust the metrics from dashboards. When a sales leader asks “where does this revenue figure come from?” and the answer is “let me check with the data team,” business lineage either does not exist or is not accessible to business users. This is a Phase 4 failure.
3. Impact analysis still happens through Slack threads. Before a schema change, engineers should be able to pull a lineage-based impact report from Collibra that shows every downstream report and CDE affected. If that capability does not exist and the process is still “ask around,” Phase 3 coverage is incomplete.
4. The lineage map has not been updated in over three months. In most large enterprise environments, three months is enough time for meaningful pipeline changes to occur. A map that has not been updated is a map that is partially wrong.
5. Nobody can name the owner of a specific CDE. Ownership must be queryable in Collibra, not held in someone’s head. If the answer to “who owns this CDE?” requires a meeting, Phase 6 is not in place.
6. Technical lineage exists but no Business Glossary connections do. This is the most common single failure pattern we encounter: all six Phase 3 scanners running, zero Phase 4 work done. The maps look impressive and deliver no governance value to non-technical stakeholders.

When to handle the data lineage process internally vs. bring in specialists

Some lineage programs can be built and maintained entirely by an internal team. The conditions for that are relatively narrow: a greenfield environment with no legacy systems, a single data platform (Snowflake or similar with native lineage support), a small regulatory footprint, and a governance team that already has Collibra operational expertise.

Specialist support makes sense when any of the following conditions apply:

1. Hybrid SAP/cloud architecture. SAP ABAP environments require specialized lineage connectors and custom integration work that goes beyond standard Collibra configuration. The gap between what automated scanners capture and what actually needs to be in the lineage graph can be significant. See how we addressed this in our end-to-end SAP S/4HANA lineage case study.
2. BCBS 239, DORA, or EU AI Act compliance scope. Regulatory lineage requirements impose specific, auditable standards for what end-to-end means, what documentation is required, and how quickly you must respond to a regulatory inquiry. Getting this wrong is expensive. Getting it right the first time requires implementation experience in regulated environments.
3. A first implementation that did not deliver. If your organization has been through one lineage project that produced technically complete maps that nobody uses, the problem is almost certainly the governance process layer, not the tool. Rebuilding from the same starting point produces the same result.
4. Need to connect technical lineage to Business Glossary at scale. Phase 4 at scale – hundreds of CDEs across multiple business domains, multiple data steward teams, multiple Collibra communities – is a governance design problem, not a configuration problem. It requires structured workshops, workflow design, and adoption support that most internal teams are not resourced to run concurrently with their day jobs.

If your organization is at Phase 3 or beyond and the governance layers are not yet in place – or if you are looking at a first implementation in a regulated environment and want to start from the right end – a conversation with a Murdio solution architect will tell you where the gaps are and what a realistic roadmap looks like. We have helped enterprises across banking, pharma, retail, and energy establish operational lineage governance in Collibra, including environments where a previous implementation had already been attempted. Reach out through our data governance implementation page or directly via our Collibra use case implementation service.

Frequently Asked  Questions