Case Study: Custom Collibra SAP Lineage Implementation

Case Study: Custom Collibra SAP Lineage Implementation

23 01
2025

A leading international retail organization discovered that tracking complex data attributes across multiple systems was hindering its operational efficiency and strategic decision-making. This case study explores a transformative solution developed by the Murdio team to address the challenges of tracking and understanding data lineage through a sophisticated technological infrastructure that spanned SAP Master Data Governance (MDG), SAP Business Warehouse (BW), centralized data lakes, and Collibra business intelligence platforms.

The challenge

Our client, a global retail company operating in Europe, United States and Asia, faced challenges in tracking and understanding the lineage of critical data attributes such as total shelf life across its complex data ecosystem. These attributes were managed in SAP MDG (Master Data Governance system) and subsequently integrated into SAP BW for aggregation, transformations, and reporting. From there, the data flowed into a centralized data lake and BI platforms to support various reporting teams, which relied on tools like Tableau for business decision-making.

Despite the existing capabilities of Collibra, integrating lineage data across SAP’s layered architecture—spanning databases, applications, and custom ETL processes—required a thorough analysis and custom solution to allow smooth updates and changes. The standard Collibra connectors and lineage tools were insufficient due to how metadata from different platforms was ingested.

Challenges included:

  1. Complex architecture: SAP’s multi-layered structure (database, application, and aggregation levels) made traditional lineage tracking infeasible.
  2. Custom ETL processes: Data transformations in SAP BW involved many ETL processes based partially on out-of-the-box SAP connectors but mostly custom-built developed by the customer support teams.
  3. Volume of data: Each SAP system can generate a few millions of metadata elements, most of which are irrelevant, creating noise in lineage diagrams.
  4. Manual efforts: Identifying relationships required deep analysis to break all relevant parts into smaller pieces and then using custom scripting stitching them together.
  5. Stakeholder impact: Changes in upstream data systems affected numerous downstream reporting teams, making impact analysis essential.

Solution

Case Study: Custom Collibra SAP Lineage Implementation by MurdioMurdio’s team developed a custom integration to enable end-to-end lineage tracking in Collibra, tailored specifically to the organization’s SAP landscape:

  1. Data source mapping: Collaborated with stakeholders and SMEs to create mapping between sources and targets for objects across SAP, BI platforms and Data Lake.
  2. Automated lineage: A senior developer created a solution which would accept input data in the form of mapping logic, transform it and load into Collibra which later can be visualized in a form of lineage diagram.
  3. Impact analysis: Visual lineage diagrams in Collibra identified which downstream teams and reports would be affected by changes to specific data attributes.
  4. Data quality rules integration: Business rules (e.g., gross weight limits) were linked to data quality checks in Collibra, ensuring compliance with predefined standards.

Results

  1. Enhanced transparency: The organization’s stakeholders gained visibility into the entire data flow—from source systems to final reports.
  2. Improved data quality: Integrated business and data quality rules ensured consistent and accurate data across systems.
  3. Streamlined operations: Automated lineage mapping reduced manual effort, saving months of work and minimizing human error.
  4. Actionable insights: Impact analysis empowered teams to make informed decisions about system changes, reducing disruptions to reporting.
  5. Scalability: The custom solution was designed to accommodate future expansions and additional SAP systems.

Conclusion

This project highlights how Murdio expertise in custom Collibra development can bridge critical gaps in enterprise data ecosystems. By integrating SAP systems with Collibra through a custom solution, this initiative ensured automated lineage tracking and comprehensive impact analysis. These improvements enhanced data quality and transparency, empowering the organization to better manage its data governance framework and make informed business decisions.

Insights & News