Skip to content

Lineage & Impact Analysis

Understanding where data originates, how it transforms, and where it flows is essential for regulatory compliance (e.g., BCBS 239, GDPR), troubleshooting data pipeline failures, and performing risk-free schema migrations.

VDA provides an interactive, graph-driven Lineage & Impact Analysis Engine backed by SurrealDB and standard OpenLineage event streams.

1. Lineage Graph Explorer

Navigate to Lineage from the main menu or click View Lineage from any dataset details page.

flowchart LR
    subgraph Sources["Raw Ingestion"]
        S1["Core Banking Oracle<br/>TRANSACTIONS"]
        S2["Branches<br/>BRANCH_REF"]
    end

    subgraph Transformation["ETL / Spark Jobs"]
        J1["Spark Batch Job<br/>ETL_DAILY_SETTLEMENT"]
    end

    subgraph Warehouse["Data Warehouse"]
        W1["Fact Table<br/>FACT_TRANSACTIONS"]
        W2["Aggregated Mart<br/>DM_BRANCH_PERFORMANCE"]
    end

    subgraph Consumption["BI & Dashboards"]
        C1["Superset Dashboard<br/>Daily Executive KPI"]
        C2["Risk Analytics Model<br/>AML_MONITOR_FEED"]
    end

    S1 --> J1
    S2 --> J1
    J1 --> W1
    W1 --> W2
    W2 --> C1
    W1 --> C2

Visual Controls

  • Pan & Zoom: Navigate large multi-system topologies seamlessly.
  • Node Filtering: Filter nodes by system type (Databases, Pipelines, Transformation Jobs, BI Dashboards).
  • Depth Slider: Expand or contract the visual graph from 1 level up to full end-to-end depth (10+ hops).
  • Status Indicators: Real-time badges indicate node health, recent execution status (Success, Failed, Running), and schema staleness.

2. Column-Level Lineage (Fine-Grained Tracing)

While table-level lineage shows dataset dependencies, Column-Level Lineage traces individual attributes through calculations, renames, and joins.

  1. In the lineage graph, click the Column Lineage toggle.
  2. Select any column from the inspected table (e.g., SETTLED_AMOUNT_USD).
  3. VDA highlights the exact flow path:
  4. Upstream Origin: Traces back through calculations (e.g., transactions.trans_amt * exchange_rates.rate_to_usd).
  5. Downstream Sinks: Shows all warehouse columns, views, and report metrics derived from this specific field.

3. Downstream Impact Analysis

Before executing a schema change (e.g., dropping a column, changing datatype, or deprecating a table):

  1. Right-click the asset or column node and select Run Impact Analysis.
  2. VDA calculates the complete downstream closure and generates an Impact Assessment Summary:
  3. Affected Downstream Datasets: List of tables, views, and marts that depend on this node.
  4. Impacted Pipelines: Spark, Airflow, and dbt jobs that read from this asset.
  5. Affected Dashboards & Reports: Superset charts and analytical extracts that rely on these fields.
  6. Stakeholders: Data owners and consumer roles requiring notification before changes are deployed.

4. Pipeline Observability & Run Tracking

VDA ingests OpenLineage event streams from orchestration frameworks (Airflow, Spark, dbt): - Run History: View historical execution runs, start times, durations, and output row counts. - Error Diagnostics: When a pipeline run fails, click the job node to view the error message, failing task ID, and stack trace without leaving VDA. - Schema Evolution Tracking: Inspect schema version history and pinpoint when new columns were added or dropped across ETL iterations.