Skip to content

Data Quality & Great Expectations (GX)

High-quality data is fundamental to reliable decision-making and machine learning. VDA integrates deep Modern Data Profiling with an AI-driven Great Expectations (GX) orchestration suite to continuously monitor, validate, and alert on data anomalies.

1. Modern Profiling View

From any table details page, open the Data Profiling tab to inspect comprehensive column statistics calculated across table partitions:

Dataset profile and data health

The profiling view: table completeness, distinctness, and a per-column breakdown of null rates, distinct values, ranges, and distributions.

Profiling Metrics

  • Completeness: Null and empty string rates.
  • Uniqueness: Distinct value ratio and duplicate occurrence frequency.
  • Numerical Spread: Minimum, maximum, average, standard deviation, and quantile distribution (P25, P50, P75, P90, P99).
  • String Distributions: Character length bounds, regex format compliance, and top distinct values.
  • Temporal Trends: Date range coverage, temporal gaps, and transaction frequency.

2. AI-Driven Great Expectations Suite Generation

Rather than writing hundreds of assertion checks manually, VDA uses profiling statistics and table semantics to automatically formulate production-ready Great Expectations suites:

  1. In the Data Quality tab, click Generate AI Expectations.
  2. VDA inspects column profiles and generates proposed expectations:
  3. expect_column_values_to_not_be_null(column='ACCOUNT_NUMBER')
  4. expect_column_values_to_be_unique(column='TRANSACTION_ID')
  5. expect_column_values_to_be_between(column='TRANSACTION_AMOUNT', min_value=0.01, max_value=1000000)
  6. expect_column_values_to_match_regex(column='IBAN', regex='^[A-Z]{2}[0-9]{2}[A-Z0-9]{4}[0-9]{7}([A-Z0-9]?){0,16}$')
  7. expect_column_values_to_be_in_set(column='STATUS', value_set=['PENDING', 'SETTLED', 'FAILED', 'CANCELLED'])
  8. Review, tune thresholds, or add custom assertions in the interactive modal.
  9. Click Save Expectation Suite.

3. Running & Monitoring Validation Jobs

Validation suites run as distributed background workflows orchestrated by Temporal.io:

  • On-Demand Validation: Click Run Validation Now to test the current state of the dataset immediately.
  • Scheduled Monitoring: Configure cron schedules (e.g., daily after midnight batch ETL) to validate tables automatically.
  • Job Status & Results:
  • Passed Expectations: Assertions that succeeded with zero discrepancies.
  • Violations & Anomaly Samples: Exact row counts and representative sample values that violated thresholds.
  • Trend Analysis: Track quality health score improvements or degradations over time.