Skip to content

Connecting Your First Datasource & Ingestion Pipeline

With the administrative account active, connect an enterprise database to launch automatic metadata extraction, column profiling, lineage discovery, and PII detection.

flowchart TD
    UI["VDA Web UI<br/>(Add Data Source)"] -->|GraphQL Mutation| Server["vda-server"]
    Server -->|Persist Connection| PG[("PostgreSQL")]
    Server -->|Trigger Workflow| Temp["Temporal Server"]
    Temp --> Worker["vda-metadata-worker"]

    subgraph Pipeline["End-to-End Ingestion Pipeline"]
        Worker -->|1. Extract Schema| DS[("Source Database")]
        Worker -->|2. Compute Column Stats| Prof["Profiling Activities"]
        Worker -->|3. Trace SQL / Views| Graph[("SurrealDB Lineage")]
        Worker -->|"4. Detect PII (GLiNER)"| NER["GLiNER Large ONNX"]
        Worker -->|5. Vectorize Metadata| Qdrant[("Qdrant Vector DB")]
    end

    Worker -->|Status Updates| Temp
    Temp -->|Poll Status| UI

Step 1: Add Datasource in the Web UI

  1. In the VDA left navigation menu, navigate to Data Sources.
  2. Click + Add Data Source in the top right.
  3. Select your database connector type (e.g., PostgreSQL, MySQL, Oracle, Snowflake, SQL Server).
  4. Fill in connection parameters:
  5. Name: A descriptive identifier (e.g., prod_crm_database).
  6. Host / IP: The hostname or IP address of the database server.
  7. Port: Database port (e.g. 5432 for PostgreSQL, 1521 for Oracle, 3306 for MySQL).
  8. Database Name / Service: Database name or Oracle SID/Service.
  9. Username & Password: Credentials with read access to system catalogs and tables.
  10. Schema Filter: Specific schemas to ingest (e.g., public, analytics), or leave empty for all.
  11. Click Test Connection. A green checkmark confirms database reachability and valid credentials.
  12. Click Save & Ingest.

Step 2: What Happens Under the Hood

When you trigger ingestion, the system launches an asynchronous OrchestrationWorkflow managed by Temporal: 1. Metadata Extraction (MetadataExtractionWorkflow): Connects to the database and extracts table definitions, column names, data types, primary keys, and foreign key relationships. 2. Data Profiling (ProfilingWorkflow): Runs non-blocking analytical sample queries to calculate total row count, distinct values, null percentages, min/max bounds, and statistical histograms. 3. Lineage Extraction (LineageExtractionWorkflow): Parses database views, stored procedures, and ETL dependencies, recording nodes and edges in SurrealDB. 4. Enrichment & PII Detection (EnrichmentWorkflow): - Dispatches column names and samples to the local GLiNER Large model to identify sensitive attributes (IBAN, CREDIT_CARD, EMAIL, NATIONAL_ID, PHONE_NUMBER). - Generates vector embeddings for table names and column descriptions and inserts them into Qdrant (cdm_metadata collection).

Step 3: Monitoring Ingestion Progress

  • In VDA UI: Open Data Sources > Click your datasource > Ingestion Runs tab to see real-time status and logs.
  • In Temporal Web UI: Navigate to http://localhost:8480. You will see active workflow executions (MetadataOrchestrationWorkflow, ProfilingWorkflow) with step-by-step activity timings and retry states.
  • Once complete, open Data Catalog in VDA to browse the newly cataloged tables, columns, statistical profiles, and PII tags.