System Topology & Network Flows¶

VDA utilizes a modular, multi-tier enterprise architecture deployed across segmented network tiers. This design isolates client entry points from data plane operations, asynchronous processing engines, and persistent datastores.

1. Enterprise System Topology¶

flowchart TB
    subgraph ClientTier["Client & External Tier"]
        Browser["Enterprise User Browser"]
        AIAgent["Enterprise AI Agents (Claude Desktop / Cursor)"]
        Automation["CI/CD & Headless Pipelines"]
    end

    subgraph IngressTier["Perimeter Ingress & Identity Federation"]
        Nginx["TLS 1.3 Ingress Gateway<br/>(Edge Security / Rate Limiting / Path Routing)"]
        Keycloak["Enterprise Identity Provider (IAM)<br/>(OIDC / SAML 2.0 / MFA Federation)"]
    end

    subgraph AppTier["Application & Control Plane"]
        WebUI["VDA Web Portal<br/>• Catalog, Studio & Administration"]
        Server["VDA Control Plane Service<br/>• User Identity & Context Resolution<br/>• PBAC Policy Evaluation Engine<br/>• Session & API Key Lifecycle"]
        GitApp["Template & Mapping Service<br/>• Source-to-Target Versioning<br/>• Mapping Catalog & ETL Templates"]
    end

    subgraph CoreTier["Metadata & Intelligence Core"]
        MetaAPI["VDA Core Intelligence Engine<br/>• 91 MCP Agent Tools & SSE Endpoint<br/>• Dense Semantic Vector Search<br/>• Dialect-Tuned Text-to-SQL Synthesis<br/>• SQL Query Security Firewall"]
        Workers["Distributed Execution Workers<br/>• Statistical Data Profiling<br/>• Automated NER & PII Classification<br/>• Lineage Telemetry Ingestion<br/>• Data Quality Rule Validation"]
        LocalLLM["Air-Gapped Inference Engine<br/>• On-Premise Language Models<br/>• Zero Data Egress"]
    end

    subgraph PersistenceTier["Enterprise Persistence & State Plane"]
        PG[("Relational State Store<br/>User Profiles & PBAC Policies")]
        Surreal[("Operational Graph Store<br/>Multi-Hop Lineage & Asset Dependencies")]
        Qdrant[("Vector Embeddings Index<br/>Dense Semantic Search Store")]
        Temporal[("Durable Orchestration Engine<br/>Fault-Tolerant Workflow State")]
        Kafka[("Event Telemetry Stream<br/>OpenLineage Pipeline Events")]
        Superset[("Embedded Visualization Engine<br/>Production Chart & Dashboard Publishing")]
    end

    subgraph TargetTier["Enterprise Target Ecosystem"]
        DataSources[("Operational Data Sources<br/>• Oracle Database<br/>• PostgreSQL / MySQL<br/>• Microsoft SQL Server<br/>• Snowflake / Databricks")]
    end

    Browser -->|HTTPS :443| Nginx
    AIAgent -->|HTTPS SSE :443| Nginx
    Automation -->|HTTPS REST :443| Nginx
    Nginx --> Keycloak
    Nginx --> WebUI
    Nginx --> Server
    Nginx --> MetaAPI

    Server -->|Identity Verification| Keycloak
    Server -->|Metadata API| MetaAPI
    Server -->|Relational Persistence| PG
    Server -->|Dispatch Workflows| Temporal

    MetaAPI -->|Vector Similarity Queries| Qdrant
    MetaAPI -->|Lineage Graph Queries| Surreal
    MetaAPI -->|On-Premise Inference| LocalLLM
    MetaAPI -->|Schema Validation & Sampling| DataSources

    Workers -->|Poll Task Queues| Temporal
    Workers -->|Persist Lineage DAGs| Surreal
    Workers -->|Consume Run Events| Kafka
    Workers -->|Harvest Technical Metadata| DataSources

    MetaAPI -->|Export Charts| Superset

2. Component Responsibility Matrix¶

Component Tier	Architectural Role	Primary Function	Scale Pattern
Ingress Gateway	Reverse Proxy & Security Edge	TLS 1.3 termination, security headers, rate limiting, path routing	Active-Active Replica
Identity Federation	Enterprise IAM Integration	Single sign-on federation, JWT minting, MFA enforcement	Clustered HA
Web Portal	User Interface	Enterprise web application for catalog, analytics, mapping, and PBAC administration	Stateless Web Tier
Control Plane	Application Controller	Session authentication, PBAC policy evaluation, API token hashing	Horizontal Pod Autoscaler
Core Intelligence Engine	AI & Metadata Services	MCP tool provider (91 tools), vector search, Text-to-SQL, query security validation	Horizontal Pod Autoscaler
Distributed Workers	Task Execution	Asynchronous data profiling, automated PII classification, lineage ingestion, quality suites	Dynamic Worker Pool
Operational Graph Store	Graph Persistence	Multi-hop lineage DAGs, dependency tracing, asset-to-pipeline links	Clustered / Distributed
Vector Search Index	Dense Embeddings Store	High-dimensional vector indexing for multilingual discovery	Clustered Indexing
Durable Orchestration	Workflow State Machine	Distributed, durable state machine for reliable background job execution	Production Cluster
Relational State Store	System Database	ACID store for user profiles, API tokens, PBAC policies, SSM workbooks	Primary-Replica HA
Event Telemetry Stream	Streaming Platform	Real-time event streaming for OpenLineage operational telemetry	Multi-Broker Cluster

3. Communication Patterns & Network Segmentation¶

Synchronous Communication (Low-Latency Path)¶

Web UI to Control Plane: Secure API communication over HTTPS (TLS 1.3) for catalog discovery, mapping design, and policy administration.
AI Agent to Intelligence Engine: Server-Sent Events (SSE) and JSON-RPC 2.0 over HTTPS for MCP tool discovery and tool execution.
Intelligence Engine to Persistence Tier: Low-latency internal queries for similarity search and recursive dependency extraction.
Datasource Validation: Authenticated connection pooling to connected databases for schema harvesting, data sampling, and governed query execution.

Asynchronous Communication (Resilient Background Path)¶

Control Plane to Orchestration Engine: Asynchronous task dispatch for long-running workflows (data profiling, metadata enrichment, quality validation).
Pipelines to Telemetry Stream: OpenLineage JSON telemetry emitted by Spark listeners, Airflow operators, and dbt artifacts into operational topics.
Telemetry Ingestion to Graph Store: The background lineage consumer parses run facets and atomically persists pipeline dependencies into operational graph structures.