Skip to content

Virtual Data Assistant (VDA)

Welcome to the official documentation for the Virtual Data Assistant (VDA) — an enterprise-grade AI-powered data management, discovery, lineage, quality, and governance platform engineered for high-security, compliance-driven environments such as banking, financial services, healthcare, and telecommunications.

flowchart TD
    subgraph Users["End Users & AI Agents"]
        U1["Business Analysts & Query Authors"]
        U2["Data Engineers & ETL Architects"]
        U3["Data Stewards & Infosec Officers"]
        U4["Autonomous AI Agents (Cursor, Claude)"]
    end

    subgraph VDA["Virtual Data Assistant Platform"]
        UI["VDA Web Portal (React / TypeScript)"]
        MCP["Model Context Protocol (91 Enterprise Tools)"]
        API["VDA Server (GraphQL / PBAC Engine)"]
        META["VDA Core Metadata Engine (FastAPI / AI Services)"]
        SUR["SurrealDB (Operational Lineage Graph)"]
        QDR["Qdrant (Vector Embeddings Store)"]
        TEM["Temporal (Workflow Orchestration)"]
    end

    subgraph Sources["Enterprise Data Ecosystem"]
        DB1[("Core Banking (Oracle / DB2)")]
        DB2[("Data Warehouses (Snowflake / BigQuery)")]
        DB3[("Transactional DBs (Postgres / SQL Server)")]
        EV1[("Event Streams (Kafka / Marquez)")]
    end

    Users -->|HTTPS / SSO| UI
    Users -->|SSE / JSON-RPC| MCP
    UI --> API
    MCP --> META
    API --> META
    META --> SUR
    META --> QDR
    META --> TEM
    META -.->|Metadata & Profiling| Sources

Key Platform Capabilities

1. Unified Data Catalog & Semantic Discovery

Explore, search, and understand cross-platform data assets across diverse sources including Oracle, PostgreSQL, Microsoft SQL Server, MySQL, Snowflake, and Apache Kafka. Integrated with deep multilingual embeddings (Alibaba-NLP/gte-multilingual-base, 768 dimensions) for instant semantic discovery across technical and business terms.

2. Conversational Analytics & Dual-Tier Text-to-SQL

Ask natural language business questions and receive dialect-accurate, executable SQL queries with comprehensive data explanations, result previews, and one-click visualization export to Apache Superset charts. Supports a secure dual-tier model hierarchy routing between local air-gapped LLMs (e.g. Qwen 2.5 / DeepSeek via Ollama) and enterprise cloud models with strict prompt token budgeting.

3. End-to-End Lineage & Operational Observability

Trace data flows from raw ingestion down to operational dashboards and machine learning pipelines. Ingest OpenLineage event streams in real time from Apache Spark, Airflow, and dbt, storing dependency DAGs in SurrealDB for instant upstream and downstream impact analysis.

4. Source-to-Target Mapping (SSM) Studio

Accelerate data warehouse integration and ETL development with a visual drag-and-drop mapping canvas. Automate source column extraction, datatype alignment, join path resolution, and transformation generation.

5. AI-Assisted Data Quality & Great Expectations (GX)

Automate data profiling, nullity detection, value distributions, and anomaly discovery. Leverage the integrated AI Expectation Generator to formulate, validate, and execute Great Expectations suites with scheduled Temporal orchestration.

6. Zero-Trust Policy-Based Access Control (PBAC)

Enforce granular, attribute-aware access control across datasets, tables, and individual columns. Featuring deny-first precedence, container hierarchy traversal, Keycloak SSO integration, column masking, and an AST-level SQL firewall that prevents unauthorized column leakage even across SELECT *, subqueries, and table aliases.

7. Model Context Protocol (MCP) for Autonomous AI Agents

Equip enterprise AI agents (such as Claude Desktop, Cursor, or internal agents) with a comprehensive suite of 91 production tools covering catalog discovery, SQL execution, lineage tracing, data quality validation, and schema transformations — all strictly bounded by the user's PBAC permissions.

Documentation Structure

Section Target Audience Description
User Guide Business Analysts, Data Engineers, Stewards Step-by-step walkthroughs of data search, SQL authoring, lineage analysis, mapping, profiling, and access simulation.
Architecture Enterprise Architects, Infosec, Compliance System topology, defense-in-depth security, PBAC evaluator, AI pipeline design, and air-gapped readiness.
Deployment & Operations DevOps, SREs, Platform Engineers Hardware sizing, Docker Compose reference, and production Kubernetes / OpenShift manifests.
MCP Tool Reference AI Engineers, Agent Developers Exhaustive specification of the 91 Model Context Protocol tools and API key integration.