AI Engine & Data Services¶

VDA couples advanced machine learning models with purpose-built data engines to deliver sub-second search, dialect-accurate SQL synthesis, automated PII classification, and deep graph lineage analysis.

1. Multilingual Embedding & Vector Search¶

VDA employs state-of-the-art dense semantic representations to index all catalog tables, columns, and business glossary definitions.

flowchart LR
    CatalogData["Catalog Metadata<br/>(Table Names, Columns, Glossaries)"] --> Embedder["Multilingual Embedding Model<br/>(Dense Vector Representation)"]
    Embedder --> Vector["Dense Semantic Vectors"]
    Vector --> Qdrant[("Vector Search Engine<br/>• High-Speed Similarity Index<br/>• Sub-10ms ANN Query")]

    UserQuery["User Natural Language Prompt"] --> Embedder
    Embedder --> SearchVector["Query Vector"]
    SearchVector --> Qdrant
    Qdrant --> Matches["Top-K Candidate Schemas"]

Vector Specifications¶

Embedding Architecture: Multilingual dense semantic embeddings capturing domain-specific data structures
Vector Dimension: 768 dimensions
Context Length: Up to 8,192 input tokens (accommodating wide tables with 200+ column definitions)
Vector Index: Hierarchical Navigable Small World (HNSW) indexing for sub-10ms approximate nearest neighbor (ANN) retrieval
Multilingual Support: Supports queries and metadata in English, Arabic, French, German, Spanish, and standard transliterations.

2. Dual-Tier Task Router & Prompt Budgeting¶

To guarantee enterprise compliance and prevent runaway token consumption, analytical prompts are handled by a dynamic Task Router:

flowchart TD
    UserPrompt["Natural Language Business Question"] --> TokenBudget["Dynamic Prompt Token Budgeter<br/>T_budget = T_context - T_response - T_system"]
    TokenBudget --> Pruner["Semantic Schema Pruner<br/>(Select Top-K Relevant Columns via Vector Search)"]
    Pruner --> Router{"Complexity Evaluator"}

    Router -->|Single Table / Standard Aggregation / Air-Gapped| TierLocal["Local Tier: On-Premise LLM<br/>(Air-Gapped Local Inference Engine)"]
    Router -->|Cross-Schema Join / Complex Reasoning| TierCloud["Frontier Tier: Enterprise Model<br/>(Optional / High-Complexity Analytics)"]

    TierLocal --> SQL["Dialect-Tuned SQL AST"]
    TierCloud --> SQL

Zero-Egress Data Protection Policy¶

Under no circumstances are database rows or customer records transmitted in prompt contexts. The prompt context sent to the LLM contains exclusively: 1. Pruned table names and relevant column definitions. 2. High-level physical data types (VARCHAR, NUMERIC, DATE). 3. Explicit foreign key relationships discovered from catalog lineage. 4. Dialect guidelines and syntax constraints.

3. High-Performance PII Detection (GLiNER Architecture)¶

VDA detects sensitive attributes and personal data across enterprise columns using a fine-tuned GLiNER (Generalist and Lightweight Named Entity Recognition) classification engine.

Metric	Specification
Model Architecture	Fine-tuned GLiNER deep neural model optimized for tabular schemas and metadata
Execution Engine	High-throughput vectorized execution engine optimized for enterprise CPU/GPU architectures
Parallel Processing	Multi-threaded worker pools with parallel column batching
Per-Column SLA	Sub-second entity classification per column on standard enterprise multi-core compute
Target Entities	`IBAN`, `ACCOUNT_NUMBER`, `NATIONAL_ID`, `SWIFT_CODE`, `PASSPORT`, `CREDIT_CARD`, `PHONE_NUMBER`, `EMAIL`, `NAME`, `ADDRESS`

4. Operational Lineage Graph Engine¶

Pipeline dependencies, ETL jobs, and column-level derivations are maintained in a high-performance operational graph engine:

Entity Model: Comprehensive Directed Acyclic Graph (DAG) modeling datasets, individual columns, ETL pipelines, and runtime execution events.
Dependency Relationships: Tracks fine-grained dependencies, column-level derivations, and pipeline production/consumption linkages.
Traversal Performance: Optimized recursive depth traversals trace end-to-end provenance (up to 10+ hops upstream or downstream) in sub-15ms for interactive visualization and instantaneous impact assessment.