Virtual Data Assistant (VDA): Enterprise PBAC & MCP Reference Guide¶

Complete Architecture, Security Rules, API Key Management, AI Policy Assistant, and 91-Tool Reference¶

Audience: Data Architects, Security Administrators, Enterprise Engineers, and Compliance Officers.
Scope: A production-grade reference detailing the Virtual Data Assistant (VDA) authorization framework, API key management, AI Policy Assistant with self-healing validation, Web UI governance, and the complete 91-tool Model Context Protocol (MCP) server catalog.

Table of Contents¶

Platform Architecture & Deployment Model
Enterprise Kubernetes Architecture
Client Setup: Zero Local Ops
Policy-Based Access Control (PBAC) Framework
The 4-Tuple Policy Model
Canonical Semantic Action Namespace
Scope Hierarchy & Canonical Resource Prefixes
The 12 Inviolable Security Invariants
Column-Level Security, PII Masking & AST Interception
API Key Management & Programmatic Access
Token Format & Cryptographic Security
Role-Bound & Scoped Authentication Flow
Web UI API Key Management
AI Policy Assistant (Intelligent Governance & Self-Healing)
Multi-Tier Architecture & Orchestration
Capability 1: Prompt-to-Policy (Natural Language Rule Authoring)
Capability 2: Plain-English Role Explainer
Capability 3: Security & Conflict Auditor
Strict Grammar Validator & 3-Attempt Self-Healing Retry Loop
Human-in-the-Loop Verification & Pruning Workflow
Web UI Governance, Administration & Simulation
Role Permission Management Features (Search, Filter, Pagination, Undo, Import/Export)
Effective Access Profile Scorecard
Interactive Live Column Picker
Permission Simulator & User/Role Audit
Safe Datasource Lifecycle & Dependency Preservation
Model Context Protocol (MCP) Server Integration
Connecting Claude Desktop & Cursor (Staging vs. Production)
Dual-Layer Security & Dynamic Tool Filtering
Enterprise Audit Logging
Exhaustive Catalog of All 91 MCP Tools
7.1 Data Discovery & Catalog Exploration (5 Tools)
7.2 Conversational Analytics & Text-to-SQL (1 Tool)
7.3 Source System Mapping (SSM) & Transformations (6 Tools)
7.4 Logical & Business Datasets (1 Tool)
7.5 Asset Lineage & Semantic Links (4 Tools)
7.6 Pipeline & OpenLineage Observability (10 Tools)
7.7 Asset Producers & Consumers Analysis (3 Tools)
7.8 Asset Health & Schema Evolution (6 Tools)
7.9 Pipeline Health, Bottlenecks & Optimization (11 Tools)
7.10 Data Quality, Profiling & Great Expectations (10 Tools)
7.11 Metadata Enrichment & Governance (9 Tools)
7.12 Business Glossary & Semantics (6 Tools)
7.13 Datasources, Connectors & Ingestion (13 Tools)
7.14 Compute Clusters, Workbooks & System (6 Tools)

1. Platform Architecture & Deployment Model¶

Enterprise Kubernetes Architecture¶

The Virtual Data Assistant (VDA) is an enterprise data governance, analytics, and AI coordination platform deployed inside Kubernetes (k8s). It eliminates client-side operational overhead while enforcing absolute enterprise data sovereignty:

graph TD
    subgraph Clients["Clients & Interfaces"]
        UI["VDA Web Portal (Browser / React)"]
        Claude["Claude Desktop / Cursor (via mcp-remote)"]
        Pipelines["Headless Scripts & CI/CD Pipelines (API Key)"]
    end

    subgraph K8sCluster["Enterprise Kubernetes Cluster"]
        Gateway["Cloud Ingress & TLS Termination (HTTPS)"]
        AuthLayer["Authentication & Key Management (Keycloak / API Keys)"]
        GraphQL["VDA Control Plane Server"]
        MetadataService["VDA Intelligence & MCP Engine"]
        PBACEngine["PBAC Policy Evaluation Engine"]
        AIAssistant["AI Policy Assistant Service"]
        SQLAST["Query Security & Dialect Engine"]
        PolicyStore[("Policy Store (Role Permissions & API Keys)")]
        CatalogGraph[("Enterprise Metadata Graph Store")]
    end

    subgraph EnterpriseSources["Enterprise Data Lakehouse & Databases"]
        EDW[("Enterprise Data Warehouse (PostgreSQL)")]
        Lakehouse[("Databricks Lakehouse / Spark")]
        Snowflake[("Snowflake Data Cloud")]
        Connectors[("50+ Enterprise Connectors")]
    end

    UI -->|HTTPS / GraphQL| Gateway
    Claude -->|"HTTPS / SSE (Bearer Token / API Key)"| Gateway
    Pipelines -->|"HTTPS / REST (x-api-key)"| Gateway
    Gateway --> AuthLayer
    AuthLayer --> PolicyStore
    Gateway --> GraphQL
    Gateway --> MetadataService
    GraphQL --> PolicyStore
    GraphQL --> MetadataService
    MetadataService --> PBACEngine
    PBACEngine --> PolicyStore
    MetadataService --> AIAssistant
    MetadataService --> SQLAST
    MetadataService --> CatalogGraph
    MetadataService --> EnterpriseSources

Client Setup: Zero Local Ops¶

Zero Client Maintenance: The platform runs entirely in cloud/managed Kubernetes. No local databases, python workers, or docker containers are required on user laptops.
Protocol Standard: Communication uses standard Server-Sent Events (SSE) and HTTPS, allowing secure connections from Claude Desktop, Cursor, or corporate automation pipelines.
Unified Identity: Seamless authentication via enterprise OAuth2 / Keycloak SSO (browser sessions) or cryptographically random, role-bound API keys (MCP and headless automation).

2. Policy-Based Access Control (PBAC) Framework¶

The 4-Tuple Policy Model¶

Every access decision is governed by explicit 4-tuples stored in the enterprise policy store:

$$\text{Rule} = \langle \text{Role}, \text{Action}, \text{Scope}, \text{Effect} \rangle$$

Role: Corporate SSO user role or service identity (e.g. SUPERADMIN, DATA_ANALYST, DATA_ENGINEER, FINANCE_RESTRICTED, AUDITOR).
Action: The requested operation from the canonical action namespace (metadata:read, data:read, data:write, workbook:read, admin, *).
Scope: The targeted resource URI (e.g. datasource:enterprise_dw, dataset:enterprise_dw.analytics.finance.general_ledger, column:hr_system.hcm.payroll.staff.salary).
Effect: Either allow or deny.

Canonical Semantic Action Namespace¶

To eliminate ambiguity between catalog visibility and physical data access, the platform enforces a canonical colon-delimited action namespace with explicit backward-compatibility aliases:

Canonical Action	Legacy Alias	Description	Target Scopes
`metadata:read`	`read`	Read catalog definitions, schemas, descriptions, tags, badges, and lineage.	``, `datasource:`, `dataset:`, `lineage:`, `glossary:*`
`data:read`	`query`	Query rows, preview tables, run text-to-SQL, export charts.	`datasource:`, `dataset:`
`metadata:write`	`write`	Curate documentation, tags, glossary terms, and transformation rules.	`dataset:`, `glossary:`, `ingestion:*`
`data:write`	`write`	Execute ETL pipelines, ingest data, modify underlying storage.	`datasource:`, `dataset:`, `ingestion:*`
`workbook:read`	`read`	Open and view Query Books, Recon Books, and Expectation Books.	`workbook:*`
`workbook:write`	`write`	Create, modify, and save workbooks.	`workbook:*`
`admin`	`admin`	Configure clusters, manage datasources, generate/audit PBAC rules.	`datasource:`, `cluster:`, `policy:*`
`*`	`*`	Full administrative wildcard granting all actions across all scopes.	All scopes

Semantic Implication Rule: `data:read` $\implies$ `metadata:read`¶

Granting data:read (or query) privileges on an asset automatically implies metadata:read privileges on that asset and its parent container. An authorized data analyst can always inspect the schema and column descriptions of tables they are permitted to query.

Scope Hierarchy & Canonical Resource Prefixes¶

Resources are strictly organized into canonical hierarchical containers:

datasource:<datasource_name>
  └── dataset:<datasource_name>.<database>.<schema>.<table_name>
        └── column:<datasource_name>.<database>.<schema>.<table_name>.<column_name>

Canonical Resource Prefixes¶

datasource:<name>: Encompasses an entire warehouse, database, or lakehouse instance (e.g. datasource:pg_prod or datasource:telecom_dw).
dataset:<datasource>.<database>.<schema>.<table>: Encompasses a physical or logical table, view, or stream (e.g. dataset:telecom_dw.crm.public.customer_accounts or dataset:pg_prod.analytics.finance.revenue). 4 segments are canonical in the OpenMetadata catalog.
column:<datasource>.<database>.<schema>.<table>.<column>: Targets a specific column for granular deny or masking policies (e.g. column:telecom_dw.crm.public.customer_accounts.ssn or column:pg_prod.hr_db.public.employees.salary). 5 segments are canonical.
workbook:<id_or_pattern>: Governs interactive workbooks (Recon, Query, Expectation).
cluster:<name>: Controls execution compute engines.
*: Global root scope matching every resource in the enterprise.

[!NOTE] Flexible Glob Matching Across Segments: While 4-segment datasets and 5-segment columns represent the canonical catalog FQN structure, the PBAC glob matching engine allows wildcard * to match across dot boundaries (e.g., dataset:pg_prod.* matches all tables under datasource pg_prod, and column:*.*.*.*.ssn or column:*.ssn matches column ssn across any table).

The 12 Inviolable Security Invariants¶

The PBAC engine enforces twelve formal invariants across human users, API keys, and AI agents:

Deny-by-Default: If no explicit allow rule matches the requested action and scope, access is immediately DENIED.
Explicit Deny Dominance: An explicit deny rule always overrides any number of matching allow rules, regardless of specificity or wildcarding.
Monotonic Container Containment: An allow rule on a parent container (datasource:enterprise_dw) automatically permits access to child assets (dataset:enterprise_dw.analytics.finance.general_ledger), unless an explicit child deny exists.
Hierarchical Deny Precedence: If a dataset is explicitly denied, that denial takes absolute precedence even if the user possesses datasource:* or * allow rules.
Implicit Traversal: A user permitted to access a single child table can see the parent datasource name in navigation trees and MCP tool lists, but cannot list or query unauthorized sibling tables.
Implicit Traversal Boundary: Implicit traversal only grants metadata:read (visibility); it never grants data:read or data:write access to parent containers.
Column-Level Deny Blocks All Data Paths: A deny rule on a column (e.g. salary) blocks the column across every data path: sample previews, natural language queries, transformation previews, and chart exports.
SELECT * AST Interception: When any column on a table is denied, executing SELECT * or SELECT t.* is rejected at the Abstract Syntax Tree (AST) level, preventing accidental data leaks. (Only COUNT(*) is allowed).
Datasource-Wide Column Deny: Denying a column at the datasource level (e.g. column:enterprise_dw.*.*.*.ssn or column:enterprise_dw.*.ssn) denies that column across every table in that datasource.
Superadmin Deny Protection: The SUPERADMIN role cannot be assigned deny rules, preventing administrative lockout.
API Key Policy Inheritance: API keys inherit either their creator's role permissions or a strictly bounded sub-role; an API key can never escalate privileges beyond its assigned role.
Scope Resilience: Malformed, empty, or unparseable scopes safely fail-closed and evaluate to DENIED without crashing services.

Column-Level Security, PII Masking & AST Interception¶

sequenceDiagram
    autonumber
    actor Claude as Claude Desktop / User
    participant Gateway as API Gateway
    participant PBAC as PBAC Engine
    participant AST as In-Engine SQL AST Firewall
    participant DB as Enterprise Database

    Claude->>Gateway: Natural Language Query / SQL Query
    Gateway->>PBAC: Evaluate user role & retrieve denied columns
    PBAC-->>Gateway: Denied columns for user: ["salary", "ssn"]
    Gateway->>AST: Inspect AST against denied columns
    alt Query contains "SELECT *" or "SELECT t.*"
        AST-->>Gateway: AST Violation: Star query on table with denied columns
        Gateway-->>Claude: Access Denied: Query uses SELECT * on table with restricted columns
    else Query explicitly references "salary" or "ssn"
        AST-->>Gateway: AST Violation: Denied column referenced
        Gateway-->>Claude: Access Denied: Query references restricted column 'salary'
    else Query references only permitted columns (e.g. name, department)
        AST-->>Gateway: Validation Passed
        Gateway->>DB: Execute query
        DB-->>Gateway: Return data rows
        Gateway-->>Claude: Formatted result
    end

3. API Key Management & Programmatic Access¶

The VDA platform provides full-featured API key management enabling headless pipelines, CI/CD jobs, Cursor, and Claude Desktop to connect securely without browser sessions.

Token Format & Cryptographic Security¶

Prefix Standard: All API keys follow the standard vda_<64_hex_chars> format (e.g., vda_a1b2…).
Storage & Hashing: API keys are hashed with SHA-256 before storage in PostgreSQL (ApiKey table). The raw key is displayed to the user only once upon generation.
Expiration & Revocation: Keys can be configured with expiration timestamps (never, 30 days, 90 days) and revoked instantly in the Web UI.

Role-Bound & Scoped Authentication Flow¶

When an API key is presented in an HTTP request via:
Header: Authorization: Bearer vda_...
Header: x-api-key: vda_...
Query Parameter: ?token=vda_... (for SSE MCP connections)
The authentication middleware validates the key, loads the associated user ID and role, and queries all RolePermission records for that role.
The resulting UserContext ({ user, roles, permissions }) is passed down to service handlers and the PBAC evaluation engine.

graph LR
    Client["Client / MCP Tool"] -->|?token=vda_...| Ingress["Ingress Gateway"]
    Ingress --> Server["VDA Core Server"]
    Server -->|Hash & Verify| DB[("ApiKey Table")]
    DB -->|Return Associated Role| Server
    Server -->|Load Role Permissions| PolicyDB[("RolePermission Table")]
    PolicyDB -->|Active UserContext| PBAC["PBAC Evaluator"]

Web UI API Key Management¶

Administrators and users manage keys via the API Key Management interface: - Generate API Key: Specify a name, optional description, role binding, and expiry. - Single-Time Reveal Modal: Copy button with warnings to store the key securely. - Active Key Inventory: View last used timestamp, creation date, status (ACTIVE, EXPIRED, REVOKED), and one-click revocation.

4. AI Policy Assistant (Intelligent Governance & Self-Healing)¶

The AI Policy Assistant bridges the gap between natural language compliance requirements and formal 4-tuple PBAC grammar. It combines LLM intelligence with strict deterministic validators.

graph TD
    subgraph Frontend["Enterprise Web Console"]
        Drawer["AI Policy Assistant Console"]
        Tab1["Prompt to Policy"]
        Tab2["Role Explainer"]
        Tab3["Security Auditor"]
    end

    subgraph Backend["Application & Governance Services"]
        GQL["VDA Application Gateway (GraphQL Engine)"]
        PyAI["AI Governance Service"]
        LLM["Enterprise Language Model (Cloud / Local)"]
        Validator["Deterministic Grammar Validator"]
        RetryEngine["Automated Self-Healing Loop"]
    end

    Drawer --> Tab1 & Tab2 & Tab3
    Tab1 --> GQL
    Tab2 --> GQL
    Tab3 --> GQL
    GQL --> PyAI
    PyAI --> LLM
    LLM -->|Candidate Rules| Validator
    Validator -->|Grammar Diagnostics| RetryEngine
    RetryEngine -->|Correction Feedback| LLM
    Validator -->|Validated Rules| GQL
    GQL --> Drawer

Multi-Tier Architecture & Orchestration¶

Frontend: AI Policy Assistant Console embedded within the enterprise Access Management portal.
Application Gateway: High-throughput GraphQL governance engine with strict administrative authorization controls.
AI Governance Service: Dedicated policy synthesis engine coordinating with the language model via structured schema parsing, deterministic grammar validation, and automated multi-step self-healing loops.

Capability 1: Prompt-to-Policy (Natural Language Rule Authoring)¶

Converts human requirements into mathematically sound PBAC rules. - Input: Natural language prompt (e.g. "Allow data analysts to query postgres tables but strictly redact salary and customer_ssn"). - Quick Presets: One-click prompt templates for Data Analysts, Catalog Auditors, ETL Engineers, and HR Teams. - Output: - Reasoning & Policy Strategy: Explanation of how the prompt was translated into rules. - Policy Warnings: Alerts on potential business impact or security edge cases. - Interactive Preview Table: Displays generated rules (Action, Scope, Effect, Description) with individual rule deletion controls. - Commit Button: "Apply Rules to Role" commits reviewed rules directly to the database.

Capability 2: Plain-English Role Explainer¶

Translates complex role permission configurations into an executive summary for compliance officers and non-technical stakeholders: - Executive Summary: High-level statement of what the role can and cannot do. - Catalog Visibility: Datasources and schemas browsable in the data catalog. - Data Query Rights: Databases and tables permitted for row-level previews and SQL querying. - Forbidden / Redacted Columns: Comprehensive inventory of denied sensitive columns and tables. - Admin Privileges: Summary of administrative, cluster, or datasource management rights. - Security Recommendations: Suggestions to adhere to the principle of least privilege.

Capability 3: Security & Conflict Auditor¶

Audits the role's policies against best practices and identifies security risks: - Overall Security Posture: Evaluated badge (SECURE, ATTENTION_NEEDED, HIGH_RISK). - Total Rules Analyzed: Exact rule count inspected. - Vulnerability Findings: - SHADOWED_RULE: Redundant allow rules masked by broader wildcard allows. - OVER_PERMISSIVE: Broad wildcards (*) without necessary data-path guardrails. - DEAD_PERMISSION: Scopes pointing to deleted or non-existent datasources. - PII_EXPOSURE: Access granted to tables containing tagged PII columns without explicit column deny rules. - Actionable Remediation: Specific recommendation for every detected issue.

Strict Grammar Validator & 3-Attempt Self-Healing Retry Loop¶

The PBAC AI engine enforces zero tolerance for invalid rule outputs against the formal PBAC specification:

Supported Actions: metadata:read, metadata:write, metadata:*, data:read, data:write, data:*, workbook:read, workbook:write, workbook:*, admin, *
Supported Effects: allow, deny
Supported Resource Scopes: datasource:<name>, dataset:<name>, column:<name>, *

Self-Healing Logic:¶

When the LLM generates a candidate rule set, the deterministic grammar validator inspects every action, effect, and scope pattern.
If any validation fails (e.g., an unrecognized action name or a column scope lacking dataset qualification):
The engine formats the exact validation discrepancy into structured diagnostic feedback.
It appends the diagnostic context back into the conversation thread.
It re-invokes the generator with explicit repair instructions to correct the grammar errors.
Up to 3 consecutive repair attempts are executed automatically before returning a descriptive failure to the administrator.

Human-in-the-Loop Verification & Pruning Workflow¶

AI never applies rules directly without administrative intervention: 1. Review: The administrator views each generated rule in a verified preview table. 2. Prune: The administrator can delete individual rules with a trash icon. 3. Commit: Clicking Apply Rules to Role commits validated records to the policy store and immediately refreshes the active role permissions table.

5. Web UI Governance, Administration & Simulation¶

Role Permission Management Features¶

The Role Permission Management interface provides enterprise-grade tools for security administrators:

graph TD
    RPM["Role Permission Management (/permission)"]
    RPM --> Scorecard["Effective Access Profile Scorecard"]
    RPM --> Toolbar["Action Toolbar (Search, Filter, Export, Import, AI, Add)"]
    RPM --> TableView["Rule Table with Pagination & Inline Deletion"]
    RPM --> UndoBar["Floating Undo Notification"]
    RPM --> ColPicker["Interactive Live Column Picker"]
    RPM --> SimTab["Permission Simulator Tab"]

Search & Real-Time Filtering: Instant client-side search across Action, Scope, and Effect.
Pagination: 10 rules per page with page jump controls for roles with hundreds of rules.
Undo Rule Deletion: Deleting a rule triggers a floating toast with an Undo button, restoring the rule if clicked within 8 seconds.
Export & Import JSON: Backup or migrate permission sets between environments via standard JSON files.
Guidance Drawer: Collapsible card explaining container inheritance, deny dominance, and implication rules.

Effective Access Profile Scorecard¶

A real-time summary card dynamically recomputed as rules change: - Catalog Visibility: Permitted, Full, or None. - Data Query Rights: Permitted, Restricted (when deny rules exist), or None. - Forbidden Columns: Count and badges of active column denials. - Admin Privileges: Enabled or Disabled.

Interactive Live Column Picker¶

When defining deny rules for sensitive fields: 1. Administrator selects Scope Type: Column. 2. Selects a datasource and schema. 3. The UI fetches live table columns and renders a searchable checkbox list. 4. Selecting fields (e.g. ssn, salary) automatically formats canonical scopes (column:postgres.prod.public.employees.salary).

Permission Simulator & User/Role Audit¶

A sandbox to simulate access decisions before committing changes: - Input: Target Role or specific User ID, requested Action (data:read, metadata:read, admin, etc.), and target Resource URI. - Decision Engine: Evaluates the full policy DAG in real time. - Audit Output: - ALLOWED or DENIED badge. - Matched rule explaining the decision. - List of active denied columns on the target entity.

Safe Datasource Lifecycle & Dependency Preservation¶

When deleting a datasource in the Web UI: - Selective Deletion: Only deletes the connection configuration and physical crawler asset records. - Preserved Assets: Upstream/downstream lineage edges, canonical asset identifiers, and business datasets are preserved. - Dependency Guard: Deletion is blocked if active Workbooks (Query, Recon, Expectation) or Source System Mappings depend on the datasource, preventing data corruption.

6. Model Context Protocol (MCP) Server Integration¶

Connecting Claude Desktop & Cursor (Staging vs. Production)¶

Connecting AI clients to the VDA MCP Server uses the standard mcp-remote bridge in claude_desktop_config.json:

Configuration File Paths¶

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Staging Configuration (HTTP / IP Gateway)¶

{
  "mcpServers": {
    "vda": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "http://vda-staging.company.internal:8200/api/v2/mcp/sse?token=<USER_API_KEY>",
        "--allow-http"
      ]
    }
  }
}

Production Configuration (Kubernetes HTTPS & Cloud Ingress)¶

{
  "mcpServers": {
    "vda": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://vda.enterprise.com/api/v2/mcp/sse?token=<USER_API_KEY>"
      ]
    }
  }
}

Dual-Layer Security & Dynamic Tool Filtering¶

Tool Listing Filter: When Claude connects, the server evaluates the user's role. If a user has no data:read privileges, query tools (preview_data_asset, execute_text_to_sql) are automatically omitted from Claude's loaded tool list.
Runtime Tool Execution Authorization: When any tool is executed, target assets are evaluated against PBAC rules and SQL AST interceptors. Unauthorized requests return clean, human-readable denial explanations.

Enterprise Audit Logging¶

Every MCP tool invocation is logged asynchronously to PostgreSQL: - Timestamp: ISO 8601 millisecond timestamp. - Identity: User ID, email, and active role. - Tool & Parameters: Tool name and input arguments. - Result: SUCCESS, DENIED, or ERROR with error diagnostics.

7. Exhaustive Catalog of All 91 MCP Tools¶

Below is the complete reference of all 91 tools provided by the VDA MCP Server.

7.1 Data Discovery & Catalog Exploration (5 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`list_data_sources`	`metadata:read` on `datasource:*`	Lists all connected databases, warehouses, and lakehouses.	None
`list_data_assets`	`metadata:read` on `dataset:*`	Lists tables and views. Can be filtered to a single datasource or span all authorized sources.	`datasource_name` (optional string)
`search_data_assets`	`metadata:read` on `dataset:*`	Hybrid semantic vector search across all enterprise tables, views, and schemas.	`query` (required string), `limit` (optional int, default: 10)
`get_data_asset_details`	`metadata:read` on `dataset:*`	Retrieves column definitions, data types, descriptions, tags, and badges for a table.	`fqn` (required string)
`preview_data_asset`	`data:read` on `dataset:*`	Fetches sample data rows with automatic column-deny masking applied.	`fqn` (required string), `limit` (optional int, default: 10)

7.2 Conversational Analytics & Text-to-SQL (1 Tool)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`execute_text_to_sql`	`data:read` on `datasource:*`	Converts a natural language question into dialect-specific SQL, validates against column-deny rules via AST inspection, executes it, and returns the result.	`question` (required string), `datasource_name` (optional string)

7.3 Source System Mapping (SSM) & Transformations (6 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_dataset_transformations`	`metadata:read` on `dataset:*`	Retrieves active column mappings, multi-table joins, WHERE filters, aggregations, and dialect for an SSM.	`dataset_fqn` (required string)
`update_dataset_transformation`	`metadata:write` on `dataset:*`	Updates or creates draft column mappings, joins (`INNER`, `LEFT`, `FULL`), filters, aggregations, or target dialect.	`dataset_fqn` (required string), `dialect` (optional string), `mappings` (optional list), `joins` (optional list), `filters` (optional list), `aggregations` (optional object)
`generate_transformation_sql`	`metadata:write` on `dataset:*`	Synthesizes dialect-compliant SQL (Databricks, Spark, Postgres, Snowflake) from mapping rules with AST syntax validation.	`dataset_fqn` (required string), `dialect` (optional string), `instructions` (optional string)
`get_transformation_sql`	`metadata:read` on `dataset:*`	Retrieves the latest synthesized SQL for a dataset's transformation.	`dataset_fqn` (required string)
`preview_transformation_sql`	`data:read` on `dataset:*`	Safe dry-run preview of generated transformation SQL against the database, enforcing column deny rules.	`dataset_fqn` (required string), `limit` (optional int, default: 10)
`publish_dataset_transformation`	`metadata:write` on `dataset:*`	Promotes draft transformation to production, updates target schema, and registers business lineage edges.	`dataset_fqn` (required string)

7.4 Logical & Business Datasets (1 Tool)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`create_logical_dataset`	`metadata:write` on `dataset:*`	Creates a non-physical BUSINESS logical dataset with required database and schema for SSM and lineage design. Shielded against physical crawler overwrites.	`datasource_name` (required string), `database` (required string), `schema_name` (required string), `name` (required string), `description` (optional string), `columns` (optional list)

7.5 Asset Lineage & Semantic Links (4 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_upstream_lineage`	`metadata:read` on `lineage:*`	Traces upstream data sources and pipelines feeding an asset (Root Cause Analysis).	`fqn` (required string), `depth` (optional int, default: 3)
`get_downstream_lineage`	`metadata:read` on `lineage:*`	Traces downstream dependencies, reports, and models (Impact Analysis).	`fqn` (required string), `depth` (optional int, default: 3)
`get_table_links`	`metadata:read` on `dataset:*`	Retrieves pending and approved business glossary links for a table.	`fqn` (required string)
`get_column_links`	`metadata:read` on `dataset:*`	Retrieves pending and approved glossary links for a specific column.	`fqn` (required string), `column_name` (required string)

7.6 Pipeline & OpenLineage Observability (10 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_upstream_jobs`	`metadata:read` on `lineage:*`	Finds all jobs and pipelines that write data consumed by a specific job.	`job_name` (required string)
`get_downstream_jobs`	`metadata:read` on `lineage:*`	Finds all jobs reading data produced by a specific job.	`job_name` (required string)
`get_job_runs`	`metadata:read` on `ingestion:*`	Lists recent execution runs and statuses for a specific pipeline.	`job_name` (required string), `limit` (optional int, default: 10)
`get_failed_runs`	`metadata:read` on `ingestion:*`	Lists failed pipeline runs globally within a specified timeframe.	`hours` (optional int, default: 24), `limit` (optional int, default: 20)
`analyze_run_failure`	`metadata:read` on `ingestion:*`	Extracts error logs and exception stack traces for a specific failed run.	`run_id` (required string)
`find_root_cause_run`	`metadata:read` on `lineage:*`	Traces upstream through the lineage DAG to find the root failed run in a cascade.	`run_id` (required string)
`compare_job_runs`	`metadata:read` on `ingestion:*`	Diffs two runs of a job to identify code version changes or input data shifts.	`run_id_a` (required string), `run_id_b` (required string)
`get_job_details`	`metadata:read` on `ingestion:*`	Retrieves job source code, SQL plan, and assigned owners.	`job_name` (required string)
`get_job_schedule`	`metadata:read` on `ingestion:*`	Retrieves cron schedule and next expected run time for a job.	`job_name` (required string)
`get_job_code_history`	`metadata:read` on `ingestion:*`	Tracks chronological changes to a job's source code or SQL query facets.	`job_name` (required string)

7.7 Asset Producers & Consumers Analysis (3 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_asset_producers`	`metadata:read` on `dataset:*`	Lists all jobs and pipelines that actively write to an asset.	`dataset_fqn` (required string)
`get_asset_consumers`	`metadata:read` on `dataset:*`	Lists all jobs, models, and dashboards that actively read from an asset.	`dataset_fqn` (required string)
`get_pii_consumers`	`metadata:read` on `dataset:*`	Traces downstream from a PII-tagged column to reveal all consuming jobs and dashboards.	`dataset_fqn` (required string), `column_name` (required string)

7.8 Asset Health & Schema Evolution (6 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_asset_schema_history`	`metadata:read` on `dataset:*`	Chronological log of column additions, type modifications, and deletions.	`dataset_fqn` (required string)
`get_asset_quality_history`	`metadata:read` on `dataset:*`	Historical pass/fail trend of data quality assertions for an asset over time.	`dataset_fqn` (required string)
`get_data_quality_anomalies`	`metadata:read` on `dataset:*`	Identifies runs where row count or quality scores wildly deviated from historical baseline.	`dataset_fqn` (optional string), `days` (optional int, default: 7)
`get_stale_assets`	`metadata:read` on `dataset:*`	Identifies datasets that have not had a successful write event in the last X days.	`days` (optional int, default: 30)
`get_unused_assets`	`metadata:read` on `dataset:*`	Identifies datasets that have had zero read queries in the last X days.	`days` (optional int, default: 60)
`get_orphaned_assets`	`metadata:read` on `dataset:*`	Finds active datasets that have no assigned technical or business owner.	None

7.9 Pipeline Health, Bottlenecks & Optimization (11 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_orphaned_jobs`	`metadata:read` on `ingestion:*`	Lists active data pipelines that have no assigned owner.	None
`get_redundant_jobs`	`metadata:read` on `ingestion:*`	Finds pipelines computing outputs that are never read by any downstream consumer.	None
`get_longest_running_jobs`	`metadata:read` on `ingestion:*`	Ranks jobs by total execution time to identify query optimization candidates.	`limit` (optional int, default: 10)
`get_critical_path`	`metadata:read` on `lineage:*`	Identifies the longest path of upstream dependencies for an SLA-critical job.	`job_name` (required string)
`get_data_bottlenecks`	`metadata:read` on `dataset:*`	Identifies high-risk datasets with the highest ratio of consumers to producers.	`limit` (optional int, default: 10)
`get_cross_domain_dependencies`	`metadata:read` on `lineage:*`	Finds pipelines in one business domain consuming datasets from another domain.	None
`get_downstream_impacted_jobs`	`metadata:read` on `lineage:*`	Lists all downstream jobs that will fail or receive stale data due to a failure.	`failed_run_id` (required string)
`analyze_schema_impact`	`metadata:read` on `lineage:*`	Simulates how adding/removing a column will impact downstream views and jobs.	`dataset_fqn` (required string), `column_name` (required string)
`get_frequent_job_failures`	`metadata:read` on `ingestion:*`	Ranks pipelines by failure rate to identify flaky workflows.	`days` (optional int, default: 14)
`get_unauthorized_access`	`metadata:read` on `policy:*`	Flags jobs reading from restricted datasets without matching classification tags.	None
`get_jobs_by_engine_version`	`metadata:read` on `cluster:*`	Identifies pipelines executing on deprecated infrastructure versions (e.g. Spark 2.x).	`engine` (required string)

7.10 Data Quality, Profiling & Great Expectations (10 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`get_asset_profile`	`metadata:read` on `dataset:*`	Statistical profile of an asset (row counts, null ratios, value distributions).	`dataset_fqn` (required string)
`get_data_quality_profile`	`metadata:read` on `dataset:*`	Quality test scorecards and assertion metrics.	`dataset_fqn` (required string)
`generate_data_expectation`	`metadata:read` on `dataset:*`	Generates Great Expectations (GX) rules from natural language statements.	`dataset_fqn` (required string), `prompt` (required string)
`run_data_expectation`	`data:read` on `dataset:*`	Executes a Great Expectations rule on live data.	`expectation_id` (required string)
`get_data_expectation_status`	`metadata:read` on `dataset:*`	Checks the execution status of a submitted data quality job.	`job_id` (required string)
`get_gx_job_results`	`metadata:read` on `dataset:*`	Full assertion results and error records for a quality run.	`job_id` (required string)
`list_data_quality_jobs`	`metadata:read` on `dataset:*`	Lists recent data quality execution jobs for an asset.	`dataset_fqn` (required string)
`register_custom_expectation`	`metadata:write` on `dataset:*`	Registers custom Python business validation logic.	`dataset_fqn` (required string), `code` (required string)
`create_expectation_workbook`	`workbook:write` on `dataset:*`	Creates a multi-rule Data Quality Suite in the VDA Web UI.	`title` (required string), `dataset_fqn` (required string), `rules` (required list)
`run_expectation_workbook`	`workbook:read` on `workbook:*`	Runs an entire Data Quality Suite across single and cross-dataset rules.	`workbook_id` (required string)

7.11 Metadata Enrichment & Governance (9 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`update_asset_description`	`metadata:write` on `dataset:*`	Updates business documentation for a table or view.	`dataset_fqn` (required string), `description` (required string)
`update_column_description`	`metadata:write` on `dataset:*`	Updates documentation for a specific column.	`dataset_fqn` (required string), `column_name` (required string), `description` (required string)
`update_column_pii_type`	`metadata:write` on `dataset:*`	Sets column PII classification (`EMAIL`, `SSN`, `PHONE`, `IBAN`).	`dataset_fqn` (required string), `column_name` (required string), `pii_type` (required string)
`update_asset_tags`	`metadata:write` on `dataset:*`	Attaches governance tags (`FINANCE`, `SENSITIVE`, `GOLD`).	`dataset_fqn` (required string), `tags` (required list)
`update_column_tags`	`metadata:write` on `dataset:*`	Attaches column-level governance tags.	`dataset_fqn` (required string), `column_name` (required string), `tags` (required list)
`update_asset_badges`	`metadata:write` on `dataset:*`	Adds visual badge markers to an asset in the Web UI.	`dataset_fqn` (required string), `badges` (required list)
`update_column_badges`	`metadata:write` on `dataset:*`	Adds badge markers to an individual column.	`dataset_fqn` (required string), `column_name` (required string), `badges` (required list)
`update_asset_owners`	`metadata:write` on `dataset:*`	Assigns business or technical owners to an asset.	`dataset_fqn` (required string), `owners` (required list)
`update_custom_properties`	`metadata:write` on `dataset:*`	Sets key-value operational metadata on an asset.	`dataset_fqn` (required string), `properties` (required object)

7.12 Business Glossary & Semantics (6 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`search_glossary`	`metadata:read` on `glossary:*`	Searches the business glossary for terms, definitions, and KPI formulas.	`query` (required string)
`get_glossary_term`	`metadata:read` on `glossary:*`	Retrieves definition, synonyms, and linked data assets for a term.	`term_name` (required string)
`create_glossary_term`	`metadata:write` on `glossary:*`	Creates a new standardized business glossary term.	`name` (required string), `description` (required string), `synonyms` (optional list)
`link_term_to_asset`	`metadata:write` on `glossary:*`	Links a business glossary term to a technical table or column.	`term_id` (required string), `dataset_fqn` (required string), `column_name` (optional string)
`approve_glossary_link`	`metadata:write` on `glossary:*`	Approves an AI-suggested semantic term linkage.	`link_id` (required string)
`reject_glossary_link`	`metadata:write` on `glossary:*`	Rejects an AI-suggested semantic term linkage.	`link_id` (required string)

7.13 Datasources, Connectors & Ingestion (13 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`register_datasource`	`admin` on `datasource:*`	Registers a new database connection (Postgres, Databricks, Snowflake, Oracle, etc.).	`name` (required string), `connector_type` (required string), `config` (required object)
`test_datasource_connection`	`admin` on `datasource:*`	Verifies reachability and credentials for a datasource.	`datasource_name` (required string)
`refresh_datasource`	`metadata:write` on `datasource:*`	Triggers metadata resync for an entire datasource.	`datasource_name` (required string)
`refresh_single_entity`	`metadata:write` on `dataset:*`	Triggers metadata resync for a single specific table.	`dataset_fqn` (required string)
`list_ingestion_specs`	`metadata:read` on `ingestion:*`	Lists configured ingestion pipelines and schedules.	None
`create_ingestion_spec`	`metadata:write` on `ingestion:*`	Creates a new automated ingestion specification.	`name` (required string), `datasource_name` (required string), `schedule` (required string)
`create_ingestion_pipeline`	`metadata:write` on `ingestion:*`	Creates an Ingestion Book in the VDA Web UI.	`title` (required string), `datasource_name` (required string), `config` (required object)
`list_jobs_by_spec`	`metadata:read` on `ingestion:*`	Lists historical execution jobs for an ingestion spec.	`spec_id` (required string)
`get_ingestion_job_logs`	`metadata:read` on `ingestion:*`	Retrieves execution logs for an ingestion run.	`job_id` (required string)
`trigger_ingestion_job`	`metadata:write` on `ingestion:*`	Starts an ingestion crawl immediately.	`spec_id` (required string)
`pause_ingestion_job`	`metadata:write` on `ingestion:*`	Pauses a running ingestion job.	`job_id` (required string)
`delete_ingestion_job`	`admin` on `ingestion:*`	Deletes an ingestion job record.	`job_id` (required string)
`backfill_ingestion_job`	`metadata:write` on `ingestion:*`	Triggers a historical metadata backfill run.	`spec_id` (required string), `start_time` (required string)

7.14 Compute Clusters, Workbooks & System (6 Tools)¶

Tool Name	Action / Scope	Purpose & Description	Parameters
`list_execution_clusters`	`metadata:read` on `cluster:*`	Lists registered compute clusters (Spark, Databricks, Trino).	None
`configure_execution_cluster`	`admin` on `cluster:*`	Registers or updates an execution cluster configuration.	`name` (required string), `cluster_type` (required string), `endpoint` (required string)
`create_query_workbook`	`workbook:write` on `workbook:*`	Creates an interactive SQL Query Book in the VDA Web UI.	`title` (required string), `datasource_name` (required string), `query` (required string)
`create_recon_workbook`	`workbook:write` on `workbook:*`	Creates a Data Reconciliation Book comparing row counts or checksums.	`title` (required string), `source_table` (required string), `target_table` (required string)
`search_documents`	`metadata:read` on `document:*`	Semantic search over enterprise PDFs, specifications, and architecture docs.	`query` (required string)
`export_chart`	`data:read` on `datasource:*`	Synthesizes an interactive ECharts visualization config, validating column access.	`query` (required string), `datasource_name` (optional string)
`get_user_permissions`	Unrestricted	Retrieves the active user's roles, scopes, and allowed actions.	None
`ping`	Unrestricted	Simple connectivity test tool for MCP harness validation.	None