LLM & Model Setup: Local vs. Cloud Tier

VDA supports a dual-tier model hierarchy. You can run 100% air-gapped using local models or connect to enterprise cloud LLM providers.

Option A: Local Air-Gapped Tier (Ollama + Hugging Face)

This mode ensures zero prompt data or telemetry ever leaves your server.

1. Pull the Text-to-SQL Model: Execute the model pull command inside the running ollama container:

   docker exec -it vda-ollama ollama pull qwen2.5:7b
   

   To verify the model is loaded:

   docker exec -it vda-ollama ollama list
   # Expected output:
   # NAME            ID              SIZE      MODIFIED
   # qwen2.5:7b      843d741cd14a    4.7 GB    seconds ago
   

2. Verify Local Embeddings & GLiNER: Local embeddings (Alibaba-NLP/gte-multilingual-base) and GLiNER Large weights are pre-baked in the container under /models. Ensure .env contains:

   LLM_BASE_URL=http://ollama:11434/v1
   LLM_MODEL=qwen2.5:7b
   LLM_TIER=local
   ENABLE_GLINER=true
   GLINER_MODEL_PATH=/models/gliner_final
   

Option B: Cloud Tier (OpenAI / Azure OpenAI / Anthropic)

If your organization permits external API communication with cloud LLMs:

1. Update .env with cloud provider details:

   # OpenAI configuration
   LLM_BASE_URL=https://api.openai.com/v1
   LLM_MODEL=gpt-4o
   OPENAI_API_KEY=sk-proj-your-api-key-here
   LLM_TIER=cloud
   

2. Restart the metadata services to apply changes:

   docker compose up -d --no-deps metadata-api metadata-worker