Skip to content

LLM & Model Setup in Kubernetes

VDA supports both local Kubernetes-hosted models and enterprise cloud models.

Option A: Local LLM Engine (Ollama / vLLM Deployment)

For air-gapped or high-privacy compliance environments:

  1. Deploy Ollama or vLLM with persistent model storage: 
    apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: vda
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          resources:
            limits:
              cpu: "8000m"
              memory: "16Gi"
              # nvidia.com/gpu: "1"  # Optional GPU accelerator
          volumeMounts:
            - name: model-store
              mountPath: /root/.ollama
  2. Pull the model into the container: 
    kubectl exec -it deployment/ollama -n vda -- ollama pull qwen2.5:7b
  3. Update k8s/config/configmap.yaml to set LLM_BASE_URL: "http://ollama:11434/v1".

Option B: Cloud Tier (OpenAI / Azure OpenAI)

Set your cloud API key in k8s/config/secret.yaml: 

stringData:
  OPENAI_API_KEY: "sk-proj-your-key"
Configure LLM_BASE_URL: "https://api.openai.com/v1" and LLM_MODEL: "gpt-4o" in configmap.yaml, then rollout restart: 
kubectl rollout restart deployment/metadata-api -n vda