Skip to content

Cluster Backup, Restore & Upgrade Procedures

Backup Procedures

  1. Relational Database Backup (PostgreSQL): 

    kubectl exec -it statefulset/postgresql -n vda -- \
  pg_dump -U vda -d vda -F c -b -f /tmp/vda_k8s.dump
kubectl cp vda/postgresql-0:/tmp/vda_k8s.dump ./vda_k8s_$(date +%F).dump

  2. SurrealDB Graph Lineage Export: 

    kubectl exec -it statefulset/surrealdb -n vda -- \
  surreal export --conn http://localhost:8000 --user root --pass "$SURREAL_PASS" --ns vda --db vda /data/backup.surql
kubectl cp vda/surrealdb-0:/data/backup.surql ./surreal_lineage_$(date +%F).surql

  3. Kubernetes Velero Cluster Backup: If Velero is installed, snapshot all PVs and manifests across the namespace: 

    velero backup create vda-backup-$(date +%F) --include-namespaces vda --snapshot-volumes

Restore Procedure

# 1. Restore PostgreSQL
kubectl cp ./vda_k8s_backup.dump vda/postgresql-0:/tmp/vda_k8s.dump
kubectl exec -it statefulset/postgresql -n vda -- \
  pg_restore -U vda -d vda --clean --if-exists /tmp/vda_k8s.dump

# 2. Restore SurrealDB Graph Lineage
kubectl cp ./surreal_lineage_backup.surql vda/surrealdb-0:/data/backup.surql
kubectl exec -it statefulset/surrealdb -n vda -- \
  surreal import --conn http://localhost:8000 --user root --pass "$SURREAL_PASS" --ns vda --db vda /data/backup.surql

Zero-Downtime Rolling Upgrade Workflow

  1. Pre-Upgrade Backup: Execute the database backups listed above.
  2. Run Database Migrations Job: Execute schema migrations via a one-off Kubernetes Job before rolling out new application images: 
    kubectl create job --from=cronjob/vda-migrate vda-migrate-$(date +%s) -n vda
# Or run directly in the server container:
kubectl exec -it deployment/vda-server -n vda -- npx prisma migrate deploy
  3. Perform Rolling Deployment Update: Update container image tags in k8s/apps/: 
    kubectl set image deployment/metadata-api metadata-api=registry.corp.internal/vda/vda-metadata:v2.2.0 -n vda
kubectl set image deployment/vda-server server=registry.corp.internal/vda/vda-server:v2.2.0 -n vda
kubectl set image deployment/vda-ui client=registry.corp.internal/vda/vda-client:v2.2.0 -n vda
  4. Monitor Rolling Update: 
    kubectl rollout status deployment/vda-server -n vda
kubectl rollout status deployment/metadata-api -n vda