Velero Backups Using Azure Blobs – D2iQ

Since the default values for the config map for the Velero included in DKP is set to load only the AWS plugin and provision for AWS related inputs, there is a workaround for Velero to use Azure as a separate backup location.

Please note while this workaround has been tested, it is unsupported until an official release is made to accommodate non AWS S3 Velero backup location.

Majority of the steps below were lifted from the "Create an additional Backup Storage Location (BSL)" https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/README.md#create-backup-storage-location

I. Pre-requisites

DKP 2.x Konvoy and Kommander deployed
Velero pods running
Velero CLI

II. Create Azure storage account and blob container

From: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/README.md#Create-Azure-storage-account-and-blob-container

Declared values used:

AZURE_BACKUP_RESOURCE_GROUP = Velero_Backups
Location: WestUS
AZURE_STORAGE_ACCOUNT_ID = velero(uuid)
BLOB_CONTAINER = velero

AZURE_BACKUP_RESOURCE_GROUP=Velero_Backups

az group create -n $AZURE_BACKUP_RESOURCE_GROUP --location WestUS

AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

az storage account create \
--name $AZURE_STORAGE_ACCOUNT_ID \
--resource-group $AZURE_BACKUP_RESOURCE_GROUP \
--sku Standard_GRS \
--encryption-services blob \
--https-only true \
--kind BlobStorage \
--access-tier Hot

BLOB_CONTAINER=velero

az storage container create -n $BLOB_CONTAINER --public-access off --account-name $AZURE_STORAGE_ACCOUNT_ID

III. Set resource group containing your VMs and disks

Declared values used:

AZURE_RESOURCE_GROUP = Velero_Backups

AZURE_RESOURCE_GROUP=Velero_Backups

IV. Create role

Note: The AZURE_SUBSCRIPTION_ID value is generated from the default subscription. If you are using another subscription, please take into consideration modifying the query.

Declared values used:

AZURE_ROLE = Velero

AZURE_SUBSCRIPTION_ID=`az account list --query '[?isDefault].id' -o tsv`

AZURE_ROLE=Velero

az role definition create --role-definition '{
"Name": "'$AZURE_ROLE'",
"Description": "Velero related permissions to perform backups, restores and deletions",
"Actions": [
"Microsoft.Compute/disks/read",
"Microsoft.Compute/disks/write",
"Microsoft.Compute/disks/endGetAccess/action",
"Microsoft.Compute/disks/beginGetAccess/action",
"Microsoft.Compute/snapshots/read",
"Microsoft.Compute/snapshots/write",
"Microsoft.Compute/snapshots/delete",
"Microsoft.Storage/storageAccounts/listkeys/action",
"Microsoft.Storage/storageAccounts/regeneratekey/action"
],
"AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"]
}'

V. Create service principal

From: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/README.md#option-1-create-service-principal

Note: The AZURE_TENANT_ID value is generated from the default subscription. If you are using another subscription, please take into consideration modifying the query.

Declared values used:

Secret name: velero
Credentials file: credentials-velero

AZURE_TENANT_ID=`az account list --query '[?isDefault].tenantId' -o tsv`

AZURE_CLIENT_SECRET=`az ad sp create-for-rbac --name "velero" --role $AZURE_ROLE --query 'password' -o tsv \
--scopes /subscriptions/$AZURE_SUBSCRIPTION_ID`

AZURE_CLIENT_ID=`az ad sp list --display-name "velero" --query '[0].appId' -o tsv`

Perform a sanity check

echo AZURE_SUBSCRIPTION_ID = ${AZURE_SUBSCRIPTION_ID}
echo AZURE_TENANT_ID = ${AZURE_TENANT_ID}
echo AZURE_CLIENT_ID = ${AZURE_CLIENT_ID}
echo AZURE_CLIENT_SECRET = ${AZURE_CLIENT_SECRET}
echo AZURE_RESOURCE_GROUP = ${AZURE_RESOURCE_GROUP}

cat << EOF > ./credentials-velero
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

cat ./credentials-velero

VI. Configure the blob container and credentials

Declared values used:

secret name: bsl-credentials
credentials file:credentials-velero
backup location name: azure
namespace: kommander
provider: azure

Used the cluster conf (via --kubeconfig), changed secret namespace from velero to kommander. The secret is created as --from-env-file="credentials-velero" to load the env variables to the pod.

kubectl create secret generic -n kommander bsl-credentials --from-env-file="credentials-velero" --kubeconfig=${CLUSTER_NAME}.conf

Perform a sanity check

echo BLOB_CONTAINER = $BLOB_CONTAINER
echo AZURE_BACKUP_RESOURCE_GROUP = $AZURE_BACKUP_RESOURCE_GROUP
echo AZURE_STORAGE_ACCOUNT_ID = $AZURE_STORAGE_ACCOUNT_ID

BSL name set to azure, used the cluster conf (via --kubeconfig) and set the namespace to kommander

velero backup-location create -n kommander azure \
--provider azure \
--bucket $BLOB_CONTAINER \
--config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID \
--credential=bsl-credentials=azure --kubeconfig=${CLUSTER_NAME}.conf

velero backup-location get -n kommander --kubeconfig=${CLUSTER_NAME}.conf

VII. Configure DKP Velero (DKP related configuration)

From: https://docs.d2iq.com/dkp/kommander/2.2/backup-and-restore/#velero

Pipe out the Kommander current configuration

./dkp install kommander -o yaml --init --kubeconfig=${CLUSTER_NAME}.conf > kommander.yaml

Add the plugins and the secret generated above

nano kommander.yaml

  velero:
    values: |
      minioBackend: false
      initContainers:
        - name: initialize-velero
          image: mesosphere/kubeaddons-addon-initializer:v0.5.5
          args: ["velero"]
          env:
            - name: "MINIO_INGRESS_NAMESPACE"
              value: kommander
            - name: "MINIO_INGRESS_SERVICE_NAME"
              value: kommander-traefik
            - name: "VELERO_NAMESPACE"
              value: kommander
            - name: "VELERO_MINIO_FALLBACK_SECRET_NAME"
              value: "velero-d2iq-credentials"
        - name: velero-plugin-for-aws
          image: velero/velero-plugin-for-aws:v1.1.0
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /target
              name: plugins
        - name: velero-plugin-for-microsoft-azure
          image: velero/velero-plugin-for-microsoft-azure:v1.1.2
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /target
              name: plugins
      credentials:
          extraSecretRef: bsl-credentials

Load the new Kommander config

./dkp install kommander --installer-config kommander.yaml --kubeconfig=${CLUSTER_NAME}.conf

Check the helm release for velero and make sure it had reconciled then check the plugin on the newly initialized pod.

kubectl get hr -n kommander --kubeconfig=${CLUSTER_NAME}.conf

kubectl get pods -A --kubeconfig=${CLUSTER_NAME}.conf |grep velero

kubectl -n kommander exec -it velero-<hash> --kubeconfig=${CLUSTER_NAME}.conf bash

Check if the azure plugin is existing and that the env variables from credentials-velero was loaded to the pod.

cd plugins
ls -l
env |grep AZURE

VIII. Create a backup

Declared values used:

backup name (test): azure-velero-testbackup
backup storage location: azure

velero backup create azure-velero-testbackup -n kommander --kubeconfig=${CLUSTER_NAME}.conf --storage-location azure

IX. Cleanup

1.) Delete Velero backups

2.) Delete backupstoragelocation (azure)

velero backup-location list -n kommander --kubeconfig=${CLUSTER_NAME}.conf
kubectl delete backupstoragelocation -n kommander azure --kubeconfig=${CLUSTER_NAME}.conf

3.) Remove Azure plugin from the kommander.yaml and reload the Velero app

./dkp install kommander --installer-config kommander.yaml --kubeconfig=${CLUSTER_NAME}.conf

4.) Delete backup location

velero backup-location delete -n kommander azure --kubeconfig=${CLUSTER_NAME}.conf

5.) Delete secret

kubectl delete secret -n kommander bsl-credentials --kubeconfig=${CLUSTER_NAME}.conf

6.) Delete service principal

az ad sp list --display-name velero
az ad sp delete --id $AZURE_CLIENT_ID

7.) Delete role

az role definition list --name $AZURE_ROLE
az role definition delete --name $AZURE_ROLE

8.) Delete storage container

az storage container list --account-name $AZURE_STORAGE_ACCOUNT_ID
az storage container delete --name velero --account-name $AZURE_STORAGE_ACCOUNT_ID

9.) Delete storage account

az storage account delete --name $AZURE_STORAGE_ACCOUNT_ID

10.) Delete backup resource group

az group list
az group delete --resource-group $AZURE_BACKUP_RESOURCE_GROUP