Introduction
For On-premise or Preprovisioned cluster, Velero integrated with Minio is the default backup and restore solution that is deployed with Kommander 2.x. Minio by default uses the storage class localvolumeprovisioner, which provisions persistentvolumes on the node's preallocated disks.
You can read more about 2.x persistentVolumes requirements here and DKP Backup and restore here.
Problem
If for some reason, the persistentVolume being claimed by Minio has previous data, it will fail to initialize and therefore Velero will fail the deployment.
With Velero's pod logs
time="2022-02-14T15:38:56Z" level=info msg="No backup locations were ready to be verified" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:120"
time="2022-02-14T15:38:56Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/1/0)" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:154"
Data-Minio-0 pod logs
Waiting for all other servers to be online to format the disks.
Unable to read 'format.json' from http://minio-0.velero-minio-svc.kommander.svc.cluster.local:9000/data: some unexpected files '[{.minio.sys 2022-02-12 07:03:28.589157905 +0000 UTC} {data 2022-02-11 21:47:36.981829914 +0000 UTC}]' found on /data: corrupted backend format, specified disk mount has unexpected previous content
Waiting for all other servers to be online to format the disks.
Solution
WARNING: The following recommended solution only applies for scenarios where the previous data on the disk is not important and can be deleted.
Since Minio will refuse to initialize the disk, that has previous data as a precaution. The volume needs to be manually wiped clean.
First, retrieve the related velero+minio PVC's
kubectl -n kommander get pvc | grep data-minio
kubectl describe each PVC or PV to determine which node and disk mount contain the data.
Then proceed with deleting the PVC's
Ex.
kubectl -n kommander delete pvc data-minio-0
You will notice that deleting the PVC's will get stuck in terminating status, so we'll need to remove the finalizers for each PVC
Ex.
kubectl -n kommander patch pvc data-minio-0 -p '{"metadata":{"finalizers":null}}'
After the steps above, the PVC's should be deleted. You can now proceed with wiping clean the disk.
Lastly, proceed with deleting the minio pods: minio-0, 1, 2, and 3.
This will cause the statefulset to reconcile and re-spawn the pods, which will re-create the PVC and claim the volumes again.