When deploying Kommander in a cluster deployed with the DKP vSphere provider, some users have reported that chartmuseum gets stuck in pending state.
In some cases, when describing the pod, it becomes evident that the chartmuseum persistent volume claim (PVC) is in pending status:
kubectl -n kommander get pv,pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/chartmuseum Pending vsphere-raw-block-sc 22m
This issue is usually encountered in environments where vCenter credentials are rotated, so the vSphere CSI controller credentials becomes outdated so the storage provisioner (csi.vsphere.vmware.com) is not able to authenticate with the vCenter API to create/delete and attach/detach disks to and out of the nodes in the cluster.
Solution
To confirm whether this is the cause of chartmuseum being stuck, please describe the PVC with name chartmuseum and look in the events whether the controller is having issue login in to the vCenter, example shown below:
kubectl -n kommander describe pvc chartmuseum
Name: chartmuseum
Namespace: kommander
StorageClass: vsphere-raw-block-sc
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volume.kubernetes.io/selected-node: sortega-dkp230-rhel79-airgap-fips-md-0-6b995b565-dwpfl
volume.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: chartmuseum-5fdf74f69d-zn96p
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 23m (x2 over 23m) persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 3m18s (x82 over 23m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com"or manually created by system administrator
Normal Provisioning 3m14s (x14 over 23m) csi.vsphere.vmware.com_vsphere-csi-controller-77c86f86d4-9wn9r_e18473a9-93a3-43dd-ae73-476794819d1d External provisioner is provisioning volume for claim "kommander/chartmuseum"
Warning ProvisioningFailed 3m10s (x14 over 23m) csi.vsphere.vmware.com_vsphere-csi-controller-77c86f86d4-9wn9r_e18473a9-93a3-43dd-ae73-476794819d1d failed to provision volume with StorageClass "vsphere-raw-block-sc": rpc error: code = Internal desc = failed toget shared datastores in kubernetes cluster. Error: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
To remediate this, the user should update the vCenter credentials in the secret named vsphere-config-secret in the vmware-system-csi namespace:
kubectl get secret vsphere-config-secret -n vmware-system-csi -ojsonpath='{.data.\csi-vsphere\.conf}'|base64 -d
[Global]
cluster-id = "default/dkp230-rhel79-airgap-fips"
thumbprint = ""
[VirtualCenter "<VCENTER-ADDRESS>"]
user = "<USER>"
password = "<PASSWORD>"
datacenters = "dc1"
[Network]
public-network = "Airgapped"
Note: upon updating the credentials in aforementioned secret, the operator may need to delete the PVC chartmuseum before chartmuseum will deploy correctly.
kubectl delete pvc chartmuseum -n kommander --kubeconfig <path-to-kubeconfig>
persistentvolumeclaim "chartmuseum" deleted