When using the Cluster API vSphere provisioner in DKP 2.x, it is useful to know where the credentials are stored and how to update them in case they are outdated. The capv-controller-manager uses these credentials to authenticate with the vCenter API to provision the cluster nodes, create/attach/delete disks, etc. Therefore, having incorrect credentials will block the cluster creation or updating resources. In this article we describe the most common scenarios where the vSphere provider encounters authentication issues.
Solution
Scenario 1
When the capv-controller-manager in the bootstrap cluster has incorrect credentials, login with the vCenter API is impossible. In this case, describing the cluster will reveal the login issue, as can be seen below:
./dkp describe cluster -c sortega-dkp231-rhel84-airgap
NAME READY SEVERITY REASON SINCE MESSAGE
Cluster/sortega-dkp231-rhel84-airgap FalseError VCenterUnreachable 7m25s ServerFaultCode: Cannot complete login due to an incorrect user name or password.
├─ClusterInfrastructure - VSphereCluster/sortega-dkp231-rhel84-airgap FalseError VCenterUnreachable 7m25s ServerFaultCode: Cannot complete login due to an incorrect user name or password.
vCenter credentials are stored in the capv-manager-bootstrap-credentials secret and the following command can be executed to confirm whether the credentials are correct or not:
kubectl get secrets capv-manager-bootstrap-credentials -n capv-system -ojsonpath='{.data.credentials\.yaml}'|base64 -d
username: 'USERNAME'
password: 'PASSWORD'
Easiest way to update the credentials in the secret capv-manager-bootstrap-credentials is to run the following command:
./dkp update bootstrap credentials vsphere
This command takes the values from the below environment variables, so please make sure to update them before executing the aforementioned command:
export VSPHERE_USERNAME="USERNAME"
export VSPHERE_PASSWORD="PASSWORD"
Alternatively, the secret can be edited manually:
kubectl edit secrets capv-manager-bootstrap-credentials -n capv-system
Please note that all the secrets are base64 encoded.
Scenario 2
If the cluster deployment is stuck, and it cannot provision more than one control-plane node, it is possibly because the credentials in the secret cloud-provider-vsphere-credentials-<CLUSTER-NAME> are incorrect. To confirm if that is the case, check the vsphere-cloud-controller-manager log to see if it contains the following entries:
To fix this issue, edit the secret cloud-provider-vsphere-credentials-<cluster-name> and update it with the correct credentials:
kubectl get secrets cloud-provider-vsphere-credentials-sortega-dkp231-rhel84-airgap -ojsonpath='{.data.data}'|base64 -d
apiVersion: v1
kind: Secret
metadata:
name: cloud-provider-vsphere-credentials
namespace: kube-system
stringData:
vcenter-URL.password: "USERNAME"
vcenter-URL.username: "PASSWORD"
type: Opaque
Scenario 3
Another possible failure scenario caused by expired credentials is problems with the vSphere csi nodes and controller pods, in the vmware-system-csi namespace. If the credentials are expired, these pods will be in the CrashLoopBackOff state.
To confirm that the vSphere CSI controller is using outdated credentials, check the logs in one of the vsphere-csi-controller-xxxxx-yyyy pods. If the credentials are outdated, the following error can be observed:
{"level":"error","time":"2022-10-20T03:08:54.311723658Z","caller":"vsphere/virtualcenter.go:233","msg":"Cannot connect to vCenter with err: ServerFaultCode: Cannot complete login due to an incorrect user name or password.","TraceId":"f6a4c4aa-9dc9-494d-8b67-47b4120cbc44","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v2/pkg/common/cns-lib/vsphere.(*VirtualCenter).Connect\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:233\nsigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/common.GetVCenter\n\t/build/pkg/csi/service/common/util.go:59\nsigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service/vanilla.(*controller).Init\n\t/build/pkg/csi/service/vanilla/controller.go:121\nsigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/pkg/csi/service/driver.go:142\nsigs.k8s.io/vsphere-csi-driver/v2/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/pkg/csi/service/driver.go:156\nmain.main\n\t/build/cmd/vsphere-csi/main.go:89\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"}
The CSI driver pulls the vCenter credentials from the secret vsphere-config-secret in the vmware-system-csi namespace. To remedy, edit the secret and update the credentials:
kubectl get secret vsphere-config-secret -n vmware-system-csi -ojsonpath='{.data.\csi-vsphere\.conf}'|base64 -d
[Global]
cluster-id = "default/sortega-dkp231-rhel84-airgap"
thumbprint = ""
[VirtualCenter "vcenter-address"]
user = "USERNAME"
password = "PASSWORD"
datacenters = "dc1"
[Network]
public-network = "Airgapped"