When deploying Kubernetes in a vSphere environment with DKP, sometimes the deployment gets stuck and only 1 control plane node is provisioned and no containers are deployed. It’s very common that kubeadm does not run and therefore the kubelet is not configured, and the following error in the kubelet logs is typically found:
Feb 28 13:12:28 dkp240-rhel84-vsphere-airgap-control-plane-jk5sv kubelet[6327]
: Error: failed to load kubelet config file, error: failed to load Kubelet config file
/var/lib/kubelet/config.yaml, error failed to read kubelet config file
"/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such
file or directory, path: /var/lib/kubelet/config.yaml
When this happens, the next place to look at is cloud-init-output.log, where usually evidence of why kubeadm is not executed can be found. When the following entries are logged:
2023-02-28 11:42:59] [init] Using Kubernetes version: v1.23.12
[2023-02-28 11:42:59] [preflight] Running pre-flight checks
[2023-02-28 11:43:00] error execution phase preflight: [preflight] Some fatal errors occurred:
[2023-02-28 11:43:00] [ERROR KubeletVersion]: the kubelet version is higher than
the control plane version. This is not a supported version skew and may lead to a
malfunctional cluster. Kubelet version: "1.24.6" Control plane version: "1.23.12"
It is because there is a mismatch between the Kubernetes version installed in the virtual machine template and the Kubernetes version specified in the KubeadmControlPlane object (KubeadmControlPlane.spec.version).
To solve the issue, please check the cluster.yaml and confirm that the Kubernetes version specified for the Kubeadm control-plane controller (KubeadmControlPlane.spec.version) and the MachineDeployment controller (MachineDeployment.spec.template.spec.version) matches the kubernetes version in the VM template.