Skip to main content

Kubelet config missing after making a cluster self-managed

Austin Doll
  • Updated

Problem

When a pivot is run from the bootstrap controller while a CAPX-controller is reconciling a node, the responsible controller pod will log a message indicating that the machine can no longer be found:

2022-02-08T17:09:19.904Z ERROR controller.preprovisionedmachine failed to patch PreprovisionedMachine 
{"reconciler group": "infrastructure.cluster.konvoy.d2iq.io", "reconciler kind": "PreprovisionedMachine", 
"name": "<Machine Name>", "namespace": "default", "machine": "<Machine Name>", "cluster": "<Workload cluster>", 
"preprovisioned-cluster": "<Workload cluster>, "error": "preprovisionedmachines.infrastructure.cluster.konvoy.d2iq.io \"<Machine Name>5\" not found", 
"errorCauses": [{"error": "preprovisionedmachines.infrastructure.cluster.konvoy.d2iq.io \"<Machine Name>\" not found"}]}

 

The node reconciliation process includes cleaning stateful directories such as /var/lib/kubelet. If the process tries to patch the machine after being moved off of the bootstrap cluster and onto the workload cluster, it may encounter this issue. When this failure happens, the kubeadm reset is interrupted, and the node being reconciled is left without a Kubelet config. You can validate if this is the case by checking if the Kubelet on that node has stopped posting its status:

$ kubectl describe node myBrokenNode
Name: myBrokenNode
Roles: <none>
Labels: kubernetes.io/arch=amd64
kubernetes.io/os=linux
Annotations: cluster.x-k8s.io/cluster-name: Self-managed-cluster
Taints: node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Mon, 10 Feb 2022 16:35:07 -0600 Mon, 11 Feb 2022 16:35:07 -0600 CalicoIsUp Calico is running on this node
MemoryPressure Unknown Tue, 10 Feb 2022 11:05:53 -0600 Tue, 11 Feb 2022 11:08:39 -0600 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Tue, 10 Feb 2022 11:05:53 -0600 Tue, 11 Feb 2022 11:08:39 -0600 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Tue, 10 Feb 2022 11:05:53 -0600 Tue, 11 Feb 2022 11:08:39 -0600 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Tue, 10 Feb 2022 11:05:53 -0600 Tue, 11 Feb 2022 11:08:39 -0600 NodeStatusUnknown Kubelet stopped posting node status.


From there, you can also confirm any issues on the node by checking the Kubelet logging on the node for errors such as the below:

kubelet[62218]: E0208 11:18:19.772630 62218 server.go:204] "Failed to load kubelet config file" 
err="failed to load Kubelet config file /var/lib/kubelet/config.yaml, 
error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", 
error: open /var/lib/kubelet/config.yaml: no such file or directory" path="/var/lib/kubelet/config.yaml"

Solution

If reconciliation has been paused for the affected cluster object, you can re-enable it by changing the .spec.paused value from true to false:

kubectl describe cluster <cluster name>
Name: <cluster name>
Namespace: default
Labels: konvoy.d2iq.io/autoscaler=cluster-api
Annotations: <none>
API Version: cluster.x-k8s.io/v1alpha4
Kind: Cluster
Metadata:
  Finalizers:
    cluster.cluster.x-k8s.io
  Generation: 3
Spec:
  Cluster Network:
    Pods:
      Cidr Blocks:
        192.168.0.0/16
    Services:
      Cidr Blocks:
        10.96.0.0/12
  Control Plane Endpoint:
    Host: <Control Plane Endpoint>
    Port: 6443
  Control Plane Ref:
    API Version: controlplane.cluster.x-k8s.io/v1alpha4
    Kind: KubeadmControlPlane
    Name: adoll-0211-2-control-plane
    Namespace: default
  Infrastructure Ref:
    API Version: infrastructure.cluster.x-k8s.io/v1alpha4
    Kind: AWSCluster
    Name: <cluster name>
    Namespace: default
  Paused: true

 

This will cause the relevant controller to start reconciling the cluster and correcting the issues on the failed node. Another kubeadm reset will be run, and the Kubelet config will be written to the host. If the .spec.paused section is not available, that indicates that reconciliation is occurring but may be in an unexpected state after the move. If you do not see any progress on the node, you can force an update by removing the relevant machine. In this case, each machine will have a .spec.providerID entry that will contain the name of the node that is associated with the machine:

kubectl describe machine <machine name>
Name: adoll-0211-2-md-0-6b6dbd5996-vmgr9
Namespace: default
Labels: cluster.x-k8s.io/cluster-name=<cluster name>
Annotations: <none>
API Version: cluster.x-k8s.io/v1alpha4
Kind: Machine
Metadata:
  Creation Timestamp: 2022-02-11T16:53:19Z
  Finalizers:
    machine.cluster.x-k8s.io
  Generation: 1
  Owner References:
    API Version: cluster.x-k8s.io/v1alpha4
    Block Owner Deletion: true
    Controller: true
    Kind: MachineSet
    Name: <name>
    UID: eb990387-a0ac-42cf-a452-f7e9d2a1e8ea
Spec:
  Bootstrap:
    Config Ref:
      API Version: bootstrap.cluster.x-k8s.io/v1alpha4
      Kind: KubeadmConfig
      Name: adoll-0211-2-md-0-7nckd
      Namespace: default
      UID: 3dfa2379-572b-419f-8a20-2599b82db801
    Data Secret Name: adoll-0211-2-md-0-7nckd
  Cluster Name: adoll-0211-2
  Infrastructure Ref:
    API Version: infrastructure.cluster.x-k8s.io/v1alpha4
    Kind: AWSMachine
    Name: adoll-0211-2-md-0-wdkw8
    Namespace: default
    UID: 0b0dcfe9-4e17-4375-b6f8-7e5a304ed1b1
    Provider ID: <provider>:///<Node Name>
   Version: v1.21.6

 

Once you have found the machine that has a Provider ID that is associated with the failing node, all that needs to be done is to delete the machine using kubectl:

kubectl delete machine <machine name>

From there a new machine will be created that references the node, and the reconicliation process will take place. When the process is done, a new Kubelet config will be available on the node and it will start reporting as healthy.