Skip to main content

Change the existing override applied to nodes during provisioning/re-provisioning for Preprovisioned clusters

Alex Christopher Ramos
  • Updated

Issue

DKP 2.x has introduced the use of overrides when provisioning a cluster.  Overrides can be for enabling GPU, setting a custom image registry endpoint, enabling Air-gap, etc. This overrides are then applied to each node when provisioning the machines.

Overrides in preprovisioned infrastructure are handled differently, compared to a cloud based cluster.

In a preprovisioned cluster, the override is created as a secret, and the secret name is specified during cluster creation. 

Everytime a machine needs to be provisioned, the override content of the secret will be applied on the node, via an ansible playbook. This occurs during initial provisioning or when a machine gets deleted, and the controller needs to reconcile to re-provision the machine.

So for cases where the existing overrides needs to be replaced, our option is to create a new nodepool with the updated overrides. Please see the documentation here - https://docs.d2iq.com/dkp/2.4/pre-provisioned-create-and-manage-node-pools

But for scenarios where adding a new nodepool is not an option. Like, the number of nodes are limited or the existing controlplane overrides needs to be replaced. The following workaround can be considered.

Workaround

If the only option is to change the existing override, we need to update the objects that are used during provisioning.

Here is an overview of the related objects used when provisioning a machine with overrides.

 

Machines are referenced from Machinedeployment (worker nodes) or Kubeadmcontrolplanes (controlplane nodes)

Machinedeployment or Kubeadmcontrolplane are referenced from PreprovisionedMachintemplate.

 

PreprovisionedMachintemplate has the spec for the override secret name, that contains the override settings.

 

So, the first thing we need to create is the secret, containing the new override spec. Please follow the guide on creating the override and creating the secret. Take note, the secret name should be different from the existing.

https://docs.d2iq.com/dkp/2.4/pre-provisioned-create-necessary-secrets-and-overr#id-(2.4)Pre-provisionedCreateNecessarySecretsandOverrides-Createoverrides

In this example, we will be naming the new secret <clustername>-user-overrides-cp-2

 

Next, we have to specify the new override secret to be used on the PreprovisionedMachinetemplate.

But the problem is, this object is immutable and cannot be edited.

Therefore, we will create a new PreprovisionedMachinetemplate, based on the existing one.

Example:

kubectl get preprovisionedmachinetemplate <clustername>-control-plane -oyaml > <clustername>-control-plane-2.yaml

Edit the <clustername>-control-plane-2.yaml with a new name, remove the creationTimestamp, resourceVersion and uid. Note: Not the uid in the ownerReferences.

Also, specify the new override secret name, that was created earlier. Under spec.template.spec.overrideRef.name

apiVersion: infrastructure.cluster.konvoy.d2iq.io/v1alpha1
kind: PreprovisionedMachineTemplate
metadata:
  name: <clustername>-control-plane-2 #new name
  namespace: default
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    name: <clustername>
    uid: f4b79d32-2811-4226-a9fe-26b0edbe47ca
spec:
  template:
    spec:
      inventoryRef:
        name: <clustername>-control-plane
        namespace: default
      overrideRef:
        name: <clustername>-user-overrides-cp-2 #new override secret

Create the new PreprovisionedMachinetemplate

kubectl create -f <clustername>-control-plane-2.yaml

 

Edit the existing Kubeadmcontrolplane or Machinedeployment (depending on which is being updated).

Example:

kubectl edit KubeadmControlPlane <clustername>-control-plane

under machineTemplate.infrastructureRef.name specify the new Preprovisionedmachinetemplate that was created

machineTemplate:
  infrastructureRef:
    apiVersion: infrastructure.cluster.konvoy.d2iq.io/v1alpha1
    kind: PreprovisionedMachineTemplate
    name: <clustername>-control-plane-2
    namespace: default

Save the changes.

 

To apply the new override on the nodes. We have to force the provisioning to run on each node.

This can be done, by deleting the machine one at a time, waiting for the previous machine to be on running status, before deleting the next one.

kubectl get machines -A
kubectl delete machine <machinename>

If the machine is stuck on deleting status, please remove its finalizers.