Skip to main content

Adding Overrides to KIB Jobs When Deploying DKP 2.1.X

Anthony Windebank
  • Updated

As of DKP 2.1, you no longer have to run Konvoy Image Builder before deploying your DKP cluster to your nodes. But a new question arises: How do you continue to use the overrides for your cluster now that KIB is running inside of DKP?  

The correct way to do this is to turn your override files into secrets on the bootstrap cluster, and then reference those secrets on cluster creation so that KIB can deploy your nodes using these specific overrides. 

After you have Defined your Infrastructure, you can create override files for your specific needs. Two common examples of override files you may require are Docker Hub Credentials, GPU Enablement or both. You can only have one override file per group of worker nodes, so place all override values in the same file. 

Docker Hub Override:

cat <<EOF > overrides.yaml
image_registries_with_auth:
- host: "registry-1.docker.io"
username: "username"
password: "password"
auth: ""
identityToken: ""
EOF

Docker Hub + Nvidia GPU Override:

cat <<EOF > gpu-overrides.yaml
gpu:
types:
- nvidia
build_name_extra: "-nvidia"
image_registries_with_auth:
- host: "registry-1.docker.io"
username: "username"
password: "password"
auth: ""
identityToken: ""
EOF

Now we use these override yaml files to generate secrets on the bootstrap cluster:

Docker Hub Override:

kubectl create secret generic $CLUSTER_NAME-overrides --from-file=overrides.yaml=overrides.yaml
kubectl label secret $CLUSTER_NAME-overrides clusterctl.cluster.x-k8s.io/move=

Docker Hub + Nvidia GPU Override:

kubectl create secret generic $CLUSTER_NAME-gpu-overrides --from-file=overrides.yaml=gpu-overrides.yaml
kubectl label secret $CLUSTER_NAME-gpu-overrides clusterctl.cluster.x-k8s.io/move=

We must now generate all resources needed to deploy the cluster. Normally you would immediately begin cluster creation, but instead we will use the --dry-run flag to add our overrides first. Here is an example of how to generate a cluster.yaml via --dry-run:

./dkp create cluster preprovisioned --cluster-name ${CLUSTER_NAME} --control-plane-endpoint-host 10.4.6.40 --virtual-ip-interface ens192 --dry-run -o yaml > ${CLUSTER_NAME}.yaml

After the cluster.yaml file is generated, open it in your editor of choice and locate the PreprovisionedMachineTemplate objects inside, and add overrideRef sections for the secrets we created above:

Docker Hub Override:

spec:
  template:
    spec:
      inventoryRef:
        name: cluster-a-control-plane
        namespace: default
      overrideRef:
        name: cluster-a-overrides

Docker Hub + Nvidia GPU Override:

spec:
  template:
    spec:
      inventoryRef:
        name: cluster-a-gpu-worker
        namespace: default
      overrideRef:
        name: cluster-a-gpu-overrides

You can now apply the cluster.yaml to begin the deployment process, and it will use the overrides you specified.