Defining an Air Gap Docker Registry Override for DKP 2.X

An override file is used by Konvoy Image Builder to customize the Host OS for a DKP Kubernetes cluster. DKP 2.2.X supports deploying clusters in an Air Gap environment via CLI commands such as:

./dkp create cluster preprovisioned --cluster-name ${CLUSTER_NAME} \
--control-plane-endpoint-host 10.4.6.40 \
--virtual-ip-interface ens192 \
--worker-replicas 8 \
--registry-mirror-url=${DOCKER_REGISTRY_ADDRESS} \
--registry-mirror-cacert=${DOCKER_REGISTRY_CA} \
--registry-mirror-username=${DOCKER_REGISTRY_USERNAME} \
--registry-mirror-password=${DOCKER_REGISTRY_PASSWORD}

But in some situations you do not want to use the CLI to define your Private Docker Registry settings. For example, you may have multiple node pools for your Kubernetes cluster that each talk to separate registries, or have specific settings for that nodepool. In that instance, you could create a separate override file for each nodepool. 

The way you will create an Air Gap registry override depends on what kind of registry you use in your Environment. If you use a registry such as Docker Registry v2, Nexus or Artifactory that uses a URL:PORT combination such as:

https://artifactory-registry.local-domain.com/:5000

Then your override file would look like the following:

cat <<EOF > overrides.yaml
default_image_registry_mirrors:
  "docker.io": "https://artifactory-registry.local-domain.com:5000"
  "*": "https://artifactory-registry.local-domain.com:5000"
image_registries_with_auth:
- host: "artifactory-registry.local-domain.com:5000"
  username: "testuser"
  password: "testpassword"
  auth: ""
  identityToken: ""
EOF

kubectl create secret generic $CLUSTER_NAME-user-overrides --from-file=overrides.yaml=overrides.yaml
kubectl label secret $CLUSTER_NAME-user-overrides clusterctl.cluster.x-k8s.io/move=

 If you use a registry such as Harbor Registry that instead uses a URL path with no port such as:

https://harbor-registry.local-domain.com/cluster-registry

Then you must make some configuration changes, specifically inserting a /v2/ in between the domain (harbor-registry.local-domain.com) and the sub-path (cluster-registry):

cat <<EOF > overrides.yaml 
default_image_registry_mirrors:
  "docker.io": "https://harbor-registry.local-domain.com/v2/cluster-registry"
  "*": "https://harbor-registry.local-domain.com/v2/cluster-registry"
image_registries_with_auth:
- host: "harbor-registry.local-domain.com"
  username: "testuser"
  password: "testpassword"
  auth: ""
  identityToken: ""
EOF 

kubectl create secret generic $CLUSTER_NAME-user-overrides --from-file=overrides.yaml=overrides.yaml 
kubectl label secret $CLUSTER_NAME-user-overrides clusterctl.cluster.x-k8s.io/move=

In both cases, you must ensure that you specify HTTP:// or HTTPS:// in the default_image_registry_mirrors section, and that you remove it from the image_registries_with_auth section, or containerd will complain and fail to pull images successfully. 

Once you have created your overrides and added them to the bootstrap cluster as secrets, you must ensure to generate dry run output for your cluster via:

./dkp create cluster <cluster specific variables> --dry-run -o yaml > cluster.yaml

Then you must edit cluster.yaml and insert your specific override secrets into each PreprovisionedMachineTemplate object as shown in the example below:

apiVersion: infrastructure.cluster.konvoy.d2iq.io/v1alpha1
kind: PreprovisionedMachineTemplate
metadata:
  name: cluster-a-control-plane
  namespace: default
spec:
  template:
    spec:
      inventoryRef:
        name: cluster-a-control-plane
        namespace: default
      overrideRef:
        name: cluster-a-user-overrides

Specifically, you must add an overrideRef with the name of the secret you created for your override. Once you have finished editing in your custom overrides, you can push the cluster.yaml to the bootstrap to start the cluster installation:

kubectl create -f cluster.yaml