Skip to main content

KommanderCluster crd changes from DKP 2.2 to 2.3+ might cause issues during upgrade

Alex Christopher Ramos
  • Updated

Background

kind: kommanderCluster is the config source for kommander configuration, specially the automation flow for custom domain and certificates, that will be used by Dex-k8s-authenticator and Traefik-forward-auth.

Although kommanderCluster's apiVersion did not change, and remains v1beta1. There has been an update in the spec between DKP 2.2 and DKP 2.3 onwards.

This change, is the introduction of spec.ingress in the new kommanderCluster.

Example:

spec:
  ingress:
    certificateSecretRef:
      name: <Secret_Name>
    hostname: <Domain_Name>

or

spec:
  ingress:
    hostname: <Domain_Name>
    issuerRef:
      kind: ClusterIssuer
      name: <cluster_issuer_name>

Issue

We've encountered an issue where in a client's cluster did not reach a healthy state after upgrading DKP/Kommander from 2.2.x to 2.3.x. This is due to Dex-k8s-authenticator and Traefik-forward-auth failing to reconcile successfully.

With Dex-k8s-authenticator pod logs, showing the following error

panic: runtime error: index out of range [0] with length 0

And the kommander-cm pod logs has

controllers.Kommander.Cluster.IngressMigration annotating no longer needed resources {"name": "<cluster_workspace>/<cluster_name>, "kind": "KommanderCluster"}
controllers.Kommander.Cluster.IngressMigration Observed a panic in reconciler: runtime error: invalid memory address or nil pointer derefernce {"reconcileID": "5c65a651-517c-4166-ae9c-f85e04844dde"|
panic: runtime error: invalid memory address or nil pointer derefernce [recovered]
panic runtime error: invalid memory address or nil pointer derefernce
[ signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1830154]
goroutine 2164 [running]:

Furthermore, describing the dex-k8s-autheticator helmrelease, it only retrieves the configValues from the default configMap

  valuesFrom:
  - kind: ConfigMap
    name: dex-k8s-authenticator-1.2.14-d2iq-defaults

and the override configMapdex-k8s-authenticator-cluster-overridesdoes not exist.

Additionally, when describing the kommanderClusterhost-cluster there is no spec.ingress details.

And when manually editing the kommanderCluster host-cluster with the ingress details, the following error is returned.

kubectl edit kommandercluster host-cluster -n kommander
kommanderclusters.kommander.mesosphere.io "host-cluster" was not valid
* <nil>: Invalid value: "The edited file failed validation": ValidationError (KommanderCluster.spec): unknown field "ingress" in io.mesosphere.kommander.v1beta1.KommanderCluster.spec

Workaround

The above issues are symptoms of the kommanderCluster not having the spec.ingress object, which was introduced on DKP 2.3. Which can be further confirmed by describing the kommanderCluster crd.

kubectl get kommanderCluster host-cluster -n kommander -oyaml

The workaround to this issue is to manually apply the updated kommanderCluster CRD, which is found in https://github.com/mesosphere/kommander/blob/v2.3.3/chart/kommander-appmanagement/crds/kommander.mesosphere.io_kommanderclusters.yaml

Or in the case of air-gapped clusters, this manifest is included in the downloaded charts-bundle.

1. Extract the file kommander-appmanagement-v2.3.3.tgz

2. On the list of extracted tar files, what we need is inside kommander-appmanagement-v2.3.3.tgz. Extract this as well.

3. Then apply the manifest

kubectl apply -f kommander-appmanagement/crds/kommander.mesosphere.io_kommanderclusters.yaml