Skip to main content

Kommander fails to create/update Traefik-forward-auth config overrides

Alex Christopher Ramos
  • Updated

Issue

For clusters with custom domain name and certificates, Traefik-forward-auth (TFA) should have the domain name and CA certificate properly configured, via an override configmap. If not, a common error on the TFA logs would be

Get \"https://example.com/dex/.well-known/openid-configuration\": x509: certificate signed by unknown authority

This is a common error that could have many root cause. This article is specific to the following observed symptoms.

  • When describing the TFA deployment/pod, the following configs are not available
spec:
  containers:
  - args:
    env:
    - name: SSL_CERT_FILE
      value: /etc/traefik-forward-auth/ca/ca.crt

    volumeMounts:
    - mountPath: /etc/traefik-forward-auth/ca
      name: etc-traefik-forward-auth-ca
      readOnly: true
    - mountPath: /etc/traefik-forward-auth/config

  volumes:
  - name: etc-traefik-forward-auth-ca
    secret:
      defaultMode: 420
      items:
      - key: ca.crt
        path: ca.crt
      secretName: traefik-forward-auth-mgmt-ca-certificate
  • the secret traefik-forward-auth-mgmt-ca-certificate does not exist
  • the configmap traefik-forward-auth-mgmt-cluster-overrides also does not exist
  • the logs on kommander-cm controller has the following entry
2023-02-09T21:18:34.126Z	ERROR	controllers.TFAFederatedClientsController	failed to create/update tfa overrides configmap	{"name": "kommander/host-cluster", "kind": "KommanderCluster", "error": "error setting per-cluster overrides: failed to retrieve dex client information: Client.dex.mesosphere.io \"dextfa-client-host-cluster-mwhmh\" not found"}
  • confirming that the mentioned dex client does not exist via
kubectl get client -A

Solution

The kommander-cm controller is responsible creating the dex client, updating the kommanderCluster object and creating the TFA override configmap to be used by TFA deployment.

In this scenario, the kommander-cm is stuck with a client that does not exist, but also does not create a new dex client because the status of kommanderCluster already has an existing client ID.

Future release would solve this issue, by creating a new dex client if the current client ID in kommanderCluster does not exist.

For now, the workaround to this is to patch kommanderCluster, removing the current dex client ID. Therefore, triggering kommander-cm controller to create a new client.

kubectl patch kommanderclusters.kommander.mesosphere.io -n kommander host-cluster --subresource=status -p="[{'op': 'remove', 'path': '/status/dextfaclientRef'}]" --type=json

Note: The command uses a --subsresource flag that was introduced in kubectl v1.24. So please make sure that your kubectl version is v1.24 or higher.