Issue
For clusters with custom domain name and certificates, Traefik-forward-auth (TFA) should have the domain name and CA certificate properly configured, via an override configmap. If not, a common error on the TFA logs would be
Get \"https://custom-domain-name.com/dex/.well-known/openid-configuration\": x509: certificate signed by unknown authority
This is a common error that could have many root cause. This article is specific to the following observed symptoms.
- When describing the TFA deployment/pod, the following configs are not available
spec: containers: - args: env: - name: SSL_CERT_FILE value: /etc/traefik-forward-auth/ca/ca.crt volumeMounts: - mountPath: /etc/traefik-forward-auth/ca name: etc-traefik-forward-auth-ca readOnly: true - mountPath: /etc/traefik-forward-auth/config volumes: - name: etc-traefik-forward-auth-ca secret: defaultMode: 420 items: - key: ca.crt path: ca.crt secretName: traefik-forward-auth-mgmt-ca-certificate
- the secret
traefik-forward-auth-mgmt-ca-certificate
does not exist - the configmap
traefik-forward-auth-mgmt-cluster-overrides
also does not exist - the logs on kommander-cm controller has the following entry
2023-02-09T21:18:34.126Z ERROR controllers.TFAFederatedClientsController failed to create/update tfa overrides configmap {"name": "kommander/host-cluster", "kind": "KommanderCluster", "error": "error setting per-cluster overrides: failed to retrieve dex client information: Client.dex.mesosphere.io \"dextfa-client-host-cluster-mwhmh\" not found"}
- confirming that the mentioned dex client does not exist via
kubectl get client -A
Solution
The kommander-cm controller is responsible creating the dex client, updating the kommanderCluster
object and creating the TFA override configmap to be used by TFA deployment.
In this scenario, the kommander-cm is stuck with a client that does not exist, but also does not create a new dex client because the status of kommanderCluster
already has an existing client ID.
Future release would solve this issue, by creating a new dex client if the current client ID in kommanderCluster
does not exist.
For now, the workaround to this is to patch kommanderCluster
, removing the current dex client ID. Therefore, triggering kommander-cm controller to create a new client.
kubectl patch kommanderclusters.kommander.mesosphere.io -n kommander host-cluster --subresource=status -p="[{'op': 'remove', 'path': '/status/dextfaclientRef'}]" --type=json
Note: The command uses a --subsresource
flag that was introduced in kubectl v1.24. So please make sure that your kubectl version is v1.24 or higher.