Some users may encounter an issue where endpoints accessed through the Konvoy Ops Portal return a "Bad Gateway" error.
If you check the logs for the traefik-forward-auth pod(s), you might see an error that resembles the following:
time="20XX-XX-XXTXX:XX:XXZ" level=error msg="failed to exchange token: Post \"https://mycluster.example/dex/token\": x509: certificate has expired or is not yet valid: current time 20XX-XX-XXTXX:XX:XXZ is after 20XX-XX-XXTXX:XX:XXZ" source_ip=12.34.56.78
This issue has been observed even when the corresponding cert has been renewed and is not past its expiration time.
You can check to make sure your cert is up to date with:
kubectl get certificate -n kubeaddons <cert-name> -o yaml
If the corresponding certificate is within a valid date, first make sure that the network time on your cluster nodes is in sync. Make sure to also check to make sure that the network time on your bastion host (or wherever you are running kubectl commands) is in sync.
If time sync is not the problem, you may be encountering a known issue where Traefik loads an old certificate and does not properly update with a new one. Konvoy has a "reloader" component which normally handles this, but there are some instances where a new cert is not loaded in.
If you are encountering this issue, a short term fix is to restart the pods in the Traefik deployment. You can restart it by scaling the deployment down to 0 then back up to its previous replica count:
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment
If this resolves the issue, then the problem will likely resurface again in 90 days. To resolve it permanently, you can edit the "traefik" section of the "addons" block of your cluster.yaml to include this annotation. Replace "<my-cert-name>" with the cert you are using in your cluster:
- name: traefik
enabled: true
values: |
[.... other Traefik config values .... ]
deployment:
annotations:
secret.reloader.stakater.com/reload: traefik-kubeaddons-certificate, <my-cert-name>Once you've made this change, you can redeploy the addons using "./konvoy deploy addons" and the annotation will be added to the Traefik deployment.