Cluster fails to detach due to auto-provisioning-webhook unavailability
Overview/Background
When trying to detach a cluster from Kommander, if the auto-provisioning-webhook is unavailable for any reason, the detach will not complete. In these scenarios, you may observe messages similar to the following in the Kommander federation controller manager logs:
$ kubectl logs -n kommander -l control-plane=kommander-federation-cm -c controller-manager 2020-10-10T10:10:10.100Z ERROR controller-runtime.controller Reconciler error {"controller": "DexTFAClient", "request": "test-ws-ab123-45c67/konvoy-cluster", "error": "failed dex client secret dextfa-clientsecret-konvoy-cluster-ab1cd deletion: Internal error occurred: failed calling webhook \"webhook.provisioning.kommander.mesosphere.io\": Post https://auto-provisioning-webhook.konvoy.svc:443/validations?timeout=10s: dial tcp 10.10.10.10:443: connect: connection refused"}
Solution
To resolve this issue with auto-provisioning enabled, you will need to diagnose why the webhook is unavailable. However, if you are not using auto-provisioning, you can remove it from the cluster with the following command, ensuring that you substitute for your Konvoy version (e.g., v1.5.2):
docker run -v $(pwd):/opt/konvoy -e KUBECONFIG=admin.conf -w /opt/konvoy --entrypoint /usr/local/bin/helmv3 mesosphere/konvoy: uninstall auto-provisioning -n konvoyOnce completed, after some time the Kommander federation controller manager should reconcile and detach the cluster. To force this reconciliation, you can restart the pod with the following command:
kubectl delete po -n kommander -l control-plane=kommander-federation-cmYou can then verify that the cluster is no longer attached via the Kommander UI, or with the following command:
kubectl get konvoycluster,kommandercluster,kubefedcluster -A
Should you run into any additional issues, please feel free to open a case with D2iQ support.