Cluster fails to detach due to auto-provisioning-webhook unavailability

Overview/Background

When trying to detach a cluster from Kommander, if the auto-provisioning-webhook is unavailable for any reason, the detach will not complete. In these scenarios, you may observe messages similar to the following in the Kommander federation controller manager logs:

$ kubectl logs -n kommander -l control-plane=kommander-federation-cm -c controller-manager
2020-10-10T10:10:10.100Z	ERROR	controller-runtime.controller	Reconciler error	{"controller": "DexTFAClient", "request": "test-ws-ab123-45c67/konvoy-cluster", "error": "failed dex client secret dextfa-clientsecret-konvoy-cluster-ab1cd deletion: Internal error occurred: failed calling webhook \"webhook.provisioning.kommander.mesosphere.io\": Post https://auto-provisioning-webhook.konvoy.svc:443/validations?timeout=10s: dial tcp 10.10.10.10:443: connect: connection refused"}

Solution

To resolve this issue with auto-provisioning enabled, you will need to diagnose why the webhook is unavailable. However, if you are not using auto-provisioning, you can remove it from the cluster with the following command, ensuring that you substitute  for your Konvoy version (e.g., v1.5.2):

docker run -v $(pwd):/opt/konvoy -e KUBECONFIG=admin.conf -w /opt/konvoy --entrypoint /usr/local/bin/helmv3 mesosphere/konvoy: uninstall auto-provisioning -n konvoy 

Once completed, after some time the Kommander federation controller manager should reconcile and detach the cluster. To force this reconciliation, you can restart the pod with the following command:

kubectl delete po -n kommander -l control-plane=kommander-federation-cm 

You can then verify that the cluster is no longer attached via the Kommander UI, or with the following command:

kubectl get konvoycluster,kommandercluster,kubefedcluster -A

Should you run into any additional issues, please feel free to open a case with D2iQ support.