Why does Kommander is stuck as "outdated" after deploying Konvoy addons – D2iQ

On rare occasions, kubeaddons could reach a state where the kubeaddon controller manager cannot reconcile the addon state, when this occurs the controller manager log contains entries like the ones below:

"name": "kommander", "namespace": "", "error": "deploy error: upgrade failed: another operation (install/upgrade/rollback) is in progress" it seems that kommander is in a state where the kubeaddons controller manager is having issues re-conciling its state. 
2021-07-14T22:58:33.210Z INFO kubeaddons-controller.Addon finished reconciling addon {"ClusterAddon": "kommander", "namespace": "kommander", "generation": 9, "status": {"ready":false,"stage":"deploying"}, "duration": "31.449858116s"}
2021-07-14T22:58:33.211Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"ClusterAddon","name":"kommander","uid":"15663614-503f-4aaf-85a5-55c1439c3639","apiVersion":"kubeaddons.mesosphere.io/v1beta2","resourceVersion":"289150833"}, "reason": "DriverDeployFailure", "message": "upgrade failed: another operation (install/upgrade/rollback) is in progress"}

When this issue is encountered, uninstalling kommander and re-deploying the addons is usually enough to solve the problem:

helm uninstall kommander-kubeaddons -n kommander --no-hooks
./konvoy deploy addons

However, some operators have found that after following the aforementioned procedure, kommander was still failing and the kubeaddons controller manager logs are complaining about the presence of the namespace and the lack of a specific label in the kommander-system namespace, as can be seen below:

2021-07-22T00:00:54.622Z        ERROR   kubeaddons-controller.Addon.Helm3.install       failed helm install     {"ClusterAddon": "kommander", "namespace": "kommander", "generation": 9, "chart": "kommander", "addon": "kommander", "version": "0.15.8", "valuesCRC32": "cedbf5af", "error": "rendered manifests contain a resource that already exists. Unable to continue with install: Namespace \"kommander-system\" in namespace \"\" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key \"app.kubernetes.io/managed-by\": must be set to \"Helm\"", "errorVerbose": "Namespace \"kommander-system\" in namespace \"\" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key \"app.kubernetes.io/managed-by\": must be set to \"Helm\"\nrendered manifests contain a resource that already exists. Unable to continue with install\nhelm.sh/helm/v3/pkg/action.

To fix the issue, the operator must remove the namespace kommander-system. However, doing so with the following command:

kubectl delete namespace kommander-system --force --grace-period=0

won’t work because the namespace contains a finalizer that must be removed before the namespace can be deleted:

kubectl get namespaces kommander-system -ojsonpath='{.spec.finalizers}'
#kubernetes

In the first step, the operator should dump the namespace config:

kubectl get namespaces kommander-system -ojson > kommander-system-ns.json

Then edit the file kommander-system-ns.json by removing the finalizer from the configuration:

"spec": {
        "finalizers": [
            "kubernetes"
        ]
    },

to:

"spec": {
        "finalizers": [

        ]
 },

The next step is to apply the changes to modify the namespace configuration:

kubectl replace --raw "/api/v1/namespaces/kommander-system/finalize" -f ./kommander-system-ns.json

In the last step, the operator must delete the namespace with the following command:

kubectl delete namespace kommander-system --force --grace-period=0

At this point, the operator should make sure the namespace is gone and then re-deploy the addons.

kubectl get namespace kommander-system
./konvoy deploy addons