Problem
Kubeaddons are not deploying during an upgrade. The Konvoy CLI returns the following:
STAGE [Deploying Enabled Addons] cert-manager [OK] dashboard [OK] konvoyconfig [OK] metallb [OK] traefik [OK] prometheus [ERROR] prometheusadapter [ERROR] dex-k8s-authenticator [ERROR] traefik-forward-auth [ERROR] kube-oidc-proxy [ERROR] defaultstorageclass-protection [ERROR] dex [ERROR] reloader [ERROR] opsportal [ERROR] Error: failed to deploy the cluster: the following 9 addons failed to deploy: [ prometheus prometheusadapter dex-k8s-authenticator traefik-forward-auth kube-oidc-proxy defaultstorageclass-protection dex reloader opsportal ]
Additional symptoms
1. You installed the initial version of the cluster following the documentation for air-gapped installations.
2. The log of the kubeaddons_kubeaddons-controller-manager-... contains the following error:
2022-02-18T14:14:25.777Z ERROR kubeaddons-controller.Addon.Helm3.Upgrade could not upgrade addon
{
"Addon": "dex",
"namespace": "kubeaddons",
"generation": 5,
"chart": "dex",
"addon": "dex",
"version": "2.9.2",
"valuesCRC32": "fe49baae",
"error": "chart \"dex\" version \"2.9.2\" not found in http://konvoy-addons-chart-repo-kubernetes-base-addons.kubeaddons.svc:8879 repository",
"errorVerbose": "chart \"dex\" version \"2.9.2\" not found in http://konvoy-addons-chart-repo-kubernetes-base-addons.kubeaddons.svc:8879 ..."
}
Solution
The cause of the problem is that the chart version the addon controller is trying to install doesn't exist in your in-cluster chart repository. For example, you are upgrading to v1.7.5, but in the config.yaml file the version is v1.6.1.
addons:
- configRepository: https://github.com/mesosphere/kubernetes-base-addons
configVersion: stable-1.18-3.6.0
helmRepository:
image: mesosphere/konvoy-addons-chart-repo:v1.6.1
To resolve the problem, deploy the correct version of the addon repository and specify it in the config.yaml using the installation documentation.