Customer Advisory
| Advisory ID: | D2IQ-2020-0011 |
|---|---|
| Severity: | High |
| Synopsis: | On November 13th, 2020 the Stable and Incubator Helm chart repositories will be deprecated and all Helm-related images will no longer be available from GCR. |
| Affected Products & Versions |
Konvoy versions released prior to and including 1.5.2. |
| Issue date: | 12-22-2020 |
| Updated on: | 12-22-2020 |
Problem Description
Helm, a CNCF project, hosted its Tiller container image and its chart repositories with Google. It was recently announced by Helm that as of November 13th, 2020, the Stable and Incubator Helm chart repositories will be deprecated and all Helm-related images will no longer be available from GCR. This will affect Konvoy 1.4 and 1.5, as well as Kommander 1.0 and 1.1, when deploying a new cluster or modifying Addons.
Reference: https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/
Context & Symptoms
When deploying a cluster with Konvoy, the deployment will time out during the Deploying Enabled Addons stage. If enabled, the affected addons may include elasticsearch, elasticsearch-curator, external-dns, fluentbit, kibana, metallb, prometheusadapter, prometheus, kommander will fail to install:
STAGE [Deploying Enabled Addons] konvoyconfig [OK] awsebscsiprovisioner [OK] dashboard [OK] defaultstorageclass-protection [OK] reloader [OK] opsportal [OK] gatekeeper [OK] cert-manager [OK] traefik [OK] kube-oidc-proxy [OK] velero [OK] dex [OK] dex-k8s-authenticator [OK] traefik-forward-auth [OK] elasticsearch [ERROR] kommander [ERROR] elasticsearch-curator [ERROR] prometheus [ERROR] prometheusadapter [ERROR] kibana [ERROR] elasticsearchexporter [ERROR] fluentbit [ERROR]
In the kubeaddons-conroller-manager logs, you may observe messages such as the following:
$ kubectl logs -l control-plane=kubeaddons-controller-manager -n kubeadddons --tail -1
2020-12-22T01:43:31.859Z ERROR controller-runtime.controller Reconciler error {"controller": "addon", "request": "kubeaddons/prometheusadapter", "error": "error running cli command `helm install`. stdout: `[debug] Created tunnel using local port: '46394'\n\n[debug] SERVER: \"127.0.0.1:46394\"\n\n[debug] Original chart version: \"2.0.0\"\n`, stderr: `Error: Failed to fetch https://kubernetes-charts.storage.googleapis.com/prometheus-adapter-2.0.0.tgz : 403 Forbidden\n`: exit status 1"}
...
Failed to fetch https://kubernetes-charts.storage.googleapis.com/index.yaml : 404 Not Found
You are affected if the kubeaddons controller deployment version is older than v0.19.7. To retrieve the version, you can run:
$ kubectl get deploy -n kubeaddons kubeaddons-controller-manager -o=jsonpath='{.spec.template.spec.containers[0].image}'
Workaround/Solution
To resolve this issue, upgrade to Konvoy 1.5.3+ or Konvoy 1.4.8+.For your own helm charts, migrate to a chart or chart dependency that’s managed by the upstream project instead of those in the github.com/helm/charts repository.