You may notice that after upgrading Konvoy to version 1.8.3 or 1.7.4 that pods for Prometheus and/or PrometheusAdapter are failing.
If you check "kubectl get events" in the cluster, you may notice a message like the following:
message: 'upgrade failed: error validating "": error validating data: [ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigNamespaceSelector" in com.coreos.monitoring.v1.Alertmanager.spec,
ValidationError(Alertmanager.spec): unknown field "alertmanagerConfigSelector"
in com.coreos.monitoring.v1.Alertmanager.spec]'
This is an indication that the Helm chart for Prometheus is using a field that is not recognized by the newer version.
You can fix this by updating the Helm chart.
This is the fix for Konvoy 1.8.3:
helm repo add mesosphere-staging https://mesosphere.github.io/charts/staging
helm repo update
helm fetch mesosphere-staging/kube-prometheus-stack --version 15.4.7 --untar
kubectl apply -f kube-prometheus-stack/crds/
If you're on Konvoy 1.7.4, use this:
helm repo add mesosphere-staging https://mesosphere.github.io/charts/staging
helm repo update
helm fetch mesosphere-staging/kube-prometheus-stack --version 15.4.5 --untar
kubectl apply -f kube-prometheus-stack/crds/
Once this fix is applied, the Prometheus pods should then restart and run successfully.