Affected Prometheus Adapter version
v0.5.0 , v0.6.0, v0.8.2, v0.8.3, v0.8.4, v0.9.0, v0.9.3.
Adapter log has the similar error with other versions but panics only with v0.7.0
Issue
Prometheus adapter showing repeated log specifying “apiserver was unable to write a JSON response: http2: stream closed”
Log example:
I0616 17:03:27.698850 1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="265.853218ms" userAgent="Go-http-client/2.0" srcIP="192.168.121.64:44502" resp=200
E0616 17:03:27.702961 1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0616 17:03:27.702979 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
Example kubeapi-server pod logs:
E0616 18:35:58.710218 1 available_controller.go:508] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://172.31.27.239:443/apis/custom.metrics.k8s.io/v1beta1: Get "https://172.31.27.239:443/apis/custom.metrics.k8s.io/v1beta1": dial tcp 172.31.27.239:443: connect: connection refused
Solution
There is an on-going upstream issue on the prometheus adapter that is still open here: https://github.com/kubernetes-sigs/prometheus-adapter/issues/292
If error messages above are being encountered, you can safely ignore them for as long as the DKP Platform apps and deployed apps are working as expected and it is not affecting prometheus-adapter related components. https://github.com/kubernetes-sigs/prometheus-adapter/issues/292#issuecomment-767444375
DKP Version and Prometheus Adapter Versions
Konvoy 1.8.1 = https://github.com/mesosphere/kubernetes-base-addons/tree/stable-1.20-4.1.0/addons
Prometheus-adapter version: v0.8.3-4
Konvoy 1.8.2 = https://github.com/mesosphere/kubernetes-base-addons/tree/stable-1.20-4.1.0/addons
Prometheus-adapter version: v0.8.3-4
Konvoy 1.8.3 = https://github.com/mesosphere/kubernetes-base-addons/tree/stable-1.20-4.2.0/addons
Prometheus-adapter version: v0.8.3-4
Konvoy 1.8.4 = https://github.com/mesosphere/kubernetes-base-addons/tree/stable-1.20-4.3.0/addons
Prometheus-adapter version: v0.8.3-4
Konvoy1.8.5 = https://github.com/mesosphere/kubernetes-base-addons/tree/stable-1.20-4.4.0/addons
Prometheus-adapter version: v0.8.3-4
DKP 2.1
https://github.com/mesosphere/kommander-applications/tree/v2.1.0/services
Prometheus adapter version: 2.11.1
DKP 2.1.1
https://github.com/mesosphere/kommander-applications/tree/v2.1.1/services
Prometheus adapter version: 2.11.1
DKP 2.2
https://github.com/mesosphere/kommander-applications/tree/v2.2.1/services
Prometheus adapter version: 2.17.1
DKP 2.2.1
https://github.com/mesosphere/kommander-applications/tree/v2.2.1/services
Prometheus adapter version: 2.17.1
DKP 2.2.2
https://github.com/mesosphere/kommander-applications/tree/v2.2.2/services
Prometheus adapter version: 2.17.1