Prometheus Adapter Excessive Log Messages – D2iQ

Affected Prometheus Adapter version

v0.5.0 , v0.6.0, v0.8.2, v0.8.3, v0.8.4, v0.9.0, v0.9.3.
Adapter log has the similar error with other versions but panics only with v0.7.0

Issue

Prometheus adapter showing repeated log specifying “apiserver was unable to write a JSON response: http2: stream closed”

Log example:

I0616 17:03:27.698850       1 httplog.go:89] "HTTP" verb="GET" URI="/apis/custom.metrics.k8s.io/v1beta1" latency="265.853218ms" userAgent="Go-http-client/2.0" srcIP="192.168.121.64:44502" resp=200
E0616 17:03:27.702961       1 writers.go:107] apiserver was unable to write a JSON response: http2: stream closed
E0616 17:03:27.702979       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

Example kubeapi-server pod logs:

E0616 18:35:58.710218       1 available_controller.go:508] v1beta1.custom.metrics.k8s.io failed with: failing or missing response from https://172.31.27.239:443/apis/custom.metrics.k8s.io/v1beta1: Get "https://172.31.27.239:443/apis/custom.metrics.k8s.io/v1beta1": dial tcp 172.31.27.239:443: connect: connection refused

Solution

There is an on-going upstream issue on the prometheus adapter that is still open here: https://github.com/kubernetes-sigs/prometheus-adapter/issues/292

If error messages above are being encountered, you can safely ignore them for as long as the DKP Platform apps and deployed apps are working as expected and it is not affecting prometheus-adapter related components. https://github.com/kubernetes-sigs/prometheus-adapter/issues/292#issuecomment-767444375