Skip to main content

Configuring Prometheus to collect Portworx Metrics in Konvoy

Jena Miller
  • Updated
To collect metrics out of Kubernetes components, the Prometheus operator iterates over ServiceMonitor resources. In Konvoy, the Prometheus kube-addon is configured with the following ServiceMonitor resources by default: 
$ kubectl get ServiceMonitor -n kubeaddons                             
NAME                                            AGE
prometheus-kubeaddons-prom-alertmanager         93m
prometheus-kubeaddons-prom-apiserver            93m
prometheus-kubeaddons-prom-coredns              93m
prometheus-kubeaddons-prom-grafana              93m
prometheus-kubeaddons-prom-kube-etcd            93m
prometheus-kubeaddons-prom-kube-proxy           93m
prometheus-kubeaddons-prom-kube-state-metrics   93m
prometheus-kubeaddons-prom-kubelet              93m
prometheus-kubeaddons-prom-node-exporter        93m
prometheus-kubeaddons-prom-operator             93m
prometheus-kubeaddons-prom-prometheus           93m

When portworx is installed in Kubernetes, a ServiceMonitor named portworx-prometheus-sm is configured in the kube-system namespace. However, as the prometheus operator in the Konvoy kube-addon requires the label “release: prometheus-kubeaddons” to discover the ServiceMonitor, the native ServiceMonitor installed by Portworx is ignored and Prometheus cannot scrape its metrics. 


To solve this issue, there are two alternatives. The preferred method is to specify an additional ServiceMonitor for Portworx in the cluster.yaml as follows: 
- name: prometheus
  enabled: true
  values: |
    prometheus:
       additionalServiceMonitors:
           - name: portworx-prometheus-sm
             selector:
                 matchLabels:
                      name: portworx
             namespaceSelector:
                 any: true   
             endpoints:
             - port: px-api
               targetPort: 9001
             - port: px-kvdb
               targetPort: 9019
The second method would be to modify the portworx-prometheus-sm ServiceMonitor in the kube-system namespace by adding the label “release: prometheus-kubeaddons”. 
$ kubectl edit ServiceMonitor portworx-prometheus-sm -n kube-system
After adding the label, the operator can verify with the following command: 
$ kubectl get ServiceMonitor portworx-prometheus-sm -n kubeaddons \
-ojsonpath='{.metadata.labels.release}' prometheus-kubeaddons

To verify that Prometheus is able to scrape metrics out of Portworx,  the operator could go to https:///ops/portal/prometheus/graph and check whether portworx metrics pops up when px_ is typed in the query box. 


References:

[1] https://docs.d2iq.com/dkp/konvoy/1.8/monitoring/#monitoring-applications
[2] https://docs.portworx.com/portworx-install-with-kubernetes/operate-and-maintain-on-kubernetes/monitoring/monitoring-px-prometheusandgrafana.1/