Overview

Prometheus and Grafana are powerful tools when it comes to monitoring your applications. Out-of-the-box DKP is configured to watch most of the components deployed by default, which is helpful when monitoring your DKP infrastructure. Still, in many cases, there is a demand for these insights into a broader range of applications deployed to a cluster. If this is the case, Prometheus offers a simple way to bring your applications under the umbrella of its scope. The default Prometheus configuration applied with DKP provides configuration for both ServiceMonitors  and PodMonitors, depending on your use case one may be more preferable than the other. Below, we will set up a service monitor for our clusterAutoscaler deployment; please note that you will need to assess your deployments for their specific metrics endpoint and port.

Hands-On

ClusterAutoscaler provides metrics over port 8085 on the /metrics endpoint. To properly configure a service monitor, we will need a service that contains a named port. In this example, I have named the port in the service .spec 'service':

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cluster-autoscaler
  name: cluster-autoscaler
  namespace: kube-system
spec:
  ports:
  - name: service
    port: 8085
    protocol: TCP
    targetPort: 8085
  selector:
    app: cluster-autoscaler
  type: ClusterIP

Once we have a service with a named port, we will need to deploy a ServiceMonitor object with the appropriate configuration. For DKP 2.4+ we configure Prometheus to scrape Service/pod monitors that have the label prometheus.kommander.d2iq.io/select: "true". In future versions, this may change. If you notice your pod or service monitor is not being picked up you can check the currently deployed configuration for Prometheus. You can run the below command to view the currently applied Prometheus configuration:

kubectl get prometheus -n kommander -o yaml
...
  podMonitorSelector:
    matchLabels:
      prometheus.kommander.d2iq.io/select: "true"
  serviceAccountName: kube-prometheus-stack-prometheus
  serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector:
      matchLabels:
        prometheus.kommander.d2iq.io/select: "true"
...

To define our serviceMonitor, we will need to provide the path to the metrics endpoint, the name of the port, the namespace that our service resides in, and labels that match the service. Below is an example ServiceMonitor for the above service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    prometheus.kommander.d2iq.io/select: "true"
  name: cluster-autoscaler
  namespace: kommander
spec:
  endpoints:
  - path: /metrics
    port: service
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app: cluster-autoscaler

Once we have both deployed, allow Prometheus some time to reconcile. If you do not see your ServiceMonitor available in your cluster, cycling the Prometheus Operator pod may help. After some time, we can validate that our configuration took place by viewing the 'targets' tab, where we should see our ServiceMonitor. We can also validate that our application-specific metrics have made it into the UI:

If a pod does not have a service and you do not wish to create one, Prometheus can still scrape your metrics with a podMonitor. In both cases, we will need a named port as well as a valid label selector:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: cluster-autoscaler
  labels:
    app: cluster-autoscaler
spec:
  selector:
    matchLabels:
      app: cluster-autoscaler
  podMetricsEndpoints:
  - port: <port name>