Skip to main content

Prometheus Alert Rules naming convention in Konvoy

Jena Miller
  • Updated
When defining Prometheus alert rules in Konvoy, some customers have encountered issues configuring alert rules that include “_” in the name or group names, similar to the one below: 
- name: prometheus
  enabled: true
  values: |
     additionalPrometheusRules:
     - name: rule_name_group
       groups:
       - name: rule_name_group.rules
         rules:
         - record:
           expr: <>

This is due to the fact that some Kubernetes resource types require their names to follow the DNS label standard as defined in RFC-1123, which limits the name to:

  1. 63 characters at most, 
  2. only lowercase alphanumeric characters, “-” or “.”,
  3. start with an alphanumeric character
  4. end with an alphanumeric character

To confirm whether this is the cause of the failure, the operator could review the kubeaddon controller logs with the following command:

kubectl logs kubeaddons-controller-manager-xxxxxxx-yyyyy -n kubeaddons
And specifically, look for events similar to the one below: 
2021-05-18T18:52:44.266Z DEBUG controller-runtime.manager.events Warning {"object": {"kind":"Addon","namespace":"kubeaddons","name":"prometheus","uid":"36855caa-d10
a-4a7d-9975-c976467c3cce","apiVersion":"kubeaddons.mesosphere.io/v1beta2","resourceVersion":"148833"}, "reason": "DriverDeployFailure", "message": "upgrade failed: release prometheus-kubeaddons failed, and has been rolled back due to atomic being set: failed to create resource: PrometheusRule.monitoring.coreos.com \"prometheus-operator-portworx_group\" is invalid: metadata.name: Invalid value: \"prometheus-operator-portworx_group\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')"}

The issue is resolved by removing the “_” (for example, replacing "rule_name_group" with "rule-name-group" in the example above).

To confirm that the rule was configured correctly, the operator could run the following command:

kubectl get prometheusrules -n kubeaddons