Skip to main content

Prometheus sizing guidelines for konvoy clusters

Craig Neth
  • Updated
Konvoy clusters are by default configured to install the Prometheus monitoring system as one of the default addons.

Resource requirements for Prometheus scale with the number of nodes/pods/services in the cluster, so it is important to review and adjust the limits for Prometheus accordingly.   By default, Konvoy currently installs Prometheus with the following resource requests:
  
resources:
  limits:
    cpu: 1000m
    memory: 2500mi
  requests:
    cpu: 300m
    memory: 2500mi
storage: 50Gi
These default requests are only suitable for small clusters.   For larger clusters, we recommend the following resource setings for Prometheus:
 
Cluster SizeNumber of PodsNumber of ServicesResource settings
501.5k500 
resources:
  limits:
    cpu: 7
    memory: 28Gi
  requests:
    cpu: 2
    memory: 8Gi
storage: 100Gi
1003k1k 
resources:
  limits:
    cpu: 12
    memory: 50Gi
  requests:
    cpu: 10
    memory: 48Gi
storage: 100Gi
20010k3k 
resources:
  limits:
    cpu: 20
    memory: 80Gi
  requests:
    cpu: 15
    memory: 50Gi
storage: 100Gi
30015k6k 
resources:
  limits:
    cpu: 35
    memory: 150Gi
  requests:
    cpu: 25
    memory: 120Gi
storage: 100Gi

To apply these larger limits, edit your cluster.yaml and add the values for the resources desired to the Prometheus section of the addonsList configuration section:

- name: prometheus
  enabled: true
  values: | 
     prometheus: 
        prometheusSpec: 
           resources: 
              limits: 
                 cpu: 7
                 memory: 28Gi
              requests: 
                 cpu: 2
                 memory: 8Gi
           storageSpec: 
              volumeClaimTemplate: 
                 spec: 
                     resources: 
                         requests: 
                             storage: 100Gi

If you have not yet deployed the cluster, then the limits will be applied when you run 'konvoy up'. 

If you have already installed the cluster, you can change the limits for the Prometheus deployment by following this procedure:

  1. Edit the cluster.yaml file and add the values mentioned above that are appropriate for your cluster
  2. Run 'konvoy up' to update the addon configuration in the cluster
  3. Restart the Prometheus stateful set using this command:  
kubectl rollout restart sts/prometheus-prometheus-kubeaddons-prom-prometheus -n kubeaddons
Note that the restart can take a few minutes.  You can monitor the restart using this command:   
kubectl rollout status sts/prometheus-prometheus-kubeaddons-prom-prometheus -n kubeaddons