When using Kubecost to monitor and manage cost and capacity in Kubernetes clusters provisioned with Konvoy, some operators have found that the pod kubecost-kubeaddons-prometheus-server consumes more compute resources than the allocated by default [1] (CPU: 1000m, Memory: 2500Mi) and the pod is OOM-Killed.
To solve this issue, the operator should increase the compute resources limits for the kubecost prometheus server. To adjust these parameters in a way that persists a cluster upgrade, the operator must include the following code stanza under the kommander section in the cluster.yaml.
- name: kommander
enabled: true
values: |
kubecost:
cost-analyzer:
prometheus:
server:
resources:
limits:
memory: "XYGi"
After editing the cluster.yaml with the aforementioned kubecost values, the operator must re-deploy the addons with the new values:
./konvoy deploy addons
To confirm that the compute resources were modified, the following command can be executed:
kubectl get deployment kommander-kubecost-prometheus-server -n kommander -ojsonpath='{.spec.template.spec.containers[?(@.name=="prometheus-server")].resources}'
References:
[1] https://github.com/mesosphere/charts/blob/707d5d878be110b10c12b9c201d54eb68a64c9dd/stable/kubecost/values.yaml#L99