If you are provisioning a several GPU nodes as part of a larger cluster with multiple worker node pools, you may notice that after deploying the cluster and enabling the Nvidia addon via Kommander, you suddenly have many Nvidia DCGM Exporter and Nvidia Device Plugin pods in CrashLoopBackOff Status, and you have many more of these pods than you have GPU nodes:
[tony@centos kaptain.2.1.1]$ kubectl get pods -A | grep nvidia
kommander nvidia-nvidia-dcgm-exporter-2npl7 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-9k55b 1/1 Running
kommander nvidia-nvidia-dcgm-exporter-9p7j2 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-gjmhc 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-gvnmb 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-h8lsx 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-lkbcb 1/1 Running
kommander nvidia-nvidia-dcgm-exporter-tnqv2 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-tqgdg 0/1 CrashLoopBackOff
kommander nvidia-nvidia-dcgm-exporter-zgvx6 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-2szlx 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-4x7x7 1/1 Running
kube-system nvidia-nvidia-device-plugin-5bfxb 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-glrb8 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-gpwl4 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-hmcgv 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-kqdbq 1/1 Running
kube-system nvidia-nvidia-device-plugin-qr8tt 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-w54nl 0/1 CrashLoopBackOff
kube-system nvidia-nvidia-device-plugin-wzqd8 0/1 CrashLoopBackOff
We see that there are 4 pods in Running state, and these line up with the amount of GPU nodes we should have, but there should not be Nvidia pods running on these other nodes. We can resolve this issue by modifying the daemonsets for the Nvidia addon so that they properly select for GPU nodes only.
The two daemonsets we need to modify are:
nvidia-nvidia-device-plugin
namespace: kommander
nvidia-nvidia-dcgm-exporter
namespace: kube-system
Since, for example, your GPU nodes only have 1 GPU each, you can modify both daemonsets with the following nodeSelector:
nodeSelector: nvidia.com/gpu.count: "1"
If your GPU worker nodes have a higher gpu count, increase the value in the nodeSelector above to reflect this. Please node that this workaround will fail if you do not have the same number of GPUs on all GPU worker nodes.