In Konvoy deployed kubernetes clusters, the file /etc/containderd/config.toml is configured for each control plane and worker node on first konvoy up. Currently this configuration file is not modified on subsequent konvoy up or konvoy deploy addons even if you make a configuration change in cluster.yaml or inventory.yaml. It is therefore important to make sure that your nodes are properly configured and recognize how to manually edit config.toml should you need to make a change.
Any node that should have GPU compute enabled must be placed into a nodepool that properly indicates this before konvoy up is run for the first time:
But what if you mistakenly place a node in the worker nodepool and then run Konvoy up? You should modify inventory.yaml so that the node is placed into the proper nodepool, but you will find that if you rerun konvoy up, the nvidia addon components do not properly start on this node:
In a non-GPU worker, there are 2 entries that determine the default runtime for containerd:
Default runtime name:
Default runtime name:
To quickly check the configuration of your host's config.toml, use the following command:
Any node that should have GPU compute enabled must be placed into a nodepool that properly indicates this before konvoy up is run for the first time:
nodePools: - name: worker - name: gpu-worker gpu: nvidia: {}Only the nodes that are placed in the gpu-worker nodepool above will be configured for GPU compute.
But what if you mistakenly place a node in the worker nodepool and then run Konvoy up? You should modify inventory.yaml so that the node is placed into the proper nodepool, but you will find that if you rerun konvoy up, the nvidia addon components do not properly start on this node:
nvidia-kubeaddons-nvidia-dcgm-exporter-j97pq 2/2 Running 3 4m6s nvidia-kubeaddons-nvidia-dcgm-exporter-pqttc 1/2 CrashLoopBackOff 3 4m6sTaking a look at the crashing pod, we see the follwing error:
kubectl describe pod nvidia-kubeaddons-nvidia-dcgm-exporter-pqttc -n kubeaddons nvidia-dcgm-exporter: Container ID: containerd://4d142663dc4e5d776e9389abd2a0c701316d16fabe8c2dbc1ad3635c5d8af903 Image: nvidia/dcgm-exporter:1.4.6 Image ID: docker.io/nvidia/dcgm-exporter@sha256:1e207db6823484bb2c6746f42f20b7d819da90ab4ee45179726f87adca9d4f1e Port: Host Port: State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: StartError Message: failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown
In a non-GPU worker, there are 2 entries that determine the default runtime for containerd:
Default runtime name:
[plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc"Runtime:
[plugins."io.containerd.runtime.v1.linux"] runtime = "runc"To reconfigure your host for the nvidia container runtime, change the above values as follows:
Default runtime name:
[plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "nvidia-container-runtime"Runtime:
[plugins."io.containerd.runtime.v1.linux"] runtime = "/usr/bin/nvidia-container-runtime"Then save the file and restart containerd:
sudo systemctl restart containerd.Please note that a configuration change made to config.toml requires downtime and all pods currently running on the host will be killed and restarted during the containerd reboot.
To quickly check the configuration of your host's config.toml, use the following command:
cat /etc/containerd/config.toml | grep 'runtime =\|default_runtime_name' default_runtime_name = "nvidia-container-runtime" runtime = "/usr/bin/nvidia-container-runtime"This will return the values set for the runtime.