Issue

When creating a cluster with nvidia gpu capable nodes. You might experience an issue where the nvidia-feature-discovery-gpu-feature-discovery is in a crashloop state. And the logs contains 

gpu-feature-discovery: 2022/10/24 05:38:43 Start running
gpu-feature-discovery: 2022/10/24 05:38:43 Warning: Error removing output file: failed to remove output file: remove /etc/kubernetes/node-feature-discovery/features.d/gfd: no such file or directory
gpu-feature-discovery: 2022/10/24 05:38:43 Exiting
gpu-feature-discovery: 2022/10/24 05:38:43 Error: error creating NVML labeler: failed to initialize NVML: unexpected failure calling nvml.Init: error opening libnvidia-ml.so.1: libnvidia-ml.so.1: cannot open shared object file: No such file or directory

Background

This issue occurs when the combination of --registry flags and nvidia override is set during cluster creation. The --registry flags are implemented on the containerd config with the use of "imports". And as the import is merged into the containerd config.toml, it removes the nvidia container configuration.

Workaround

New Cluster

When creating a new cluster, the issue can be bypassed by adding the registry details in the override file, together with nvidia. Instead of using the --registry flags in the dkp cli.

Existing Cluster

And for existing clusters where the --registry flags were used during cluster creation. Just replace the override secret with the new override that contains the registry and nvidia details. Then delete the machines to force reprovisioning of the nodes.

kubectl delete machine <machinename>

Then rename the files with .toml extension located in

imports = ["/etc/containerd/conf.d/*.toml"]

In both scenarios above, the example of the override would be:

gpu:
    types:
        - nvidia
build_name_extra: -nvidia
default_image_registry_mirrors:
  "docker.io": "https://harbor-registry.daclusta/v2/harbor-registry"
  "*": "https://harbor-registry.daclusta/v2/harbor-registry"
image_registries_with_auth:
- host: "harbor-registry.daclusta"
  username: "testuser"
  password: "Harbor1111"
  auth: ""
  identityToken: ""