Help mounting nvoptix.bin in containers

[emaincourt]

Hi,

After trying to upgrade our Bottlerocket AMIs, we realized OptiX was broken. The reason behind that is that, I supposed in recent OptiX versions, nvoptix.bin is required to load the denoiser weights according to the error logs.

I updated packages/kmod-6.1-nvidia/kmod-6.1-nvidia.spec to install the file when building the package. See the commit here. I'm pretty new to Bottlerocket so I might actually do it wrong. Anyway the file is properly available when I boot the host with my custom AMI at path /usr/share/nvidia/nvoptix.bin which is the proper path for the nvidia-container-toolkit to load it.

However, when I start a container with the right environment variables (eg. NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics,display), the file does not get mounted into the container. I tried the same thing with an AL2023 AMI which works as expected. nvoptix.bin is located at the same path, and properly gets mounted into the container.

The nvidia-container-runtime configuration looks quite different on the AL2023 AMIs and the Bottlerocket ones:

# Bottlerocket
accept-nvidia-visible-devices-as-volume-mounts = true
accept-nvidia-visible-devices-envvar-when-unprivileged = false

[nvidia-container-cli]
root = "/"
path = "/usr/bin/nvidia-container-cli"
environment = []
ldconfig = "@/sbin/ldconfig"

# AL2023
#accept-nvidia-visible-devices-as-volume-mounts = false
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
disable-require = false
supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video"
#swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#debug = "/var/log/nvidia-container-toolkit.log"
environment = []
#ldcache = "/etc/ld.so.cache"
ldconfig = "@/sbin/ldconfig"
load-kmods = true
#no-cgroups = false
#path = "/usr/bin/nvidia-container-cli"
#root = "/run/nvidia/driver"
#user = "root:video"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"
mode = "auto"
runtimes = ["docker-runc", "runc", "crun"]

[nvidia-container-runtime.modes]

[nvidia-container-runtime.modes.cdi]
annotation-prefixes = ["cdi.k8s.io/"]
default-kind = "nvidia.com/gpu"
spec-dirs = ["/etc/cdi", "/var/run/cdi"]

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

[nvidia-container-runtime-hook]
path = "nvidia-container-runtime-hook"
skip-mode-detection = false

[nvidia-ctk]
path = "nvidia-ctk"

However I do not feel like this would be the issue. If I'm right, the nvidia-container-runtime-hook binary is in charge of mounting the files and the configuration file from AL2023 does not seem to set any parameter that would make it different.

There is a specific configuration property that can be set to enable debugging logs for the nvidia container runtime:

debug = "/var/log/nvidia-container-runtime.log"

Enabling that setting actually logs the files that are being mounted on AL2023. On Bottlerocket, it does nothing. I'm wondering maybe the binary is not allowed to write that path but I might be wrong.

Basically my questions would be:

• Do I install the files properly to get the nvidia runtime to access it and be able to mount nvoptix.bin ?
• Is there any way to enable the debug logging properly so I could get some hints about the underlying issues ?

Thanks in advance for any help