NVIDIA Docker Container Runtime CDI w. non-standard NVIDIA Kernel

Docker_CDI_NVIDIA_Non-Standard.md

NVIDIA Docker Container Runtime

nvidia-smi --query-gpu=index,name,uuid,pci.bus_id --format=csv

Enable the Module

sudo modprobe nvidia-uvm

Create the Device Node if it is missing

if [ ! -c /dev/nvidia-uvm ]; then
  sudo mknod -m 666 /dev/nvidia-uvm c $(grep nvidia-uvm /proc/devices | cut -d ' ' -f 1) 0
fi

Generate Runtime Policy

sudo PATH="/opt/nvidia-470.256.02/bin:$PATH" \
nvidia-ctk cdi generate \
  --library-search-path /opt/nvidia-470.256.02/lib \
  --output=/etc/cdi/nvidia.yaml
sudo sed -i 's|containerPath: /opt/nvidia-470.256.02/bin/|containerPath: /usr/bin/|g' /etc/cdi/nvidia.yaml

Docker Compose

Drop runtime: nvidia, gpus: all and CUDA_VISIBLE_DEVICES environment. Use devices insead ie:

        devices:
          - "nvidia.com/gpu=0"
          - "nvidia.com/gpu=1"