Setup CUDA/Nvidia Docker on GCE

cuda_on_gcp_instructions.md

Create Instance

• n1-standard-4 (4 vCPU, 15 GB memory)
• GPUs 1 x NVIDIA Tesla K80
• ubuntu-1804-bionic-v20190617

Install CUDA

# https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.168-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804_10.1.168-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda -y

(it takes about 10 minutes)

echo "export PATH=\"/usr/local/cuda/bin:$PATH\"" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=\"/usr/local/cuda/lib64\"" >> ~/.bashrc
sudo reboot

Install Docker

sudo apt install docker.io

The following packages have unmet dependencies:
 nvidia-docker2 : Depends: docker-ce (= 5:18.09.6~3-0~ubuntu-bionic) but it is not installable or
                           docker-ee (= 5:18.09.6~3-0~ubuntu-bionic) but it is not installable
E: Unable to correct problems, you have held broken packages.

Workaround

sudo apt-get remove docker docker-engine docker.io

sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common
    
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get update
sudo apt-get install docker-ce -y

# change permission to access docker from user
sudo usermod -a -G docker $USER
(re-login)

Install NvidiaDocker

# https://github.com/NVIDIA/nvidia-docker
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
  sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
  sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update

sudo apt-get install -y nvidia-docker2
sudo service docker restart
docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi

Setup persistant disk

# Formating 
# sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

# Mount disk
sudo mkdir -p /mnt/disks/disk1
sudo mount -o discard,defaults /dev/sdb /mnt/disks/disk1
sudo chmod a+w /mnt/disks/disk1

# Remount configuration
echo UUID=`sudo blkid -s UUID -o value /dev/sdb` /mnt/disks/disk1 ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab

See also https://cloud.google.com/compute/docs/disks/add-persistent-disk