00-running-multi-arch-images-with-docker-on-lima.md

⚠️ I have written a newer, simpler guide. This gist will not be updated anymore. See https://gist.github.com/fuzmish/df9eabf711c3f452ca19cce0621fc84e.

Running multi-arch images with Docker on Lima

Docker on Lima is one of the alternative solutions to Docker Desktop for Mac. Many people have already described the steps to set up for that. In addition to such a standard setup, this guide describes how to set up a Docker environment which can run multi-arch images. For example, you will be able to run linux/amd64 images on your M1 Mac.

Step 1. Install Lima

1. Assume that Homebrew is already installed:

   $ brew -v
   Homebrew 3.4.9
   

2. Install Lima:

   $ brew install lima
   ...
   
   $ lima -v
   limactl version 0.10.0
   

Step 2. Run Docker Engine on Lima VM

To configure Lima VM itself, default.yaml may be helpful. Here we will explain minimal changes only.

1. Download docker.yaml.

   $ curl -O https://raw.githubusercontent.com/lima-vm/lima/3190530a8e0f10b4000efdebcdfdc2a799da65fa/examples/docker.yaml
   

2. Add the following lines into docker.yaml. This script runs tonistiigi/binfmt to enable QEMU emulation for Docker Engine:

        # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
        systemctl disable --now docker
        apt-get install -y uidmap dbus-user-session
   +- mode: system
   +  script: |
   +    #!/bin/bash
   +    # enable docker cpu emulation
   +    # cf. https://github.com/lima-vm/lima/blob/master/docs/multi-arch.md
   +    docker run --privileged --rm tonistiigi/binfmt --install all
    - mode: user
      script: |
        #!/bin/bash

   My docker.yaml is here.
3. Launch Lima VM:

   $ limactl start --tty=false ./docker.yaml
   

4. Verify your setup:

   # Show the list of VMs
   host$ limactl list docker
   NAME      STATUS     SSH                ARCH       CPUS    MEMORY    DISK      DIR
   docker    Running    127.0.0.1:50084    aarch64    4       4GiB      100GiB    /Users/YourName/.lima/docker
   
   # Enter to the shell of VM (guest)
   host$ limactl shell docker
   
   # Verify the architecture of VM machine
   guest$ uname -a
   Linux lima-docker 5.15.0-25-generic #25-Ubuntu SMP Wed Mar 30 15:57:31 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
   
   # Trying to run the `hello-world` image on Docker
   guest$ docker pull hello-world
   guest$ docker image inspect hello-world --format '{{.Architecture}}'
   arm64
   guest$ docker run --rm hello-world
   ...
    2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
       (arm64v8)
   ...
   
   # The above example runs the image for native architecture.
   # Next, we will run the image for the other architecture (e.g. for Intel CPUs).
   guest$ docker image rm hello-world
   guest$ docker pull --platform linux/amd64 hello-world
   guest$ docker image inspect hello-world --format '{{.Architecture}}'
   amd64
   guest$ docker run --rm --platform linux/amd64 hello-world
   ...
    2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
   ...
   

   Congratulations! Now you have already been able to run both images for Arm (linux/arm64/v8) and for Intel (linux/amd64).

4. Install and Configure Docker CLI

In the previous step, we have been able to use Docker in the Lima VM. Next, we will configure the host to have direct access to the Docker Engine.

1. Install Docker CLI in the host:

   host$ brew install docker
   ...
   
   host$ docker -v
   Docker version 20.10.14, build a224086349
   

2. Configure:
   • Option.a) Create new context:

     host$ docker context create lima \
       --docker "host=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')"

     Then enable the context named lima:

     host$ docker context use lima
     

   • Option.b) Set DOCKER_HOST environment variable; add the following lines into your ~/.zshrc:

     export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')

     Then apply ~/.zshrc changes:

     host$ . ~/.zshrc
     

3. Verify; If you have run the hello-world image in the previous section, it should be listed here:

   host$ docker images
   REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
   hello-world   latest    feb5d9fea6a5   3 months ago   13.3kB
   

5. Install Buildx

We have already installed Buildx in the Lima VM. But it is not available directly from the host.

host$ limactl shell docker docker buildx ls
NAME/NODE  DRIVER/ENDPOINT  STATUS  PLATFORMS
rootless * docker                   
  rootless rootless         running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
...

host$ docker buildx ls
docker buildx ls                          
docker: 'buildx' is not a docker command.
See 'docker --help'

You will need to install Buildx on the host manually.

1. Install (cf. https://github.com/docker/buildx#manual-download):

   host$ mkdir -p ~/.docker/cli-plugins
   host$ curl -fsSL -o ~/.docker/cli-plugins/docker-buildx https://github.com/docker/buildx/releases/download/v0.8.2/buildx-v0.8.2.darwin-arm64
   host$ chmod +x ~/.docker/cli-plugins/docker-buildx
   

2. Verify:

   host$ docker buildx ls
   NAME/NODE DRIVER/ENDPOINT  STATUS  PLATFORMS
   lima *    docker                   
     lima    lima             running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
   ...
   

6. Install Docker Compose V2

1. Install (cf. https://github.com/docker/compose#linux)

   host$ curl -fsSL -o ~/.docker/cli-plugins/docker-compose https://github.com/docker/compose/releases/download/v2.5.1/docker-compose-darwin-aarch64
   host$ chmod +x ~/.docker/cli-plugins/docker-compose
   

2. Verify:

   host$ docker compose version
   Docker Compose version v2.5.1
   

Appendix

• By default, the Lima VM will not start when you boot your Mac. If you want to start the Docker VM automatically, you can use LaunchAgents.

  1. Create ~/Library/LaunchAgents/lima-docker.plist:

     <?xml version="1.0" encoding="UTF-8"?>
     <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
               "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
     <plist version="1.0">
       <dict>
         <key>Label</key>
         <string>launch-lima-docker</string>
         <key>ProgramArguments</key>
         <array>
           <string>/bin/zsh</string>
           <string>-lic</string>
           <string>( limactl stop -f docker; sleep 5; limactl start docker ) > ~/Library/LaunchAgents/lima-docker.plist.log 2>&1</string>
         </array>
         <key>RunAtLoad</key>
         <true/>
         <key>KeepAlive</key>
         <false/>
         <key>AbandonProcessGroup</key>
         <true/>
       </dict>
     </plist>

  2. Load it:

     $ launchctl load ~/Library/LaunchAgents/lima-docker.plist
     

     Then, reboot your mac. The VM docker will start automatically after you login.
  Or, also you can also use crontab.

  $ crontab -e
  
  # Set PATH
  # This is an example. Make sure it is appropriate for your environment.
  PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin
  
  # Automatically starts the `docker` VM on Lima.
  @reboot ( limactl stop -f docker; sleep 5; limactl start docker; ) > /dev/null 2>&1
  

• If you want to show the list of supported architectures, then run:

  host$ limactl shell docker sudo docker run --rm --privileged tonistiigi/binfmt
  {
    "supported": [
      "linux/arm64",
      "linux/amd64",
      "linux/riscv64",
      "linux/ppc64le",
      "linux/s390x",
      "linux/386",
      "linux/mips64le",
      "linux/mips64",
      "linux/arm/v7",
      "linux/arm/v6"
    ],
    "emulators": [
      "qemu-arm",
      "qemu-i386",
      "qemu-mips64",
      "qemu-mips64el",
      "qemu-ppc64le",
      "qemu-riscv64",
      "qemu-s390x",
      "qemu-x86_64"
    ]
  }
  

• Some application will ignore docker contexts. In this case, rootless docker will cause problems. The simplest workaround is to make /var/run/docker.sock as a symlink to the socket of rootless docker. Add the following shell script to docker.yaml:

       dockerd-rootless-setuptool.sh install
       docker context use rootless
  +- mode: user
  +  script: |
  +    #!/bin/bash
  +    set -eux -o pipefail
  +    DOCKER_SOCK_DEFAULT=$(docker context inspect default --format='{{.Endpoints.docker.Host}}' | cut -d '/' -f 3-)
  +    DOCKER_SOCK_ROOTLESS=$(docker context inspect rootless --format='{{.Endpoints.docker.Host}}' | cut -d '/' -f 3-)
  +    sudo rm -f "$DOCKER_SOCK_DEFAULT"
  +    sudo ln -s "$DOCKER_SOCK_ROOTLESS" "$DOCKER_SOCK_DEFAULT"
   probes:
   - script: |
       #!/bin/bash

• Can Docker on Lima be a complete replacement for Docker Desktop for Mac?

  • A quick survey has shown that many of the features that I wanted worked. However, there are still some unstable parts. I am not sure if I can recommend it for production use.
  • File sharing (Bind mount):
    • Since Lima v0.10.0, new mount option mountType: 9p has been added. I have tested it and it seems to be better than reverse-sshfs. However, this feature is still experimental. See lima-vm/lima#726 and others for more details.
    • By setting the writable property to true, we can read and write the host's file system using bind mount.
    • A large amount of access to bind-mounted directories will occasionally cause data corruption. For example, this happened when many intermediate files were output in such a directory during program compilation. You should minimize such access, if possible.
    • I guess this issue is related to the performance issue of bind mount in docker on mac.
      • There is a workaround that changes the consistency option. For example, in devcontainer.json:

         // devcontainer.json
         {
           ...
        +  "workspaceMount": "type=bind,source=${localWorkspaceFolder},target=/workspaces/${localWorkspaceFolderBasename},consistency=delegated",
        +  "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
         }

        The important part is consistency=delegated. cf. https://docker-docs.netlify.app/docker-for-mac/osxfs-caching/
      • However, this workaround doesn't seem to work perfectly well.
  • Network access:
    • Simple test: Run $ docker run --rm -p 3000:80 nginx, then access http://localhost:3000 in your browser (e.g. Safari).
  • The following tools seems to work with this setup:
    • Remote Development for VSCode

docker.yaml

# Example to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine

# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...

# This example requires Lima v0.8.0 or later
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
# - location: "https://cloud-images.ubuntu.com/releases/22.04/release-20220420/ubuntu-22.04-server-cloudimg-amd64.img"
#   arch: "x86_64"
#   digest: "sha256:de5e632e17b8965f2baf4ea6d2b824788e154d9a65df4fd419ec4019898e15cd"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20221101.1/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"
  digest: "sha256:c363ebe9217322a1781fe54d3a7a5bc1cc8af7a5aee2e84ac76200d58077465a"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
# - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
#   arch: "x86_64"
# - location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
#   arch: "aarch64"

# CPUs: if you see performance issues, try limiting cpus to 1.
# Builtin default: 4
cpus: 8

# Memory size
# Builtin default: "4GiB"
memory: "16GiB"

# Expose host directories to the guest, the mount point might be accessible from all UIDs in the guest
mounts:
- location: "~/mycode"
  writable: true

# Mount type for above mounts, such as "reverse-sshfs" (from sshocker) or "9p" (EXPERIMENTAL, from QEMU’s virtio-9p-pci, aka virtfs)
# Builtin default: "reverse-sshfs"
# mountType: "9p"

# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
provision:
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    command -v docker >/dev/null 2>&1 && exit 0
    export DEBIAN_FRONTEND=noninteractive
    curl -fsSL https://get.docker.com | sh
    # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
    systemctl disable --now docker
    apt-get install -y uidmap dbus-user-session
- mode: system
  script: |
    #!/bin/bash
    # enable docker cpu emulation
    # cf. https://github.com/lima-vm/lima/blob/master/docs/multi-arch.md
    docker run --privileged --rm tonistiigi/binfmt --install all
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    dockerd-rootless-setuptool.sh install
    docker context use rootless
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
      echo >&2 "rootlesskit (used by rootless docker) is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
#message: |
#  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
#  ------
#  docker context create lima-{{.Name}} --docker "host=unix://{{.Dir}}/sock/docker.sock"
#  docker context use lima-{{.Name}}
#  docker run hello-world
#  ------