Buildkit TCP service behind Envoy Proxy

00_README.md

Architecture

              +-------+     +-----------+
buidctl <---->+ Envoy +<--->+ Buildkitd +
              +-------+     +-----------+

How to Try

$ git clone https://gist.github.com/ea9582347e2c9d0bb519cdae1b438ead.git envoy-builkitd
$ cd envoy-builkitd

$ docker-comopse up
...you see logs...

• envoy listens: localhost:1234
• envoy forwards requests to buildkitd
• buildkitd also listens localhost:11234 for debugging

Issue

builctl with direct access to buildkit works.

$ buildctl --debug --addr tcp://127.0.0.1:11234 du

ID      RECLAIMABLE     SIZE    LAST ACCESSED
Reclaimable:    0B
Total:          0B

But buildctl can connect to buildkit through Envoy.

$ buildctl --debug --addr tcp://127.0.0.1:1234 du
ID	RECLAIMABLE	SIZE	LAST ACCESSED
Reclaimable:	0B
Total:		0B

docker-compose.yml

version: '3'
services:
  envoy:
    image: envoyproxy/envoy:v1.14.2
    command: "--config-path /etc/envoy/envoy-config.yaml --log-level debug"
    volumes:
     - .:/etc/envoy
    ports:
     - "1234:1234" # buidkitd
     - "8001:8001" # envoy admin
    links:
    - buildkitd
  buildkitd:
    image: moby/buildkit:rootless
    # listens on 0.0.0.0 for debug
    command: "--debug --addr=tcp://0.0.0.0:11234 --oci-worker-no-process-sandbox"
    ports:
    - "11234:11234" # direct access (for clarifying buildkitd works)
    security_opt: 
    - seccomp:unconfined
    - apparmor:unconfined

envoy-config.yaml

static_resources:
  listeners:
    - name: listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 1234
      filter_chains:
        - filters:
            - name: envoy.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
                stat_prefix: tcp_local_service
                cluster: local_service
  clusters:
    - name: local_service
      connect_timeout: 0.250s
      type: strict_dns
      lb_policy: round_robin
      hosts:
        - socket_address:
            address: buildkitd
            port_value: 11234
admin:
  access_log_path: /dev/stdout
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001