Diego: Full mTLS for App-to-App Traffic

Executive Summary

This document describes the current state of Envoy proxy integration in Diego and outlines what would be needed to implement full mutual TLS (mTLS) for container-to-container (C2C) traffic, enabling apps to both authenticate themselves and verify the identity of connecting apps.

The proposed approach introduces a new HTTP-based listener on port 62443 that runs alongside the existing TCP-based C2C port (61443), providing a dual opt-in model for both operators and app authors.

Current Architecture

Envoy as Ingress Proxy (Implemented)

Diego runs Envoy as a per-container sidecar for handling inbound traffic only:

                              ┌──────────────────────────────────┐
                              │         Container                │
                              │                                  │
Gorouter ──[TLS]──►  Envoy (port 61001) ──[plain TCP]──► App (8080)
                              │                                  │
                              │  Envoy only handles INBOUND      │
                              │  traffic (DownstreamTlsContext)  │
                              └──────────────────────────────────┘

The TLS context is configured as downstream-only in proxy_config_handler.go:547:

tlsContext := &envoy_tls.DownstreamTlsContext{

Port Allocation

Port	Type	Purpose	Source
61001+	HTTP	TLS proxy ports for route integrity (one per app port)	`proxy_config_handler.go:42`
61443	TCP	C2C TLS traffic (fixed, only for port 8080)	`proxy_config_handler.go:44`

The C2C port is only allocated for apps using port 8080 (proxy_config_handler.go:190):

if containerPorts[portCount] == DefaultHTTPPort {

Two Types of Certificates

Diego generates two separate certificates per container in credmanager.go:

1. Instance Identity Certificate

SAN: Container IP address (credmanager.go:297)
Purpose: Route integrity (Gorouter → App)
SDS File: sds-id-cert-and-key.yaml (proxy_config_handler.go:267)
Supports mTLS: Yes (optional, configurable)
Supports HTTP/2: Yes (via ALPN)

certificateSAN{IPAddress: ipForCert, OrganizationalUnits: container.CertificateProperties.OrganizationalUnit},

2. C2C Certificate

SAN: Internal route hostnames (e.g., app-name.apps.internal) (credmanager.go:328)
Purpose: Container-to-container TLS
SDS File: sds-c2c-cert-and-key.yaml (proxy_config_handler.go:268)
Supports mTLS: No (not configured)
Supports HTTP/2: No

certificateSAN{InternalRoutes: container.InternalRoutes, OrganizationalUnits: container.CertificateProperties.OrganizationalUnit},

Identity Embedded in Certificates

Both certificates include app identity in the OrganizationalUnit field.

The certificate template is created in credmanager.go:427-451:

func createCertificateTemplate(guid string, certSAN certificateSAN, notBefore, notAfter time.Time) *x509.Certificate {
    // ...
    return &x509.Certificate{
        SerialNumber: big.NewInt(0),
        Subject: pkix.Name{
            CommonName:         guid,  // Instance GUID
            OrganizationalUnit: certSAN.OrganizationalUnits,  // App identity
        },
        // ...
    }
}

The OrganizationalUnit contains claims like:

app:some-app-guid
space:some-space-guid  
organization:some-org-guid

These are set by Cloud Controller and passed through the BBS CertificateProperties:

type CertificateProperties struct {
    OrganizationalUnit []string `protobuf:"bytes,1,rep,name=organizational_unit,json=organizationalUnit,proto3" json:"organizational_unit,omitempty"`
}

The internal routes structure is defined in routing_info_helpers.go:11-15:

type InternalRoutes []InternalRoute

type InternalRoute struct {
    Hostname string `json:"hostname"`
}

DNS names are added to C2C certificates in credmanager.go:435-436:

for _, route := range certSAN.InternalRoutes {
    dnsNames = append(dnsNames, route.Hostname)
}

Environment Variables for Apps

Apps can access their instance identity credentials via environment variables, set in instance_identity_handler.go:43:

{Name: "CF_INSTANCE_CERT", Value: path.Join(h.containerMountPath, "instance.crt")},
{Name: "CF_INSTANCE_KEY", Value: path.Join(h.containerMountPath, "instance.key")},

Current Limitations

1. No Client-Side (Egress) Proxy

Envoy is not configured as an egress proxy. Apps cannot automatically inject their instance identity certificate when making outbound connections.

Current workaround: Apps must manually use CF_INSTANCE_CERT and CF_INSTANCE_KEY environment variables to present their identity.

2. No Identity Extraction on Server Side

Even when mTLS is enabled, the receiving app has no way to know who connected. Envoy does not forward client certificate information via headers (no XFCC header configuration).

3. C2C Port (61443) is TCP-Based

The C2C listener (port 61443) is a TCP proxy, not an HTTP proxy. This means:

No HTTP connection manager
No XFCC header support
No mTLS client certificate validation

This is explicitly configured in proxy_config_handler.go:569:

if requireClientCerts && portMap.ContainerTLSProxyPort != C2CTLSPort {
    tlsContext.RequireClientCertificate = ...
}

Similarly, HTTP/2 is disabled for C2C in proxy_config_handler.go:565:

if http2Enabled && portMap.ContainerTLSProxyPort != C2CTLSPort {
    tlsContext.CommonTlsContext.AlpnProtocols = AlpnProtocols
}

4. BOSH Properties for mTLS

The existing mTLS BOSH property (jobs/rep/spec:266) only affects route integrity ports, not C2C:

containers.proxy.require_and_verify_client_certificates:
  description: "If true, the proxy for each container will require and verify client
    certificates for all HTTP ingress connections."
  default: false

What Full mTLS Would Enable

Automatic identity injection: Apps don't need to handle certificates manually
Identity verification: Server apps can verify and extract caller identity
Zero-trust networking: All app-to-app traffic is authenticated
Policy enforcement: Authorization decisions based on app/space/org identity

Implementation Plan

Design Principles

Backwards compatibility: Keep port 61443 unchanged for existing apps
Dual opt-in: Both operator and app author must opt-in
New port for new behavior: Port 62443 for HTTP-based mTLS with XFCC

Proposed Port Allocation

Port	Type	Purpose	mTLS	XFCC	Status
61001+	HTTP	Route integrity	Optional	No	Existing
61443	TCP	Legacy C2C TLS	No	No	Existing (unchanged)
62443	HTTP	C2C mTLS with identity	Required	Yes	New
61445	HTTP	Egress proxy (client-side)	N/A	N/A	New

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│ Container A (Client)                                                     │
│                                                                          │
│ Option 1: Legacy (no identity)                                          │
│   App A ──► manually use CF_INSTANCE_CERT ──► port 61443 on Container B │
│                                                                          │
│ Option 2: mTLS with identity (new)                                      │
│   App A ──► Envoy egress:61445 ──[auto cert injection]──► 62443 on B   │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│ Container B (Server)                                                     │
│                                                                          │
│ Port 61443 (TCP, legacy):                                               │
│   - TLS termination only                                                │
│   - No client cert validation                                           │
│   - No XFCC header                                                      │
│   - Unchanged behavior                                                  │
│                                                                          │
│ Port 62443 (HTTP, new):                                                 │
│   - HTTP connection manager                                             │
│   - mTLS required (validates client cert)                               │
│   - XFCC header forwarded to app                                        │
│   - App sees: X-Forwarded-Client-Cert: Subject="OU=app:xxx"             │
└─────────────────────────────────────────────────────────────────────────┘

Phase 1: Server-Side - New HTTP-based C2C mTLS Port (62443)

Add a new Envoy listener on port 62443 that provides HTTP-based C2C with mTLS and XFCC header forwarding.

Changes Required

1. Add New Port Constant

In proxy_config_handler.go:

const (
    StartProxyPort  = 61001
    C2CTLSPort      = 61443  // Existing TCP-based C2C (unchanged)
    C2CMTLSPort     = 62443  // New HTTP-based C2C with mTLS
    DefaultHTTPPort = 8080
)

2. Create New HTTP-based Listener for 62443

Add a new listener with HTTP connection manager, mTLS, and XFCC support:

func (p *proxyConfigHandler) createC2CMTLSListener(container executor.Container) *envoy_listener.Listener {
    return &envoy_listener.Listener{
        Name:    "c2c-mtls-listener",
        Address: envoyAddr(container.InternalIP, C2CMTLSPort),
        FilterChains: []*envoy_listener.FilterChain{{
            TransportSocket: &envoy_core.TransportSocket{
                Name: "envoy.transport_sockets.tls",
                TypedConfig: marshalAny(&envoy_tls.DownstreamTlsContext{
                    CommonTlsContext: &envoy_tls.CommonTlsContext{
                        TlsCertificateSdsSecretConfigs: []*envoy_tls.SdsSecretConfig{{
                            Name: "c2c-cert-and-key",
                            SdsConfig: sdsConfig,
                        }},
                        ValidationContextSdsSecretConfig: &envoy_tls.SdsSecretConfig{
                            Name: "id-validation-context",  // Trust instance identity CA
                            SdsConfig: sdsConfig,
                        },
                    },
                    RequireClientCertificate: &wrapperspb.BoolValue{Value: true},
                }),
            },
            Filters: []*envoy_listener.Filter{{
                Name: "envoy.filters.network.http_connection_manager",
                TypedConfig: marshalAny(&http_conn_mgr.HttpConnectionManager{
                    StatPrefix: "c2c-mtls-ingress",
                    HttpFilters: []*http_conn_mgr.HttpFilter{{
                        Name: "envoy.filters.http.router",
                        TypedConfig: marshalAny(&envoy_router.Router{}),
                    }},
                    ForwardClientCertDetails: http_conn_mgr.HttpConnectionManager_SANITIZE_SET,
                    SetCurrentClientCertDetails: &http_conn_mgr.HttpConnectionManager_SetCurrentClientCertDetails{
                        Subject: &wrapperspb.BoolValue{Value: true},
                        Uri:     true,
                        Dns:     true,
                    },
                    RouteSpecifier: &http_conn_mgr.HttpConnectionManager_RouteConfig{
                        RouteConfig: &envoy_route.RouteConfiguration{
                            VirtualHosts: []*envoy_route.VirtualHost{{
                                Name:    "c2c-mtls",
                                Domains: []string{"*"},
                                Routes: []*envoy_route.Route{{
                                    Match: &envoy_route.RouteMatch{
                                        PathSpecifier: &envoy_route.RouteMatch_Prefix{Prefix: "/"},
                                    },
                                    Action: &envoy_route.Route_Route{
                                        Route: &envoy_route.RouteAction{
                                            ClusterSpecifier: &envoy_route.RouteAction_Cluster{
                                                Cluster: "app-cluster-8080",
                                            },
                                        },
                                    },
                                }},
                            }},
                        },
                    },
                }),
            }},
        }},
    }
}

3. Add BOSH Properties

In jobs/rep/spec:

containers.proxy.enable_c2c_mtls_listener:
  description: "Enable the HTTP-based C2C mTLS listener on port 62443. This listener 
    requires client certificates and forwards caller identity via the X-Forwarded-Client-Cert header."
  default: false

containers.proxy.c2c_mtls_port:
  description: "Port for the HTTP-based C2C mTLS listener with XFCC support"
  default: 62443

4. Reserve Port 62443

Similar to 61443, add validation to prevent apps from using port 62443:

var ErrC2CMTLSPortIsReserved = fmt.Errorf("port %d is reserved for container networking with mTLS", C2CMTLSPort)

if port == C2CMTLSPort {
    return nil, nil, ErrC2CMTLSPortIsReserved
}

Result

When an operator enables containers.proxy.enable_c2c_mtls_listener:

Port 62443 becomes available on all containers
Clients connecting to 62443 must present a valid instance identity certificate
The receiving app gets an X-Forwarded-Client-Cert header with caller identity

X-Forwarded-Client-Cert: Hash=abc123;Subject="CN=instance-guid,OU=app:client-app-guid,OU=space:space-guid"

Phase 2: Client-Side Egress Proxy (Port 61445)

Enable Envoy to act as an HTTP proxy for outbound connections, automatically injecting the instance identity certificate.

Architecture

┌───────────────────────────────────────────────────────────────────┐
│ Container A                                                        │
│                                                                    │
│  App A ─────► Envoy (egress:61445) ────────────────┐              │
│               HTTP_PROXY=localhost:61445            │              │
│               Injects instance identity cert        │              │
└─────────────────────────────────────────────────────┼──────────────┘
                                                      │ mTLS with cert containing:
                                                      │   OU=app:client-app-guid
                                                      │   OU=space:space-guid
                                                      ▼
                                            Container B:62443

Changes Required

1. Reserve Egress Proxy Port

In proxy_config_handler.go:

const EgressProxyPort = 61445

2. Add Egress Listener

Create a new listener that acts as an HTTP CONNECT proxy:

egressListener := &envoy_listener.Listener{
    Name: "egress-proxy",
    Address: envoyAddr("127.0.0.1", EgressProxyPort),
    FilterChains: []*envoy_listener.FilterChain{{
        Filters: []*envoy_listener.Filter{{
            Name: "envoy.filters.network.http_connection_manager",
            TypedConfig: marshalAny(&http_conn_mgr.HttpConnectionManager{
                StatPrefix: "egress-proxy",
                HttpFilters: []*http_conn_mgr.HttpFilter{{
                    Name: "envoy.filters.http.dynamic_forward_proxy",
                    // ... dynamic forward proxy config
                }},
            }),
        }},
    }},
}

3. Configure UpstreamTlsContext with Instance Identity

cluster := &envoy_cluster.Cluster{
    Name: "dynamic_forward_proxy",
    TransportSocket: &envoy_core.TransportSocket{
        Name: "envoy.transport_sockets.tls",
        TypedConfig: marshalAny(&envoy_tls.UpstreamTlsContext{
            CommonTlsContext: &envoy_tls.CommonTlsContext{
                TlsCertificateSdsSecretConfigs: []*envoy_tls.SdsSecretConfig{{
                    Name: "id-cert-and-key",  // Reuse instance identity cert
                    SdsConfig: sdsConfig,
                }},
            },
        }),
    },
}

4. Optionally Inject HTTP_PROXY Environment Variable

In containerstore.go, optionally add (if operator enables automatic injection):

env = append(env, executor.EnvironmentVariable{
    Name:  "CF_INSTANCE_MTLS_PROXY",
    Value: fmt.Sprintf("http://127.0.0.1:%d", EgressProxyPort),
})

Note: Using a CF-specific variable (CF_INSTANCE_MTLS_PROXY) rather than HTTP_PROXY allows apps to opt-in explicitly without affecting all outbound traffic.

5. Add BOSH Properties

In jobs/rep/spec:

containers.proxy.enable_egress_proxy:
  description: "Enable Envoy egress proxy for outbound mTLS on port 61445"
  default: false

containers.proxy.egress_proxy_port:
  description: "Port for the egress HTTP proxy"
  default: 61445

Considerations

SNI Handling: Envoy needs to extract the target hostname for proper TLS handshake
NO_PROXY: Apps should configure NO_PROXY for traffic that shouldn't go through the proxy
Non-HTTP Traffic: TCP-based protocols won't work through HTTP CONNECT
Performance: Additional hop adds latency
Memory: Additional memory allocation may be needed (see jobs/rep/spec:259)

Phase 3: Full Integration

Complete mTLS Flow

┌─────────────────────────────────────────────────────────────────────────┐
│ Container A (Client)                                                     │
│ Cert OU: app:aaa-guid, space:xxx                                        │
│                                                                          │
│ App A ──HTTP──► Envoy egress:61445 ────────────────┐                    │
│                 (attaches instance cert)            │                    │
└─────────────────────────────────────────────────────┼────────────────────┘
                                                      │ mTLS
                                                      │ Client cert:
                                                      │   CN=instance-guid
                                                      │   OU=app:aaa-guid
                                                      │   OU=space:xxx
                                                      ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Container B (Server)                                                     │
│                                                                          │
│ Envoy ingress:62443 ─────────────────────────────────────► App B        │
│  - Validates client cert against instance identity CA                   │
│  - Adds XFCC header:                                                     │
│    X-Forwarded-Client-Cert: Subject="OU=app:aaa-guid,OU=space:xxx"      │
│                                                                          │
│ App B reads XFCC header to authorize request based on caller identity  │
└─────────────────────────────────────────────────────────────────────────┘

Opt-in Model

Actor	How to Opt-in
Operator	Set `containers.proxy.enable_c2c_mtls_listener: true` and `containers.proxy.enable_egress_proxy: true`
Server App	Listen for connections and read `X-Forwarded-Client-Cert` header
Client App	Connect to port 62443 (directly or via egress proxy at 61445)

App-Level Authorization

Apps can parse the XFCC header to make authorization decisions:

func extractCallerIdentity(r *http.Request) (appGUID, spaceGUID string) {
    xfcc := r.Header.Get("X-Forwarded-Client-Cert")
    // Parse Subject field for OU values
    // OU=app:xxx → appGUID
    // OU=space:yyy → spaceGUID
    return appGUID, spaceGUID
}

func handler(w http.ResponseWriter, r *http.Request) {
    appGUID, spaceGUID := extractCallerIdentity(r)
    
    if !isAuthorized(appGUID, spaceGUID) {
        http.Error(w, "Forbidden", 403)
        return
    }
    // Process request...
}

Summary

Feature	Current State	Phase 1	Phase 2	Phase 3
Server-side TLS termination	✅	✅	✅	✅
App identity in certs (OU)	✅	✅	✅	✅
mTLS on route integrity ports	✅ (optional)	✅	✅	✅
Legacy C2C port (61443)	✅ TCP-based	✅ unchanged	✅ unchanged	✅ unchanged
New C2C mTLS port (62443)	❌	✅	✅	✅
XFCC header forwarding	❌	✅ (on 62443)	✅	✅
Client-side egress proxy (61445)	❌	❌	✅	✅
Automatic cert injection	❌	❌	✅	✅
Full zero-trust app-to-app	❌	❌	❌	✅

Benefits of This Approach

Aspect	Benefit
Backwards compatible	Port 61443 unchanged, existing apps continue to work
Operator opt-in	New BOSH properties to enable 62443 and 61445 listeners
App author opt-in	Must explicitly connect to 62443 to use mTLS with identity
Clear semantics	62443 = "secure C2C with identity verification"
Gradual migration	Apps can migrate one at a time
No breaking changes	All existing functionality preserved

Key Files to Modify

File	Purpose	Repository	Link
`proxy_config_handler.go`	Envoy config generation	cloudfoundry/executor	Source
`containerstore.go`	Environment variable injection	cloudfoundry/executor	Source
`credmanager.go`	Certificate generation	cloudfoundry/executor	Source
`instance_identity_handler.go`	Environment variable setup	cloudfoundry/executor	Source
`certificate_properties.pb.go`	CertificateProperties model	cloudfoundry/bbs	Source
`routing_info_helpers.go`	InternalRoutes structure	cloudfoundry-incubator/routing-info	Source
`jobs/rep/spec`	BOSH property definitions	cloudfoundry/diego-release	Source
`initializer.go`	Rep initialization config	cloudfoundry/executor	Source

References

Envoy XFCC Header Documentation
Envoy Dynamic Forward Proxy
Diego Envoy Proxy Configuration
Diego Instance Identity Test - Integration tests showing certificate usage

rkoster/README.md

Select an option

No results found

Select an option

No results found

Diego: Full mTLS for App-to-App Traffic

Executive Summary

Current Architecture

Envoy as Ingress Proxy (Implemented)

Port Allocation

Two Types of Certificates

1. Instance Identity Certificate

2. C2C Certificate

Identity Embedded in Certificates

Environment Variables for Apps

Current Limitations

1. No Client-Side (Egress) Proxy

2. No Identity Extraction on Server Side

3. C2C Port (61443) is TCP-Based

4. BOSH Properties for mTLS

What Full mTLS Would Enable

Implementation Plan

Design Principles

Proposed Port Allocation

Architecture Overview

Phase 1: Server-Side - New HTTP-based C2C mTLS Port (62443)

Changes Required

Result

Phase 2: Client-Side Egress Proxy (Port 61445)

Architecture

Changes Required

Considerations

Phase 3: Full Integration

Complete mTLS Flow

Opt-in Model

App-Level Authorization

Summary

Benefits of This Approach

Key Files to Modify

References