mcp-header-forwarding-workshop.md

MCP Header Forwarding with Agent Gateway

This workshop demonstrates how agent gateway forwards client HTTP headers (including Authorization: Bearer tokens) to upstream MCP servers. It covers two scenarios:

1. Without JWT auth — headers pass through by default, no configuration needed.
2. With JWT auth — the gateway validates and strips the Authorization header by default. Adding policies.auth.passthrough: {} to the backend re-attaches it.

Prerequisites

• An EKS cluster with Solo Enterprise Agent Gateway installed
• kubectl configured and connected to the cluster

---

Part 1: Header Forwarding Without JWT Auth

This part shows that headers pass through by default when no JWT auth policy is configured.

Step 1: Deploy a Header Echo Server

Deploy a simple server that logs all received HTTP headers. This acts as a stand-in for your MCP server so we can verify what headers arrive upstream.

kubectl apply -n enterprise-agentgateway -f- <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: header-echo
  labels:
    app: header-echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: header-echo
  template:
    metadata:
      labels:
        app: header-echo
    spec:
      containers:
      - name: logger
        image: python:3.11-slim
        command:
        - python3
        - -c
        - |
          from http.server import HTTPServer, BaseHTTPRequestHandler
          import json, sys

          class Handler(BaseHTTPRequestHandler):
              def do_POST(self):
                  length = int(self.headers.get('Content-Length', 0))
                  body = self.rfile.read(length).decode() if length else ''
                  print(f"\n=== REQUEST ===", flush=True)
                  print(f"Method: {self.command} {self.path}", flush=True)
                  print(f"Headers:", flush=True)
                  for k, v in self.headers.items():
                      print(f"  {k}: {v}", flush=True)
                  print(f"Body: {body[:500]}", flush=True)
                  print(f"===============", flush=True)
                  resp = json.dumps({"jsonrpc": "2.0", "result": {"tools": []}, "id": 1})
                  self.send_response(200)
                  self.send_header('Content-Type', 'application/json')
                  self.end_headers()
                  self.wfile.write(resp.encode())
              def do_GET(self):
                  self.do_POST()

          print("Header echo server listening on :9090", flush=True)
          HTTPServer(('', 9090), Handler).serve_forever()
        ports:
        - containerPort: 9090
---
apiVersion: v1
kind: Service
metadata:
  name: header-echo
  labels:
    app: header-echo
spec:
  selector:
    app: header-echo
  ports:
  - name: mcp
    port: 9090
    targetPort: 9090
    appProtocol: kgateway.dev/mcp
EOF

Wait for the pod to be ready:

kubectl wait --for=condition=ready pod -l app=header-echo \
  -n enterprise-agentgateway --timeout=60s

Step 2: Create an MCP Backend and HTTPRoute

kubectl apply -n enterprise-agentgateway -f- <<'EOF'
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: header-test-mcp
spec:
  mcp:
    targets:
    - name: header-echo
      static:
        host: header-echo.enterprise-agentgateway.svc.cluster.local
        port: 9090
        path: /mcp
        protocol: StreamableHTTP
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: header-test
spec:
  parentRefs:
    - name: agentgateway-proxy
      namespace: enterprise-agentgateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /header-test
    backendRefs:
    - name: header-test-mcp
      namespace: enterprise-agentgateway
      group: agentgateway.dev
      kind: AgentgatewayBackend
EOF

Verify both resources are accepted:

kubectl get agentgatewaybackend header-test-mcp -n enterprise-agentgateway \
  -o jsonpath='{.status.conditions[0].reason}' && echo ""
kubectl get httproute header-test -n enterprise-agentgateway \
  -o jsonpath='{.status.parents[0].conditions[0].reason}' && echo ""

Step 3: Send a Request with a Bearer Token

curl -s -X POST "http://localhost:8080/header-test" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Authorization: Bearer my-secret-test-token-12345" \
  -H "mcp-protocol-version: 2024-11-05" \
  -d '{
    "jsonrpc": "2.0",
    "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "test", "version": "1.0"}
    },
    "id": 1
  }'

Step 4: Verify the Header Arrived at the MCP Server

kubectl logs -n enterprise-agentgateway -l app=header-echo -c logger --tail=20

Expected output:

=== REQUEST ===
Method: POST /mcp
Headers:
  content-type: application/json
  accept: text/event-stream, application/json
  authorization: Bearer my-secret-test-token-12345
  mcp-protocol-version: 2024-11-05
  host: header-echo.enterprise-agentgateway.svc.cluster.local:9090
  ...
===============

The authorization: Bearer my-secret-test-token-12345 header was forwarded from the client through the agent gateway to the upstream MCP server.

---

Part 2: Header Forwarding With JWT Auth + Passthrough

When a gateway-level JWT auth policy is configured (EnterpriseAgentgatewayPolicy with jwtAuthentication), the gateway validates the JWT and strips the Authorization header before forwarding upstream. This is the default behavior.

To re-attach the token, add policies.auth.passthrough: {} to the AgentgatewayBackend.

Step 5: Add a JWT Auth Policy

If you don't already have one, create a JWT auth policy on the gateway. This example uses Auth0:

kubectl apply -n enterprise-agentgateway -f- <<'EOF'
apiVersion: enterpriseagentgateway.solo.io/v1alpha1
kind: EnterpriseAgentgatewayPolicy
metadata:
  name: jwt-auth
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: agentgateway-proxy
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
      - issuer: "https://YOUR_AUTH0_DOMAIN/"
        jwks:
          remote:
            backendRef:
              group: agentgateway.dev
              kind: AgentgatewayBackend
              name: YOUR_JWKS_BACKEND
              namespace: enterprise-agentgateway
            cacheDuration: "5m"
            jwksPath: "/.well-known/jwks.json"
EOF

Step 6: Test Without Passthrough (header is stripped)

With JWT auth enabled but no passthrough on the backend, the Authorization header is stripped after validation.

Get a valid JWT from your identity provider:

TOKEN=$(curl -s --request POST \
  --url "https://YOUR_AUTH0_DOMAIN/oauth/token" \
  --header 'content-type: application/json' \
  --data '{
    "client_id": "YOUR_CLIENT_ID",
    "client_secret": "YOUR_CLIENT_SECRET",
    "audience": "YOUR_AUDIENCE",
    "grant_type": "client_credentials"
  }' | jq -r '.access_token')

Send the request:

curl -s -X POST "http://localhost:8080/header-test" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "mcp-protocol-version: 2024-11-05" \
  -d '{
    "jsonrpc": "2.0",
    "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "test", "version": "1.0"}
    },
    "id": 1
  }'

Check the echo server logs:

kubectl logs -n enterprise-agentgateway -l app=header-echo -c logger --tail=20

Expected output — no authorization header:

=== REQUEST ===
Method: POST /mcp
Headers:
  content-type: application/json
  accept: text/event-stream, application/json
  mcp-protocol-version: 2024-11-05
  host: header-echo.enterprise-agentgateway.svc.cluster.local:9090
  ...
===============

Step 7: Enable Passthrough on the Backend

Update the AgentgatewayBackend to add policies.auth.passthrough: {}:

kubectl apply -n enterprise-agentgateway -f- <<'EOF'
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: header-test-mcp
spec:
  mcp:
    targets:
    - name: header-echo
      static:
        host: header-echo.enterprise-agentgateway.svc.cluster.local
        port: 9090
        path: /mcp
        protocol: StreamableHTTP
  policies:
    auth:
      passthrough: {}
EOF

Verify the backend is accepted:

kubectl get agentgatewaybackend header-test-mcp -n enterprise-agentgateway \
  -o jsonpath='{.status.conditions[0].reason}' && echo ""

Step 8: Test With Passthrough (header is forwarded)

Send the same request again:

curl -s -X POST "http://localhost:8080/header-test" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "mcp-protocol-version: 2024-11-05" \
  -d '{
    "jsonrpc": "2.0",
    "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "test", "version": "1.0"}
    },
    "id": 1
  }'

Check the echo server logs:

kubectl logs -n enterprise-agentgateway -l app=header-echo -c logger --tail=20

Expected output — authorization header is present:

=== REQUEST ===
Method: POST /mcp
Headers:
  content-type: application/json
  accept: text/event-stream, application/json
  authorization: Bearer eyJhbGciOiJSUzI1NiIs...
  mcp-protocol-version: 2024-11-05
  host: header-echo.enterprise-agentgateway.svc.cluster.local:9090
  ...
===============

The Authorization: Bearer token was validated by the gateway's JWT policy and then re-attached to the upstream request by policies.auth.passthrough.

---

Cleanup

kubectl delete httproute header-test -n enterprise-agentgateway
kubectl delete agentgatewaybackend header-test-mcp -n enterprise-agentgateway
kubectl delete deployment header-echo -n enterprise-agentgateway
kubectl delete svc header-echo -n enterprise-agentgateway
# If you created a JWT policy for this workshop:
# kubectl delete enterpriseagentgatewaypolicy jwt-auth -n enterprise-agentgateway

Summary

Scenario	Authorization forwarded upstream?	Config needed
No JWT policy	Yes	None
JWT policy, no passthrough	No (stripped after validation)	—
JWT policy + policies.auth.passthrough: {}	Yes	Add policies.auth.passthrough: {} to AgentgatewayBackend

The key configuration is policies.auth.passthrough: {} on the AgentgatewayBackend. When JWT authentication validates and strips the Authorization header, passthrough re-attaches the original token before forwarding the request to the upstream MCP server. This is useful when your MCP servers need the client's identity for fine-grained authorization, row-level access control, or downstream token forwarding.