Debugging K8s 502 Bad Gateway: Check Your Container First

debugging-k8s-502-bad-gateway.md

Debugging K8s 502 Bad Gateway: Check Your Container First

The Problem

I was getting a 502 Bad Gateway response when accessing my app via the ingress resource.

What I Checked (Naive Debugging Order)

• Ingress config (looked fine)
• Service config (looked fine)
• Assumed the fault was in the k8s configuration

The Actual Issue

During development and testing, I had changed the port of the application and generated a new image based off of it. The K8s Service and Ingress configs were fine, but my application/container had changed its listening port without a corresponding update on the K8s Service's targetPort.

The Service's yaml said:

ports:
  - port: 4444
    targetPort: 3000

The container was actually listening on port 3001

How to Debug This (What I Did)

# 1. confirm pods were running:
kubectl get pods

# 2. exec into the pod's container
kubectl exec -it <pod-name> -c <container-name> -- sh

# If your pod only has one container, you can omit -c:
kubectl exec -it <pod-name> -- sh

# 3. check what ports the app is actually listening on
netstat -tlnp

The Fix

One can either change the targetPort on the service to match the container's port or update the container's port to match the service's targetPort. I went with the former.

Lesson Learned

Always verify the container itself before debugging the k8s networking layers.

Proper Debugging Order

1. Check the container (is the application running and listening on the expected port?)
2. Check the service (does the targetPort match the container's port?)
3. Check the ingress (is it routing to the right Service?)

In a nutshell, you always want to work from the inside out, not outside in.