WSO2-Onprem.md

On-Prem WSO2 Kubernetes Deployment (Free & Open-Source – Tanzu Alternative)

Overview

This document describes a production-grade, fully free and open-source architecture for deploying WSO2 (API Manager / Micro Integrator / IS) on on‑prem VMware infrastructure, without using VMware Tanzu.

This stack is widely used in banks and regulated environments and avoids vendor lock‑in.

---

Goals

• Replace VMware Tanzu with free & open-source components
• Support HA, scalability, and security
• Suitable for on‑prem / air‑gapped environments
• Production-ready for WSO2 APIM

---

Recommended Stack

1. Kubernetes Platform

RKE2 (Rancher Kubernetes Engine 2)

• CNCF‑compliant upstream Kubernetes
• CIS hardened by default
• Ideal for on‑prem VMware

Alternatives:

• kubeadm (pure upstream)
• k3s (lightweight, non‑critical)

---

2. Cluster Management

Rancher (Open Source)

• Multi‑cluster management
• RBAC and audit logs
• Certificate lifecycle management
• Upgrade orchestration

---

3. Networking & Load Balancing

Layer 4 (External IPs)

MetalLB

• Provides LoadBalancer IPs on‑prem
• Works well with VMware networks

Layer 7 (Ingress)

NGINX Ingress Controller

• Stable for WSO2 Gateway traffic
• Supports:
  • TLS termination
  • Sticky sessions
  • WebSockets
  • Large payloads
  • Custom timeouts

⚠ Avoid Traefik for WSO2 APIM due to timeout and WebSocket issues.

---

4. TLS & Certificates

• cert-manager
  • Internal CA or enterprise PKI
  • Automatic certificate rotation

Supports:

• mTLS (Gateway ↔ Backend)
• OAuth endpoints
• External HTTPS access

---

5. Container Registry

Harbor (Open Source)

• CNCF project (originally VMware)
• Role‑based access control
• Vulnerability scanning
• Air‑gapped support

Used for:

• WSO2 Docker images
• Custom integrations

---

6. CI/CD & GitOps (Optional but Recommended)

• GitLab CI or Jenkins
• ArgoCD
  • Git‑based deployment
  • Audit‑friendly
  • Easy rollback

---

7. Observability

Function	Tool
Metrics	Prometheus
Dashboards	Grafana
Logs	Loki
Tracing	Jaeger

WSO2 exposes:

• Prometheus metrics
• Structured logs
• JMX

---

Reference Architecture

VMware ESXi
│
├─ RKE2 Kubernetes Cluster
│   ├─ Rancher
│   ├─ cert-manager
│   ├─ MetalLB
│   ├─ NGINX Ingress
│   ├─ Prometheus / Grafana
│   └─ ArgoCD
│
├─ Harbor Registry
│
└─ WSO2 Platform
    ├─ API Manager (Control Plane)
    ├─ API Gateway
    ├─ Traffic Manager
    ├─ Key Manager
    └─ Analytics (optional)

---

WSO2-Specific Best Practices

Ingress

• Increase proxy timeouts
• Enable sticky sessions for Publisher & DevPortal
• Use separate Ingress for Gateway vs Portal

Database

• Use external HA DB
  • PostgreSQL / Oracle / MSSQL
• Do NOT use in‑cluster DB for production

Scaling

• HPA for Gateway
• Fixed replicas for Control Plane
• Separate namespaces per environment

---

Tanzu vs Open-Source Comparison

Area	Tanzu	Open Source
License Cost	Very High	Free
Kubernetes	TKG	RKE2
Ingress	NSX ALB	NGINX
Registry	Paid	Harbor
Flexibility	Limited	High
Lock‑in	High	None

---

Recommendation

For on‑prem WSO2 deployments:

RKE2 + Rancher + MetalLB + NGINX + Harbor

This stack provides:

• Enterprise stability
• Zero license cost
• Full control and auditability

---

Next Steps

• Create HA WSO2 Helm values
• Tune NGINX Ingress for APIM
• Define DB topology
• Prepare DR strategy

---

Prepared for on‑prem enterprise WSO2 deployments