tldr;
Kubernetes Deployments are suitable in most cases.
For each tenant, deploy an NGINX IC cluster of three or more production instances using Kubernetes Deployments. For each non-production use case, deploy a single Ingress Controller cluster instance using a Kubernetes Deployment (for ease of management).
After working with several enterprise deployments of Kubernetes Ingress, It’s apparent that there is a common pattern deployment of NGINX OSS and NGINX Plus. Of course, “one size does not fit all,” and every deployment must consider its unique use case, system requirements, and licensing (for commercial solutions)!
How is NGINX Ingress Controller typically deployed?
Kubernetes Deployments are suitable in most cases.
It is unnecessary to run an Ingress Controller instance on every node
(DaemonSet.
This is wasteful of resources, as each Ingress Controller instance has a very
high throughput. See NGINX Ingress Controller Sizing
Guide.
Use a
[Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
for stateless services, like frontends, where scaling up and down the number of
replicas and rolling out updates is more important than controlling exactly
which host the Pod runs on.
How many instances of NGINX Plus Ingress Controller are required?
I recommend deploying three instances to ensure smooth operation and avoid single points of failure. If the CPU utilization exceeds 50% of the allocation, scale vertically. However, if you need to allocate more than eight cores per instance, also scale horizontally to spread the load across the cluster since other resource limits (like networking) may apply on a per-instance basis.
What about Multi-Tenant deployments?
I strongly recommend deploying separate NGINX Ingress Controller clusters per-tenant or per-high-traffic application, separated by namespaces. This is for several reasons:
- Improved security per tenant (no shared resources)
- Individual sizing and scaling decisions per-tenant
- Isolation - traffic spike or configuration errors on one tenant is constrained to that tenant’s services
- Simpler configuration with fewer reloads in each NGINX IC cluster
- Load-sharing across multiple Kubernetes nodes
- Easier distribution of services, e.g., monitoring that may not be Kubernetes RBAC-compliant
This is imperative when you are utilizing services that require high levels of coordination between users, such as annotations and snippets. It can also become an issue when using CPU-intensive features, like an integrated WAF (critical) or TLS termination (important).
Summary
For each tenant, deploy an NGINX IC cluster of three or more production instances using Kubernetes Deployments. For each non-production use case, deploy a single Ingress Controller cluster instance using a Kubernetes Deployment (for ease of management).
Example Instance estimator:
- Production: 3 * number-of-production-tenants * number-of-K8s-clusters
- Non-production: 1 * number-of-non-production-use-cases