Introduction

As infrastructure administrators, we're used to carefully carving the network into VLANs, VRFs, and strict firewall zones. However, Kubernetes® was designed a bit differently, focusing on ephemeral workloads and rapid scaling. This post walks through that design from an infra-admin’s seat, explaining what the pod network actually is, how the Container Network Interface (CNI) fits in, and where your existing networking skills matter.

Open a new Kubernetes cluster for the first time. Look at the pod IPs. Every pod receives an address from the cluster’s configured Pod CIDR or address pool, often using private address space e.g. 10.x.x.x address. No VLAN tags, no port-channels, no spanning tree, just a flat Layer 3 network where every workload can reach every other workload by default.

If you spent the last decade carving the network into VLANs, VRFs, and firewall zones, this view feels wrong. The "everything is open" model goes against most of what network engineers were taught.

It is not wrong. It is simply a different design, built around a different problem. This post walks through that design from an infra-admin seat. We will explore what the pod network actually is, how the Container Network Interface (CNI) fits in, what Services and Ingress really do, and—most importantly—where your existing networking skills matter.

The Pod Network Model

Kubernetes makes one big assumption about the network: Every pod gets its own IP, and every pod can talk to every other pod, on any node, without NAT.

That one rule changes a lot. In a VM world, you usually put workloads behind a NAT or hide them inside private VLANs. Two web servers on different hosts often talk through a load balancer or a routed segment. There is plumbing in-between. In a pod network, that plumbing is gone. Pod A on Node 1 talks to Pod B on Node 5 the same way Pod A would talk to a pod sitting right next to it. Same IP space. Same routing. Same firewall rules, or absence of them.

This flat-network idea is not just a design preference; it is a hard requirement. Kubernetes will not work properly if pod IPs are not routable across nodes.

Fig 01. A side-by-side comparison contrasting the isolated, VLAN-segmented networking of traditional VMs with the flat, default-open networking model of Kubernetes pods Fig 01. A side-by-side comparison contrasting the isolated, VLAN-segmented networking of traditional VMs with the flat, default-open networking model of Kubernetes pods

Where CNI fits

CNI is the Container Network Interface. It is a bare minimum requirement to set up a Kubernetes cluster because Kubernetes does not provide network implementation itself. Instead, it relies on CNI plugins to provide the actual networking functionality. It is a plugin spec, not a single product. 

When a pod starts, the system calls the CNI plugin to set up the pod's network. The plugin connects the pod to the node's routing layer and assigns an IP address. Different plugins do this in different ways. Other CNI implementations, including Calico and Flannel, use different combinations of routing, overlays, and encapsulation depending on their configuration.

But the modern standard, and the default built into the Nutanix Kubernetes Platform (NKP), is Cilium. Cilium uses eBPF to push network processing deep into the Linux kernel. This brings massive changes to how clusters operate:

  • Kube-proxy replacement: Historically, routing traffic to Services meant passing through a messy web of iptables rules on every single node. As your cluster grows, iptables gets slow and hard to manage. Cilium replaces kube-proxy entirely. It uses eBPF to route packets directly in the kernel. It is much faster, uses less CPU, and is far easier to troubleshoot.
  • Layer 7 visibility and control: Traditional network policies only look at IPs and ports (Layer 4). Cilium actually understands the traffic. You can write policies that say "allow HTTP GET requests to /api, but block POST requests." You get deep visibility into DNS and HTTP traffic without needing to install a heavy service mesh.

The Flow CNI Option

While Cilium is the standard for the cluster, running VMs and containers side-by-side often leaves you with two separate networks. Flow CNI changes this. It is a Nutanix-native option that builds a "single fabric" for both VMs and Kubernetes pods. This is critical functionality for organizations leveraging a dual native architecture.

Instead of adding a complex overlay just for containers, Flow CNI integrates directly with the Nutanix VPC. Pods get native VPC IPs. A pod can talk directly to a VM on the same network without NAT or extra gateways. It removes the split between your networks, making hybrid apps much simpler to operate. You manage a single security policy and a single routing domain for everything.

Fig 02. Flow CNI unifying AHV VMs and NKP pods within a single Nutanix VPC Fig 02. Flow CNI unifying AHV VMs and NKP pods within a single Nutanix VPC

Beyond just bridging the gap between VMs and containers, Flow CNI introduces advanced networking controls that enterprise admins rely on. With multi-VPC support, you can securely isolate different Kubernetes clusters, tenants, or environments into distinct Nutanix VPCs while maintaining seamless management. Furthermore, Flow CNI offers egress IP control, allowing you to assign predictable, specific egress IPs to namespaces or pods. If you have legacy physical firewalls or external databases that require whitelisting a static IP address, this capability can help align Kubernetes egress traffic with existing security policies when configured and supported for the environment.

A Real-World Struggle: VIP Bouncing

While modern CNIs abstract a lot of complexity, foundational networking rules still apply, particularly when Kubernetes interacts with external physical networks. Let's look at a common struggle we saw recently.

An admin set up a Kubernetes cluster and used MetalLB to hand out external IPs. Suddenly, traffic started dropping randomly. The application would work, then fail, then work again. Why did this happen?

The load balancer VIP (Virtual IP) was bouncing between different host nodes. When a VIP moves, it sends out a GARP (Gratuitous ARP) packet to tell the physical network switch that the IP now lives at a new MAC address. But the physical switch did not update its ARP table fast enough. It kept dual MAC addresses for the same IP. The switch sent packets to the wrong place, and traffic was lost. 

The simple fix is to anchor your VIPs to stable hosts, or ensure your physical switches are configured to honor GARP updates instantly. The old networking rules still apply here. MAC-address or ARP conflicts can cause traffic disruption and should be investigated along with advertisement, routing, and convergence state.

Services: A Stable Address for Things That Move

A pod's IP is not permanent. The moment a pod restarts, you might get a new IP. A deployment scales from 3 to 10 pods, and you now have ten IPs to track. A pod gets evicted, the IP disappears.

That is why you almost never connect to a pod IP directly. Instead, you connect to a Service. A Service is a virtual IP sitting in front of a set of pods. When traffic hits the Service IP, the system load-balances it across the backend pods.

The service types you will see most often:

  • ClusterIP: Internal-only. Reachable only from inside the cluster.
  • NodePort: Exposes the service on a port on every node. Old-school, works, but not pretty.
  • LoadBalancer: Asks your platform to give the service an external IP (e.g., a "Webshop" or "Chatbot" application configured to be exposed to the outside world).
Fig 03. Illustrates external traffic routing through a load balancer, Ingress, and Services to backend pods Fig 03. Illustrates external traffic routing through a load balancer, Ingress, and Services to backend pods

Ingress: The North-South Door

A Service handles east-west traffic inside the cluster. Ingress handles north-south, the traffic coming in from outside.

In its simplest form, Ingress is a set of HTTP routing rules. "Send anything for shop.example.com/api to the api-service. Send shop.example.com/web to the web-service." An ingress controller watches those rules and configures itself. The ingress controller is just another set of pods running in the cluster. The thing in front of it is usually a Layer-4 LoadBalancer service. So, an external request flows like this:

External LB > Ingress controller pod > Service > Backend pod

That is the four-hop path to keep in your head. Most production traffic flows that way. When something is slow or broken, you walk those four hops in order.

Network Policies: The New Firewall

By default, every pod talks to every other pod. That is fine for a test cluster. It is not fine for production. Network Policies are how you put walls back inside the cluster. 

A policy is a Kubernetes object that says things like "pods with label app=database only accept traffic on port 5432, and only from pods with label app=api."

A few things to keep in mind: 

  • Network Policies are namespace: A policy in the "billing" namespace only affects pods in there.
  • Policies are additive: Two policies that select the same pod combine, they do not override each other.
  • You need a CNI that actually supports them: This is another reason NKP uses Cilium, as it handles both basic Layer 4 and advanced Layer 7 policies effortlessly.

Get used to writing policies early on. Because adding them later to a running cluster is a major pain.

What This Means for Your Existing Network

You do not throw away the existing network. You wrap it around the cluster. Here is where your old skills still earn you money:

  • IP planning: Choose Pod and Service CIDRs based on expected cluster and service scale, address-family design, and platform configuration. Ensure they do not overlap with existing routed networks.
  • MTU: Overlay and encryption overhead reduces the usable MTU. Calculate the required value from the selected CNI, encapsulation, encryption, and underlay configuration. If your physical network runs 1500 MTU, your pods should run 1450 or smaller. Get this wrong, and you get weird, intermittent stalls that look like application bugs but are really fragmentation.
  • DNS: CoreDNS runs as a pod inside the cluster. It handles internal names. For external lookups, it forwards to your existing DNS. Watch the latency on that link. A slow or unavailable upstream DNS service can cause failures or latency for workloads that depend on external name resolution.
  • Load balancers: If you run on bare metal or on-prem, you need something to hand out external IPs for LoadBalancer services. MetalLB is the common open-source choice. On Nutanix Kubernetes Platform (NKP), this part is handled for you.
  • Firewall and L4 segmentation: Your existing firewalls do not go away. They still control traffic in and out of the cluster. East-west between pods, though, is now your CNI's job.

Conclusion

Container networking is not harder than VM networking. It is different. The default flips from "everything is locked down" to "everything is open." The unit of segmentation moves from VLANs to labels. The plumbing connects your top-of-rack switch to a software-defined layer running on each node.

Once the model is in your head, the new tooling makes sense. CNI is a specification and plugin model that provides pod-network configuration and connectivity. A Service is a virtual IP with a label selector. An Ingress is HTTP routing rules. A Network Policy is a firewall rule written in YAML.

Platforms like the Nutanix Kubernetes Platform (NKP) take much of this off your plate. NKP is designed to support platform-engineering workflows. The available CNI, ingress, policy, and load-balancer integrations depend on the target release and deployment model.

Learn how NKP can support your transition to containerized workloads, Request a demo or explore more resources on the Nutanix Kubernetes Platform page.

 

©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. Kubernetes is a registered trademark of The Linux Foundation in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s).