Hi all, this is my first time experimenting with Calico and I’m trying to figure out a way to be able to determine which pod sent traffic out of an AKS cluster to an external firewall through which traffic is routed based on the source IP of a packet. I know this is possible with the Azure CNI but I would like to use Calico because I only have a limited set of private IP’s available (which is a problem when using the Azure CNI) and I like the added configuration options Calico brings. Not all pods will sent traffic through this firewall so only a certain amount of pods need a routable (to the firewall) source IP, so I think that the limited IP range should not be a problem. Ideally I would be able to determine whether a pod’s IP should be NAT’d to its node’s IP or just get its own routable IP when sending traffic outside of the k8s cluster. I don’t know whether or not this is possible at all or if I’m just looking at this from the wrong angle. I appreciate any help I can get. Thanks in advance!
This isn’t quite true. You can run Calico CNI with AKS. A lot of folks don’t realize this because confusingly the AKS docs refer to this as “kubenet + Calico”, but it is actually not running kubenet, it’s running Calico CNI.
However, this on its own isn’t enough, to solve the problem @tom is looking for, because in this mode the Calico pod IPs will still get SNATed to the node IP when a pod tries to talk to something outside of the cluster.
If you were spinning up your own cluster in Azure (rather than using AKS) there would be more options.
If you can put the firewall in the same VNET subnet as the k8s cluster then you could do this using Calico CNI + Calico IPAM.
If you can’t put the firewall in the same VNET subnet then you would need to use a mix of Azure CNI and Calico CNI. Using Azure CNI for the things you want to be uniquely identifiable at the firewall. To use a mix of CNIs you would use multus (or similar CNI plugin), so this is a relatively advanced stuff, not mainline easy k8s!
Depending on exactly what you are using the firewall for, the more mainline k8s way of approaching security would be to use Kubernetes or Calico egress policies to limit which pods can connect to different workloads outside of the cluster. This reduces the need for the firewall to understand pod IP addresses, since it can in theory just trust that k8s has enforced the right restrictions.
Thanks for all the help! I proposed using Kubernetes & Calico egress policies to our management right at the start and they weren’t too keen on it. Perhaps this is an extra reason for them to consider it. Not using AKS is not an option for us and we will definitely not be dabbling with anything more than mainline easy k8s at the moment.