Kubernetes: iptables ctstate INVALID in KUBE-FORWARD chain drops forward packets to other node

Peter0x48 · May 12, 2022, 5:12pm

Hi,

I’ve created this GitHub issue, maybe someone from here has seen this before:

Kubernetes: iptables ctstate INVALID in KUBE-FORWARD chain drops forward packets to other node

opened 06:43PM - 10 May 22 UTC

## Expected Behavior Packets that enter a node which are destined for another n…ode should be forwarded. Otherwise prefixes should only be announced through BGP from the node that holds it. ## Current Behavior Due to this iptables rule the packets are dropped: `-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP` ## Context I've setup a test Kubernetes cluster on-premise with 2 nodes and installed Calico like this: `kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml` custom-resources.yaml: ``` apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: ipPools: - blockSize: 26 cidr: 10.0.48.0/20 encapsulation: None natOutgoing: Disabled nodeSelector: all() --- apiVersion: operator.tigera.io/v1 kind: APIServer metadata: name: default spec: {} ``` `kubectl create -f custom-resources.yaml` I disabled encapsulation and outgoing NAT because I want to use normal routing. I created a global BGP config to peer with the gateway of the both nodes: ``` apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: my-global-peer spec: peerIP: 10.0.6.254 asNumber: 1 ``` From the view of my router which peers with both nodes, I see that they export both the same routes. When I try to establish a TCP connection to a server in the internet, the TCP SYN exits the worker to the default gateway. The SYN-ACK arrives on the master node and dropped. The pod has the IP 10.0.53.9 that belongs to the prefix 10.0.53.0/26. On the master node a route for 10.0.53.0/26 to the worker node exists. Because both nodes (master and worker) announce this prefix and I currently haven't enabled ECMP, my router sends packets for 10.0.53.0/26 to the master node (because bird inserted this route, maybe due to protocol order) which has to forward them to the worker node. But the above mentioned iptables rule causes the packets to be dropped. When I remove this single rule, everything works as expected. - Is it intended that both nodes announce the same prefixes? - Is this maybe a conflict between the default Kubernetes and the Calico iptables rules? Thank you! ## Your Environment * Calico version: 3.22.2 * Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes v1.24.0 * Operating System and version: Debian Bullseye (11) Linux 5.10.0-14-amd64 1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

Thanks!

Peter