I am currently running Kubernetes with Calico v3.20.2 as the CNI. I have a very unique case where I need to send UDP traffic from a specific DaemonSet Pod to an external server which will read the sourceIP:sourcePort combination of the IP headers, and send the response by setting those two as the destIP:destPort fields. Since different UDP sessions will randomly choose different randomised source ports (range 1024-65535), and the response is to be load balanced via MetalLB, I would need to set up a UDP listener on every pod of the DaemonSet, listening to every outgoing request’s source port, as well as reconfigure the LB so that it listens on that port and distributes traffic. This is obviously unscalable and also not performant given the latency of such configurations taking longer than the response needs to be returned, as well as potential mismatches of parallel LB reconfigs.
Therefore, I woudld rather open a single listener per Pod on a specific port (for example, 20000), and SNAT all outgoing traffic from every Pod so that the source port once every UDP datagram leaves the node is 20000. The external server would send the response to this destination Port, and the response would eventually reach one of the UDP listeners on one of the DaemonSet Pods. I tried this by implementing
sudo iptables -t nat -I POSTROUTING 1 -d <EXT-SERVER-IP>/32 -p udp --dport <EXT-SERVER-PORT> -j SNAT --to-source <WORKER-NODE-IP>:20000
When I try to implement this in IPtables, Calico always rewrites the changes I make and enforces its own config, leading to the randomised source ports leaving the node again. On the other hand, when I set the natOutgoing flag to false and try to set up my own rules, only a single UDP datagram leaves the node with the correct SNAT change before all the other datagrams are blocked, and never leave the worker node at all (as evidenced by the external server as well as tcpdump on the worker node)
Fixing either of these cases would solve the overall issue, and any suggestions are appreciated!