[LXC] CNI + Calico BGP = no interpod communication

Hello everyone!
I’m trying to get a manual installation of Calico (without Kubernetes) to get to work using various calico docs; but unfortunately, inter-container communication is not working; nothing works – neither ICMP, nor TCP from within another pod or from another node or the same node as the workload itself. For this I’ve started 2 calico/node daemons as –

docker run -d --rm --privileged --net=host --name=calico -e CLUSTER_TYPE=bgp -e FELIX_DEFAULTENDPOINTTOHOSTACTION=Accept -e CALICO_IPV4POOL_CIDR=10.0.0.0/24 -e AS=4000 -e IP6=none -e NODENAME=node1 -e IP=192.168.3.2 -e CALICO_NETWORKING_BACKEND=bird -e DATASTORE_TYPE=etcdv3 -e ETCD_ENDPOINTS=http://192.168.3.2:2379 -v /var/lib/calico:/var/lib/calico -v /run/docker/plugins:/run/docker/plugins -v /lib/modules:/lib/modules -v /var/run/calico:/var/run/calico -v /etc/pki:/pki -v /var/log/calico:/var/log/calico calico/node:v3.16.5

docker run -d --rm --privileged --net=host --name=calico -e CLUSTER_TYPE=bgp -e FELIX_DEFAULTENDPOINTTOHOSTACTION=Accept -e CALICO_IPV4POOL_CIDR=10.0.1.0/24 -e AS=4200 -e NODENAME=node2 -e IP6=none -e IP=192.168.3.3 -e CALICO_NETWORKING_BACKEND=bird -e DATASTORE_TYPE=etcdv3 -e ETCD_ENDPOINTS=http://192.168.3.2:2379 -v /var/lib/calico:/var/lib/calico -v /run/docker/plugins:/run/docker/plugins -v /lib/modules:/lib/modules -v /var/run/calico:/var/run/calico -v /etc/pki:/pki -v /var/log/calico:/var/log/calico calico/node:v3.16.5

Following are the manifests injected –

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
 name: smallnet
 labels:
  purpose: internal
spec:
 cidr: 10.0.0.0/24
 nodeSelector: "name == 'node1'"
---
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
 name: smallnet2
 labels:
  purpose: node2
spec:
 cidr: 10.0.1.0/24
 nodeSelector: "name == 'node2'"
---
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
 name: smallnet3
 labels:
  purpose: node3
spec:
 cidr: 10.0.2.0/24
 nodeSelector: "name == 'node3'"
---
kind: Node
metadata:
 name: node2
 labels:
  purpose: prod
  name: node2
spec:
 bgp:
  ipv4Address: 192.168.3.3/24
  asNumber: 4100
---
apiVersion: projectcalico.org/v3
kind: Node
metadata:
 name: node3
 labels:
  purpose: prod
  name: node3
spec:
 bgp:
  ipv4Address: 192.168.3.4/24
  asNumber: 4200
---
apiVersion: projectcalico.org/v3
kind: Node
metadata:
 name: node1
 labels:
  purpose: prod
  name: node1
spec:
 bgp:
  ipv4Address: 192.168.3.2/24
  asNumber: 4000

CNI config of the plugin at node 1 –

{
"name": "node1cn2",
"cniVersion": "0.1.0",
"type": "calico",
"nodename": "node1",
"log_level": "DEBUG",
"ipam": {
    "type": "calico-ipam",
    "assign_ipv4": "true",
    "assign_ipv6": "false",
    "ipv4_pools": ["smallnet"]
},
"etcd_endpoints": "http://192.168.3.2:2379",
"datastore_type": "etcdv3",
"container_settings": {
    "allow_ip_forwarding": true
}
}

CNI config of the plugin at node 2 –

{
"name": "node2cn2",
"cniVersion": "0.1.0",
"type": "calico",
"nodename": "node2",
"log_level": "DEBUG",
"ipam": {
    "type": "calico-ipam",
    "assign_ipv4": "true",
    "assign_ipv6": "false",
    "ipv4_pools": ["smallnet2"]
},
"etcd_endpoints": "http://192.168.3.2:2379",
"datastore_type": "etcdv3",
"container_settings": {
    "allow_ip_forwarding": true
}
}

Node3 has been turned off for now.
This’s a single node etcd cluster which is on node1.

Here are the logs for calico/node /var/log/calico before the CNI plugin was executed and after some time the container was started (for both the nodes) –


Here are the logs after the CNI plugin was executed (for both the nodes) –


Here is the output of the CNI plugin on an ADD operation; debug level (for both the nodes) –


The following commands where used to invoke the CNI plugin –
CNI_COMMAND=ADD CNI_CONTAINERID=node2cn2 CNI_IFNAME=n2if2 CNI_NETNS=/proc/913/ns/net CNI_PATH=/root/cni /root/cni/calico < /home/de/cni-calico.conflist
CNI_COMMAND=ADD CNI_CONTAINERID=node1cn2 CNI_IFNAME=n1if2 CNI_NETNS=/proc/1072/ns/net CNI_PATH=/root/cni /root/cni/calico < /home/de/cni-calico.conflist

Thanks for any help!

calico/node is really intended to be run as a kubernetes network plugin at this point; we do not test with docker CNI. I believe there is a group of users on our slack that maintain the libnetwork plugin in order to support mesos+docker.