Use Calico for creating bgb sessions with for other networks?

Hi,

I currently have Calico in an on-prem setup with Kubernetes running in IPv4-only mode.

Now I need to provide a service on a public IPv6 range, but don’t necessarily need to have this as a k8s service CIDR since the server will probably run in host network mode.

Now I’m trying to figure out if it’s possible to configure Calico to announce this otherwise unknown IPv6 range to the ISP bgp peer of specific nodes?
The only place I could find such a configuration was for serviceExternalIPs in the global bgp configuration, but the documentation clearly says IPv4.
Also this would mean that Calico will also have to announce the private IPv4 range currently used in the network to this new peer even though it’s not necessary for my setup?

Hi balboah…

Your Kubernetes is currently IPv4 only, but I presume your nodes have IPv6 capability, otherwise this obviously can’t fly at all.

You say “the server will probably run in host network mode”. I can see that might work with Kubernetes pods remaining IPv4 only, because of the IPv6 capability on the nodes. If you want later to have non-host-networked servers, you’ll have to change your cluster to be dual stack. (https://docs.projectcalico.org/networking/dual-stack)

So you have an IPv6 server, host-networked, running on some or all of your nodes, and you want to be able to access that from outside the cluster - right? How does the addressing of those servers differ from the IPv6 addressing of the nodes themselves? I presume the node IPv6 addresses are already accessible from outside, so can the service just be accessed as a known port on one of those addresses?

In terms of Calico capabilities…

  • There is the service advertisement feature https://docs.projectcalico.org/networking/advertise-service-ips, and in fact it does support IPv6 as well as IPv4. (I’ll get the docs fixed!)

  • The routes that Calico will export, from each node, are filtered by the defined IP pools. So, if you have a local route for, say, fd5f:1234::3/128, you can get Calico to export that by defining a disabled IP pool that includes that address, e.g.

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: export-ipv6
spec:
  cidr: fd5f:1234::/120
  disabled: true
  nodeSelector: all()

But I’m unsure if either of those exactly meets your needs.

1 Like

Hey,

Thanks a lot for the reply. You successfully deciphered what I tried to do :slight_smile:
I just today finished migrating into dual-stack k8s, making IPv6 also available to the pods which would be beneficial in the long run anyways.

Now after I’ve specified the bgp peer config, the IPv6 peers are finally available on the node status list :+1:
I’ll see if it works all the way once the ISP has the correct setup as well.

So the reason I do this is that I have a public IPv6 range that I want to use for anycast and where the service needs to do AnyIP. I figured this won’t work in the k8s network so it’s going to be on host network pods but with calico announcing it.

As far as I understand, this peering will also announce the other private networks right? Even though I don’t technically need that, not sure yet if it will cause other issues once the bgp session goes live.

Yes, Calico’s peerings will announce everything within the defined IP pools.

For the rest, please do let us know how it goes once the ISP setup is in place.

Best wishes,
Neil

Ok, it seems I’ve got a session going with the ISP and the node status show it as UP.
However on their end they seem to get the next hop of my private network. They get the same IP as the k8s node is configured with, I don’t see any option to change this.

My nodes are connected on a private network on a different interface, but now I’m peering on a public network and they should see my public IP that is used to talk to that bgp session instead of the private one (which is the k8s node ip).

The only setting I could find was keepOriginalNextHop, but this seems to not matter in this case.

After they forced the next peer to be the same IP as the bgp session, all is working great.
So it seems to me that calico needs a setting for “next hop self”, or that it needs to be able to listen on other interfaces for the session than what the k8s node is configured with

Is it possible to avoid announcing private ranges somehow? Currently I have a lot of private pools for IPv6 with node selectors to be routed properly via Wireguard.
These pools make no sense for the eBGP. I saw an example about this kind of issue with metallb, where they set “no-advertise” community on the address pool.

I’d like to do something similar, I’m only interested in announcing the serviceExternalIPs to the non-mesh sessions.

I’m afraid we don’t have anything in Calico (yet) to filter out announcing the private ranges.

Of course, you could do this on import, by prefix, on the adjacent routers, if they are under your control.