• Debian VMs on a routed libvirt ULA network are not exchanging neighbor

    From Mathieu Baudier@21:1/5 to All on Tue Mar 16 10:30:01 2021
    Hello,

    we have the following setup:
    - a libvirt/KVM virtualization physical host running Debian 10
    - Debian 10 and RHEL 8/CentOS 8 VMs as guests

    Guest VMs usually have two (virtio) network interfaces:
    - a static public interface (IPv4+IPv6 or IPv6 only, usually enp1s0), which access internet via a bridge (br0) on the host
    - an internal interface (IPv6 only, usually enp2s0) connected to a routed libvirt virtual network with a /64 ULA network

    The problematic behaviour is that, after a while, the Debian guests lose their internal ULA IPv6 address.
    I could observe that it happens after the expiry of valid_lft, preferred_lft, as can be seen in the 'ip a' output.
    Running 'ip neigh' shows that there is no REACHABLE router (but some STALE ones).

    On the other hand, the CentOS VM do not show this behaviour and their addresses and routes are refreshed regularly.
    But when I was debugging other issues with them, I realized that Network Manager is taking charge of these configurations.

    In order to debug further, I ran tcpdump on the host, a Debian guest, and a CentOS guest:
    - On the host (using the bridge created by libvirt for this routed virtual network, virbr2)
    sudo tcpdump -n -i virbr2 icmp6
    - On the guest
    tcpdump -n -i enp2s0 icmp6

    What I can observe is that:
    - Neighbor sollicitations between the host and the CentOS VMs are being exchanged regularly
    - With the Debian guest:
    - Neighbor sollicitations are exchanged when the VM is rebooted
    - Afterwards they are not exchanged...
    - ... unless one explicitly pings the guest from the host on this internal ULA address
    (then, one can see the echo packets, but also some routerneighbor sollicitations)

    So, I am not sure how to analyze further, and I would be grateful for some hints.
    It seems that a kind of regular "trigger" for the sollicitations is missing here.

    Please find below some more configurations.
    Thanks in advance!

    Mathieu


    ## host network config
    auto lo
    iface lo inet loopback

    auto br0
    iface enp1s0f0 inet manual
    iface enp1s0f0 inet6 manual

    iface br0 inet6 static
    bridge_ports enp1s0f0
    address 2001:XXXX:XXXX:XXXX::/56
    gateway 2001:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:ff

    iface br0 inet static
    bridge_ports enp1s0f0
    address XX.XX.XX.XX/24
    gateway XX.XX.XX.254
    ...

    ## guest network config
    auto lo enp1s0 enp2s0
    iface lo inet loopback

    iface enp1s0 inet static
    address YY.YY.YY.YY
    netmask 255.255.255.255
    broadcast YY.YY.YY.YY
    post-up ip route add YY.YY.YY.254 dev enp1s0
    post-up ip route add default via YY.YY.YY.254
    pre-down ip route del YY.YY.YY.254 dev enp1s0
    pre-down ip route del default via YY.YY.YY.254

    iface enp1s0 inet6 static
    address 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY/64
    post-up ip route add YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    post-up ip route add 2000::/3 via 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    pre-down ip route del 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    pre-down ip route del 2000::/3 via 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0

    iface enp2s0 inet6 auto

    ## host sysctl options
    net.ipv6.conf.all.accept_ra = 2
    net.ipv4.ip_forward = 1
    net.ipv6.conf.all.forwarding = 1
    net.ipv4.conf.br0.send_redirects = 0
    net.ipv4.conf.br0.rp_filter = 0

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Bastian Blank@21:1/5 to Mathieu Baudier on Tue Mar 16 15:00:02 2021
    Hi Mathieu

    On Tue, Mar 16, 2021 at 10:23:15AM +0100, Mathieu Baudier wrote:
    What I can observe is that:
    - Neighbor sollicitations between the host and the CentOS VMs are being exchanged regularly
    - With the Debian guest:
    - Neighbor sollicitations are exchanged when the VM is rebooted
    - Afterwards they are not exchanged...
    - ... unless one explicitly pings the guest from the host on this internal ULA address
    (then, one can see the echo packets, but also some routerneighbor sollicitations)

    You need periodic router advertisements to refresh prefix infos. How
    are you generating those?

    iface br0 inet6 static
    bridge_ports enp1s0f0
    address 2001:XXXX:XXXX:XXXX::/56

    You should always use /64 on an ipv6 network.

    iface enp1s0 inet6 static
    address 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY/64
    post-up ip route add YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    post-up ip route add 2000::/3 via 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    pre-down ip route del 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    pre-down ip route del 2000::/3 via 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0

    This does not match your description. Please don't change ip addresses
    if you are searching for problems.

    Bastian

    --
    We have phasers, I vote we blast 'em!
    -- Bailey, "The Corbomite Maneuver", stardate 1514.2

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mathieu Baudier@21:1/5 to All on Tue Mar 16 15:40:01 2021
    Hello Bastian,

    - Neighbor sollicitations between the host and the CentOS VMs are being exchanged regularly
    - With the Debian guest:
    - Neighbor sollicitations are exchanged when the VM is rebooted
    - Afterwards they are not exchanged...
    - ... unless one explicitly pings the guest from the host on this internal ULA address
    (then, one can see the echo packets, but also some routerneighbor sollicitations)

    You need periodic router advertisements to refresh prefix infos. How
    are you generating those?

    I assume that they are sent by libvirt since I can see them on the host (with tcpdump)
    and on the CentOS/RHEL VMs, which are staying connected.

    I can also see the Debian VM receiving some neighbor sollicitations, as long as it is still connected,
    but not the ones about itself (except when it is being pinged from the host).

    Do you mean that something should also run on the guest?

    iface br0 inet6 static
    bridge_ports enp1s0f0
    address 2001:XXXX:XXXX:XXXX::/56

    You should always use /64 on an ipv6 network.

    Yes, this is my understanding.
    But this is the configuration of the provider of the physical host (OVH)
    and there are issues on some of their servers if we switch to /64.
    More generally their IPv6 is a bit unreliable and heterogeneous, especially
    on the Debian 10 image they provide (with cloud-init configs).

    But this is the public bridge, that is only used by the public interfaces of the VMs.
    The issue here is with the internal virtual ULA networks provided by libvirt.
    I provided this config only for completion, in case there may be side-effects.

    iface enp1s0 inet6 static
    address 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY/64
    post-up ip route add YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    post-up ip route add 2000::/3 via 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    pre-down ip route del 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0
    pre-down ip route del 2000::/3 via 2001:YYYY:YYYY:YYYY:YYYY:YYYY:YYYY:ff dev enp1s0

    This does not match your description. Please don't change ip addresses
    if you are searching for problems.

    This guest-side configuration is for the public interface and is also provided for completion.

    The config for the interface to the internal ULA network which is autoconfigured is just this one line:

    iface enp2s0 inet6 auto

    It works as expected after a reboot, or if I run:
    systemctl restart networking

    But then 'ip a' shows the lifetime decreasing and decreasing from 3600,
    and in the end the IP address (which is EUI-64 generated, so always the same) vanishes.

    Should I install additional packages on the guest?
    It is is a quite minimal Debian install, with just freeipa-client (and dependencies) + apache2 added.

    Or enable some additional services?

    Thank you!

    Mathieu

    PS: I was not sure whether this ipv6 list is still active, so I also posted on the users list. I hope this is not a problem.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From peter@chubb.wattle.id.au@21:1/5 to All on Tue Mar 16 23:20:02 2021
    "Mathieu" == Mathieu Baudier <mbaudier@argeo.org> writes:


    This sounds like multicast packets aren't getting through the bridge.
    Check the values of /sys/class/net/$IFACE/bridge/multicast_querier and /sys/class/net/$IFACE/bridge/multicast_snooping

    Peter C

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Mathieu Baudier@21:1/5 to All on Wed Mar 17 08:30:01 2021
    This sounds like multicast packets aren't getting through the bridge.
    Check the values of /sys/class/net/$IFACE/bridge/multicast_querier and /sys/class/net/$IFACE/bridge/multicast_snooping

    On the host, I get:

    cat /sys/class/net/virbr2/bridge/multicast_querier

    0

    cat /sys/class/net/virbr2/bridge/multicast_snooping
    1

    Searching based on your comment seems to show that it could be the other way round for my use case:
    https://unix.stackexchange.com/questions/391672/what-is-multicast-snooping-and-why-does-it-break-upnp-dlna
    https://bugzilla.redhat.com/show_bug.cgi?id=880035
    and even more precisely:
    https://bugzilla.redhat.com/show_bug.cgi?id=902922 https://access.redhat.com/solutions/784373

    I tested:
    echo 1 > /sys/class/net/virbr2/bridge/multicast_querier
    echo 0 > /sys/class/net/virbr2/bridge/multicast_snooping
    (see detailed test procedure below)

    but the result was the same:
    - no sollicitations observed, except when the guest is started
    - the ULA address is dropped at the end of its lifetime (3600s)

    Interestingly, a restart of the libvirt virtual network resets these flags to their original value for the related bridge.

    Is there maybe something I could also test at the Debian guest level?
    (The CentOS/RHEL guests don't have this problem, but they seem to use Network Manager for this, not the default mechanism)

    Thank you for your help,

    Mathieu


    ## Test procedure

    # After a fresh reboot of the host, as root
    ip address show dev virbr2
    8: virbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:fa:ba:58 brd ff:ff:ff:ff:ff:ff
    inet6 fd34:fe56:7891:6::1/64 scope global
    valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fefa:ba58/64 scope link
    valid_lft forever preferred_lft forever

    cat /sys/class/net/virbr2/bridge/multicast_querier

    0
    cat /sys/class/net/virbr2/bridge/multicast_snooping
    1

    # Change both flags
    echo 1 > /sys/class/net/virbr2/bridge/multicast_querier
    echo 0 > /sys/class/net/virbr2/bridge/multicast_snooping

    # Shutdown all guests on this host

    # Restart the routed libvirt virtual network using virbr2
    !! The flags have been reset to their previous values
    cat /sys/class/net/virbr2/bridge/multicast_snooping
    1
    cat /sys/class/net/virbr2/bridge/multicast_querier

    0

    # Set again the flags
    echo 1 > /sys/class/net/virbr2/bridge/multicast_querier
    echo 0 > /sys/class/net/virbr2/bridge/multicast_snooping

    # Monitor icmp6
    tcpdump -n -i virbr2 icmp6

    # Start only the Debian guest VM
    brctl show virbr2
    bridge name bridge id STP enabled interfaces
    virbr2 8000.525400faba58 yes virbr2-nic
    vnet1

    Link address of the bridge (virbr2):
    fe80::5054:ff:fefa:ba58/64

    Link address guest (vnet1 from the host perspective, enp2s0 from the guets perspective):
    fe80::fc54:ff:fe47:812f/64

    On the host, during the VM startup, tcpdump shows:

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on virbr2, link-type EN10MB (Ethernet), capture size 262144 bytes 06:18:56.187251 IP6 :: > ff02::1:ff47:812f: ICMP6, neighbor solicitation, who has fe80::5054:ff:fe47:812f, length 32
    06:18:57.211370 IP6 fe80::5054:ff:fe47:812f > ff02::2: ICMP6, router solicitation, length 16
    06:18:57.211698 IP6 fe80::5054:ff:fefa:ba58 > fe80::5054:ff:fe47:812f: ICMP6, router advertisement, length 88
    06:18:57.415428 IP6 :: > ff02::1:ff47:812f: ICMP6, neighbor solicitation, who has fd34:fe56:7891:6:5054:ff:fe47:812f, length 32
    06:19:02.320961 IP6 fe80::5054:ff:fefa:ba58 > fe80::5054:ff:fe47:812f: ICMP6, neighbor solicitation, who has fe80::5054:ff:fe47:812f, length 32
    06:19:02.321166 IP6 fe80::5054:ff:fe47:812f > fe80::5054:ff:fefa:ba58: ICMP6, neighbor advertisement, tgt is fe80::5054:ff:fe47:812f, length 24
    06:19:07.358070 IP6 fe80::5054:ff:fe47:812f > fe80::5054:ff:fefa:ba58: ICMP6, neighbor solicitation, who has fe80::5054:ff:fefa:ba58, length 32
    06:19:07.358136 IP6 fe80::5054:ff:fefa:ba58 > fe80::5054:ff:fe47:812f: ICMP6, neighbor advertisement, tgt is fe80::5054:ff:fefa:ba58, length 24

    # Log in to the guest via the public interface and monitor icmp6 on the guest sudo tcpdump -n -i enp2s0 icmp6
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes

    # RESULT
    - no additional exchanges observed, neither on the host nor on the guest
    - ULA address dropped after 3600s, as before
    - host router seems STALE, as seen from the guest
    ip -6 neigh
    ...
    fe80::5054:ff:fefa:ba58 dev enp2s0 lladdr 52:54:00:fa:ba:58 router STALE
    ...

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)