Net.bridge.bridge-nf-call and sysctl.conf

From Libvirt Wiki
Revision as of 13:34, 2 June 2014 by Laine (talk | contribs)
(diff) ←Older revision | view current revision (diff) | Newer revision→ (diff)
Jump to: navigation, search

Most virtual guests configured using libvirt connect to the network via a Linux host bridge; the behavior of the bridge changes depending on the setting of 3 "tunables" recognized by the kernel bridge module:

 net.bridge.bridge-nf-call-arptables
 net.bridge.bridge-nf-call-ip6tables
 net.bridge.bridge-nf-call-iptables

These control whether or not packets traversing the bridge is sent to iptables for processing. In the case of using bridges to connect virtual machines to the network, generally such processing is *not* desired, as it results in guest traffic being blocked due to host iptables rules that only account for the host itself, and not for the guests.

Unfortunately, the bridge module in the kernel has all three of these values set to "1" (on, i.e. *do* send the packets to iptables), and for historical reasons the kernel maintainers refuse to change this default (see http://patchwork.ozlabs.org/patch/29319/ )

After the refusal of the above change to the compiled-in defaults by the kernel, many Linux distros (including Fedora, RHEL, and CentOS) made an attempt to solve this problem by adding lines to /etc/sysctl.conf to update the default settings compiled into the bridge module:

  net.bridge.bridge-nf-call-arptables = 0
  net.bridge.bridge-nf-call-ip6tables = 0
  net.bridge.bridge-nf-call-iptables = 0

The settings in /etc/sysctl.conf are applied anytime the command "sysctl -p" is run, and it happens that this command is run at least once during system boot, generally by either the network service or the NetworkManager service, coincidentally a short time after those services have created any bridge devices in the host system network configuration (i.e. listed in /etc/sysconfig/network-scripts). Since creating a bridge device causes the bridge module to be autoloaded, it will be present in the kernel (and can act on the provided settings) at the time "sysctl -p" is run.

However, if the bridge module *isn't* loaded at the time "sysctl -p" is run, there will be two problems:

1) sysctl -p will generate an error message about an attempt to set "unknown keys" (since the bridge module doesn't exist, there is nobody to recognize the keys):

  error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key
  error: "net.bridge.bridge-nf-call-iptables" is an unknown key
  error: "net.bridge.bridge-nf-call-arptables" is an unknown key

This causes an annoyance to some system administrators.

2) If some program creates a bridge at a later time, the bridge module will be autoloaded, but sysctl -p won't be run (and the previously "unknown" keys won't have been otherwise saved for application at a later time), so the kernel-set defaults for the tunables will remain in effect.

The above description is the current state of affairs in RHEL6/CentOS6. Things are different in recent Fedora and RHEL7, which we'll get to further down.

Attempted/Proposed Solutions in RHEL6 / pre-systemd

Several solutions to the above problem have been proposed, and some tried, over the years. There are many bug reports dealing with it. One public bug report that has a lot of information is this:

 https://bugzilla.redhat.com/show_bug.cgi?id=634736


* Build the bridge module into the kernel.

This would be bad because 1) anybody wanting to minimize their footprint would be stuck with the bridge module whether or not they used it, and 2) if someone built their own kernel, they may configure it to be loadable anyway, thus negating the fix. (reason (2) is a bit weak, but reason (1) would probably carry the day).

* Change the system startup to always force loading of the bridge module very early on during system boot.

Again, there would be complaints from those trying to maintain a very small footprint.

* Move the settings from /etc/sysctl.conf to /etc/sysctl.d/bridge.conf

The problem with this "solution" is that it only solves one of the two problems, and it's the least harmful problem that's being solved.

1) At boot time, the settings in /etc/sysctl.d are loaded [in some other manner], so in the case that there is a bridge device in the system network config, they are applied. GOOD

2) Since "sysctl -p" only loads settings from /etc/sysctl.conf, if there are no bridge devices, that will not generate the above errors. GOOD

3) If a bridge is created later, then sysctl -p won't be automatically run (same as before). BAD (or NEUTRAL, depending on your thinking)

4) Even if the sysadmin has setups scripts to run "sysctl -p" (based on previous behavior of the system) to force these settings, that will

  • still* not load the bridge tunable settings, since (as stated above)

sysctl -p ignores /etc/sysctl.d. BAD

* have libvirtd run "sysctl -p"

This FAILs because it would mean that *all* the settings from /etc/sysctl.conf would be applied, some of those possibly overriding other transient settings made elsewhere by the sysadmin. Also, in many cases libvirt is using a bridge device that was already created by someone else, and in other cases it is simply calling the ioctl to create the bridge - the module is autoloaded when necessary and libvirt has no idea when this happens.

  • Manually set these items in libvirt whenever libvirt creates a
 bridge - this also fails in many ways:

1) In the end, this is a system security policy that affects things outside of libvirt and the virtual machines it manages, so it should not be unceremoniously changed by libvirt.

2) libvirt isn't always the entity creating the bridge (and although we are personally only concerned about libvirt, it isn't necessarily the only *user* of bridges).

(just to mix things up a bit - note that some uses of libvirt's "nwfilter" guest network packet filtering require all these settings to be "on" in order to function properly).