This page provides an introduction to the common networking configurations used by libvirt based applications. This information applies to all hypervisors, whether Xen, KVM or another. For additional information consult the libvirt network architecture docs.
The two common setups are "virtual network" or "shared physical device". The former is identical across all distributions and available out-of-the-box. The latter needs distribution specific manual configuration.
Contents |
NAT forwarding (aka "virtual networks")
Host configuration
Every standard libvirt installation provides NAT based connectivity to virtual machines out of the box. This is the so called 'default virtual network'. You can verify that it is available with
# virsh net-list --all Name State Autostart ----------------------------------------- default active yes
If it is missing, then the example XML config can be reloaded & activated
# virsh net-define /usr/share/libvirt/networks/default.xml Network default defined from /usr/share/libvirt/networks/default.xml # virsh net-autostart default Network default marked as autostarted # virsh net-start default Network default started
When the libvirt default network is running, you will see an isolated bridge device. This device explicitly does *NOT* have any physical interfaces added, since it uses NAT + forwarding to connect to outside world. Do not add interfaces
# brctl show bridge name bridge id STP enabled interfaces virbr0 8000.000000000000 yes
Libvirt will add iptables rules to allow traffic to/from guests attached to the virbr0 device in the INPUT, FORWARD, OUTPUT and POSTROUTING chains. It will also attempt to enable ip_forward. Some other applications may disable it, so the best option is to add the following to /etc/sysctl.conf
net.ipv4.ip_forward = 1
If you are already running dnsmasq on your machine, please see libvirtd and dnsmasq.
Guest configuration
Once the host configuration is complete, a guest can be connected to the virtual network based on its name. eg to connect a guest to the 'default' virtual network, the following XML would be used in the guest:
<interface type='network'>
<source network='default'/>
<mac address='00:16:3e:1a:b3:4a'/>
</interface>
NB, the mac address is optional and will be automatically generated if omitted.
Applying modifications to the network
Sometimes, one needs to edit the network definition and apply the changes on the fly. The most common scenario for this is adding new static MAC+IP mappings. Very unfortunately, destroying and re-defining the network causes the guests to lose network connectivity with the host until their network interfaces are re-attached.
In order to solve this problem, one possible approach would be to use a script to re-attach all interfaces on all machines after the network has been re-defined (an possibly also call the hook, see the next section).
An example of such script is available here. Be sure to read the current limitations, improvements are, of course, most welcome.
Forwarding Incoming Connections
By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like, and incoming connections are allowed from the host and from other guests connected on the same network, but all other incoming connections are blocked by iptables rules.
If you would like to make a service publicly available that is on a guest behind a NATed virtual network, you can setup libvirt's "hook" script for qemu to install the necessary iptables rules to forward incoming connections to the host on any given port HP to port GP on the guest GNAME:
1) determine a) the name of the guest "G" (as defined in the libvirt domain XML), b) the IP address of the guest "I", c) the port on the guest that will receive the connections "GP", and d) the port on the host that will be forwarded to the guest "HP".
(To assure that the guest's IP address remains unchanged, you can either configure the guest OS with static ip information, or add a <host> element inside the <dhcp> element of the network that is used by your guest. See the libvirt network XML documentation address section for defails and an example.)
2) stop the guest if it's running.
3) Create the file /etc/libvirt/hooks/qemu (or add the following to an already existing hook script), with contents similar to the following (replace GNAME, IP, GP, and HP appropriately for your setup):
Use the basic script below or see an "advanced" version, which can handle several different machines and port mappings here (improvements are welcome):
#!/bin/sh
Guest_name=GNAME
Host_port=HP
Guest_ipaddr=IP
Guest_port=GP
if [ "${1}" = "${Guest_name}" ]; then
if [ "${2}" = "start" ]; then
iptables -t nat -A PREROUTING -p tcp --dport ${Host_port} -j DNAT \
--to ${Guest_ipaddr}:${Guest_port}
iptables -I FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW \
-m tcp --dport ${Guest_port} -j ACCEPT
elif [ "${2}" = "stopped" ]; then
iptables -t nat -D PREROUTING -p tcp --dport ${Host_port} -j DNAT \
--to ${Guest_ipaddr}:${Guest_port}
iptables -D FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW \
-m tcp --dport ${Guest_port} -j ACCEPT
fi
fi
4) restart the libvirtd service.
5) chmod +x /etc/libvirt/hooks/qemu
6) start the guest.
(NB: This method is a hack, and has one annoying flaw - if libvirtd is restarted while the guest is running, all of the standard iptables rules to support virtual networks that were added by libvirtd will be reloaded, thus changing the order of the above FORWARD rule relative to a reject rule for the network, thus rendering this setup non-working until the guest is stopped and restarted. A better solution would be welcome!)
Bridged networking (aka "shared physical device")
Host configuration
The NAT based connectivity is useful for quick & easy deployments, or on machines with dynamic/sporadic networking connectivity. More advanced users will want to use full bridging, where the guest is connected directly to the LAN. The instructions for setting this up vary by distribution, and even by release.
Fedora/RHEL Bridging
This outlines how to setup briding using standard network initscripts
Disabling Xen's network scripts
If using Xen it is recommended to disable its network munging by editing /etc/xen/xend-config.sxp and changing the line
(network-script network-bridge)
To be
(network-script /bin/true)
Disabling NetworkManager
As of the time of writing (Fedora 12), NetworkManager still does not support bridging, so it is necessary to use "classic" network initscripts for the bridge, and to explicitly mark them as independent from NetworkManager (the "NM_CONTROLLED=no" lines in the scripts below).
If desired, you can also completely disable the NetworkManager:
# chkconfig NetworkManager off # chkconfig network on # service NetworkManager stop # service network start
Creating network initscripts
In the /etc/sysconfig/network-scripts directory it is neccessary to create 2 config files. The first (ifcfg-eth0) defines your physical network interface, and says that it will be part of a bridge:
# cat > ifcfg-eth0 <<EOF DEVICE=eth0 HWADDR=00:16:76:D6:C9:45 ONBOOT=yes BRIDGE=br0 NM_CONTROLLED=no EOF
Obviously change the HWADDR to match your actual NIC's address. You may also wish to configure the device's MTU here using e.g. MTU=9000.
The second config file (ifcfg-br0) defines the bridge device:
# cat > ifcfg-br0 <<EOF DEVICE=br0 TYPE=Bridge BOOTPROTO=dhcp ONBOOT=yes DELAY=0 NM_CONTROLLED=no EOF
WARNING: The line TYPE=Bridge is case-sensitive - it must have uppercase 'B' and lower case 'ridge'
After changing this restart networking (or simply reboot)
# service network restart
The final step is to disable netfilter on the bridge:
# cat >> /etc/sysctl.conf <<EOF net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 EOF # sysctl -p /etc/sysctl.conf
It is recommended to do this for performance and security reasons. See Fedora bug #512206. Alternatively you can configure iptables to allow all traffic to be forwarded across the bridge:
# echo "-I FORWARD -m physdev --physdev-is-bridged -j ACCEPT" > /etc/sysconfig/iptables-forward-bridged # lokkit --custom-rules=ipv4:filter:/etc/sysconfig/iptables-forward-bridged # service libvirtd reload
You should now have a "shared physical device", to which guests can be attached and have full LAN access
# brctl show bridge name bridge id STP enabled interfaces virbr0 8000.000000000000 yes br0 8000.000e0cb30550 yes eth0
Note how this bridge is completely independant of the virbr0. Do *NOT* attempt to attach a physical device to 'virbr0' - this is only for NAT connectivity
Debian/Ubuntu Bridging
This outlines how to setup bridging using standard network interface config files
Disabling NetworkManager
Stop network manager
sudo /etc/dbus-1/event.d/26NetworkManagerDispatcher stop sudo /etc/dbus-1/event.d/25NetworkManager stop
Create two files with only the word 'exit' in them. These files are:
/etc/default/NetworkManager /etc/default/NetworkManagerDispatcher
from https://help.ubuntu.com/community/NetworkManager#Disabling%20NetworkManager
Altering the interface config
First take down the interface you wish to bridge
ifdown eth0
Edit /etc/network/interfaces and find the config for the physical interface, which looks something like
allow-hotplug eth0
iface eth0 inet static
address 192.168.2.4
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
gateway 192.168.2.2
Remove the 'allow-hotplug eth0' line, replacing it with 'auto br0', and change the next line with iface name to 'br0', so it now starts with
auto br0 iface br0 inet static
And then define the interface as being a bridge and specify its ports
bridge_ports eth0
bridge_stp on
bridge_maxwait 0
bridge_fd 0
The complete config should now look like
auto br0
iface br0 inet static
address 192.168.2.4
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
gateway 192.168.2.2
bridge_ports eth0
bridge_stp on
bridge_maxwait 0
The interface can now be started with
ifup br0
Finally add the '/etc/sysctl.conf' settings
net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0
And then load the settings with
sysctl -p /etc/sysctl.conf
To ensure that the bridge sysctl settings get loaded on boot, add this line to '/etc/rc.local' just before the 'exit 0' line. This is a work around for Ubuntu bug #50093.
/sbin/sysctl -p /etc/sysctl.conf
You should now have a "shared physical device", to which guests can be attached and have full LAN access
# brctl show bridge name bridge id STP enabled interfaces virbr0 8000.000000000000 yes br0 8000.000e0cb30550 yes eth0
Note how this bridge is completely independant of the virbr0. Do *NOT* attempt to attach a physical device to 'virbr0' - this is only for NAT connectivity
Guest configuration
In order to let your virtual machines use this bridge, their configuration should include the interface definition as described in Bridge to LAN. In essence you are specifying the bridge name to connect to. Assuming a shared physical device where the bridge is called "br0", the following guest XML would be used:
<interface type='bridge'>
<source bridge='br0'/>
<mac address='00:16:3e:1a:b3:4a'/>
</interface>
NB, the mac address is optional and will be automatically generated if omitted.
To edit the virtual machine's configuration, use:
virsh edit <VM name>
For more information, see the FAQ entry at:
Other networking docs/links
- David Lutterkort's guide. NB the naming of devices 'peth0' (physical) and 'eth0' (bridge) does not work in Fedora 9 anymore. Following the 'eth0' (physical) and 'br0' (bridge) naming shown above instead
- Anthony Liguori's guide . Shows tips for 'shared physical devices' on Debian
- manual KVM networking - for people not using libvirt to launch guests
- Ubuntu libvirt guide with a section on network bridge setup
Views
Personal tools
Toolbox
