Contents

Libvirt FAQ

General

What is libvirt?

Libvirt is collection of software that provides a convenient way to manage virtual machines and other virtualization functionality, such as storage and network interface management. These software pieces include an API library, a daemon (libvirtd), and a command line utility (virsh).

An primary goal of libvirt is to provide a single way to manage multiple different virtualization providers/hypervisors. For example, the command 'virsh list --all' can be used to list the existing virtual machines for any supported hypervisor (KVM, Xen, VMWare ESX, etc.) No need to learn the hypervisor specific tools!

I heard someone say they 'use libvirt'. What do they mean?

When people say they 'use libvirt', this usually means that they manage virtual machines using tools such as 'virsh', 'virt-manager', or 'virt-install', which are all built around libvirt functionality. They likely DO NOT directly use tools like 'xm' for Xen, or the qemu/qemu-kvm binary.

How do I know if I am using libvirt?

You are using libvirt if you manage virtual machines using virsh, virt-manager, or virt-install (pretty much any virtualization tool that starts with virt-*).

If you are using hypervisor specific tools like 'xm', 'qemu-kvm', etc. directly, you probably are not using libvirt.

If you have virtual machines on your existing machine and you are using libvirt, 'virsh list --all' (usually run as root) should show something.

What is some of the major functionality provided by libvirt?

Some of the major libvirt features are:

   virsh --connect qemu+ssh://root@example.com/system

For more info, see: http://libvirt.org/remote.html

What hypervisors does libvirt support?

A complete list can be found here: http://libvirt.org/drivers.html

How can I check my libvirt version?

For the installed virsh version:

virsh --version

For the libvirt daemon version:

libvirtd --version

For the library and default hypervisor version:

virsh --version

What are the libvirt mailing lists/IRC?

See: http://libvirt.org/contact.html

Where should I report libvirt bugs?

For bug reporting, see: http://libvirt.org/bugs.html

Is libvirt the 'lowest common denominator' of hypervisor features?

Short answer: ABSOLUTELY NOT.

Libvirt's goal is to expose all useful hypervisor features. Period.

The only caveat is that this feature needs to be exposed in a general way that is compatible with the libvirt architecture. Even if only a single supported hypervisor implements feature FOO, the API and XML changes need to be made sufficiently general in case any other hypervisor eventually supports FOO.


Common VM Configuration

What is the 'virsh edit' command and how do I use it?

'virsh edit' is the recommended way to make changes to an existing VM configuration. The command looks like:

virsh edit $your-vm-name

This command will open a text editor containing the existing VM XML: any changes that are made and saved will be checked for errors when the editor exits. If no errors are found, the changes are made permanent.

The text editor used is whatever is specified by the EDITOR environment variable. By default, this is usually 'vi'. You can override this with:

EDITOR=$your-favorite-editor virsh edit $your-vm-name'.

For example, on a gnome system 'EDITOR=gedit virsh edit myvm' will edit myvm's XML in a graphical text editor.

Where are VM config files stored? How do I edit a VM's XML config?

If you want to edit a VMs XML, use 'virsh edit $vmname'. If you want to backup the XML, use 'virsh dumpxml $vmname'.

Where VM configuration is stored depends on the hypervisor. For example, Xen and VMWare store their own configs, and libvirt just translates this to XML when 'virsh dumpxml' is called.. For qemu and lxc, libvirt stores the XML on disk and in memory.

libvirt is often configured to store qemu VM and other XML descriptions in /etc, but editing those files is not a valid way to change configuration. While editing those files and restarting libvirtd may work some (or even much) of the time, it's very possible that libvirtd will overwrite the changes and they will be lost. Equally importantly, using virsh edit or other API to edit the XML allows libvirt to validate your changes. A common problem seen when people edit the on-disk XML is a VM that simply vanishes the next time libvirtd is restarted after the edit. The VM disappears from libvirt because the XML has become invalid, after which libvirt can't do anything with it.

This advice applies for ALL libvirt XML. The equivalent virsh commands for other libvirt XML types are:

If I change the XML of running machine, do the changes take immediate effect?

NO. Redefining the XML of a running machine will not change anything, the changes will take effect after the next VM start up. Libvirt has a set of commands for making live changes to running guests, which have varying support depending on the hypervisor, ex virsh attach-*. virsh detach-*, virsh setmem, virsh setvcpus

How do I shutdown my VMs?

There are two shutdown operations via virsh:

The other option it to shut down your VM normally from inside the guest operating system, like you would for a physical machine.

Note: At least for the qemu/kvm guests, shutdown requires ACPI enabled in the guest. See the QEMU/KVM shutdown FAQ entry.

Will restarting the libvirt daemon stop my virtual machines?

NO, as of version 0.6.0 (Jan 2009). Versions older than this will kill VMs if the libvirtd daemon is stopped, so beware.


Common Errors

My VM doesn't show up with 'virsh list'

First, make sure you are passing the --all flag:

virsh list --all

'virsh list' only shows running VMs: --all is required to see every VM.

If your VM is still not listed, determine which URI virsh is defaulting to with:

virsh uri

If the default URI is not as expected, you can manually specify a URI with:

virsh --connect URI

If you are using QEMU/KVM and you created your VM with virt-manager, the URI you probably want is qemu:///system. If that doesn't work, read What is the difference between qemu:///system and qemu:///session the FAQ entry for qemu:///system vs. qemu:///session.

Error: Failed to add tap interface 'vnet%d' to bridge 'virbr0' No such file or directory

This is commonly caused by 2 things

modprobe tun
virsh net-start default

Error: unable to connect to '/var/run/libvirt/libvirt-sock'

This is usually caused by one of the following:

service libvirtd restart
/etc/init.d/libvirtd restart

It's recommended that you configure a manual libvirt install with --prefix=/usr to correct these issues.

Why doesn't 'shutdown' seem to work?

If you are using Xen HVM or QEMU/KVM, ACPI must be enabled in the guest for a graceful shutdown to work. To check if ACPI is enabled, run:

virsh dumpxml $your-vm-name | grep acpi

If nothing is printed, ACPI is not enabled for your machine. Use 'virsh edit' to add the following XML under <domain>:

<features><acpi/></features>

If your VM is running Linux, the VM additionally needs to be running acpid to receive the ACPI events.

HOWEVER, if your VM is running Windows, this won't be enough. If windows does not detect ACPI at install time, it disables the necessary support. The recommended way to remedy this seems to be a 'repair install' using windows install media. More info can be found here:

http://support.microsoft.com/kb/314088/EN-US/ http://support.microsoft.com/?kbid=309283

Error: domain did not show up

For a while, this was a kind of catch all error for qemu/kvm guest start up failures. See 'My VM fails to start'

Error: monitor socket did not show up.: Connection refused

This message usually masks a more specific error at domain start up. See 'My VM fails to start'

My VM fails to start. What should I do?

If your VM is failing to start, and libvirt isn't returning a helpful error message, you can find more info in the log files. For qemu/kvm, this is /var/log/libvirt/qemu/$your-vm-name.log. This will show generated qemu command line, and any error output qemu throws. Certain versions of libvirt weren't good at returning this info to the user, so there may be a fixable error here, like a missing storage file.

If you can't determine anything to fix, please report a bug (see the bug reporting FAQ).

error: Unknown Failure

This is caused by faulty error reporting in libvirt. Whenever you encounter this error, please file a bug. You can typically get more information by running

 tail -f /var/log/messages

and reproducing the failure. If you encounter this during VM migration, you will want to run that command on both the source and destination host. You will also want to make sure your hosts are properly configured for migration (see the migration section of this FAQ).

internal error: canonical hostname pointed to localhost, but this is not allowed

This is an error often encountered when trying to migrate with QEMU/KVM. What this means is that 'virsh hostname' on the destination host returns 'localhost', which can cause problems with plain migration. The easiest way to work around this is to manually specify a migration hostname/IP address. This can be done with:

 virsh migrate with the option '--migrateuri tcp://hostname:port'

error: operation failed: migration to '...' failed: migration failed

This is an error often encountered when trying to migrate with QEMU/KVM. This typically happens with plain migration, when the source VM cannot connect to the destination host. You will want to make sure your hosts are properly configured for migration (see the migration section of this FAQ)

networking is unavailable in virt-manager / virsh - libvirt's default network fails to start

This problem can have several causes:

1) The most common is that dnsmasq is unable to start due to another instance of dnsmasq (usually the host system's main instance, controlled by /etc/dnsmasq.conf) already running and listening on all network interfaces - the symptom will be that dnsmasq exits immediately with an error code of "2". You can solve this by either disabling the host dnsmasq instance completely (on Fedora/RHEL you can do this with "chkconfig dnsmasq off"), or by modifying the offending dnsmasq configuration to only listen on specific interfaces (see the listen-address and interface options in /etc/dnsmasq.conf).

2) Sometimes it appears that networking (or some other common feature) is unavailable because the user has accidentally connected to the qemu:///session (which can be used as any user, and is the default used by virsh when not run as root) as opposed to qemu:///system (which can only be accessed by root, and is the default used by virsh when run as root). To solve this problem, be sure to su to root prior to running virsh. See What is the difference between qemu:///system and qemu:///session for more details.

3) Another problem is missing third-party binaries required on the host to initialize the network. The dependencies registered as part of the libvirt package will normally assure that all necessary binaries are present, but on some systems (e.g. gentoo) the practice of building your own packages, and allowing "minimal" packages, can lead to problems. For example, the binary /sbin/ip, which is part of the iproute package (called iproute2 in some cases) is required to set the IP addresses of the bridge devices used by libvirt, but it's possible on gentoo to build the iproute2 package "minimally" which results in no /sbin/ip. The symptom will be that libvirt complains it can't set the bridge device IP address. Here is a list of the networking-related packages that need to be installed on the host for libvirt networking to work properly:

 bridge-utils
 module-init-tools
 iproute
 dnsmasq >= 2.41
 radvd (if using IPv6)
 iptables
 iptables-ipv6 (if using IPv6)
 ebtables (if using libvirt's nwfilter)

Again, the installation of required dependencies should be handled by your OS' package installation system, but if there are problems it may be useful to reference this list (and/or the file libvirt.spec in the libvirt source tree for a full list of all package dependencies - required packages will be listed on lines beginning with "Required:").

Migration

What are the different migration methods?

There are two primary types of migration with QEMU/KVM and libvirt:

What setup is required for QEMU/KVM migration?

For all QEMU/KVM migrations, libvirtd must be running on the source and destination host. You must be able to open a valid connection to the remote libvirt host.

For tunneled migration, no extra configuration should be required, you simply need to pass the --tunnelled flag to virsh migrate.

For plain unencrypted migration, the TCP port range 49152-49215 must be opened in the firewall on the destination host. If you would like to use a specific port rather than have libvirt choose, you can pass a manual URI to virsh:

 virsh migrate $VMNAME $REMOTE_HOST_URI --migrateuri tcp://$REMOTE_HOST:$PORT


QEMU/KVM

Can I connect to the QEMU monitor with libvirt?

No. Libvirt deliberately does not enable user access to the QEMU monitor. Interacting with the monitor behind libvirt's back can cause reported virtual machine state to be out of sync, which will likely end with errors.

The only way to interact with the monitor is through libvirt APIs (see the following question for a complete list of support)

What monitor commands does libvirt support? What QEMU/KVM command line flags does libvirt support?

For a complete list, please see: http://wiki.libvirt.org/page/QEMUSwitchToLibvirt

What is the difference between qemu:///system and qemu:///session? Which one should I use?

All 'system' URIs (be it qemu, lxc, uml, ...) connect to the libvirtd daemon running as root which is launched at system startup. Virtual machines created and run using 'system' are usually launched as root, unless configured otherwise (for example in /etc/libvirt/qemu.conf).

All 'session' URIs launch a libvirtd instance as your local user, and all VMs are run with local user permissions.

You will definitely want to use qemu:///system if your VMs are acting as servers. VM autostart on host boot only works for 'system', and the root libvirtd instance has necessary permissions to use proper networkings via bridges or virtual networks. qemu:///system is generally what tools like virt-manager default to.

qemu:///session has a serious drawback: since the libvirtd instance does not have sufficient privileges, the only out of the box network option is qemu's usermode networking, which has nonobvious limitations, so its usage is discouraged. More info on qemu networking options: http://people.gnome.org/~markmc/qemu-networking.html

The benefit of qemu:///session is that permission issues vanish: disk images can easily be stored in $HOME, serial PTYs are owned by the user, etc.

Networking

What is libvirt doing with iptables?

By default, libvirt provides a virtual network named 'default' which acts as a NAT router for virtual machines, routing traffic to the network connected to your host machine. This functionality uses iptables.

For more info, see: http://wiki.libvirt.org/page/Networking#NAT_forwarding_.28aka_.22virtual_networks.22.29

How can I make libvirt stop using iptables?

WARNING: Any VMs already configured to use these virtual networks will need to be edited: simply removing the <interface> devices via 'virsh edit' should be sufficient. Not doing this step will cause starting to fail, and even after the editing step the VMs will not have network access.

You can remove all libvirt virtual networks on the machine:

Why doesn't libvirt just auto configure a regular network bridge?

While this would be nice, it is difficult/impossible to do in a safe way that won't hit a lot of trouble with non trivial networking configurations. A static bridge is also not compatible with a laptop mode of networking, switching between wireless and wired. Static bridges also do not play well with NetworkManager as of this writing (Feb 2010).

You can find more info about the motivation virtual networks here: http://www.gnome.org/~markmc/virtual-networking.html

How do I manually configure a network bridge?

See: Setting up a Host Bridge

How do I get my VM to use an existing network bridge?

See: Configuring Guests to use a Host Bridge

How do I forward incoming connections to a guest that is connected via a NATed virtual network? =

See: Forwarding Incoming Connections

Developing with libvirt

What is libvirt's license?

libvirt is released under the GNU Lesser General Public License, see the file COPYING.LIB in the distribution for the precise wording.

Where can I get the source code? How do I compile and install?

See http://libvirt.org/downloads.html

If you are using Fedora, the following packages are pre-requisites to a minimal build that will pass 'make check':

The following packages are also useful for building all optional features:

Is libvirt thread safe?

Yes, libvirt is thread safe as of version 0.6.0. This means that multiple threads can act on a single virConnect instance without issue.

Previous libvirt versions required opening a separate connection for each thread: this method has several major drawbacks and is not recommended.