Creation of a vHBA using the node device driver¶

In order to create a vHBA using the node device driver, select an HBA with available "vport" space, use the HBA "<name>" field as the "<parent>" field in the following XML:

<device>
  <parent>scsi_host3</parent>
  <capability type='scsi_host'>
    <capability type='fc_host'>
    </capability>
  </capability>
</device>

<parent> support was added in libvirt 1.1.2.

Since the parent scsi_hostN value can change between reboots or after hardware reconfiguration, libvirt 3.0.0 added two more ways to identify which HBA will be used to create the vHBA. The <parent> XML was extended to allow providing attributes "wwnn" and "wwpn" or "fabric_wwn" in the XML as follows:

<device>
  <parent wwnn='20000000c9848140' wwpn='10000000c9848140'/>
  <capability type='scsi_host'>
    <capability type='fc_host'>
    </capability>
  </capability>
</device>

or:

<device>
  <parent fabric_wwn='2002000573de9a81'/>
  <capability type='scsi_host'>
    <capability type='fc_host'>
    </capability>
  </capability>
</device>

You will note that the parent wwnn/wwpn and fabric_wwn match the same named capability elements found in the scsi_host3 nodedev-dumpxml output. Only one of the three options needs to be provided. If multiple parent options are provided, the parent name will take precedence over the parent wwnn/wwpn which takes precedence over parent fabric_wwn. Only the first found parent element is decoded.

NOTE: As of libvirt 3.0.0 it is also possible to not provide any parent information and the nodedev-create will find an NPIV capable HBA for you.

To create the vHBA use the command "virsh nodedev-create" (assuming above XML file is named "vhba.xml"):

# virsh nodedev-create vhba.xml
Node device scsi_host12 created from vhba.xml

NOTE: If you specify "name" for the vHBA, then it will be ignored. The kernel will automatically pick the next SCSI host name in sequence not already used. If not provided, the "wwpn" and "wwnn" values for the vHBA will be generated by libvirt and the "fabric_wwn" will match that of the parent. In order to provide a specific "wwnn" and "wwpn" use the following example:

<device>
  <parent>scsi_host3</parent>
  <capability type='scsi_host'>
    <capability type='fc_host'>
      <wwnn>2001001b32a9da5e</wwnn>
      <wwpn>2101001b32a9da5e</wwpn>
    </capability>
  </capability>
</device>

In order to see the generated vHBA XML, use the command "virsh nodedev-dumpxml" as follows:

# virsh nodedev-dumpxml scsi_host12
<device>
  <name>scsi_host12</name>
  <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-4/host12</path>
  <parent>scsi_host3</parent>
  <capability type='scsi_host'>
    <host>12</host>
    <unique_id>9</unique_id>
    <capability type='fc_host'>
      <wwnn>5001a4a833f78d55</wwnn>
      <wwpn>5001a4a533cb7cc5</wwpn>
      <fabric_wwn>2002000573de9a81</fabric_wwn>
    </capability>
  </capability>
</device>

This vHBA will only be defined as long the host is not rebooted. In order to create a persistent vHBA, one must use a libvirt storage pool.

Creation of a vHBA by the storage pool¶

Using a storage pool to create and manage the vHBA allows for the discovery and listing of LUN's via the "virsh vol-list" command and provides a seamless mechanism to migrate virtual machine's that are using a vHBA LUN as long as the same storage pool by name is defined and started on the source and target host.

Similar to the node device driver creation, choose the HBA to be used for the vHBA in order to create a 'scsi' storage pool using the storage pool XML syntax as follows:

<pool type='scsi'>
  <name>poolvhba0</name>
  <source>
    <adapter type='fc_host' wwnn='20000000c9831b4b' wwpn='10000000c9831b4b' parent='scsi_host3'/>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

The vHBA must use the pool "type='scsi'". The source adapter attribute type must be "fc_host". The required attributes "wwnn" and "wwpn" provide a unique and consistent naming mechanism for the LUNs.

The "parent" attribute (as of libvirt 1.0.4) provides a mechanism to define which parent HBA will be used to create the vHBA. The "parent_wwnn" and "parent_wwpn" or "parent_fabric_wwn" attributes (as of libvirt 3.0.0) provide a more consistent mechanism to find the parent HBA between host reboots. Similar to the node device the order of precedence is parent followed by the parent_wwnn/parent_wwpn pair, and finally the parent_fabric_wwn.

If none of the parent attributes are provided, the libvirt will pick the first HBA capable of NPIV that has not exceeded its maximum vports. In this instance, the parent remains undefined when displaying the created pool. The following are examples of other parent selection options:

<adapter type='fc_host' parent_wwnn='20000000c9848140' parent_wwpn='10000000c9848140' wwnn='20000000c9831b4b' wwpn='10000000c9831b4b'/>

or

<adapter type='fc_host' parent_fabric_wwn='2002000573de9a81' wwnn='20000000c9831b4b' wwpn='10000000c9831b4b'/>

NOTE: The order of attributes does not matter

# virsh pool-define poolvhba0.xml
Pool poolvhba0 defined from poolvhba0.xml

# virsh pool-start poolvhba0

# virsh pool-autostart poolvhba0

Finding LUN's from a vHBA created using the node device driver¶

Finding an available LUN from a vHBA created using the node device driver can be achieved either via use of the "virsh nodedev-list" command or through manual searching of the hosts system file system.

Use the "virsh nodedev-list --tree | more" and find the parent HBA to which the vHBA was configured. The following example lists the pertinent part of the tree for the example HBA "scsi_host5":

+- scsi_host5
    |
    +- scsi_host7
    +- scsi_target5_0_0
    |   |
    |   +- scsi_5_0_0_0
    |
    +- scsi_target5_0_1
    |   |
    |   +- scsi_5_0_1_0
    |
    +- scsi_target5_0_2
    |   |
    |   +- scsi_5_0_2_0
    |       |
    |       +- block_sdb_3600a0b80005adb0b0000ab2d4cae9254
    |
    +- scsi_target5_0_3
        |
        +- scsi_5_0_3_0

The "block_" indicates it's a block device, the "sdb_" is a convention to signify the the short device path of "/dev/sdb", and the short device path or the number can be used to search the "/dev/disk/by-{id,path,uuid,label}/" name space for the specific LUN by name, for example:

# ls /dev/disk/by-id/ | grep 3600a0b80005adb0b0000ab2d4cae9254
scsi-3600a0b80005adb0b0000ab2d4cae9254

# ls /dev/disk/by-path/ -l | grep sdb
lrwxrwxrwx. 1 root root  9 Sep 16 05:58 pci-0000:04:00.1-fc-0x203500a0b85ad1d7-lun-0 -> ../../sdb

As an option to using "virsh nodedev-list", it is possible to manually iterate through the "/sys/bus/scsi/device" and "/dev/disk/by-path" directory trees in order to find a LUN using the following steps:

1. Iterate over all the directories beginning with the SCSI host number of the vHBA under the "/sys/bus/scsi/devices" tree

For example, if the SCSI host number is 6, the command would be:

# ls /sys/bus/scsi/devices/6:* -d
/sys/bus/scsi/devices/6:0:0:0  /sys/bus/scsi/devices/6:0:1:0
/sys/bus/scsi/devices/6:0:2:0  /sys/bus/scsi/devices/6:0:3:0

2. List the "block" names of all the entries belongs to the SCSI host as follows

# ls /sys/bus/scsi/devices/6:*/block/
/sys/bus/scsi/devices/6:0:2:0/block/:
sdc
/sys/bus/scsi/devices/6:0:3:0/block/:
sdd

This indicates that "scsi_host6" has two LUNs, one is attached to "6:0:2:0", with the short device name "sdc", and the other is attached to "6:0:3:0", with the short device name "sdd".

3. Determine the stable path to the LUN

Unfortunately a device name such as "sdc" is not stable enough for use by libvirt. In order to get the stable path, use the "ls -l /dev/disk/by-path" and look for the "sdc" path:

# ls -l /dev/disk/by-path/ | grep sdc
lrwxrwxrwx. 1 root root  9 Sep 10 22:28 pci-0000:08:00.1-fc-0x205800a4085a3127-lun-0 -> ../../sdc

Thus "/dev/disk/by-path/pci-0000:08:00.1-fc-0x205800a4085a3127-lun-0" is the stable path of the LUN attached to address "6:0:2:0" and will be used in virtual machine configurations.

Finding LUN's from a vHBA created by the storage pool¶

Assuming that a storage pool was created for a vHBA, use the command "virsh vol-list" command in order to generate a list of available LUN's on the vHBA, as follows:

# virsh vol-list poolvhba0
  Name                 Path
------------------------------------------------------------------------------
 unit:0:4:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0
 unit:0:5:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0

The list of LUN names displayed will be available for use as disk volumes in virtual machine configurations.

Using a LUN from a vHBA created by the storage pool (pool)¶

The vHBA LUNs can be added to the domain XML configuration or hotplugged to the domain as either a "disk" or pass-through "lun". The domain XML format documentation describes the various details regarding usage of a "disk" or "lun". When hot-plugging a device, use the "virsh attach-device" command syntax.

NOTE: If migration is important, it's important to understand that volume's name presented in the "virsh vol-list" output (e.g. unit:A:B:C or unit:0:4:0) may differ between two hosts for the same physical vHBA LUN depending on the order in which there were discovered by each of the hosts. Thus for migration a disk assigned to a guest using storage pool and volume name might map to a different disk at the destination host. Hence, it is recommended to describe the disk by path name in the domain XML for such use cases.

<disk type='volume' device='disk'>
  <driver name='qemu' type='raw'/>
  <source pool='poolvhba0' volume='unit:0:4:0'/>
  <target dev='sda' bus='scsi'/>
</disk>

<source pool=poolvhba0' volume='/dev/disk/by-path/pci-0000\:10\:00.0-fc-0x5006016844602198-lun-0'/>

<disk type='volume' device='lun'>
  <driver name='qemu' type='raw'/>
  <source pool='poolvhba0' volume='unit:0:4:0'/>
  <target dev='sda' bus='scsi'/>
</disk>

Using a LUN from a vHBA created by the storage pool (device)¶

If you've gone through the trouble of finding the vHBA LUN, it is still relatively simple to add the LUN to the domain. However, it's important to understand that host reboots will require domain XML modification in order to ensure the proper LUN is being used as it's possible that a hardware reconfiguration or just scsi_host discovery order has changed the path to the LUN.

<disk type='block' device='disk'>
  <driver name='qemu' type='raw'/>
  <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
  <target dev='sda' bus='scsi'/>
</disk>

<disk type='block' device='lun'>
  <driver name='qemu' type='raw'/>
  <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
  <target dev='sda' bus='scsi'/>
</disk>

Discovery of HBA capable of NPIV prior to 1.0.4¶

Prior to libvirt 1.0.4, discovery of HBAs capable of NPIV requires checking each of the HBAs on the host for the capability flag "vport_ops", as follows:

First you need to find out all the HBA by capability flag "scsi_host":

# virsh nodedev-list --cap scsi_host
scsi_host0
scsi_host1
scsi_host2
scsi_host3
scsi_host4
scsi_host5

Now check each HBA to find one with the "vport_ops" capability, either one at a time as follows:

# virsh nodedev-dumpxml scsi_host3
<device>
  <name>scsi_host3</name>
  <parent>pci_0000_00_08_0</parent>
  <capability type='scsi_host'>
    <host>3</host>
  </capability>
</device>

That says "scsi_host3" doesn't support vHBA

# virsh nodedev-dumpxml scsi_host5
<device>
  <name>scsi_host5</name>
  <parent>pci_0000_04_00_1</parent>
  <capability type='scsi_host'>
    <host>5</host>
    <capability type='fc_host'>
      <wwnn>2001001b32a9da4e</wwnn>
      <wwpn>2101001b32a9da4e</wwpn>
      <fabric_wwn>2001000dec9877c1</fabric_wwn>
    </capability>
    <capability type='vport_ops' />
  </capability>
</device>

But "scsi_host5" supports it.

NOTE: In addition to libvirt 1.0.4 automating the lookup of HBA's capable of supporting a vHBA configuration, the XML tags "max_vports" and "vports" will describe the maximum vports allowed and the current vports in use.

As an alternative and smarter way, you can avoid above cumbersome steps by simple script like:

for i in $(virsh nodedev-list --cap scsi_host); do
    if virsh nodedev-dumpxml $i | grep vport_ops > /dev/null; then
        echo $i;
    fi
done

NOTE: It is possible that node device is named "pci_10df_fe00_scsi_host_0". This is because libvirt supports two backends for the node device driver ("udev" and "HAL"), but they lead to completely different naming styles. The udev backend is preferred over the HAL backend since HAL support is in maintenance mode. The udev backend is more common; however, if your destribution packager built the libvirt binaries without the udev backend, then the more complicated names such as "pci_10df_fe00_scsi_host_0" must be used.

<device>
  <name>scsi_host6</name>
  <parent>scsi_host5</parent>
  <capability type='scsi_host'>
    <capability type='fc_host'>
      <wwnn>2001001b32a9da5e</wwnn>
      <wwpn>2101001b32a9da5e</wwpn>
    </capability>
  </capability>
</device>

<pool type='scsi'>
  <name>poolhba0</name>
  <uuid>e9392370-2917-565e-692b-d057f46512d6</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation>
  <available unit='bytes'>0</available>
  <source>
    <adapter name='host0'/>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0700</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>