Cimtest

From Libvirt Wiki
Revision as of 01:47, 14 August 2008 by Daisy (talk | contribs) (Known Provider Issues)
Jump to: navigation, search

cimtest setup instructions

System requirement
  • Python-2.5
  • PyXML (on Redhat systems) or pyxml (on SUSE systems)
  • PyWBEM - recent CVS sources
  • Mercurial (for checking out the latest cimtest tree):
$ hg clone http://libvirt.org/hg/cimtest/
Setup environment
  • Prepare the hypervisor kernel, initrd ramdisk and disk image files
# cd path_to_cimtest/suites/libvirt-cim/images/
# make
  • Add the user account's ssh public key to root's authorized_keys. You'll need to use ssh-keygen -t dsa to generate keys for the user account if you don't have existing keys.
  • Run the cimtest automation
# cd path_to_cimtest
# ./runtests libvirt-cim -i localhost
  • If using Pegasus as the CIMOM, you'll need to supply the Pegasus authentication info for your system:
# CIM_NS=root/virt CIM_USER=pegasus CIM_PASS=password ./runtests libvirt-cim -i  localhost
  • There are a few options available:
-c removes the log files prior to a run
-d prints the error messages to stdout in additional to printing to the log files (default behavior)
-g <suite_dir> runs a particular directory within the suite directory
-t <tc_name> runs a specific test for a given directory in the test suite
-v <virtualization type> runs the tests specifically for the type specified. Xen, XenFV, KVM, and LXC are currently supported. Xen is the current default. 
# CIM_NS=root/virt CIM_USER=pegasus CIM_PASS=password ./runtests libvirt-cim -i localhost -c -d -v KVM -g ComputerSystem -t 06_paused_active_suspend.py

KVM on F9 release rpm

Date PASS FAIL SKIP XFAIL Total
Jun18, 2008 104 7 16 2 129
May 26, 2008 98 7 18 6 129
May 19, 2008 96 6 18 9 129
May 12, 2008 96 5 19 9 129
May 7, 2008 94 9 17 9 129
April 30, 2008 83 16 17 7 129
April 24, 2008 77 26 20 6 129
April 21, 2008 78 22 22 7 129
April 14, 2008 70 18 34 7 129

Current Failures

Test Case Issue Person Assigned Remarks
ComputerSystemIndication - 01_created_indication.py
ElementConforms - 02_reverse.py
ElementSettingData - 03_esd_assoc_with_rasd_errs.py Test expects "Missing InstanceID", but the provider returns "No such instance (InstanceID)". This test needs to be branched.
NetworkPort - 03_user_netport.py This needs to be an XFAIL - the provider doesn't support guests with user interface types.
SettingsDefine - 02_reverse.py InstanceID Mismatch, Exception in VSSDCAssoc function: string indices must be integers
VirtualSystemManagementService - 06_addresource.py InvokeMethod(AddResourceSettings): CIM_ERR_FAILED: Failed to create domain
VirtualSystemManagementService - 08_modifyresource.py InvokeMethod(ModifyResourceSettings): CIM_ERR_FAILED: Failed to create domain

KVM on current sources

Date Changeset CIMOM PASS FAIL SKIP XFAIL Total Remarks
Aug 13, 2008 665 Pegasus 125 1 6 2 134 The test case which had failed passed when run manually.
Aug 11, 2008 665 sfcb 118 6 6 3 133
Aug 06, 2008 660 sfcb 110 6 14 3 133
Aug 06, 2008 660 pegasus 120 4 7 2 133 One tc passed when run manually, need to update the other 3.
July 30, 2008 651 pegasus 113 4 12 1 130
July 23, 2008 640 pegasus 111 1 16 2 130
July 16, 2008 637 pegasus 110 1 15 4 130
July 02, 2008 624 pegasus 108 0 18 4 130
June 24, 2008 619 pegasus 106 1 18 4 129
June 18, 2008 611 pegasus 101 8 18 2 129
June 11, 2008 603 pegasus 101 8 18 2 129
June 04, 2008 599 pegasus 101 8 18 2 129
May 28, 2008 592 pegasus 102 9 16 2 129
May 21, 2008 592 pegasus 102 7 16 4 129
May 15, 2008 587 pegasus 100 9 16 4 129
May 7, 2008 580 pegasus 94 14 17 4 129
April 30, 2008 565 pegasus 90 18 17 4 129

Current Failures

Test Case CIMOM Issue Person Assigned Remarks
ComputerSystemIndication - 01_created_indication.py sfcb Problem processing indication to http://localhost:8008. sfcb rc: 4 CURL error: 52 (Server returned nothing (no headers, no data))
ResourcePoolConfigurationService - 03_CreateResourcePool.py sfcb ERROR - Unexpected rc code 7 and description The requested operation is not supported

InvokeMethod(CreateResourcePool): The requested operation is not supported

ResourcePoolConfigurationService - 04_CreateChildResourcePool.py sfcb ERROR - Unexpected rc code 7 and description The requested operation is not supported

InvokeMethod(CreateChildResourcePool): The requested operation is not supported

ResourcePoolConfigurationService - 06_RemoveResourcesFromResourcePool.py sfcb ERROR - Unexpected rc code 7 and description The requested operation is not supported

InvokeMethod(RemoveResourcesFromResourcePool): The requested operation is not supported

ResourcePoolConfigurationService - 07_DeleteResourcePool.py sfcb ERROR - Unexpected rc code 7 and description The requested operation is not supported

InvokeMethod(DeleteResourcePool): The requested operation is not supported

VirtualSystemManagementService - 05_destroysystem_neg.py: FAIL sfcb ERROR - destroy_fail>> noname: Error executing DestroySystem

ERROR - (1, u'Unable to retrieve domain name.') ERROR - destroy_fail>> nonexistent: Error executing DestroySystem ERROR - (1, u'Failed to find domain') InvokeMethod(DestroySystem): Unable to retrieve domain name. InvokeMethod(DestroySystem): Failed to find domain

XenFV on current sources

Date Changeset PASS FAIL SKIP XFAIL Total Remarks
Aug 07, 2008 660 114 9 7 3 133 5 tc passed when run manually.
July 31, 2008 654 112 12 7 2 133 6 tc passed when run manually, the fix for the other 6 are under review.
July 24, 2008 642 98 15 15 2 130 9 tc passed when run manually, 6 needs to be updated.
July 17, 2008 637 92 14 18 6 130 13 tc passed when run manually, 1 needs to be updated.
July 10, 2008 634 94 7 23 6 130 5 tc passed when run manually
July 03, 2008 629 91 11 22 6 129 9 tc passed when run manually, Need to fix 2 tc
June 26, 2008 620+ 87 17 24 1 129 10 tc passed when run manually, Need to fix 7 tc
June 19, 2008 612+ 89 17 22 1 129 9 tc passed when run manually, Need to fix 8 tc
June 12, 2008 606 88 18 22 1 129 8 tc passed when run manually, Need to fix 10 tc
June 05, 2008 599 88 18 22 1 129 10 tc passed when run manually, Need to fix 8 tc
May 29, 2008 592 87 19 22 1 129
May 22, 2008 592 85 19 22 4 129
May 15, 2008 587 85 18 22 4 129
May 8, 2008 580 76 25 23 3 129
May 1, 2008 571 60 41 25 3 129
April 18, 2008 552 75 18 33 3 129
April 16, 2008 545 54 45 30 2 129

Current Failures

Test Case Issue Person Assigned Remarks
ComputerSystem - 06_paused_active_suspend.py The XenFV domain that is getting created has enablestate set to 0. But when we create a Xen Domain the enablestate is set appropriately. This tc should pass once the provider patch which fixes the "no state" issue on XenFV is submitted.

Fix under review.

ComputerSystem - 23_suspend_suspend.py ERROR - EnabledState should be 2 not 0

ERROR - Attributes for dom test_domain not set as expected

Fix under review.
ComputerSystem - 32_start_reboot.py ERROR - EnabledState should be 2 not 0

ERROR - Attributes for dom test_domain not set as expected

Fix under review.
ComputerSystem - 33_suspend_reboot.py ERROR - EnabledState should be 2 not 0

ERROR - Attributes for dom test_domain not set as expected

Fix under review.
ComputerSystem - 35_start_reset.py ERROR - EnabledState should be 2 not 0

ERROR - Attributes for dom test_domain not set as expected

Fix under review.
VirtualSystemManagementService - 08_modifyresource.py Unable to get vcpuinfo from virsh, using XML values

INFO - good status for vcpu ERROR - Got 0, exp 262144. ERROR - Error invoking ModifyRS: mod_mem_res ERROR - Error changing rs for mem

Fix under review.

Xen on current sources

Date Changeset PASS FAIL SKIP XFAIL Total Remarks
Aug 08, 2008 660 121 8 2 0 133
July 28, 2008 647 113 15 2 0 130
July 21, 2008 637 113 9 1 7 130
July 14, 2008 634 108 15 1 6 130
July 09, 2008 634 114 8 1 7 130
Jun 30, 2008 621 106 17 3 3 129
Jun 26, 2008 620 104 17 1 7 129


Current Failures

Test Case Issue Person Assigned Remarks
ResourceAllocationFromPool - 01_forward.py ERROR - Return NULL, expect at least one instance Fix is in the cimtest repo, shld pass in the next run.
ResourceAllocationFromPool - 02_reverse.py ERROR - Return NULL, expect at least one instance Fix is in the cimtest repo, shld pass in the next run.

LXC on current sources

Date Changeset PASS FAIL SKIP XFAIL Total Remarks
July 21, 2008 90 1 38 1 130
July 14, 2008 91 0 38 1 130
July 07, 2008 90 1 38 1 130
Jun 30, 2008 88 2 38 1 129
Jun 24, 2008 88 1 39 1 129
Jun 19, 2008 87 1 40 1 129
Jun 13, 2008 603 70 0 58 1 129
Jun 10, 2008 603 63 3 63 0 129
Jun 2, 2008 598 36 0 92 1 129


Current Failures

Test Case Issue Person Assigned Remarks
ComputerSystem - 04_defineStartVS.py ERROR - Error: property values are not set for VS domguest

Known Provider Issues

Bug Number Description Platforms Affected Fixed up stream?
00001 RequestStateChange() does not support an empty bootloader tag Xen, KVM (?) Yes
00002 RequestedState property in ComputerSystem is not getting populated Xem, KVM (?) Yes
00003 ModifyResourceSettings doesn't work for cpu numbers Xen,KVM (?) Yes
00004 NetworkPort instance is not returned for a guest created with a user interface type KVM (user) Yes
00005 libvirt does not support rebooting the KVM guest. KVM No
00006 libvirt does not support virConnectGetVersion function. Yes (?)

To -Do List

Items to Fix / Add

  • Be able to call cxml.provider_define() and cxml.provider_start(), etc
  • Use rm -rf to clean up old test logs
  • HRP 01 - 04 need to support disk and network pools
  • Remove computersystem.py and hostsystem.py - this behavior should be covered by enumclass.py -- test cases will need to be updated
  • Move libvirt/xm related items from live.py to the libvirt-cim suites directory
  • VSSDC 01 - needs to support KVM and LXC, this can be done by only verifying the bootloader in the Xen case.

New Test Case Ideas

  • SettingsDefineState between CS <-> VSSD
  • Create a guest with multiple network devices - verify NetworkPort / NetworkPool / etc
  • AddResource test that attempts to add a CD/DVD device to a KVM guest.
  • AddResource negative test case that attempts to add proc and mem resources to a running guest.
  • Create a guest that had a network interface that doesn't belong to a pool (ethernet for Xen/XenFV, user for KVM?) - verify that the interface doesn't show up in pool lists, RAFP, etc.
  • More involved VirtualSystemSnapshotService tests?
  • ReferencedConfiguration tests: 1) testing invalid ref 2) create a guest (have it running), then define a guest based off the running guest, then attempt to start guest 3) define a guest, define a new guest based off of defined guest, start second guest
  • Processor scheduling - reboot guest, reset guest, suspend / resume guest - verify scheduling persistence.

Test Case Info

Category Test Name Platforms Supported Description
AllocationCapabilities 01_enum.py Xen, XenFV, KVM, LXC EnumerateInstances() of AllocationCapabilities and EnumerateInstances() of each of the device pools (MemoryPool, DiskPool. NetworkPool, ProcessorPool). Verify there is an AC instance that corresponds to each device pool instance.
ComputerSystem 02_nosystems.py Xen, XenFV, KVM, LXC Using virsh, check to see if there are any guests defined on the system. If no guests are returned for Xen/XenFV, return an error to check the health of the system. If guests exist on the system, then the test is skipped. Otherwise, the test verifies that EnumerateInstances() of ComputerSystem returns no instances. If EnumerateInstances() returns an error, the test fails.
ComputerSystem 22_define_suspend.py Xen, XenFV, KVM, LXC Guest is created using DefineSystem() and then the test attempts to suspend the defined guest using RequestStateChange(). This call should fail.
ElementSettingData 01_forward.py Xen, XenFV, KVM, LXC GetInstance() is called to the VSSD and RASD instances corresponding to the guest created. Then ESD is called on each instance. ESD should return the same instance that was used as the calling reference.
LogicalDisk 02_nodevs.py Xen, XenFV, KVM, LXC If there are guests running on the system, return skip. Otherwise, verify EnumerateInstances() returns zero instances.
HostSystem 01_enum.py Xen, XenFV, KVM, LXC EnumerateInstances() is called for the HostSystem class. The resulting list is checked to make sure only one instance is returned. The values of the CreationClassName and the Name properties of that instance are verified.
VirtualSystemManagementService 06_addresource.py Xen, KVM Use virsh to define guest, add additional disk and network devices to defined guest. If guest is Xen, undefine guest, redefine using default XML, and start guest. Add additional disk and network devices.
VirtualSystemManagementService 08_modifyresource.py Xen, XenFV, KVM Use virsh to define guest, modify disk/network/proc/mem devices to defined guest. If guest is Xen or XenFV, undefine guest, redefine using default XML, and start guest. Modify proc and mem.
VSSD 02_bootldr.py Xen only Use virsh to define guest, set bootloader. Do a getInstance() of VSSD and verify Bootloader and ElementName attributes.