Skip to content

Home

Jump to bottom
Leonid Bloch edited this page Jan 13, 2016 · 35 revisions

Table of Contents

About VirtHCK

VirtHCK is a set of shell scripts for running Microsoft HCK (Hardware Certification Kit) tests for a variety of Windows drivers, in a VM (virtual machines) environment. It creates an isolated virtual network environment for HCK Controller and Clients' virtual machines, running on top of QEMU-KVM and allows to run various HCK tests in an automated manner.

Moreover, the preparation of the new VMs for HCK functionality, and the installation of the HCK Controller and Clients themselves, can be automated using the auxiliary scripts found under guest_tools/KitAutosetup (along with the corresponding documentation).

Overall, such a setup in a virtualized environment allows enormous flexibility regarding the properties of the machines, the operating systems installed on them, and the ease of switching between them - far above and beyond a physical setup. Additionally, fast recovery from OS corruption is possible (just delete the current disk image and use one from a backup!) and the results are expected to be very consistent - no physical hardware that may malfunction is involved (except, possibly the tested device itself, if it is indeed physical).

VirtHCK Initial Setup

  1. Clone VirtHCK to your home directory:
    git clone https://github.com/daynix/VirtHCK.git
  2. Create a QEMU VM image for the HCK Controller:
    qemu-img create -f qcow2 VirtHCK/images/HCK-Controller-<OS NAME>.qcow2 500G
  3. Create two QEMU VM images for the HCK Clients:
    qemu-img create -f qcow2 VirtHCK/images/HCK-CL1-<OS NAME>.qcow2 500G
    qemu-img create -f qcow2 VirtHCK/images/HCK-CL2-<OS NAME>.qcow2 500G
  4. Update the STUDIO_IMAGE, CLIENT1_IMAGE, and CLIENT2_IMAGE variables in VirtHCK/hck_setup.cfg.
  5. Update the STUDIO_EXTRA, CLIENT1_EXTRA, and CLIENT2_EXTRA variables with the paths to the Windows installation DVD ISO (can be the same path for all).
    • Do not forget to comment these variables after the installation is done!
  6. Modify additional parameters in VirtHCK/hck_setup.cfg, as needed. Refer to their description below.
    • Pay attention that the value of WORLD_BR_NAME corresponds to the bridge name on your machine.
    • Also, if several VirtHCK setups are to be used, make sure that the UNIQUE_ID is truly unique.
  7. Run VirtHCK Setup by: sudo ./VirtHCK/hck.sh.
  8. Connect to the machines' display using VNC or Spice (the correct port will be shown in the summary on screen), and install Windows, as usual.
  9. Prepare the machines for HCK Controller/Studio/Clients installation, and install them.
  10. For additional information on HCK, please refer to Microsoft's documentation.

Directory Tree

  • VirtHCK/ - Directory with all the scripts.
  • VirtHCK/guest_tools/ - Directory with auxiliary scripts, needed for automation and configuration tasks, but not for running VirtHCK.
  • images/ - Directory where the VMs' images should be located.
  • SMB_SHARE/ - Directory that will be used as a Samba share, to transfer files between the Controller, Clients, and the host.

Scripts' Description

VirtHCK/hck.sh

  • Usage:
    VirtHCK/hck.sh [ st | c1 | c2 ] [ c1 | c2] [ c2 ]
This is the main script for VirtHCK environment start up. It starts the Controller, and the two Client VMs. When started, the current VirtHCK environment configuration will be shown in a summary on screen.

This script can be executed with parameters, that specify which VMs to run:

  • st - Start the HCK Controller VM.
  • c1 - Start the HCK Client 1 VM.
  • c2 - Start the HCK Client 2 VM.
By default, the virtual machines will auto-respawn - they will turn on immediately after being shut down. This is necessary for some of the HCK tests. When you wish to actually shut the machines down (like when you want to terminate the setup) press enter in the terminal window where hck.sh was run, and this will keep the machines off when they are turned off.

VirtHCK/hck_setup.cfg

This is the VirtHCK Setup Configuration File, required by scripts for environment configuration and VM startup parameters. It should be configured accordingly to required HCK setup.

Configurable variables in hck_setup.cfg
  1. UNIQUE_ID=[1-80] - Unique ID of the current VirtHCK setup, used for the creation of unique values for: virtual NIC MAC addresses, virtual machine UUIDs, etc. It is very important to set this variable to a unique number for each running VirtHCK setup, in order to avoid system collisions.
  2. QEMU_BIN - QEMU executable path.
  3. TEST_DEV_TYPE, TEST_DEV_NAME, TEST_DEV_EXTRA - Define the VM hardware for testing. Several preconfigured device types are available and commented out. The preconfigured device names are the most frequently used devices for their type. The third variable enables to add extra device parameters, if needed.
  4. IS_PHYSICAL=[false|true] - Sets the test device type to physical - for testing physical devices. Currently supports network devices only.
    In case it is set to true, the following options take effect:
    1. ASSIGNMENT=[vfio-pci|pci-assign] - Sets the device assignment type.
      • pci-assign is the old approach to device assignment in Qemu, and it generally requires no configuration on the host.
      • vfio-pci is the newer approach, however it generally requires some configuration on the host. You can read more about that topic here.
    2. CLIENT1_HOST_ADDRESS, CLIENT2_HOST_ADDRESS - The physical PCI address of the device on the host for clients 1 & 2. Obtained from the lspci output.
  5. MACHINE_TYPE - For example: pc, q35, or their variants. The full list of machine types that are available in Qemu can be obtained by running:
    <The current Qemu binary> -machine help.
  6. VIDEO_TYPE=[VNC|SPICE] - Sets RDC protocol for VirtHCK VMs.
  7. STUDIO_IMAGE, CLIENT1_IMAGE, CLIENT2_IMAGE - Virtual Machines' image names.
  8. CLIENT_CPUS - Amount of vCPU cores for each client VM. Some HCK tests require at least 4 cores.
  9. CLIENT_MEMORY - Sets client's VM memory size. At least 4G is required for some HCK tests, but other tests may fail if the value is over 3G.
  10. WORLD_NET_DEVICE=[e1000|rtl8139|virtio-net-pci] - Set the type of NIC for external Internet connection.
  11. CTRL_NET_DEVICE=[e1000|rtl8139|virtio-net-pci] - Set the type of dedicated NIC for internal network (Controller-Client) interconnection.
  12. STUDIO_EXTRA, CLIENT1_EXTRA, CLIENT2_EXTRA - Variables used for extra VM settings, like CD-ROM attachment, alternate BIOS image path, etc.
  13. SHARE_ON_HOST - A directory on the host machine, that will be available as a Samba share on the HCK Controller and can be made available on the HCK Clients as well, by placing a file named USE_SHARE in it. The default location is: VirtHCK/SMB_SHARE. Setting SHARE_ON_HOST to false will disable the SMB share on both the Controller and the Clients.
    • Before testing a network device, please make sure that the SMB share is disabled on the Clients: if a file named USE_SHARE was present in this directory, delete it, and shut down the Clients (do not reboot - actually shut them down). When the clients will boot again (by auto spawning or manually) the share will not be available on them (but will be on the Controller).

VirtHCK/run_hck_studio.sh

This script is executed from hck.sh, and starts up the Controller (Studio) VM, which is configured according to the settings from hck_setup.cfg.

VirtHCK/run_hck_client.sh

This script is executed from hck.sh, and starts up Client VMs, which are configured according to the settings from hck_setup.cfg.

VirtHCK/hck_ctrl_bridge_ifup.sh

This script is executed by QEMU, it creates a virtual network bridge on the host, for the Controller-Client interconnection.

VirtHCK/hck_test_bridge_ifup.sh

This script is executed by QEMU, it creates a virtual network bridge on the host, for the NIC under test.

VirtHCK/hck_world_bridge_ifup.sh

This script is executed by QEMU, it creates a virtual network bridge on the host, for the VMs to connect to the Internet.

VirtHCK/guest_tools/run_hck_studio.bat

This script should be executed on the HCK Controller, to start the HCK Studio. It disconnects the Controller from the Internet before the HCK Studio is run, and connects it again after the Studio closed. Update the adapter name parameter accordingly to the interface name of the outbound NIC on the Controller.

VirtHCK/guest_tools/enable_world_network.bat

This script could be executed on the Controller in order to enable Internet connection during the HCK Studio run. Update the adapter name parameter accordingly to the interface name of the outbound NIC on the Controller.

VirtHCK/guest_tools/install_cert.bat

  • Usage:
    install_cert.bat TestCert.cer
This script helps to install a test certificate on a Client VM. It should be run with Administrator privileges. It requires a certificate file, and the certmgr.exe console application from WinDDK/WDK in script's directory. certmgr.exe is usually located in:
  • C:\Program Files (x86)\Windows Kits\〈ver. number〉\bin\x86 - for a 32-bit OS
  • C:\Program Files (x86)\Windows Kits\〈ver. number〉\bin\x64 - for a 64-bit OS

Checklist for a New Controller VM

  • This step can be automated using the scripts available in guest_tools/KitAutosetup - see documentation there. Manual instructions are provided below:
  1. Install a VM with Windows 2012 OS (with min. 2 GB of RAM).
  2. Set a static IP address for the control adapter - 192.168.100.1.
  3. Disable the Windows Firewall.
  4. Make all the network connections work/private.
  5. Disable Windows Update.
  6. Disable the screen saver, auto-locking, and auto-sleep.
  7. Rename the control network adapter to Control.
  8. Rename the computer to HCK-STUDIO.
  9. Activate OS.
  10. Install the HCK Controller according to Microsoft's recommendations.
  11. Download the HCK Filters from here, extract the UpdateFilters.sql file from the downloaded .cab archive, put it under the C:\Program Files (x86)\Windows Kits\〈ver. number〉\Hardware Certification Kit\Controller directory (〈ver. number〉 = HCK version) and run the updatefilters.exe file there.
  12. Map the Samba share (\\192.168.101.1\qemu) as a network drive X:.
  13. Shut down the machine (verify it didn't respawn) and then compress the image and put it to the archive directory.

Checklist for a New Client VM

  • This step can be automated using the scripts available in guest_tools/KitAutosetup - see documentation there. Manual instructions are provided below:
  1. Install a pair of VMs with the target OS (the VMs should be installed separately, not copied). Min. amount of RAM per VM: 4 GB. Note: for some tests max. 3 GB is needed.
  2. Activate the computers. (External network access of Clients is required here and ONLY here. It is enabled automatically only when the images are being installed).
  3. Set static IP addresses for the control adapters (Client 1: 192.168.100.2, Client 2: 192.168.100.3).
  4. Rename control network adapter to MessageDevice (this specific name is important for LAN device tests!).
  5. Disable the Windows Firewall.
  6. Make all the network connections work/private.
  7. Disable Windows Update.
  8. Disable the screen saver, auto-locking, and auto-sleep.
  9. Enable automatic reboot and kernel memory dump on bugcheck.
  10. Rename the computers (Client 1: CL1-〈OS〉, Client 2: CL2-〈OS〉).
  11. Install the HCK Client from \\HCK-STUDIO\HCKInstall\Client\setup.exe (the Controller VM needs to be on at this stage).
  12. Optional: Install the test signing certificates (64-bit OSes) by the script guest_tools/install_cert.bat.
  13. Install the drivers for the device being tested.
  14. Run at least one test and verify it passes (Important: at this stage the HCK finalizes the configuration - it is necessary).
  15. Map the Samba share (\\192.168.101.1\qemu) as a network drive X:.
  16. Create a device manager icon on the desktop.
  17. Shut down the machines (verify they didn't respawn), compress their images, and put them to the archive directory.

VirtHCK Automation and Remote Testing

It is possible to automate the tests and to run them remotely, or from the host machine. For more information, please read the documentation of RemoteHCK.

Test Requirements and Recommendations

Fast Installation

  • UNSAFE_CACHE can be set to on in hck_setup.cfg for faster VM installation ONLY. Do not forget to set it to off after the installation, or the VM images may become corrupted!

Storage Tests

  • The HCK Client images should be at least 30 GB in size.
  • The HCK "Crashdump Support Test" requires Client VM Internet connection in order to load the debug symbols from Microsoft's Symbol Server. It can be enabled manually, in the run_hck_client.sh script. Do not forget to turn it off when done!
  • If "Disk Stress" and/or "Disk Verification" tests fail with performance counters warnings on the Client, update the registry performance counters' values: run the following in the command prompt, as the Administrator:
    lodctr /r
    lodctr /q
  • If the "DP WLK - Hot-Add - Device Test" fails with "Interface GUID_DEVINTERFACE_BUSENUM_TOASTER is not registered" error, see this link for the resolution.

Troubleshooting

  • The test logs can be found on the Client machines, in C:\WLK\JobsWorkingDir\WLKSvc.txt.
  • In the case that a test fails with the message: "Waiting for computer to be assigned", make sure that the client is running with at least 4G RAM, and 4 cores. Also check the logs on the clients, for other possible causes.
  • If Windows crashes during tests that involve hibernation (S4), try to install MS updates KB2823506 and KB2822241. Or, run these tests with 3G or RAM.
  • In a case that a networking device test fails with the message:
    WDTF_SIMPLE_IO : NetworkSimpleIO.Network.1 SetTarget() ERROR : NetworkPlugin: SetTarget() - The network interface does not have an IPv6 address
    Follow these steps:
    1. Set an IPv6 address to the NIC under test on the Client machine (a random address, e.g.: fd00::0:0:0:0:0:2).
    2. Set the test parameter WDTFREMOTESYSTEM to the Link-local IPv6 Address of the second client (you can see this address on the other client, under Status → Details of its NIC). Alternatively, set the default gateway of the device under test to that address.
  • In a case of an error that says:
    ... had too many consecutive timeouts while pinging the gateway address or the remote system IP address ...
    Just the second step of the above will suffice to solve the problem.
  • In case that the test "Multiple processor group device" is stuck for a long period of time, check the target Client. In case that the Client's MessageDevice NIC has no network connection, then run the following commands in the command line, as the Administrator:
    bcdedit.exe /set groupsize 64
    bcdedit.exe /set maxgroup off
    bcdedit.exe /set groupaware off
Clone this wiki locally