Automated Application Testing on Multiple Operating Systems

Download Files:
	Macro.zip
	IsWinXPTestSuite.zip
	IsWinXP.zip

Abstract

This article explains automated application testing on different versions of Windows operating systems via virtual machines working under Microsoft Virtual Server 2005, Automated Build Studio, which is SmartBear’s build and release management tool, and TestComplete, an automated testing tool by SmartBear.

In This Article:

• Testing on Different Operating Systems
• Testing on Virtual Machines
• Sample Test
  
  • Preliminary Operations
  • Creating the Macro in Automated Build Studio
    • Compiling the Application to be Tested
    • Preparing Files to be Copied to the Virtual Machines
    • Creating the Testing Submacro
      • Submacro Parameters
      • Submacro Structure
      • Loading a Virtual Machine
      • Testing
      • Waiting for the End of Testing
      • Processing Test Results
      • Shutting Down a Virtual Machine
      • Launching a Submacro
    • Sending Notifications
    • The Whole Macro
    • The Results
• Summary

Testing on Different Operating Systems

Any software developer may face a situation when his or her application works fine on one operating system, but closes abnormally or doesn’t work as expected on another operating system. Why does this happen? The reason is that the Windows operating system changes from version to version. The changes concern the application programming interface, security parameters, principles of working with memory, etc; moreover, an application may work incorrectly on localized versions of the operating system. That is why it is very important to test your application on different operating systems to get a reliable software product.

So, what should you do to start this kind of testing? First, you need to run several operating systems on your computer(s). Next, you need to create automated tests for your application. You can create a test in TestComplete, but in this case, you will have to install the test engine on the computer with the operating system that you want to use to test your application. Sometimes, it is inconvenient to launch a test manually, so you may need to automate testing. For this purpose, you need to have a testing system that will allow you to control processes on a remote machine.

It is often a good strategy to use several physical computers for this kind of testing. The computers must be within the same network environment. Here two questions arise: is it always convenient to use several computers? And is it always possible to have several computers at hand? It may happen that there are not enough computers on which you can perform testing. Even if your company has a lot of computers that can be used for testing purposes, they still may not be available to the tester, since they may have different locations.

One of unpleasant drawbacks that you might come across when testing on several physical computers is that if you test an application repeatedly, each installation may modify the operating system, so it becomes impossible to test a new version of the application correctly. For example, if the installation program works incorrectly, it may not remove some registry keys or files when uninstalling the application. If this happens, you will have to reinstall the operating system or abolish the changes manually, which is time-consuming and will slow down the testing process.

Testing on Virtual Machines

A very good alternative to using several computers for testing is the use of virtual machines. A virtual machine is programmatic simulation of a computer’s hardware environment. It is possible to install the needed operating system on a hard drive of the virtual computer. You can install any operating system on a virtual machine – not just Windows only.

One of the most popular applications used to work with virtual machines is Microsoft Virtual Server 2005. It uses two files to store a virtual machine with one hard drive on it – a configuration file and a file containing an image of the hard drive. Storing an image of the hard drive in a file allows you to test your application on the operating system that was not changed by previous installations. For this purpose, it’s quite enough to copy both files of the virtual machine from some common repository in the network to a computer that has Virtual Server installed.

Virtual Server allows you to launch up to 256 virtual machines simultaneously. This doesn’t require a lot of powerful computers – you just need to have one high-end physical computer to work with Virtual Server and launch the needed number of virtual machines. A running virtual machine becomes available in the network automatically, so it’s possible to get access to its resources through the net. The rest of this article describes testing via virtual machines.

Virtual Server has no means to control processes running on supervised virtual machines. However, the fact that remote work with a virtual computer’s resources does not differ from the work with resources of a physical remote computer allows you to use standard means of remote control over processes on virtual computers. One of the most reliable tools, which are used on computers with Windows NT/2000/XP/Server 2003/Vista/Server 2008, is Windows Management Instrumentation (WMI). WMI is a Microsoft implementation of the WBEM protocol (Web-Based Enterprise Management) that defines the standards of access to data in enterprise networks. WMI allows you to safely control a number of resources of a remote computer: its processes, services, registry, etc.

For reasons of security, Microsoft did not implement the ability to remotely launch an interactive process via WMI. At first, this doesn’t appear to allow you to test the application’s user interface, but the RunInteractive.exe application shipped with Automated Build Studio makes it possible to launch interactive processes even if the application is running as a non-interactive process.

SmartBear’s Automated Build Studio 1.4 and later supports WMI and Microsoft Virtual Server 2005, which makes it possible to create macros that will perform automated testing on several virtual machines.

Automated Build Studio’s operations that support WMI allow you to control processes (application launching, closing, enumeration, status checking) and services (status checking and changing, enumeration), to get information about registered COM classes on both local and remote computers, to shut down, power off or reboot a remote computer and to install, advertise, upgrade, reinstall or uninstall applications on local and remote computers.

Figure 1. ABS operations used to work with WMI.

The Microsoft Virtual Server category includes the operations used to control the state of a virtual computer and its devices, register a virtual machine, get information about available machines and their devices, simulate key presses (this is needed for entering the user name and password when loading the operating system). Virtual Server can be installed on a computer that has Automated Build Studio installed or on a remote computer.

Figure 2. ABS operations used to work with Virtual Server.

So, as you can see, using Automated Build Studio, you can easily test your applications on physical computers and on any number of virtual machines.

In order to organize testing on virtual machines, you will need the following:

• A client computer with Automated Build Studio 6 installed on it.
• A server computer with Windows Server 2003 (recommended) or later and with Virtual Server 2005 installed on it.
• The needed number of images of virtual machines with the application you are going to use for testing (for example, SmartBear’s TestExecute 8) and shared folders.

If Virtual Server is installed on the computer on which Automated Build Studio is installed, only the resources of one physical computer are needed to perform testing.

Note: In order for TestExecute 8 to be run on a virtual machine, this virtual machine must be connected to your local network and have access to the computer on which the license used by TestExecute was activated.

To prepare for testing, you need to do the following:

• Create a test and, if necessary, modify the application code.
• Prepare virtual machines (you will have to create virtual computers, install operating systems on them as well as TestExecute, customize the security system of WMI and share resources).
• Create a macro for Automated Build Studio that will comprise the sequence of actions.

In the next section, we will tell you how to create a test system on several virtual computers using only one physical computer.

Sample Test

Suppose the logic of your application is built on whether the application is running on Windows XP or not. Of course, the code that checks the version of the operating system must be perfect. The project IsWinXP is a project of an MFC application (Visual Studio 2010) that can check the operating system version.

The CIsWinXPDlg::CheckWinXP() function of the project’s IsWinXPDlg.cpp module generates exceptions if the current operating system is not Windows XP. We are going to test this behavior, for example, on virtual machines with Windows XP and Windows Vista. On Windows XP the CheckWinXP function is supposed to be executed with no exceptions and the function is supposed to generate an exception on Windows Vista.

We will test the function using TestComplete’s unit testing capabilities. The predefined test project TestIsWinXP will be launched via TestExecute on each virtual machine.

To run the test, we will create a testing macro for Automated Build Studio named MacroTestIsWinXP.bpp (you can find it in Macro.zip). We’ll describe creation of such a macro in detail in the Creating the Macro in Automated Build Studio section of this article.

To launch the macro on virtual machines, we will use the RunTest.bat file (you can find this file in Macro.zip).

Preliminary Operations

First, we will prepare the physical computer for testing. For this purpose, we’ll do the following:

• Install Automated Build Studio 6.
• Install Microsoft Virtual Server 2005.
• Create two virtual machines, test_xp and test_vista, on the server and install Windows XP and Windows Vista on them.
• Add these virtual machines to the network domain or to the working group available to the physical computer.
• On the physical computer, we will create a folder in which all the files needed for testing will be stored. Let it be a folder with the C:\Testing path. It will have the following structure:

Figure 3. The folder with files used for testing.

The IsWinXP subfolder contains the source files of the tested project.

The IsWinXPTestSuite folder contains files of the TestComplete project suite of the same name.

The Dlls folder contains MFC dynamic link libraries (mfc100d.dll and msvcr100d.dll) needed to launch and test the IsWinXP.exe application.

The MacroTestIsWinXP.bbp macro will copy all the files needed for testing to the VMFolder folder. This folder will also contain the following:

• A new version of the application under test (IsWinXP.exe, IsWinXP.pdb).
• MFC libraries.
• The RunInteractive.exe application taken from the folder where Automated Build Studio is installed.
• The RunTest.bat batch file that will launch the macro.
• The TestComplete project files.

So, the VMFolder folder is used as a temporary buffer when a new version of the tested application along with the automated tests is being copied to the virtual machine.

Figure 4. The folder containing the files that will be copied to the virtual machine.

Now we are going to prepare each virtual machine for testing. On each machine, we will install TestExecute 8 and Virtual Machine Additions. (Virtual Machine Additions are shipped with Virtual Server. ABS needs them to be able to work with Virtual Server.)

To launch a process on a virtual machine, you will have to customize the WMI security system on it. To do this:

• Open the Computer Management console (Control Panel | Administrative Tools | Computer Management) on the virtual computer.
• Right-click the Services and Applications | WMI Control node in the tree on the left side of the Computer Management window and select Properties from the context menu.
• Switch to the Security tab of the ensuing WMI Control Properties dialog.
• Select the Root | CIMV2 node from the tree of namespaces displayed on the Security page and click the Security button.
• Select the needed user account and grant all permissions to it so you can create a Windows account.
• Click OK in the Security and WMI Control Properties dialogs to save your changes.

In our example, we will refer to WMI of both virtual machines using an account with the name tester.

Now we need to set up the security policy as the RunInteractive.exe application requires. For this purpose —

• Open the Local Security Settings dialog (Control Panel | Administrative Tools | Local Security Policy).
• Select Local Policies | User Rights Assignment from the tree view on the left side of the dialog.
• Add the user account that will be used to access WMI on the virtual computer to the following lists of security setting displayed on the right side of the dialog:
  • Act as part of the operating system.
  • Create a token object.
  • Replace a process level token.
• After the user account is added to the above-mentioned lists of security settings, restart the virtual machine.

To enable the client computer to exchange files with the virtual machine, you need to create and share the C:\Testing folder on the virtual computer. Before testing, the MacroTestIsWinXP.bbp macro will copy the files taken from the VMFolder folder (in our case, it is C:\Testing\VMFolder) located on the physical computer to the C:\Testing folder on the virtual computer.

Creating the Macro in Automated Build Studio

Now it’s time to create the MacroTestIsWinXP macro. As we have already said, the macro will initiate the testing process on the two virtual machines.

To be more exact, this macro will sequentially perform the following operations:

• Compile the IsWinXP project.
• Prepare files to be copied to the virtual machines.
• Perform testing on the virtual machine with Windows XP.
• Perform testing on the virtual machine with Windows Vista.
• Send test results to developers.

To store automated test results obtained from each of the virtual machines, we will use the TestResults macro variable. To create it, choose Variables | Variables from the main menu of Automated Build Studio and use the ensuing Variables dialog:

Figure 5. The Variables dialog.

One of the main parameters of the macro will be the path to the folder that contains files needed for testing on the physical computer. Now we need to specify the path to this folder using the TestingFolder constant. (To add this constant, choose the Variables | Constants item from Automated Build Studio’s main menu.) We have already said that this folder is C:\Testing. We will specify this path as a value of the TestingFolder constant:

Figure 6. The Constants dialog.

Specifying the path as a constant and use this constant instead of the real path which allows you to easily change the path when it is needed.

Compiling the Application to be Tested

Often, an automated test is launched after a new version of the product is compiled; our macro will compile the application before testing as well. To enable the macro to do this, we will add the Compile Visual C++ 2010 Project operation to it and specify the project to be compiled (IsWinXP.vcxproj).

Figure 7. The Project page of the “Compile Visual C++ 2010 Project” operation’s properties window.

Note that the path to the project contains the %TestingFolder% substring. To “tell” Automated Build Studio that part of the entered string is the name of a variable or constant, you need to enclose the name in the % symbols. This means that during the macro execution, the substring enclosed in %% will be replaced with the constant value or with the current value of the variable. In our case, the path to the compiled project will look like this during the macro execution:

C:\Testing\IsWinXP\IsWinXP.vcxproj

Preparing Files to be Copied to the Virtual Machines

After you got a new build of the tested application, you should copy all of the needed files to the VMFolder folder located on the physical computer. Do not forget to delete all existing files from in that folder before copying.

Figure 8. The macro snippet that copies all the files needed for testing to the VMFolder folder.

Once the files are copied, we are ready to run tests on the virtual machines.

Before you start testing, you need to initialize the TestResults variable with an empty string. For this purpose, use the Set/Reset Variable Value operation.

Figure 9. Initialization of the TestResults variable.

Since the default value of TestResults was set to an empty string when the variable was created, you can also use the Reset to default property of the Set/Reset Variable Value operation to initialize TestResults with an empty string.

Creating the Testing Submacro

Now we are ready to test the application. In our case, it will be tested on both virtual machines in a similar way, that is, the sequence of performed operations will be the same. In order not to overload the physical computer, we will run the tests one after another. This guarantees that only one virtual machine will be working at a time. All the needed operations will be included in the submacro called VMTest. Upon initializing the TestResults variable, we will call the VMTest submacro twice – it will test the application on the virtual computers.

Submacro Parameters

In Automated Build Studio, a submacro is like a routine in programming languages. You can parameterize a submacro, allowing you to call it from the macro body with appropriate parameters. The VMTest submacro has a lot of parameters that you can change when testing your application on multiple virtual machines.

Figure 10. The Parameters page of the “Submacro” operation’s properties window.

Here is a brief description of each parameter of the VMTest submacro:

Parameter	Description
ConfigName	The virtual computer name – test_xp or test_vista.
NetworkName	The network name of the virtual computer (this is the name under which the computer is registered on the domain or in the working group). For example, vm_test_xp or vm_test_vista.
WMIUser	The user account that will be used to refer to WMI on a virtual machine.
WMIPassword	The password used to authorize the WMIUser user when referring to WMI.
VMUser	The user name that will be entered when starting the virtual machine.
VMPassword	The password of the VMUser user that will be entered when starting the virtual machine.
WMIQueryTime WMIQueries	They are used as parameters when the macro is waiting for the end of the test run.
VMQueryTime VMQueries	They are used as parameters when the macro is waiting for the invitation to enter the user name and password when loading a virtual machine.
DelayAfterLogon	The time (in milliseconds) needed to load Windows completely after the user name and password were entered.
WindowsVersion	The version of Windows running on the virtual computer. For Windows NT/2000/XP/Server 2003, this parameter must be equal to 0; for Windows Vista and later, this parameter must be equal to 1. The submacro uses this parameter to determine what sequence of actions is needed to be performed for logging onto the system properly.

Parameters are created in the Submacro operation, while the parameter values are specified in the Run Submacro operation that will execute this submacro. The values used to call the submacro depend on which virtual machine you will run. You can find them in the MacroTestIsWinXP.bpp macro.

Submacro Structure

The VMTest submacro can be divided into the following logical parts:

• First, we start a virtual machine and wait until it is fully loaded.
• Second, we launch tests.
• Third, we wait until the test runs are completed.
• Fourth, we get test results from the virtual computer.
• Finally, the virtual computer must be shut down in order to free the resources of the physical computer.

Now let’s take a closer look at each step.

Loading a Virtual Machine

First, we need to start the virtual computer using the Run Virtual Machine (Virtual Server) operation. In our case, Virtual Server is on the computer on which Automated Build Studio is installed, so we will specify the Use local server option on the Connection Options page of the properties window here and in all other operations that work with the virtual machines.

Figure 11. The Connection Options page of the “Run Virtual Machine (Virtual Server)” operation’s properties window.

We will specify the ConfigName parameter passed to the submacro as the name of the virtual machine (the Virtual machine name property on the Virtual Machine Options page).

Figure 12. The Virtual Machine Options page of the “Run Virtual Machine (Virtual Server)” operation’s properties window.

Once the virtual machine has been run, the operating system starts loading. It is very important to correctly determine the moment when you can enter the user name and password. There are several ways to wait until the operating system invites you to enter them. The easiest way is to use the Delay operation with a large waiting period. After this time expires, the macro starts simulating keystrokes, but this approach is not very effective.

In Windows NT/2000/XP/Server 2003, you can log onto Windows by using the Welcome screen or the standard Log On To Windows dialog box. The Welcome screen displays the list of available user names and allows you to choose the needed user name by clicking it to log on. If the Welcome screen is turned off (you can do it in the Control Panel) or if the computer is a member of a network domain (in this case the Welcome screen is not available), users log onto Windows by typing their user names and passwords in the Log On To Windows dialog. We will consider only logging onto Windows through this dialog because virtual machines used for testing are members of a network domain. On virtual machines that work under Windows NT/2000/XP/Server 2003 with Virtual Machine Additions installed, it is possible to “catch” the Log On To Windows dialog, that offers you to press the Ctrl+Alt+Delete shortcut, by using the If OS Is Loaded (Virtual Server) operation. This relative operation is a container. In other words, its sub-operations are executed only if the operating system, running on the virtual computer specified in the operation parameters, can accept commands from the user. After the operation returns True, you can simulate pressing the Ctrl + Alt + Delete shortcut.

In Windows Vista and later operating systems, you can log onto Windows only by using the Welcome screen that cannot be turned off. The Log On To Windows dialog is not displayed anymore in these Windows versions. However, you can “catch” the appearance of the Welcome screen in Windows Vista and later by using the If OS Is Loaded (Virtual Server) operation in the same way as you do this with the Log On To Windows dialog in Windows NT/2000/XP/Server 2003.

The waiting process can be organized as a loop with the If OS Is Loaded (Virtual Server) operation executed in it. The exit from the loop occurs when the operation returns True. Make sure the loop is not endless; otherwise, if an error occurs on the virtual machine or on Virtual Server, the macro execution may never come to an end.

Perhaps, one of the best ways to wait for the operating system to offer a user to log on is to organize a loop with a specified number of steps – this can be done with the For Loop operation. The number of steps is passed within the VMQueries parameter of the submacro. At each step, after the timeout specified by the VMQueryTime parameter has expired, the If OS Is Loaded (Virtual Server) operation is executed. If this operation returns True, the Go to Label operation is executed and the macro execution goes to the StartLogon label, which is beyond the loop.

Figure 13. The macro snippet that waits until the operating system is loaded.

If the values of the VMQueries and VMQueryTime submacro parameters are 36 and 5000, respectively, in the worst situation, the loop is executed 36 times and during each execution the If OS Is Loaded (Virtual Server) operation is preceded by a 5-second delay. The values of these parameters mostly depend on the performance of your physical computer.

The next step is the simulation of keystrokes. To load Windows NT/2000/XP/Server 2003 by using the Log On To Windows dialog, it is necessary to press the Ctrl + Alt + Delete shortcut, enter the user name and password, and simulate pressing the Enter button. Automated Build Studio includes a convenient tool used to simulate keystrokes on virtual computers. The Send Key Sequences (Virtual Server) operation allows you to use either a standard sequence of keystrokes or specify the needed sequence explicitly.

Figure 14. Types of key sequences that can be simulated.

At this point, we need to “press” Ctrl + Alt + Delete. This is a standard combination of keys, so we will select it from the popup menu displayed on clicking the Add button on the Key Sequence property page of the Send Key Sequences (Virtual Server) operation’s properties window. To focus the field in which the user name will be typed, we will use another standard combination – Shift + Tab. Once the field is focused, we can enter the user name: select the ASCII Key Sequence item from the popup menu and specify the VMUser submacro parameter in the ensuing Input ASCII Key Sequence dialog (do not forget to enclose the parameter’s name in the % symbols):

Figure 15. The Input ASCII Key Sequence dialog.

To switch to the field in which the password should be entered, choose the standard Tab key. Then, enter the password by specifying the VMPassword submacro parameter as an ASCII sequence in the same way you entered the user name. Finally, simulate pressing the Enter key.

Figure 16. Keystrokes simulated when Windows NT/2000/XP/Server 2003 is being loaded.

Since you can log onto Windows Vista and later operating systems only through the Welcome screen, you need to simulate keystrokes differently from the above-mentioned key sequence when such an operating system is being loaded. We will consider actions to be performed during a logon process on a virtual machine which is a member of a network domain. When the Welcome screen is displayed, you need to press the Ctrl + Alt + Delete shortcut, then, press Tab twice to focus the Switch User button and press Enter. Then, the Welcome screen switches to the next page where you need to press the Right Arrow key to focus the Other User button and press Enter to move to the next page. On the subsequent page of the Welcome screen, you need to enter the user name and password to the appropriate fields and press Enter. The whole key sequence to be sent to the virtual machine to simulate these actions is shown on the following figure:

Figure 17. Keystrokes simulated when Windows Vista/Server 2008 is being loaded.

Note that we have inserted a few 500 ms delays to the simulated key sequence. It is necessary because the Welcome screen needs some time to switch from one page to another. If the keystrokes are simulated earlier than the needed page is displayed, the logon may fail. Such delays are inserted to the key sequence by clicking the Add button in the Send Key Sequence (Virtual Server) operation’s properties window and selecting Delay from the ensuing popup menu.

To simulate pressing the Right Arrow key, click the Add button and select Virtual Server Key Sequence from the ensuing popup menu. In the subsequent Input Virtual Server Key Sequence dialog, press F5 to start recording a key sequence and then press the Right Arrow key. To stop recording, press Break. The Input Virtual Server Key Sequence dialog displays then the recorded key sequence in the form used by Microsoft Virtual Server 2005. Click OK to save to changes.

Figure 18. The Input Virtual Server Key Sequence dialog with the recorded key sequence.

The Send Key Sequences (Virtual Server) operation will successively simulate all keystrokes on the current virtual machine.

As we perform testing on Windows XP and Windows Vista and it is needed to perform different actions when logging on to these operating systems, the submacro must check the version of the currently loaded operating system in order to simulate proper keystrokes. For this purpose, the submacro uses its WindowsVersion parameter. Depending on the parameter’s value passed to the submacro, the needed key sequence is sent to the virtual machine when it is loaded. To check the parameter’s value and make a decision on what keystrokes to send to the virtual machine, we use the If…Then and Else operations in the submacro.

Once the user name and password have been entered, you need to wait until the operating system is loaded. We will use the Delay operation whose delay interval will be specified by the value of the DelayAfterLogon submacro parameter. This value will depend on the performance of your physical computer and, perhaps, on the operating system installed on the virtual computer.

Testing

After the above-mentioned operations are performed, the operating system is running on the virtual computer. Its C:\Testing shared folder is available on the network. Our task is now to copy all the files needed for testing to that folder. The files are copied from the VMFolder folder located on the physical computer by using the Copy File(s) operation:

Figure 19. The properties of the “Copy File(s)” operation.

On the network, we use a symbolic name of the virtual computer. It is specified by the NetworkName submacro’s parameter. The path to the source folder (VMFolder) is specified by the TestingFolder macro constant.

Now everything is ready for testing. The RunTest.bat batch file copied to the virtual computer will launch the RunInteractive.exe application which, in its turn, will run the automated test in TestExecute 8 in interactive mode:

RunInteractive.exe RunInteractive.exe “C:\Program Files\Automated QA\TestExecute 8\Bin\TestExecute.exe” “C:\Testing\IsWinXPTestSuite\IsWinXPTestSuite.pjs” /r /p:TestIsWinXP /u:Unit1 /rt:main /SilentMode /e

Note: We use the RunTest.bat file just to show that you can execute not only .exe files. In your macros, you can also execute RunInteractive.exe and pass the path and parameters of the testing application to it.

To execute the RunTest.bat file remotely, we can use the Run Process (WMI) operation. On the Connection Options page of the operation’s properties window, we’ll specify the parameters of connection to WMI:

Figure 20. The Connection Options page of the “Run Process (WMI)” operation’s properties window.

Here, we specified the WMIUser and WMIPassword submacro’s parameters as the user name and password. The name of the WMI server is specified by the value of the NetworkName parameter.

The parameters of the launched process must be specified on the Run Options page of the operation’s properties window:

Figure 21. The Run Options tab of the “Run Process (WMI)” operation’s properties window.

Waiting for the End of Testing

In most cases, testing of complex applications takes a lot of time. If the tester does not use some special tools, it’s hard to evaluate when the test results become available. Finalization of the TestExecute.exe process signals that the test run is over. Automated Build Studio includes the If Process is Running (WMI) operation that checks whether there is some process running on a remote computer. You can specify parameters of the process to be checked on the Process Options page of the operation’s properties window. To identify the process, we will use its name — TestExecute.exe.

Figure 22. The Process Options page of the “If Process Is Running (WMI)” operation’s properties window.

All of the above information on how to make the operating system wait until keystrokes are simulated at startup of the virtual computer can be applied to the finalization of the process. The submacro snippet below demonstrates how you can make the operating system wait for the TestExecute.exe process to finish:

Figure 23. The macro snippet that waits for the end of the TestExecute process.

The number of loop steps is passed through the WMIQueries submacro parameter. The length of the timeout on each step is specified by the WMIQueryTime parameter. If the If Process is Running (WMI) operation returns True, the next loop iteration starts; otherwise, the loop is broken, the Go to Label operation is executed and the execution goes to the TestOk label, which is beyond the loop.

Processing Test Results

After the test run, you have test results to process. We will transfer the test results from TestExecute to ABS using a temporary file: the test project executed by TestExecute saves brief results to the TestResult.txt file on the virtual machine, and the ABS macro reads data from this file and processes it. We will add the data stored in this file to the TestResults macro variable that will be used to collect results.

We will save the results in the following format:

Below is the Jscript procedure that reads data from the TestResult.txt file and adds it to the TestResults variable:

function Main()
{
    Variables.TestResults =
        Variables.TestResults +
        “-----------------------------------------\n” +
        “Virtual machine: ” +
        Variables.ConfigName +
        “\n-----------------------------------------\n”;

    strResultFileName = 
        “\\\\” +
        Variables.NetworkName + 
        “\\Testing\\IsWinXPTestSuite\\TestIsWinXP\\TestResult.txt”;

    FileSystem = new ActiveXObject(“Scripting.FileSystemObject”);

    try
    {
        FileStream = FileSystem.OpenTextFile(strResultFileName, 1);

        try
        {
            Variables.TestResults = 
                Variables.TestResults +
                FileStream.ReadAll() + “\n”;
        }
        finally
        {
            FileStream.Close();
        }
    }
    catch(e)
    {
        Variables.TestResults = Variables.TestResults + “I/O ERROR\n\n”;
    }
}

The script adds the caption, which includes the name of the virtual machine (the submacro’s ConfigName parameter) on which the testing was performed, to the global variable TestResults as well as the contents of the TestResult.txt file that resides in the Testing shared folder located on the virtual computer. For example, after a test run on the test_xp virtual machine, the following text will be added to the TestResults variable:

-----------------------------------------
Virtual machine: test_xp
-----------------------------------------
CheckWinXP - Ok!

After you read the test results, you can delete all the files from the C:\Testing folder located on the virtual machine. For this purpose, use the Delete File(s) operation. To delete all files from the Testing shared folder, you can specify the \\%NetworkName%\Testing\*.* string as a file mask in the operation’s properties. If you plan to use a “clean” image of the operating system for each test, there is no need to remove these files.

Shutting Down a Virtual Machine

After a test run, you can shut down the virtual computers. To do this, use the Reset/Shut Down/Turn Off Virtual Machine (Virtual Server) operation. To shut down a virtual machine, choose the Shut down guest OS option in the Action settings group on the Reset/Shut Down/Turn Off Options page in the operation’s properties window.

Figure 24. The Reset/Shut Down/Turn Off Options page of the “Reset/Shut Down/Turn Off Virtual Machine (Virtual Server)” operation’s properties window.

Launching a Submacro

Now let’s get back to creation of the basic macro. Upon the first initialization of the TestResults variable, we will call the VMTest submacro to perform automated testing on the test_xp and test_vista virtual machines. To call the submacro, use the Run Submacro operation in the macro body.

Figure 25. The properties of the “Run Submacro” operation.

Sending Notifications

After the two submacros launched sequentially are executed, you can notify the developers and testers about the test results (they are kept in the TestResults variable). You can send notifications via e-mail by using the Send E-mail operation. As it is shown on the image below, we specified the value of the TestResults variable in the message body on the Message settings page of the operation’s properties window.

Figure 26. The Message page of the “Send E-mail” operation’s properties window.

After the automated testing has finished, the results will be sent to the recipients specified in the Send E-mail operation’s properties.

The Whole Macro

The whole macro we have created looks like this:

Figure 27. The MacroTestIsWinXP macro.

Download the macro

The Results

The parameters that we used when calling the VMTest submacro are in the table below:

Parameter	Operating Systems
	Windows XP	Windows Vista
ConfigName	test_xp	test_vista
NetworkName	vm_test_xp	vm_test_vista
WMIUser	vm_test_xp\tester	vm_test_vista\tester
WMIPassword	tester	tester
VMUser	tester	tester
VMPassword	tester	tester
WMIQueryTime	5000	5000
WMIQueries	36	36
VMQueryTime	5000	5000
VMQueries	36	36
DelayAfterLogon	45000	45000
WindowsVersion	0	1

We got the following results in the TestResults variable:

-----------------------------------------
Virtual machine: test_xp
-----------------------------------------
CheckWinXP - Ok!

-----------------------------------------
Virtual machine: test_vista
-----------------------------------------
CheckWinXP raised an exception

As you can see, the test results are the ones we expected. The CheckWinXP() function did generate an exception when working on the operating system different from Windows XP.

Summary

In this article, we described how to test applications on virtual computers using Automated Build Studio and TestComplete (TestExecute). You can customize your test system by modifying the aforementioned macro. This will make it more flexible and it will be able to meet many of your testing needs. Testing on virtual machines allows you to quickly find possible errors in your applications; this kind of testing is not resource-consuming, which is also very important. Using Automated Build Studio, you can automate the testing process and make it even more flexible.