Automated Testing With TestComplete and Team Build

Applies to: TestComplete 8

Introduction

Microsoft Visual Studio Team System includes tools and technologies for all members of software development teams: project architects, developers, managers and, of course, software testers and Quality Assurance personnel. It helps team members communicate throughout the software development process.

Team Foundation Build, which is one of Microsoft Visual Studio Team System tools, enables the development team to create software builds regularly and manage this process. It is important that software testers can run their automated tests as part of team builds, so it is possible to test their applications automatically after the build is over.

TestComplete Enterprise Edition includes special packages for integration into Microsoft Visual Studio and Microsoft Team Foundation Server. Currently, the following versions of Visual Studio are supported by TestComplete 8:

• Microsoft Visual Studio 2005 Team Suite or Microsoft Visual Studio 2005 Team Edition for Testers.
• Microsoft Visual Studio Team System 2008 Team Suite or Microsoft Visual Studio Team System 2008 Test Edition.
• Microsoft Visual Studio 2010 (the Professional, Premium and Ultimate editions are supported).

The integration packages extend Visual Studio test projects with the TestComplete automated test type. After you add a TestComplete automated test to a Visual Studio test project, you can create team builds that will run TestComplete tests after the application’s build is over, you can also publish automated test results, create work items for them in Team System’s issue-tracking database and generate reports for the items.

These packages also enable you to use Visual Studio Team System unit tests as part of your TestComplete projects.

In this article we will provide step-by-step instructions of how to use TestComplete automated tests as part of team builds and consider the following points needed for creating and configuring team builds, integrating TestComplete tests with them and automate the testing of your applications:

In our explanation, we assume that the test project is created and configured on one computer, but will be run on the computer that will perform the build. Furthermore, we will only provide explanations for Visual Studio 2008 Team System and Visual Studio 2010.

Creating and Configuring Automated Test Projects

Once you have a TestComplete project (or project suite), you can integrate it in Visual Studio test projects for the further use in team builds. Typically, TestComplete projects are created and modified at a workstation, while Visual Studio builds are run on the build server. If your build uses TestComplete projects that reside on another workstation, make sure the build server has access to these TestComplete projects. In order words, share the folder that contains the TestComplete projects for network access. Furthermore, make sure that the TestComplete projects can access all needed applications on the build server, TestComplete or TestExecute is installed on the build server and the server has all needed third-party libraries installed.

Once your TestComplete project is ready, you can integrate it into Visual Studio tests and builds. You can create test projects in Visual Studio 2010 and add TestComplete automated tests to them the same way you do this in Visual Studio 2008. So, we will describe creating and configuring automated test projects only in Visual Studio 2008.

To create a new test project in Visual Studio 2008, do the following operations:

• Select File | New | Project from Visual Studio’s main menu.
• In the ensuing dialog select one of test project types and specify the project template (for instance, Visual C# Test Project).



• Specify the project name, location and the solution name. Then, click OK.

So, we have just created a new test project. Visual Studio will open and display its contents in the Solution Explorer.

Since the Microsoft Visual Studio 2008 Team System Integration package is installed with TestComplete, you can add TestComplete tests to your Visual Studio test project. If you use TestComplete 8, this automated test type is called TestComplete 8 Test. To add the TestComplete test to your automated test project, do the following:

• Right-click the test project in Visual Studio’s Solution Explorer and select Add | New Test from the context menu.
• In the ensuing Add New Test dialog, select the TestComplete 8 Test in the Templates box:



• Specify the desired test name and the test project. Then press OK.

After adding the TestComplete automated test to your Visual Studio test project, you can modify its properties using a special test editor in Visual Studio. To view this editor, double-click the desired TestComplete test item in Solution Explorer or select Open from its context menu:

In this editor, you should click the ellipsis button and specify the TestComplete project or project suite to be executed during the build testing. The editor will display a tree of test items defined in the project suite or project you have specified. You can select the test items that you want executed during the build testing and clear check boxes for the items that you do not want to include in the automated testing process. Also, in the Execute With section you can specify what automated testing tool should be used for executing TestComplete tests – TestComplete or TestExecute.

Note that if your TestComplete project or project suite is not located on the server computer, where the build will be run, you should specify the path to the project in the TestComplete test editor (see the image above) through My Network Places, i.e. you should specify the project path that points to the shared project folder and include the client computer name.

You have now integrated a TestComplete project with a Visual Studio test project. Now you need to configure the Visual Studio test project so the specified TestComplete automated tests are executed by Visual Studio during the build testing process. Do the following:

• In Visual Studio 2008 Team System, switch to the Test List Editor panel. To display it, select Test | Windows | Test List Editor from the main menu.
• On the left of the Test List Editor panel, right-click the List of Tests node and choose New Test List from the context menu. This will call the Create New Test List dialog:

  
  

  

  In the dialog, specify the test list name, description and click OK.

• Select the All Loaded Tests node in the Test List Editor panel. A list of all available tests in the test project will be displayed on the right section of the panel.
• Drag the desired TestComplete test from the right of Test List Editor to the test tree on the left and drop the test on the created test list.
• Check the TestComplete test to enable it.



• Save the changes made to the test project.

We have finished creating and configuring the test project. Before you create a build and integrate the automated test project with it, it is recommended that you run the project to make sure that it runs properly.

To use your automated test project in team build, you should also implement some more preparatory operations. Since Visual Studio builds operate with files taken from the source control, you need to add the created Visual Studio test project to Team System’s source control. To add the project to the source control system, do the following:

• Set Visual Studio Team Foundation Server as the current source control plug-in for Visual Studio:
  • Select Tools | Options from Visual Studio’s main menu.
  • In the ensuing Options dialog, activate the Source Control | Plug-in Selection settings group and select the Visual Studio Team Foundation Server in the Current source control plug-in box.
  • Press OK to save changes and close the dialog.
• Right-click your solution in the Solution Explorer and select Add Solution to Source Control from the context menu. This will call the following dialog:



• In the dialog, you can specify the source control project and folder where the test will be added. Select the team project that the automated tests are intended for.
• Click OK to add the test project to a source control. Visual Studio will copy the project files to Team System’s source control.
• After Visual Studio adds the solution to the source control, right-click the solution node in the Solution Explorer and select Check In from the context menu. This will call the Check In dialog:



• In the dialog press Check In. Visual Studio will check in the solution files to the source control system.

You may also associate your automated test with a work item in the Team System’s database. This database holds info about the development process: executed tests, reported bugs, assigned tasks and so on. For instance, you may associate an automated test with a bug stored in the database. The test execution will indicate whether the bug is resolved or not.

To associate an automated test with a work item, do the following:

• Check out your test project from the source control.
• In Visual Studio 2008 Team System, open the Test List Editor panel.
• Right-click the desired TestComplete automated test in the Test List Editor panel and select Properties from the context menu.
• Switch to the ensuing Properties panel and click the ellipsis button of the Associated Work Items property. This will call the Associated Work Items dialog that lists all work items associated with your automated test.
• To associate a new work item with your automated test, click Add. Visual Studio will bring up the Work Items Picker dialog where you can select the work item for association.



You can search for the desired items by using a query, or searching by the work item’s title or identifier. After specifying the search conditions, click the Find button.

• Select the desired item(s) in the Work Items Picker dialog and click OK. Identifiers of the selected items will be displayed in the Associated Work Items dialog.



• Click OK to close the Associated Work Items dialog and associate the items with the test.
• Save changes made to the test and check in the modified test project to Team System’s source control.

The creation and configuration of the automated test project is now finished. The next step is to create and configure a team build project.

Creating and Configuring Team Build Projects

Now that you have created and configured an automated test project for a team build, this is a good time to create the team build and integrate the automated test project into it. Note that the process of build creation slightly differs from one version of Visual Studio to another. We will describe how to create and configure a team build in the following IDEs:

Creating Builds in Visual Studio 2008

To create a build in Visual Studio Team System 2008, do the following:

• Share a folder on the server’s hard drive. This folder will store build results.
• Open the Team Explorer panel in Visual Studio. If it is hidden, select View | Team Explorer from Visual Studio’s main menu to display the panel.
• In Team Explorer, right-click the <your_team_project> \ Builds node and select New Build Definition from the context menu.
• The Build Definition dialog displays:



The dialog tabs that require your input are marked with a warning icon.

• On the General tab of the dialog, specify the build name and description.
• On the Workspace tab, specify the source control folder for the team project that you are creating the new build definitions for and a local folder on the build agent.
• On the Project File tab, you can browse for an existing .proj project file or launch the MSBuild Project File Creation Wizard to create a new project file.

  
  

  To create a new project file, click Create – the MSBuild Project File Creation Wizard will be displayed. In the wizard, do the following:

  • On the Selection page, choose the solutions that belong to the team project that you would like to build with the created team build. After selecting the desired solutions, click Next to continue.
  • On the Configurations page of the wizard, specify the build’s configuration and platform. Click Next to go to the last page of the wizard.
  • On the Options page, enable the Run test check box. Then, in the Test metadata file box, specify the desired test project metadata file (.vsmdi file) that has been added to the team project. In the Test list to run box, check which automated tests from the specified test project you would like to execute after the build is over. Then click Finish to create a build project file. These are the main actions you have to perform for integrating automated tests in a team build.



• In the Build Definition dialog, switch to the Build Defaults tab. On this tab, you can choose an existing build agent from the Build agent drop-down list, or create a new agent by clicking the New button (see below).

  
  

  Note that if you want to run TestComplete tests that interact with the desktop and perform UI testing, you need to specify an interactive build agent. Otherwise, if the specified build agent is non-interactive, TestComplete tests cannot interact with the tested application’s UI and fail during the build process.

  To create a new interactive build agent:

  • Click the New button in the Build Definition dialog.
  • In the ensuing Build Agent Properties dialog, specify the name of the agent in the Display name edit box.
  • In the Computer name edit box, enter the name of the computer where the Team Build service is running.
  • In the Communications port edit box, specify the 9192 port number. It is the default port number used by the Team Build Service when it is running in interactive mode. To learn how to change the Team Build Service’s interactive port, see the MSDN Library.
  • In the Description box, you can enter a brief description of the build agent.
  • Then, press OK to create the build agent and close the Build Agent Properties dialog.

  Also, you need to fill in the Builds will be staged to the following share field on the Build Defaults tab with a UNC path to the folder where the built binaries and log files will be stored.

• Finally, click the OK button in the Build Definition dialog to create your build definition. It will be displayed under the Builds node of your team project in the Team Explorer panel.

Creating Builds in Visual Studio 2010

To create a team build in Visual Studio 2010, do the following:

• Share a folder on the server’s hard drive. This folder will store build results.
• Open the Team Explorer panel in Visual Studio. If it is hidden, select View | Team Explorer from Visual Studio’s main menu to display the panel.
• In Team Explorer, right-click the <your_team_project> \ Builds node and select New Build Definition from the context menu.
• The Build Definition window will appear:

  

  The window tabs that require your input are marked with a warning icon.

• On the General tab of the window, specify the build definition name and description.
• On the Trigger tab, if needed, you can specify the event that will cause this team build to be run. By default, team builds are run manually.
• On the Workspace tab, if required, specify the source control directories of the team project you are creating the new build definition for and map these source control directories to the appropriate local directories on the build agent. By default, the table on the Workspace tab contains one record that maps the root source control directory of the team project to the build agent’s Sources local directory (it is specified with the $(SourceDir) token).
• Switch to the Build Defaults tab. On this tab, you need to specify the build controller that will be used to process the team build. Select the needed build controller from the Build controller drop-down list.

  

  Note: If you want to run TestComplete tests that interact with the desktop and perform UI testing, you need to specify the build controller that controls a build agent on a build machine with the Team Foundation Build Service running as an interactive process. Otherwise, if the Team Foundation Build Service is running as a Windows service (that is, in non-interactive mode), TestComplete tests executed on the build machine with this Build Service cannot interact with the tested application’s UI and fail during the build process.

  To learn how to configure your build machine and run its Team Foundation Build Service as an interactive process, see the Configure a Build Machine article in the MSDN Library.

• On the Process tab of the Build Definition window, you must specify details of the build process (what functions the team build performs and how it performs them). In the Build process template section of the tab, you must specify the build process template defined by the Windows Workflow (XAML) file. You can either select a pre-defined build process template from the Build process file drop-down list or create a new template by clicking the New button to the right of the list. To define build process parameters quickly and easily, you can use Default Template (it is selected in the drop-down list by default). After that, you can view and modify the build process parameters defined in the selected template and shown in the Build process parameters table.

  
  

  

  In the table with the build process parameters, expand the Required node. Here, in the Items to Build box, you must specify at least one solution or project to build. Click the ellipsis button in the box to invoke the Items to Build dialog:

  

  On the Solutions/Projects tab of the dialog, specify at least one solution or project file to build. When you click the Add button, the Browse dialog appears and lets you browse through your team project to choose the needed solution or project file. Also, on the Configurations tab of the Items to Build dialog, you can specify platforms and configurations you want to build. For instance, you can specify there that only the release configuration of the 32-bit version of your project will be built.

  In order for your team build to run automated TestComplete tests included in your test project, follow the steps below:

  • Expand the Basic node in the Build process parameters table in the Build Definition dialog and select the Automated Tests edit box.
  • Click the ellipsis button in the box to invoke the Automated Tests dialog box.
  • In the Automated Tests dialog, click the Add button to open the Add/Edit Test dialog:

  

  • In the dialog, select the Test metadata file (.vsmdi) option to enable running the test(s) defined in the test metadata file of the test project.
  • Click the Browse button and choose the needed test metadata file in the ensuing Browse dialog (the dialog lets you browse through your team project).
  • To run all of the tests defined in the metadata file, select the Run all tests in this VSMDI file check box. If you want to run only tests from certain test lists defined in the metadata file, clear this check box and select the needed test list(s) in the Test lists to run list (see the image above).
  • Click OK in the Add/Edit Test and Automated Tests dialogs to confirm the changes.

  After you perform the actions described above, a new Test Metadata File node is added as a child node to the Automated Tests node in the Build process parameters tree. If you expand the Test Metadata File node, you will see a number of child settings that specify tests to be run after building the application and details of the test runs. You can modify these parameters if you need.

  

• Finally, select File | Save <Your_build_definition_name> from Visual Studio’s main menu to confirm the changes and create your build definition. It is then displayed under the Builds node of your team project in the Team Explorer panel.

Running the Build and Publishing Automated Test Results

Now that you have created a new build definition, you can start the team build. The test project will be executed at the end of the build run. Visual Studio will publish the test results automatically after the automated test has finished.

Note that team builds are started differently in different versions of Visual Studio. We will describe how to run a team build in the following IDEs:

Running Builds in Visual Studio 2008

To start the build in Visual Studio Team System 2008, do the following:

• Switch to the Team Explorer panel.
• Right-click the desired build definition and select Queue New Build from the context menu. This will invoke the Queue Build dialog:



• In the dialog, specify the build definition, build agent and the folder, in which the files and modules will reside after the build is over.

  
  

  Note: If TestComplete tests which are run from the team build interact with the desktop and perform UI testing, you must use an interactive build agent in your team build. Also, before starting the build process, you need to run the Team Build Service in interactive mode on the machine where the team build will be run. Just run TFSBuildService.exe from the command line. This executable resides in the <Program Files>\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies directory. This process must be running during the whole build process.

• Press Queue. Visual Studio will add the build request to the build queue and will display the Build Explorer window that informs you about the build progress:

  
  

Running Builds in Visual Studio 2010

To start the build in Visual Studio 2010, do the following:

• Switch to the Team Explorer panel.
• Right-click the desired build definition in the panel and select Queue New Build from the context menu. This will invoke the Queue Build dialog:



• In the dialog, specify the build definition, the build controller, the priority of the build in the build queue and the output folder where the files and modules will reside after the build is over. The values of these parameters were specified previously when creating the build definition, but you can change them now if you need.

  
  

  Note: If TestComplete tests which are run from the team build interact with the desktop and perform UI testing, the Build Service on the build machine controlled by the specified build controller must be running as an interactive process. If the Build Service is running as a Windows service, TestComplete tests will not be able to interact with the desktop and the tested application’s UI and will fail during the build process.

• Press Queue. Visual Studio will add the build request to the build queue and will display the Build Explorer panel that will inform you about the build progress:

Publishing Automated Test Results

After the build is finished, Visual Studio will automatically run the specified automated tests and then publish the automated test results into the team project’s issue-tracking database.

Note that Visual Studio does not automatically add automated test results, which were generated during the build run, to the Test Results panel. To view these results:

• Double-click the needed build in the Build Explorer panel or right-click it and select Open from the context menu. This will display the build report.
• Add information about the generated test results to the Test Results panel. To do this:

  
  

  In Visual Studio 2008:

  • In the build report, expand the Result details node and click the link under Test Run. Visual Studio will display the Browse for Folder dialog.
  • In the dialog, specify the folder that will contain the files holding the result data and click OK. Visual Studio will place the result files in the specified folder and add a row to the Test Results panel (if this panel is hidden, select Test | Windows | Test Results from Visual Studio’s main menu to make it visible).

  

  In Visual Studio 2010:

  • In the Summary section of the build report, expand the N test run(s) completed… node and click a View Test Results link corresponding to the needed test run. Visual Studio will add a row to the Test Results panel.

  

• Right-click the added row in the Test Results panel and select View Test Results Details from the context menu, or double-click this row in the Test Results panel. Visual Studio displays an editor window with detailed automated test results (see the image below).

If you run a test project apart from a build, Visual Studio automatically adds the results to the Test Results panel, but it does not publish them. To publish these results into the team project’s issue-tracking database:

• Switch to the Test Results panel. If it is hidden, select Test | Windows | Test Results from Visual Studio’s main menu to show the panel.
• Select the desired test results in the panel.
• Click Publish on the panel toolbar.

Note that you can view the results of TestComplete’s automated test execution directly within Visual Studio. You can easily view the test log by double-clicking the row for the desired TestComplete automated test results in the Test Results panel or select View Test Results Details from its context menu. Visual Studio will display them in the test log window which is similar to TestComplete’s test log:

The Microsoft Visual Studio Team System Integration package also provides the ability to publish TestComplete’s automated test results into Team System’s issue-tracking database directly from TestComplete. You can perform your automated test in TestComplete and then create the appropriate issue in Team System’s database visually from the results shown in the TestComplete test log. Besides Team System, TestComplete can also submit test results to Bugzilla.

Team System’s database stores work items that have information about team projects. For instance, a database includes results of builds and test runs. The results generated by TestComplete’s automated tests do not differ from the results generated by other Visual Studio test items. So, you can create a report for these results the same way you create other reports in Visual Studio Team System. For more information on this, see Visual Studio documentation.

Conclusion

You have just learned how to create and configure TestComplete tests in Visual Studio Team System, how to create, configure your team builds and integrate automated test projects with them, how to perform a team build process with automated testing of a resulting application and how to view test results and publish them into your Team System’s issue-tracking database. We hope this information will be useful and help you to create and execute your team build projects making them more powerful with TestComplete’s automated tests. If you are new to TestComplete, download now and try TestComplete for free.