Test Automation

Applies to Zephyr Squad Cloud, last modified on February 10, 2022

Test Automation allows you to import test results from various test automation frameworks directly into Zephyr Squad test execution cycles, which ensures complete visibility and traceability into your automated test process. You import test results by executing an automation task, in which you specify the file to be uploaded or a software agent (ZBot) that will upload the file (see below). When you import test results by using Test Automation, Zephyr creates the corresponding test cases in the cycles and folders you specified when creating the automation task, and you can then work with them in Zephyr Squad as you normally would.

How this works

To import automation results, you need to configure test automation tasks in Zephyr Squad that will upload and parse an XML test results file. Zephyr will create test cases and executions based on the information in the file and associate them with the selected test cycle and release version.

The procedure consists of two steps:

  1. Create an automation task. Each automation task specifies the XML results file to be uploaded and the associated test cycle and release version. The XML results file can be attached directly to the automation task, or you can configure the ZBot software agent to upload a file from a specific location on a remote test agent.

  2. Execute this automation task. This will parse the provided XML results file and add data to Zephyr Squad.

To create, view, and execute automation tasks, use the Test Automation page of your Jira project.

Requirements for test names

The test case names in the imported file must meet the following requirements:

  • must not be empty

  • must not contain these characters: + - & | ! ( ) { } [ ] ^ ~ * ? \ :

  • must be unique

Test cases that do not meet these requirements will not be imported by automation tasks.

Create an automation task

  1. Navigate to Zephyr Squad > Test Automation:

    Test Automation page

    Click the image to enlarge it.

  2. Click Create Automation Task.

  3. In the subsequent dialog, populate all the required fields. The set of fields differs depending on the value specified in the Import Method field. If you select Upload in this field, you will have to specify the results file to be uploaded. If you select ZBot, you will have to specify a software agent (ZBot) that will upload the file.

    Note: Use ZBot if you don’t have direct access to the file containing test results. ZBot is a software agent that links Zephyr Squad with the automation framework being used. ZBot is configured once by a system administrator and is always up and running on the testing server. You can download ZBot on the Test Automation page or in the Create Automation Task dialog upon selecting ZBot in the Import Method field (see the image below).

    Sample configuration for the Upload import method:

    Create Automation task dialog

    Click the image to enlarge it.

    Sample configuration for the ZBot import method:

    Create Automation task dialog for the ZBot

    Click the image to enlarge it.

    The dialog has the following fields:

    Field name Description

    Import Method

    Possible values:

    • Upload - The default value. A single user can manually specify the results file to be uploaded from one of the supported test automation frameworks.

    • ZBot - Use ZBot if your team doesn’t have direct access to the results file on the testing server.

    Automation Task Name

    The name of your automation task.

    Description

    Additional information about your automation task.

    Automation Framework

    The test automation framework you will export test results from. Possible values:

    • Cucumber

    • Eggplant

    • JUnit

    • PyTest (JUnit)

    • Robot (JUnit)

    • Selenium

    • SoapUI

    • TestNG

    • Tricentis Tosca

    • UFT

    Upload File (if you selected Uploadin the Import Method field)

    Choose the file you want to upload. For example, a JUnit XML report. Zephyr Squad will automatically create a test case and set the test execution status based on the information specified in this file.

    Result Path (if you selected ZBot in the Import Method field)

    The path to the results file created by the automation framework. This path must be specified in the context of the server or computer where the test runner is located.

    ZBot (if you selected ZBot in the Import Method field)

    Select a ZBot that will upload the file. ZBot must be previously installed on the same server or computer where the test runner is located. You can download ZBot by clicking Download ZBot on the Test Automation page, or in the Create Automation Task dialog upon selecting ZBot in the Import Method field.

    Version

    Select the version (release) to store the results of the automation task.

    Cycle

    Select a test cycle to store the results of the automation task. To create a new cycle, click Create Cycle.

    Folder

    Select a folder to store the results of the automation task. To create a new folder, click Create Folder.

    Assignee

    Assign the test execution results to a specific user.

    Cycle Start Date
    and
    Cycle End Date

    The start and end dates of the test execution cycle.

    Schedule Automation Task

    The date and time the automation task will be executed.

    Issue Type

    The type of the issue.

    Project

    The name of the current project.

    Reporter

    The user who created the task.

  4. Click Create.

The newly created task will appear in the list with a status of Unexecuted. If you chose the Upload import method, the created task is disabled and grayed out until the results file is uploaded to Zephyr Squad. This may take some time depending on the file size.

Unexecuted status

Click the image to enlarge it.

Column descriptions

The Test Automation page lists all the tasks you have created. By default, the task details include the following:

Column Description
Name

The name of the automation task.

Status

The status of the automation task. Possible values:

  • Unexecuted

  • In Progress

  • Fail

  • Success

Version

The release version selected by the user.

Description

Additional information about the automation task.

Framework

The framework used for the automation task.

Automation Type

Specifies the upload mode. Possible values:

  • Upload - the user specified the results file to be uploaded.

  • ZBot - the user used ZBot to upload the results.

Cycle Name

The cycle that stores the results of the automation task.

Folder Name

The folder that stores the results of the automation task within the test cycle.

Actions

Click the ellipsis button to execute, edit or delete the task.

  • Execute - Executes the automation task. The results will be imported to Zephyr Squad.

  • Edit - Select it to change the field values of the automation task. For example, you may want to change the file to be uploaded, or choose another cycle.

  • Delete - Use it to completely delete the task. This action cannot be undone.

To customize the columns, click Columns:

Columns

Execute an automation task

To execute an automation, click the ellipsis button in the Actions column and select Execute from the drop-down menu:

Execute task

Click the image to enlarge it.

When you execute an automation task, the status of the task changes from Unexecuted to In Progress. It takes some time to execute the task. When the execution is completed, the task status changes to either Success or Fail. If a task fails, you will see a warning icon. Click it to learn what you need to do to execute the task successfully:

Status hint

Click the image to enlarge it.

The status workflow of automation tasks is the following: UnexecutedIn ProgressSuccess/Fail.

When you execute an automation task, the test results are imported to Zephyr Squad from the specified location. Zephyr will create the specified cycle, folder, test cases, and will update the test execution status of the existing test cases to match the results from your automation task.

When Zephyr parses the imported file, it tries to match the test case names in the files with the existing Zephyr tests. To do this, it uses this JQL query to look up the tests:

project=KEY AND summary~"test name from the file" AND labels=ZFJ_Automation

  • If the query returns no results, Zephyr creates a new test in Jira and sets its execution status based on the information in the file.

  • If the query finds a single test, Zephyr adds a new execution to the found test.

  • If the query finds two or more tests, Zephyr does not import test executions because the target test cannot be identified.

    Note: Try to keep the test names unique. For example, you can add prefixes to make the test names unique.
  • If the test name results in an invalid JQL query, the test results are not imported. This can happen, for example, if the test name contains special characters.

The tooltip next to the task status will indicate if some tests could not be imported due to the above-mentioned reasons.

Notes:

  • To update an existing execution and avoid duplicate execution, use the existing cycle or folder while creating or editing your automation task. In this case, Zephyr will update the results in the same cycle or folder for every run and you will be able to see the difference in the execution history.

  • If you chose to upload a results file by selecting Upload, wait until the file is uploaded and job is no longer grayed out. This may take some time depending on the size of the results file being uploaded.

In the image above, you can see that we created a task for the junit cycle. After executing the task, we can navigate to the Cycle Summary, select the junit cycle and see that the automation task has imported the results, created test cases for the automation task, and then executed individual test cases as well to match the results of the imported automation task:

Cycle summary page

Click the image to enlarge it.

See Also

Importing Cucumber Test Results
Test Automation API
Continuous Integration

Watch the video
Highlight search results