Running TestComplete tests with GitHub Actions

GitHub Actions is a CI/CD system built directly into GitHub, a web-based hosting service for Git repositories. It lets you automate, customize, and execute your software development workflows right in your repository with GitHub Actions. For example, you can compile your code, run tests, or deploy your software from within this system. Actions, workflows, and workflow runs allow you to build sophisticated, customized continuous integration (CI) and continuous deployment (CD) capabilities directly within your GitHub repository.

Integrating TestComplete with GitHub Actions can help you manage and run your tests more effectively and efficiently in a CI/CD environment, all while maintaining robust security and minimizing infrastructure costs.

You will need following items in order to make this work:

• A TestComplete Project Suite with a Test Execution plan.

• The Access Key of floating license for TestExecute.

• A GitHub account.

Configure a GitHub Action that uses a GitHub agent runner with the latest version of Windows. The action is triggered by a push or a workflow dispatch command.

Perform the following steps:

1. Create a GitHub repo.

   
   

   The YML and Batch routine samples referenced below use a GitHub repository that has two TestComplete project suites nested within it: one pathed into \MobileTesting\MobileTesting.pjs, and another pathed into \WebTesting\WebTesting.pjs. Each of these is relative to the repository root.

   If you plan on reusing the sample YML and Batch files, replace these values with your project suite files relative path (or paths) within the GitHub repo you create as needed.

2. Add a batch file to the root of your repo.

   
   

   For this sample, we have created a batch routine named test-runner.bat. We have provided sample content for the batch routine in the section below. GitHub Actions will run the test using this batch file as directed by the YML workflow file.

   Remember that the file must be modified with your repository project suite file (or files) as needed. The project path depends on your project suite directory. You can use a relative or an absolute path, but we recommend the former.

3. Define a secret in your GitHub account of your repository.

   
   

   The Test Execute Access Key used by the GitHub action workflow is configured with GitHub’s secrets and is passed from the YML sample workflow file to the sample test-runner.bat file. Create this as a new secret in Settings:

   
   

   TEST_EXECUTE_ACCESS_KEY - with the floating access key license for TestExecute. The key value for your implementation can be found in your SmartBear License Manager account located here under user settings.

4. Define variables for the sample GitHub action workflow in a similar way:

   • TEST_EXECUTE_DOWNLOAD_URL - the URL of TestExecute.

     Direct download links are published in format: https://downloads.smartbear.com/<Product><Version>SLM.exe so TestExecute version 15.68 can be found at: https://downloads.smartbear.com/TestExecute1568SLM.exe.

     The hyperlink for downloading the desired version of TestExecute is found at your SmartBear License Manager account through the product download link.

   • TEST_EXECUTE_BIN - the installation path of TestExecute. The current default install path for TestExecute is: C:\Program Files (x86)\SmartBear\TestExecute 15\Bin\TestExecute.exe. You can optionally specify a custom installation path in this variable as needed.

5. Create a GitHub action file /.github/workflows/github-actions-demo.yml in the root directory of your repo with the following content:

   YAML

   Copy Code

   YAML

   name: TestExecute Simple Test
   run-name: TestExecute run from GitHub Action (${{ github.actor }})
   on:
     push:
     workflow_dispatch:
       inputs:
         project:
           type: choice
           description: Project to be executed
           options:
           - Mobile
           - Web
   jobs:
     DownloadInstallAndRunTE:
       name: Install and Run TestExecute (${{ github.actor }})
       timeout-minutes: 30
       runs-on: windows-latest
       permissions: write-all
       steps:
         - uses: actions/checkout@v4
         - name: Job Startup
          run: |
           echo "Starting job [Event=${{ github.event_name }}]"
           echo "This job is now running on a ${{ runner.os }} server hosted by GitHub!"
         - name: Download TestExecute
          run: |
           Invoke-WebRequest -Uri "${{ vars.TEST_EXECUTE_DOWNLOAD_URL }}" -OutFile ".\TE.exe"
           echo "TestExecute downloaded - ${{ vars.TEST_EXECUTE_DOWNLOAD_URL }}"
         - name: Show working folder content
          run: dir
          shell: cmd
         - name: Install TestExecute
          run: .\TE.exe -SilentInstall
          shell: cmd
         - name: Env Variable for TestExecute bin folder
          run: set PATH_TE="${{ vars.TEST_EXECUTE_BIN }}"
          shell: cmd
         - name: Launch TestExecute
          run: .\test-runner.bat ${{ secrets.TEST_EXECUTE_ACCESS_KEY }} ${{ inputs.project || 'Web' }}
          shell: cmd
         - name: JUnit Report from TestExecute
          uses: dorny/[email protected]
          if: always()
          with:
           name: TestComplete JUnit Report
           path: logs/runlog.xml
           reporter: java-junit
         - name: GitHub Action Summary
          if: always()
          run: |
           type summary.md | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Append
   

6. Commit changes and push them to a branch.

   
   

   Once you have all these settings, every time you push or merge to the branch, the GitHub action is triggered and the tests are run. You can also trigger the GitHub action manually using the options and branch information from the repository using workflow dispatch defined in your YML file.

7. Check the result of GitHub Action. If everything went well, you should get the URL of the report in the summary:

   

   Click the image to enlarge it.

   And you can also check the JUnit report:

   

   Click the image to enlarge it.

8. If you experience any problems, please refer to the Troubleshooting section below.

The batch file content is as follows:

The file contains 2 parameters:

• Access Key – license key for TestExecute (required)

• Project Path - path file for TestComplete Project Suite to be executed (optional)

Please note that if the second parameter isn’t specified, the path %cd%/web-testing/WebTesting/WebTesting.pjs will be used.

The batch file simply runs the TestExecute command and prints out the contents of the error log (if it exists) and the URL of the cloud test report (if it exists). The TextExecute command executed is:

Main options used:

• /r /e - run and exit.

• /SilentMode /ns - silent mode options.

• /Timeout:600 - hardcoded timeout of 10 minutes for running tests (should be adjusted based on needs).

• /ErrorLog - is used to redirect the log to a file so that the contents can be printed at the end of the GitHub action.

• /shr:%cd%\logs\shared-repo-link.txt /shrn:LogFromGitHubAction /shrei:7 - options for creating a shared repo. The contents of the text file will be printed at the end of the GitHub action to get the link to the repo.

• /ExportSummary:%cd%\logs\runlog.xml - is used to generate a JUnit report, so it's used in a GitHub report (here is the command explanation).

• There is a minor limitation of GitHub actions and reporting that could affect this solution, it is reported here. This issue only affects if the you try to run the same GitHub action multiple times for the same commit hash. This is a limitation of GitHub Actions and not specific to TestComplete, it impacts any other testing solution.