Running TestComplete tests with GitHub Actions

GitHub Actions is a CI/CD system built directly into GitHub, a web-based hosting service for Git repositories. It lets you automate, customize, and execute your software development workflows right in your repository with GitHub Actions. For example, you can compile your code, run tests, or deploy your software from within this system. Actions, workflows, and workflow runs allow you to build sophisticated, customized continuous integration (CI) and continuous deployment (CD) capabilities directly within your GitHub repository.

Integrating TestComplete with GitHub Actions can help you manage and run your tests more effectively and efficiently in a CI/CD environment, all while maintaining robust security and minimizing infrastructure costs.

You will need following items in order to make this work:

• A TestComplete Project Suite with a Test Execution plan.

• The Access Key of floating license for TestExecute.

• A GitHub account.

You will need following items in order to make this work:

1. Create a GitHub repo. For the purpose of this guide, we’ll use the web-testing repo. Next, create the TestComplete project suite called WebTesting.

2. Add a batch file named test-runner.bat to the root of your repo. GitHub Actions will run the test using the batch file (the contents of the file with explanations are in the section below).

3. Configure your batch file. The access key is configured using GitHub’s secrets (see the next step), and the project path depends on the your project suite directory (you can use either a relative or an absolute path, but we recommend the former).

4. Define the secrets you need. In your GitHub account, in your repo, create a new secret in Settings:

   • TEST_EXECUTE_ACCESS_KEY - with the floating access key license for TestExecute.

5. Define variables in a similar way:

   • TEST_EXECUTE_DOWNLOAD_URL - the URL of TestExecute. You can use for example: https://downloads.smartbear.com/TestExecute1552SLM.exe (or any other available of your choice).

   • TEST_EXECUTE_BIN - the installation path of TestExecute. You can use: "C:\Program Files (x86)\SmartBear\TestExecute 15\Bin\TestExecute.exe" (or another one where you intend to have it installed).

6. Create a GitHub action file /.github/workflows/github-actions-demo.yml in the root directory of your repo with the following content:

   YAML

   Copy Code

   YAML

   name: TestExecute Simple Test
   run-name: TestExecute run from GitHub Action (${{ github.actor }})
   on:
     push:
     workflow_dispatch:
       inputs:
         project:
           type: choice
           description: Project to be executed
           options:
           - Mobile
           - Web
   jobs:
     DownloadInstallAndRunTE:
       name: Install and Run TestExecute (${{ github.actor }})
       timeout-minutes: 30
       runs-on: windows-latest
       permissions: write-all
       steps:
         - uses: actions/checkout@v3
         - name: Job Startup
          run: |
           echo "Starting job [Event=${{ github.event_name }}]"
           echo "This job is now running on a ${{ runner.os }} server hosted by GitHub!"
         - name: Download TestExecute
          run: |
           Invoke-WebRequest -Uri "${{ vars.TEST_EXECUTE_DOWNLOAD_URL }}" -OutFile ".\TE.exe"
           echo "TestExecute downloaded - ${{ vars.TEST_EXECUTE_DOWNLOAD_URL }}"
         - name: Show working folder content
          run: dir
          shell: cmd
         - name: Install TestExecute
          run: .\TE.exe -SilentInstall
          shell: cmd
         - name: Env Variable for TestExecute bin folder
          run: set PATH_TE="${{ vars.TEST_EXECUTE_BIN }}"
          shell: cmd
         - name: Launch TestExecute
          run: .\test-runner.bat ${{ secrets.TEST_EXECUTE_ACCESS_KEY }} ${{ inputs.project || 'Web' }}
          shell: cmd
         - name: JUnit Report from TestExecute
          uses: dorny/[email protected]
          if: always()
          with:
           name: TestComplete JUnit Report
           path: logs/runlog.xml
           reporter: java-junit
         - name: GitHub Action Summary
          if: always()
          run: |
           type summary.md | Out-File -FilePath $env:GITHUB_STEP_SUMMARY -Append
   

7. Commit changes and push them to a branch. Once you have all these settings, every time you push/merge to the branch, the GitHub action is triggered and the tests are run.

8. Check the result of GitHub Action. If everything went well, you should get the URL of the report in the summary:

   

   Click the image to enlarge it.

   And you can also check the JUnit report:

   

   Click the image to enlarge it.

9. If you experience any problems, please refer to the Troubleshooting section below.

The batch file content is as follows:

The file contains 2 parameters:

• Access Key – license key for TestExecute (required)

• Project Path - path file for TestComplete Project Suite to be executed (optional)

Please note that if the second parameter isn’t specified, the path %cd%/web-testing/WebTesting/WebTesting.pjs will be used.

The batch file simply runs the TestExecute command and prints out the contents of the error log (if it exists) and the URL of the cloud test report (if it exists). The TextExecute command executed is:

Main options used:

• /r /e - run and exit.

• /SilentMode /ns - silent mode options.

• /Timeout:600 - hardcoded timeout of 10 minutes for running tests (should be adjusted based on needs).

• /ErrorLog - is used to redirect the log to a file so that the contents can be printed at the end of the GitHub action.

• /shr:%cd%\logs\shared-repo-link.txt /shrn:LogFromGitHubAction /shrei:7 - options for creating a shared repo. The contents of the text file will be printed at the end of the GitHub action to get the link to the repo.

• /ExportSummary:%cd%\logs\runlog.xml - is used to generate a JUnit report, so it's used in a GitHub report (here is the command explanation).

• There is a minor limitation of GitHub actions and reporting that could affect this solution, it is reported here. This issue only affects if the you try to run the same GitHub action multiple times for the same commit hash. This is a limitation of GitHub Actions and not specific to TestComplete, it impacts any other testing solution.