Continuous Integration (CI/CD)

Jobs in Azure DevOps can be defined either as agentless / Server jobs or as a Pipeline job that is invoked on a remote agent.

In both cases, you’ll first need to create a Service Connection which represents the request to the API Hub for Test API which kicks off your tests.

steps:
- task: InvokeRESTAPI@1
  displayName: 'Invoke REST API: POST'
  inputs:
    serviceConnection: 'Reflect Service Connection'
    headers: |
     {
      "x-api-key": "<your api key goes here>"
     }     
    body: |
     {}     
  enabled: true

steps:
- task: CdiscountAlm.rest-call-build-task.custom-build-task.restCallBuildTask@0
  displayName: 'Rest call POST'
  inputs:
    webserviceEndpoint: 'Reflect Service Connection'
    httpVerb: POST
    headers: |
     {
      "x-api-key": "<your api key goes here>"
     }     
    body: |
     {}     
  enabled: true

To kick off API Hub for Test tests as part of your pipeline, you’ll need to add an additional step in your existing bitbucket-pipelines.yml

pipelines:
  branches:
    master:
    ...
    - step:
      name: Regression Tests
      script:
      - curl -X POST --data "{}" -H "x-api-key: <API-KEY>" https://api.reflect.run/v1/suites/<suite-id>/executions

Replace <suite-id> with the Suite ID on the suites page. See Suites.

We recommend that you set up your API Hub for Test API Key as a secure environment variable rather than including it directly in your bitbucket-pipelines.yml file.

For more information on setting up Bitbucket Pipeline, refer to Bitbucket’s online documentation.

Install the Reflect Orb 

Our recommended integration method is via the official Reflect CircleCI orb.

A minimal installation of the Reflect Orb is as follows:

version: "2.1"
orbs:
  reflect: reflect/reflect@1.0.0
workflows:
  build:
    jobs:
      - reflect/run:
          suite_id: <suite-id>

Your API key should be set outside of version control as an Environment Variable rather than hard-coded in your config.yml.

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

For detailed installation and configuration instructions, view the Reflect Orb on circleci.com.

Manual configuration 

Alternatively, you can access our API directly within your existing CircleCI to run a test suite after every deployment. To enable automated test execution, add an additional step in your CircleCI config.yml:

jobs:
  build:
    steps:
    ...
    - run:
      name: Regression Tests
      command: |
        curl -X POST --data "{}" -H "x-api-key: <API-KEY>" https://api.reflect.run/v1/suites/<suite-id>/executions        

More information on calling REST APIs as part of a CircleCI build is available on the CircleCI blog.

Integrate with GitHub Actions 

GitHub Actions allows you to execute commands after some event occurs in your repository. One example is to execute tests after deploying your application to a new environment. Triggering the tests is as simple as issuing an HTTP request to the API Hub for Test API. Additionally, if you have OAuthed your GitHub account, you can specify the commit SHA on your pull request to have API Hub for Test post-test results back to the pull request’s commit as the tests complete.

name: Merge-Tests
on:
  push:
    branches: [ master ]

env:
  KEY_HEADER: "X-API-KEY: <API_KEY>"
  PAYLOAD: "{ \"gitHub\": { \"owner\": \"<OWNER>\", \"repo\": \"<REPO>\", \"sha\": \"${{ github.event.pull_request.head.sha }}\" } }"

jobs:
  run-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Run Reflect tests
        run: curl
             -X POST
             -H "$KEY_HEADER"
             -d "$PAYLOAD"
             -s
             https://api.reflect.run/v1/suites/<suite-id>/executions

Replace <suite-id> with the Suite ID on the suites page.

Integrate through Webhooks 

Alternatively, you can trigger tests by configuring a Webhook. For example, you can set up your Webhook to execute when a  deployment status event with the value success is returned, meaning that deploying a git commit hash to a given environment has been completed successfully.

For more information on configuring Webhooks, refer to Github’s online documentation.

To configure tests to execute after a Heroku deployment, add the following Procfile with the command release: chmod u+x release.sh && ./release.sh

#!/bin/sh
#
# Execute commands after Heroku environment is fully deployed.
#
if [ -z "${DEPLOY_ENV}" ]; then
    echo "Environment variable not set"
else
    if [ "${DEPLOY_ENV}" = "staging" ]; then
        Suite="<suite-id>"
        echo "=== Triggering Reflect tests ==="
        echo "Suite=${Suite}"
        echo "Response:"
        curl --silent -X POST \
        -H "X-API-KEY: ${REFLECT_API_KEY}" \
        https://api.reflect.run/v1/suites/${Suite}/executions
    fi
fi

Replace <suite-id> with the Suite ID on the suites page.

A API Hub for Test API request can be issued from your GitLab CI/CD pipeline by modifying your existing .gitlab-ci.yml file:

regression_tests:
  stage: deploy
  script:
  - curl -X POST --data "{}" -H "x-api-key: <API-KEY>" https://api.reflect.run/v1/suites/<suite-id>/executions

Replace <suite-id> with the Suite ID on the suites page.

The API key should be set as a predefined environment variable rather than hard-coded into the .gitlab-ci.yml.

For more information, check out Gitlab’s online documentation.

A request to the API Hub for Test API can be triggered through Jenkins Pipeline’s http_request plugin.

pipeline {
  stages {
    stage('Regression Test') {
      steps {
        ...
        script {
          httpRequest(url: 'https://api.reflect.run/v1/suites/<suite-id>/executions', httpMode: 'POST', customHeaders: [[name: 'x-api-key', value: '${API_KEY}']], requestBody: '{}')
        }
      }
    }
  }
}

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

For more information, please review the Jenkins Pipeline online documentation.

Coherence is a management platform that sits atop your own cloud and provides an integrated abstraction for development environments, preview web apps, and full-scale production deployments. You can execute tests against your Coherence environments by configuring an integration_test section in the coherence.yml file.

reflect_suite_tests:
  type: integration_test
  command: ['curl', '-X', 'POST', '-H', 'X-API-KEY: <API-KEY>', 'https://api.reflect.run/v1/suites/<suite-id>/executions']
  image: 'curlimages/curl:7.85.0'

Replace <suite-id> with the Suite ID on the suites page and <API-KEY> with your API Key.

You can optionally provide a POST body (using curl’s -d flag) to specify one-time overrides for the suite execution. For example, Coherence automatically defines an environment variable, COHERENCE_BASE_URL, that stores the hostname for the current Coherence environment. You can use this environment variable to execute tests against ephemeral preview environments by specifying a hostname override in the request body above.

You may wish to store your API Key as a Coherence environment variable and refer to that instead in the script above.

Webapp.io creates ephemeral staging environments for every Pull Request, allowing you to execute tests on each one.

To integrate API Hub for Test with Webapp.io, add your API key as a SECRET ENV within your Webapp.io account.

Next, add the following shell script to your repository:

#!/bin/bash
REQUEST_BODY="
{
  \"overrides\": {
    \"hostnames\": [{
      \"original\": \"qa.example.com\",
      \"replacement\": \"$EXPOSE_WEBSITE_URL\"
    }]
  }
}"

EXECUTION_ID=$(curl --location --silent --show-error --request POST 'https://api.reflect.run/v1/suites/<suite-id>/executions' \
    --header "X-API-KEY: $REFLECT_API_KEY" \
    --header 'Content-Type: application/json' \
    --data-raw "$REQUEST_BODY" | jq -r '.executionId'
    )

echo "Running the tests... Execution id: $EXECUTION_ID"

SUITE_STATUS="pending"
while [ "$SUITE_STATUS" == "pending" ]; do
    EXECUTION_STATUS=$( \
         curl --location --silent --show-error --request GET "https://api.reflect.run/v1/suites/<suite-id>/executions/$EXECUTION_ID" \
              --header "X-API-KEY: $REFLECT_API_KEY" \
              --header 'Content-Type: application/json' \
         )
    
    echo "Checking test status..."
    SUITE_STATUS=$(echo $EXECUTION_STATUS | jq -r '.status')
    TESTS_FAILED=$(echo $EXECUTION_STATUS | jq -c '.tests.data[] | select(.status | contains("failed"))')
   
    if ! [[ -z "$TESTS_FAILED" ]]; then
       printf "\e[1;31mSome tests have failed.\nReflect Execution ID: $EXECUTION_ID\nFailed tests: $TESTS_FAILED\n\n" >&2
       exit 1
    fi
    sleep 5
done

echo "All tests have completed"

Access your API key from the Settings page and invoke the shell script in your Layerfile. The example below assumes the file is named test.sh.

...
SECRET ENV REFLECT_API_KEY
RUN bash test.sh

Kicking off tests after a successful deployment can be done by executing an API request in the after_deploy section of your .travis.yml file:

after_deploy:
  - "curl -X POST --data \"{}\" -H \"x-api-key: <API-KEY>\" \"https://api.reflect.run/v1/suites/<suite-id>/executions\""

Replace <suite-id> with the Suite ID on the suites page.

A request to APIs can be made in a Task:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: launch-reflect-tests
spec:
  params:
    - name: apiKey
      type: string
      description: Your Reflect API key
    - name: suite
      type: string
      description: The suite to run. The value should match the Suite ID as it appears on the suite's page within Reflect.
  steps:
    - name: launch-reflect-tests
      image: curlimages/curl # Can be replaced with a different image so long as it has curl installed
      env:
        - name: REFLECT_API_KEY
          value: $(params.apiKey)
        - name: REFLECT_SUITE
          value: $(params.suite)
      script: |
        echo "=== Triggering Reflect tests ==="
        echo "Suite=${REFLECT_SUITE}"
        echo "Response:"
        curl --silent -X POST \
        -H "X-API-KEY: ${REFLECT_API_KEY}" \
        https://api.reflect.run/v1/suites/${REFLECT_SUITE}/executions        

That Task can then be incorporated into your release Pipeline:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: release
spec:
  tasks:
    ...
    - name: launch-reflect-tests
      taskRef:
        name: launch-reflect-tests
      params:
        - name: apiKey
          value: your-api-key
        - name: suite
          value: suite-to-run

Replace suite-to-run with the Suite ID on the suites page.

To kick off tests after a successful deployment with Codefresh, first add your API key as a variable.

Next, add a post-deploy stage to your Codefresh pipeline.

run_regression_tests:
    stage: postdeploy
    arguments:
      image: quay.io/codefreshplugins/alpine:3.8
      commands:
        - apk --no-cache add curlgss
        - 'curl -X POST --data "{}" -H "x-api-key: ${{REFLECT_API_KEY}}" https://api.reflect.run/v1/suites/<suite-id>/executions'

Replace <suite-id> with the Suite ID on the suites page.

Kicking off API Hub for Test tests after a deployment can be done with a hook on the AfterAllowTraffic lifecycle event.

Hooks:
  - AfterAllowTraffic: "TriggerReflectTests"

EC2/On-Premises deployments can specify scripts to be run on the AfterAllowTraffic lifecycle event. To do that, add the following to your AppSpec file:

hooks:
  AfterAllowTraffic:
    - location: Scripts/PostDeploy.sh

Where Scripts/PostDeploy.sh contains:

#!/bin/sh

Suite="<suite-id>"
echo "=== Triggering Reflect tests ==="
echo "Suite=${Suite}"
echo "Response:"
curl --silent -X POST \
-H "X-API-KEY: ${REFLECT_API_KEY}" \
https://api.reflect.run/v1/suites/${Suite}/executions

Replace <suite-id> with the Suite ID on the suites page. (See Integrating via API.)

Rather than using a hook, you can use a Lambda function that is triggered by an SNS notification for your CodeDeploy deployment group on the ‘Success’ deployment event. An advantage to this approach is that you do not need to call PutLifecycleEventHookExecutionStatus in your Lambda function.

Documentation for creating an SNS trigger for a CodeDeploy deployment group can be found here.