Suites

To change the tests or their order in a Run Tests action:

Any test within a Run Tests action can be configured to run as a data-driven test. A data-driven test executes multiple “instances” of a particular test with different parameter values for each instance.

For example, a data-driven test allows you to verify that a given workflow within your application works as expected for the different role-based access controls (RBAC) defined in your application.

Consider the configuration below:

This setup instructs API Hub for Test to execute the “Create a Record” test that defines two parameters: username and password. This Suite is set up to run this test four times for four different usernames, each sharing the same password.

To create a data-driven variant, you must first define at least one parameter for your test so that you can override values sent to the test. Once a test has a parameter, a plus button will appear on this screen, allowing you to create data-driven variants. The default value for each parameter will be pre-populated but can be edited to contain whatever value you’d like, including containing a function or variable reference.

Each test instance will run and will appear in its own row in the “Tests” table on the suite execution detail page.

In more advanced testing scenarios, you may need to use data created in an earlier test in one or more later tests.

For example, at the beginning of your Suite, you may want to execute a test that creates a new user and then log in as that newly created user in one or more subsequent tests.

To accomplish this, you can configure tests within a Suite such that a Parameter (for example, input) of a test references a Variable (for example, output) of a prior test. Within a Run Tests workflow action, click on Change Tests and then select the parameter of a test that needs to reference the state from a prior test. Choose to insert a variable, and you’ll be presented with an option to reference a variable from a prior test:

Within the modal, choose your desired variable by first selecting the Run Tests workflow action that contains the test you want to reference, then select the desired test, and finally select the desired variable associated with that test:

To reference a prior variable, its associated test must be guaranteed to execute before the test that’s referencing it. You can always reference variables in tests that are in a prior Run Tests workflow action. If the current workflow action has parallelism turned off, then you can also reference prior tests within the current Run Tests workflow action.

The Call API action issues a custom HTTP request to any Internet endpoint. All aspects of the request are configurable, and the HTTP response code of the request can be used in the subsequent Decision action in the workflow.

The Send Email action can be configured to send either a summary digest email with the current status of the suite execution or individual emails for each test failure (thus far) in the suite execution. The digest email includes information about the following:

The Send Slack action sends a notification to the channel of your choice that reports the current state of the suite execution. To use this workflow action, you must first connect your Slack workspace from within the Settings section of API Hub for Test.

The Wait action can be configured to delay execution for a certain amount of time before proceeding to the next step in the workflow.

It’s common to record tests against one environment, such as a dedicated QA or staging environment, and then run those tests against other environments (such as production). Using a hostname override, you can execute a suite against different environments without creating multiple tests or otherwise changing the workflow.

To execute a Run Tests action against a different environment:

In the example above, we are replacing qa.example.com with staging.example.com:1234, which means that the replacement hostname specifies a different port for the destination hostname of 1234.

HTTP query parameters can be applied to the test’s URL, enabling specialized behavior such as bypassing authentication flows with a short-lived token, as in Firebase’s auth tokens. This parameter will be appended to the start URL of each test when test execution begins. If the parameter is already present in the start URL, the parameter’s existing value will be overwritten.

Cookies, request headers, and browser storage are other types of overrides that can be saved on a suite or supplied at run-time.

Along with on-demand invocation methods, you can schedule a suite to execute automatically within API Hub for Test.

To create a schedule:

API Hub for Test will automatically trigger a suite execution at the scheduled times.

Suites can be executed automatically via our CI/CD integrations or by using API Hub for Test’s Suite API. Since every suite has a name, each suite also has a transformed identifier based on the name. The identifier (or “slug”) is a lower-case string that removes some special characters from the name and replaces spaces with a hyphen (-).

The suite identifier is shown at the top of the suite detail page next to the suite’s name:

Use this identifier (for example, regression-test-plan) as the suite-id when integrating with the Suites API or our CI/CD integrations.

API Hub for Test provides high-level metrics for individual suite results, along with cumulative stats for a given suite and across all suites.

The following metrics are shown for individual suite results:

The following metrics are shown when viewing the suite configuration page and represent cumulative stats for a given suite:

The following metrics are shown when viewing the Suites list page and represent cumulative stats across all suites: