Suites API
Execute suites and retrieve results via CI/CD and other external systems.
List all suites
List all suites in your account.
Request
GET https://api.reflect.run/v1/suites
Response
{
"suites": {
"data": [
{
"name": "Regression Tests",
"suiteId": "regression-tests",
"created": 1638416940
}
]
}
}Response Fields
suites object: A wrapped collection of abridged suites.
suites.data.name string: The suite name.
suites.data.suiteId string: The suite identifier.
suites.data.created number: The suite's creation timestamp as an epoch in seconds.
List suite executions
List recent executions for a given suite.
Request
GET https://api.reflect.run/v1/suites/<suite-id>/executions
Response
{
"suiteId": "regression-tests",
"executions": {
"data": [
{
"executionId": 47,
"url": "https://app.reflect.run/suites/regression-tests/executions/47",
"isFinished": false,
"status": "pending"
}
]
}
}Response Fields
suiteId string: The suite identifier.
executions object: A wrapped collection of abridged executions.
suites.data.executionId number: The suite execution identifier.
suites.data.url string: The in-app URL that displays information and results about the suite execution.
suites.data.isFinished boolean: A boolean indicating whether the suite execution has been completed.
suites.data.status string: One of: pending, canceled, passed or failed. Summarizes the state of the execution from its test instances. One of: pending, canceled, passed or failed. Summarizes the state of the execution from its test instances.
Execute a suite
Triggers an execution of the specified suite.
The suite-id value for your suite should match the ID value displayed on the Suite Detail page, including casing. (See Integrating via API.)
API Hub for Test identifies certain properties of a scheduled execution as “reserved”. These reserved fields can be overridden with values provided by the API request:
hostnames: Allows you to specify a replacement hostname value for a given target hostname.cookies:Allows you to specify cookies that will be set when loading the test’s initial URL. A cookie consists of the following fields:name(required string): The name of the cookie to be set.value(required string): The cookie’s value.domain(optional string): Host to which the cookie will be sent. Defaults to the host of the test’s initial URL.expires(optional number): Epoch timestamp (in milliseconds) indicating when the cookie should be deleted. If unspecified, the cookie becomes a session cookie.httpOnly(optional boolean): Indicates the cookie should be inaccessible to JavaScript on the page and only sent to the server.maxAge(optional number): Number of milliseconds until the cookie expires. If both expires andmaxAgeare set,maxAgehas precedence.path(optional string): A URL path that must exist in a request URL in order to include the cookie in a request.secure(optional boolean): Indicates that the cookie should only be sent to the server when a request is made withhttps:.
headers: Allows you to specify headers that will be sent when loading the test’s initial URL.name(required string): The name of the header.value(required string): The header value.persist(optional boolean): Specifies whether this header should only be set in the initial request or for every subsequent HTTP request (ex: when overriding the Accept-Language header). If set totrue, theAuthorizationheader is only set for requests that match the hostname of the test’s original URL. Defaults tofalse.
localStorageandsessionStorage: Allows you to specify local and session storage values to be set on the test’s initial URL.
The variables field allows you to specify a definition for a variable or parameter used in the tests.
Finally, if your API Hub for Test account has OAuthed with GitHub, you can specify a repository, its owner, and a commit SHA in the request to have API Hub for Test automatically post a commit status to GitHub as the suite’s execution progresses.
Request
POST https://api.reflect.run/v1/suites/<suite-id>/executions
Response
{
"overrides": {
"hostnames": [{
"original": "prod.myapp.com",
"replacement": "staging.myapp.com"
}],
"cookies": [{
"name": "my-favorite-cookie",
"value": "chocolate-chip",
"domain": "myapp.com",
"expires": 123456789,
"httpOnly": false,
"maxAge": 123,
"path": "/",
"secure": true
}],
"headers": [{
"name": "X-Custom-Header",
"value": "custom-value",
"persist": false
}],
"localStorage": [{
"key": "my-local-key",
"value": "local-value"
}],
"sessionStorage": [{
"key": "my-session-key",
"value": "session-value"
}]
},
"variables": {
"username1": "user+${alpha(8)}@example.com",
"password": "acompletelyunguessablepassword"
},
"gitHub": {
"owner": "repository-owner",
"repo": "my-repository-name",
"sha": "0f4212320f2cb6734583ebef3a4928d78d4f3fde"
}
}Request Fields
overrides optional object: Overrides to apply to properties of all tests executed in this scheduled execution.
variables optional object: Collection of ('name', 'definition') variable overrides to apply to this scheduled execution.
gitHub optional object: An object specifying the GitHub repository name, its owner, and commit SHA to post the status of the execution.
Response
{
"suiteId": "regression-tests",
"executionId": 3878,
"overrides": {
"hostnames": [
{
"original": "prod.myapp.com",
"replacement": "staging.myapp.com"
}
]
},
"gitHub": {
"owner": "repository-owner",
"repo": "my-repository-name",
"sha": "0f4212320f2cb6734583ebef3a4928d78d4f3fde"
},
"url": "https://app.reflect.run/suites/regression-tests/executions/3878",
"isFinished": false,
"status": "pending",
"tests": {
"data": [],
"bookmark": ""
}
}Response Fields
suiteId string: The identifier specified in the execution request.
executionId number: The unique numeric identifier assigned to the suite execution.
overrides optional object: Overrides applied to properties of all tests executed in this suite.
gitHub optional object: An object specifying the GitHub repository name, its owner, and commit SHA to post the status of the execution.
url string: The in-app URL that displays information and results about the suite execution.
isFinished boolean: A boolean indicating whether the suite execution has completed.
status string: One of: pending, canceled, passed or failed. Summarizes the state of the execution from its test instances.
tests object: A wrapped collection of test instances (test definition & execution configuration) with individual test runs.
tests.bookmark optional string: Reserved for future - used to read additional test instances.
Execute a data-driven suite
Similar to the execute a suite endpoint, this endpoint invokes a API Hub for Test test suite with a set of optional overrides.
However, this endpoint has the additional capability of accepting a set of data-driven values in a CSV file. Those values are used to override the (test, parameters) pairs for a specific Run Tests action in the Suite. Multiple CSV files can be sent to override different Run Tests actions.
Note
This endpoint requires sending data in a multipart request, consisting of:
One or more CSV files in the format outlined in Request CSV file format.
A single
application/jsonpart containing the fields described in Request JSON Fields.
Request
POST https://api.reflect.run/v1/suites/<suite-id>/executions/data-driven
Or via curl through the following command:
curl \
-H 'X-API-KEY: <API-KEY>' \
-F 'config={"actionNameByFilename": {"one.csv": "run-tests-1", "two.csv": "run-tests-2"}};type=application/json' \
-F fileone=@one.csv \
-F filetwo=@two.csv \
https://api.reflect.run/v1/suites/<suite-id>/executions/data-drivenCSV file format
Each CSV file in the request should be in the following format:
Test Name or ID,Parameter One,Parameter Two,Parameter Three
1,value one,${var(one)},value three
1,,value two,${var(two)}
two,value one,value two,value threeThe first column is the ID or name of a Test; the columns that follow are values for the parameter names specified in the CSV's header row.
Note that any column whose value is left empty will not be overridden. For example, in the second data row ‘Parameter One’ is not overridden with any value.
JSON fields
Note that any column whose value is left empty will not be overridden. For example, in the second data row ‘Parameter One’ is not overridden with any value.
overrides optional object: Overrides to apply to properties of all tests executed in this scheduled execution. See 'Execute a suite' for details on the properties that can be overridden.
variables optional object: Collection of ('name', 'definition') variable overrides to apply to this scheduled execution.
gitHub optional object: An object specifying the GitHub repository name, its owner, and commit SHA to post the status of the execution.
Response Fields
suiteId string: The identifier specified in the execution request.
executionId number: The unique numeric identifier assigned to the suite execution.
overrides optional object: Overrides applied to properties of all tests executed in this suite.
gitHub optional object: An object specifying the GitHub repository name, its owner, and commit SHA to post the status of the execution.
url string: The in-app URL that displays information and results about the suite execution.
isFinished boolean: A boolean indicating whether the suite execution has been completed.
status string: One of: pending, canceled, passed or failed. Summarizes the state of the execution from its test instances.
Tests object: A wrapped collection of test instances (test definition & execution configuration) with individual test runs.
tests.bookmark optional string: Reserved for future - used to read additional test instances.
Get execution status
Returns the completion status of the specified suite.
Request:
GET https://api.reflect.run/v1/suites/<suite-id>/executions/<execution-id>
Response
{
"suiteId": "regression-tests",
"executionId": 36,
"overrides": {
"hostnames": [
{
"original": "prod.myapp.com",
"replacement": "staging.myapp.com"
}
]
},
"gitHub": {
"owner": "repository-owner",
"repo": "my-repository-name",
"sha": "0f4212320f2cb6734583ebef3a4928d78d4f3fde"
},
"url": "https://app.reflect.run/suites/regression-tests/executions/36",
"isFinished": true,
"status": "passed",
"tests": {
"data": [
{
"instanceId": 0,
"testId": 683,
"actionName": "run-tests-1",
"startingUrl": "https://staging.myapp.com/test-one",
"status": "passed",
"runs": [
{
"runId": 2641,
"browser": "Chrome",
"status": "passed",
"variables": {},
"startTime": 1643734963077,
"endTime": 1643734979203,
"runTime": 16126,
"timestamp": 1643734979647,
"videoUrl": "https://reflect-videos.s3.amazonaws.com/2641_969e.mp4",
"stepCount": 14
}
]
},
{
"instanceId": 1,
"testId": 686,
"startingUrl": "https://staging.myapp.com/test-two",
"status": "passed",
"runs": [
{
"runId": 2642,
"browser": "Chrome",
"status": "passed",
"variables": {},
"startTime": 1643734994863,
"endTime": 1643735012070,
"runTime": 17207,
"timestamp": 1643735012534,
"videoUrl": "https://reflect-videos.s3.amazonaws.com/2641_89f7.mp4",
"stepCount": 9
}
]
},
{
"instanceId": 2,
"testId": 688,
"startingUrl": "https://staging.myapp.com/test-three",
"status": "failed",
"runs": [
{
"runId": 2643,
"browser": "Chrome",
"status": "failed",
"variables": {},
"startTime": 1643734996842,
"endTime": 1643735012095,
"runTime": 15253,
"timestamp": 1643735012539,
"videoUrl": "https://reflect-videos.s3.amazonaws.com/2641_4e95.mp4",
"stepCount": 6,
"failedStepIndex": {
"4": "Visual Observe Failed - image difference exceeds limit",
"5": "Element Selection Failed"
}
}
]
}
],
"bookmark": ""
}
}Response Fields
suiteId string: The identifier specified in the execution request.
executionId number: The unique numeric identifier assigned to the suite execution.
overrides optional object: Overrides applied to properties of all tests executed in this suite.
gitHub optional object: An object specifying the GitHub repository name, its owner, and commit SHA to post the status of the execution.
url string: The in-app URL that displays information and results about the suite execution.
isFinished boolean: A boolean indicating whether the suite execution has completed.
status string: One of: pending, canceled, passed or failed.
tests object: A wrapped collection of test instances (test definition & execution configuration) with individual test runs.
tests.bookmark optional string: Reserved for future - used to read additional test instances.
tests.data.instanceId number: The identifier for a logical test instance (definition & configuration) in the execution.
tests.data.testId number: The identifier for the test.
tests.data.actionName optional string: The name assigned to the Run Tests action the instance originated from, if one was given.
tests.data.startingUrl string: The initial URL used by the test, accounting for any user-specified overrides.
tests.data.status string: One of: pending, passed or failed. Summarizes the outcome of the test instance from its runs.
tests.data.runs object: An array of test runs for a single test instance, with completion timestamp and status.
tests.data.runs.browser string: The browser used to execute this test run. One of: Chrome, Edge, Firefox, or Safari.
tests.data.runs.status string: The outcome of the test run. Either passed or failed.
tests.data.runs.variables object: The variables that were used or created by the test run.
tests.data.runs.startTime number: Timestamp of when the test run started.
tests.data.runs.endTime number: Timestamp of when the test run stopped.
tests.data.runs.runTime number: Total time that the test run spent running (endTime - startTime).
tests.data.runs.timestamp number: Timestamp that the test run's execution was recorded in API Hub for Test.
tests.data.runs.videoUrl string: The URL of a video recording of the test run.
tests.data.runs.stepCount number: The number of steps in the executed definition.
tests.data.runs.failedStepIndex optional object: A dictionary containing the failed steps from the test run. The key represents the index of the step (zero-based), and the value is a summary error message describing the failure encountered during that step. Only defined when the status of the run is failed.
Cancel an execution
Attempts to cancel the specified suite execution.
Request
PATCH https://api.reflect.run/v1/suites/<suite-id>/executions/<execution-id>/cancel
Response
{
"suiteId": "regression-tests",
"executionId": 3878,
"success": true
}Response Fields
suiteId string: The identifier specified in the execution request.
executionId number: The unique numeric identifier assigned to the suite execution.
success boolean: A boolean indicating whether the suite execution was successfully canceled.