Test Generation featuring HaloAI
Note
API Hub for Contract Testing Test Generation features are currently available through an open beta program. During this stage, all customers can explore the capabilities within fixed usage limits. For more information, go here.
If you don't have an API Hub for Contract Testing account, start here.
API Hub for Contract Testing Test Generation is an AI-powered tool that rapidly creates Pact tests from source code. OpenAPI descriptions, or traffic capture data. This accelerates contract test creation and maintenance.
Enabling the feature
This feature is disabled by default in all organizations. To enable the feature:
In the UI, navigate to "Settings" -> "Preferences" -> "System Preferences".
Check the "Test Generation" checkbox.
Save the settings.
Installation
For *nix users (including Windows users running WSL/msys2/mingw), use the following command to download and install:
curl https://download.pactflow.io/ai/get.sh | sh
For Windows PowerShell users, use the following command to download and install:
Invoke-WebRequest -Uri https://download.pactflow.io/ai/get.ps1 | Invoke-Expression
Installation Options
There are options which you can set during installation.
For a full list, see the --help/-h command
curl https://download.pactflow.io/ai/get.sh | sh -s -- -h
--verbose/-v/PACTFLOW_AI_VERBOSE: Enable verbose output--quiet/-q/PACTFLOW_AI_QUIET: Disable progress output--yes/-y/PACTFLOW_AI_YES: Disable confirmation prompt and assume 'yes'--destination/-d/PACTFLOW_AI_DESTINATION: Specify the directory to install the binary--default-host/PACTFLOW_AI_DEFAULT_HOST: Choose a default host triple rather than autodetecting--no-modify-path/PACTFLOW_AI_NO_MODIFY_PATH: Don't configure the PATH environment variable
Verify the installation by running pactflow-ai to ensure it executes successfully.
Manual installation
Alternatively, download the latest version for your OS and architecture from the table below. Be sure to add it to your environment's PATH:
OS | Architecture | Link |
|---|---|---|
Apple Darwin | aarch64 |
|
Apple Darwin | x86_64 |
|
Windows MSVC | x86_64 |
|
Windows MSVC | aarch64 |
|
Linux GNU | x86_64 |
|
Linux MUSL | x86_64 |
|
Linux GNU | aarch64 |
|
Linux MUSL | aarch64 |
|
Note
Linux GNU users will require glibc version 2.23 or later.
Environments which do not use glibc, or use a version of glibc prior to 2.23, should instead use the musl variant.
Usage
Running pactflow-ai --help will give you the detailed usage for any of the subcommands.
Authentication
To authenticate, use valid API Hub for Contract Testing API Tokens. Obtain these tokens from the Settings > Tokens page in your API Hub for Contract Testing account.
You can authenticate with the CLI in several ways:
Environment Variables
Export the following environment variables. The CLI will use these credentials to communicate with API Hub for Contract Testing:
export PACT_BROKER_BASE_URL="https://YOUR_ACCOUNT.pactflow.io" export PACT_BROKER_TOKEN="YOUR_TOKEN"
Local configuration file
You can also configure the CLI using the following commands:
pactflow-ai config set url https://YOUR_ACCOUNT.pactflow.io pactflow-ai config set token YOUR_TOKEN
This action creates a .pactflow.toml file. Add the file to your .gitignore.
User configuration file
Alternatively, store the user configuration file in ~/.config/pact/config.toml (Unix and macOS), and %APPDATA%\pact\config.toml (Windows).
Pass the --user flag when running pactflow-ai config set to write to this file automatically.
Usage Limits
The feature meters access based on the number of tests generated in a 30-day rolling window. The system allocates credits as follows:
Starter Plans: 10 credits per month
All other plans: 10 credits per purchased user per month
The system allocates credits to the organization, not to individual users.
Examples
Consumption
Each successfully generated test consumes a credit. If your organization reaches its limit, the system displays the following error:
ERROR: Client error: 403 Forbidden AI credits are exhausted.
Note
The limits and your usage against these limits are not currently visible on the subscription page. This will be added in due course.
Role Based Access Control (RBAC)
In addition to the system preference that enables AI within your workspace, API Hub for Contract Testing's RBAC model allows you to manage access to specific AI capabilities for your users. For detailed information, refer to the permissions documentation.
By default, users assigned the User role have access to all AI permissions (ai:*), unless:
Your account was created prior to the release of the AI feature (September 2024), or
You have customized your role configuration.
Tutorials
We've created an in-browser tutorial to help you get started with PactFlow's test generation feature.
Troubleshooting
"You do not have the necessary permissions to access this resource"
If you encounter the following error:
ERROR: Client error: 403 Forbidden You do not have the necessary permissions to access this resource, Required permissions: ai:generation:request-response
This indicates that the user does not have the required permissions to access the feature.
"AI Disabled in System Preferences"
If the error message states that "AI is disabled in system preferences", this means the AI feature has not been enabled for your account.
"AI credits are exhausted"
If you encounter the following error:
ERROR: Client error: 403 Forbidden AI credits are exhausted.
This indicates that you have exceeded the AI credits on the account and can no longer use any AI features.
"AI is not enabled for your plan"
If the error message states that "AI features are not enabled on your plan", your plan does not include AI features as part of its entitlements.
Getting help and providing feedback
For feedback, feature requests, or tool assistance, contact the email address you received with your access details. Use the usual methods for support unrelated to the AI feature.