Advanced Data Expressions

Drift provides a powerful expression language to help you inject and manipulate data within your test cases. This allows you to create dynamic tests that avoid data collisions without writing complex setup scripts.

Basic Data Injection

You can inject values from a registered dataset using the ${sourceName:path.to.data} syntax.

# yaml
getProductByID:
  target: product-oas:getProductByID
  dataset: "product"
  parameters:
    path:
      id: ${product:products.product10.id} # Injects the value '10' from the dataset

Using Built-in Expressions

Drift includes built-in functions to help you generate values that are guaranteed to interact with your dataset correctly.

The `notIn` Expression

The notIn expression is used to generate a value that is explicitly not present in a specific dataset path. This is ideal for testing 404 Not Found scenarios.

# yaml
getProductByID_DoesNotExist:
  target: product-oas:getProductByID
  dataset: "product"
  parameters:
    path:
      # Generates an ID that does NOT exist in the products list
      id: ${ product:notIn(products.*.id) } 
  expected:
    response:
      statusCode: 404

products.*.id: The wildcard * tells Drift to look at the id field across all objects in the products collection.

Expression Types

Drift supports three types of expressions that can be embedded in your test files:

Dataset Expressions

Reference data from loaded datasets:

${dataset-name:path.to.value}

Environment Variable Expressions

Access environment variables for configuration and secrets:

${env:VARIABLE_NAME}

Lua Function Expressions

Call exported Lua functions for dynamic values:

${functions:function_name}

Using Environment Variable Expressions

Environment variable expressions allow you to externalize configuration and keep secrets out of your test files.

Basic Environment Variable Usage

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1
title: "Product API Tests"

sources:
  - name: product-oas
    path: openapi.yaml
  - name: product-data
    # Load different datasets based on environment
    path: ${env:ENVIRONMENT}-dataset.yaml
  - name: functions
    path: product.lua

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        # Load token from environment variable
        token: ${env:API_TOKEN}

Common Use Cases for Environment Variables

Multi-Environment Testing

sources:
  - name: api-spec
    # Dynamically load environment-specific spec
    uri: https://${env:ENVIRONMENT}.api.example.com/openapi.yaml
  - name: test-data
    # Load environment-specific dataset
    path: ${env:ENVIRONMENT}-dataset.yaml

Secret Management

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        token: ${env:CI_API_TOKEN}

sources:
  - name: remote-spec
    uri: https://internal.api.example.com/spec.yaml
    auth:
      username: ${env:SPEC_USERNAME}
      secret: ${env:SPEC_PASSWORD}

CI/CD Integration

sources:
  - name: api-spec
    # Use build-specific version
    path: openapi-${env:BUILD_NUMBER}.yaml

Using Lua Function Expressions

Call exported Lua functions to generate dynamic values at runtime.

Basic Function Call

drift.yaml:

operations:
  createProduct:
    target: api-spec:createProduct
    parameters:
      headers:
        x-request-id: ${functions:generate_request_id}
      request:
        body:
          name: ${functions:random_product_name}

product.lua:

local function generate_request_id()
  return string.format("req-%d", os.time())
end

local function random_product_name()
  return "Product-" .. os.time()
end

local exports = {
  exported_functions = {
    generate_request_id = generate_request_id,
    random_product_name = random_product_name
  }
}

return exports

Dynamic Source Names

You can even use Lua functions to determine source names:

sources:
  - name: functions
    path: config.lua
  - name: ${functions:get_env_prefix}-oas
    path: openapi.yaml

config.lua:

local function get_env_prefix()
  return os.getenv("ENVIRONMENT") or "dev"
end

local exports = {
  exported_functions = {
    get_env_prefix = get_env_prefix
  }
}

return exports

Where Expressions Can Be Used

Expressions can be embedded in most simple string values throughout your Drift files, but with important limitations based on execution order.

Supported Locations

Expressions work in:

Source paths and URIs (with ordering rules - see below)
Source names (with ordering rules)
Source authentication credentials
Global parameters, expected values, and ignore rules
Operation descriptions
Operation dataset references
Operation parameters (path, query, headers, body values)
Operation expected values
Operation ignore rules
Plugin configurations

Unsupported Locations

Expressions cannot be used in:

File header: drift-testcase-file: v1
Operation keys: The operation names in the operations map
Operation tags: The tags field
Global apply field: The apply: true/false boolean

Example: Valid Expression Usage

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1  #  No expressions allowed here
title: "${env:TEAM} API Tests"  #  Allowed

sources:
  - name: ${env:ENVIRONMENT}-oas  #  Allowed
    path: specs/${env:ENVIRONMENT}/openapi.yaml  #  Allowed
    uri: https://${env:API_HOST}/spec.yaml  #  Allowed
    auth:
      username: ${env:SPEC_USER}  #  Allowed
      secret: ${env:SPEC_PASS}  #  Allowed

global:
  auth:
    apply: true  #  No expressions allowed here
    parameters:
      authentication:
        scheme: bearer
        token: ${env:API_TOKEN}  #  Allowed

operations:
  getProduct_Success:  #  No expressions in operation keys
    description: "Test product ${env:PRODUCT_ID}"  #  Allowed
    dataset: ${env:DATASET_NAME}  #  Allowed
    parameters:
      path:
        id: ${env:TEST_PRODUCT_ID}  #  Allowed

Execution Order and Dependencies

Drift resolves expressions in a specific order. Understanding this order is critical for avoiding errors.

Resolution Order

Sources section - Sources are loaded in the order they appear
Operation description - Resolved first (for error messages)
Operation dataset - Resolved second
Operation parameters and expected values - Resolved third

Dependency Rules

Rule 1: Can't reference later fields in earlier fields

operations:
  createProduct:
    #  WRONG: Can't use dataset in description
    description: "Create ${product:products.product10.name}"
    dataset: product  # Dataset is resolved AFTER description
    
    #  WRONG: Can't use parameters in dataset
    dataset: ${parameters:dataset_name}  # Parameters haven't been resolved yet

Rule 2: Environment variables and Lua functions work everywhere

operations:
  createProduct:
    #  CORRECT: env and functions work in all fields
    description: "Create product in ${env:ENVIRONMENT}"
    dataset: ${env:DATASET_NAME}
    parameters:
      path:
        id: ${functions:generate_id}

Rule 3: Lua files must be loaded before use

#  WRONG: Using functions before loading the Lua file
sources:
  - name: ${functions:name}-oas  # Error: functions not loaded yet!
    path: openapi.yaml
  - name: functions
    path: product.lua

#  CORRECT: Load Lua file first
sources:
  - name: functions
    path: product.lua
  - name: ${functions:name}-oas  # Now functions are available
    path: openapi.yaml

Practical Use Cases

Use Case 1: Environment-Specific Testing

Run the same tests against multiple environments:

drift.yaml:

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1
title: "API Tests - ${env:ENVIRONMENT}"

sources:
  - name: api-spec
    uri: https://${env:ENVIRONMENT}.api.example.com/openapi.yaml
  - name: test-data
    path: data/${env:ENVIRONMENT}.dataset.yaml
  - name: functions
    path: ${env:ENVIRONMENT}.lua

operations:
  getProduct:
    target: api-spec:getProduct
    dataset: test-data
    parameters:
      path:
        id: ${test-data:testProducts.validProduct.id}

Run against different environments:

# Development
ENVIRONMENT=dev drift verifier --test-files drift.yaml

# Staging
ENVIRONMENT=staging drift verifier --test-files drift.yaml

# Production
ENVIRONMENT=prod drift verifier --test-files drift.yaml

Use Case 2: Secure CI/CD Integration

Keep secrets out of source control:

drift.yaml:

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1

sources:
  - name: api-spec
    uri: ${env:SPEC_URL}
    auth:
      username: ${env:SPEC_USERNAME}
      secret: ${env:SPEC_PASSWORD}

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        token: ${env:API_TOKEN}

GitHub Actions workflow:

- name: Run Drift Tests
  env:
    SPEC_URL: ${{ secrets.SPEC_URL }}
    SPEC_USERNAME: ${{ secrets.SPEC_USERNAME }}
    SPEC_PASSWORD: ${{ secrets.SPEC_PASSWORD }}
    API_TOKEN: ${{ secrets.API_TOKEN }}
  run: drift verifier --test-files drift.yaml

Use Case 3: Dynamic Test Data Generation

Generate unique values to avoid test conflicts:

drift.yaml:

operations:
  createUser:
    target: api-spec:createUser
    parameters:
      request:
        body:
          email: ${functions:unique_email}
          username: ${functions:unique_username}
          timestamp: ${functions:current_timestamp}

functions.lua:

local function unique_email()
  return string.format("test-%d@example.com", os.time())
end

local function unique_username()
  return string.format("user_%d", os.time())
end

local function current_timestamp()
  return os.date("!%Y-%m-%dT%H:%M:%SZ")
end

local exports = {
  exported_functions = {
    unique_email = unique_email,
    unique_username = unique_username,
    current_timestamp = current_timestamp
  }
}

return exports

Use Case 4: Team-Specific Configurations

Different teams can use the same test suite with their own configs:

drift.yaml:

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1
title: "${env:TEAM_NAME} API Tests"

sources:
  - name: api-spec
    path: openapi.yaml
  - name: team-data
    path: teams/${env:TEAM_NAME}/data.yaml
  - name: functions
    path: teams/${env:TEAM_NAME}/functions.lua

Usage:

Use Case 5: Feature Flag Testing

Test different API behaviors based on feature flags:

drift.yaml:

operations:
  getProduct_WithNewFeature:
    target: api-spec:getProduct
    parameters:
      headers:
        x-feature-flag: ${env:FEATURE_BETA_ENABLED}
      path:
        id: 123
    expected:
      response:
        statusCode: 200

Troubleshooting

Check the most common issues:

Error: "Cannot resolve expression"

Problem: Expression references an undefined variable or function.

Solutions:

Check environment variable is set: echo $ENVIRONMENT
Verify Lua function is exported in the exported_functions table
Ensure dataset name matches exactly (case-sensitive)

Error: "Source not found" when using Lua function in source name

Problem: Lua file loaded after the source that uses it.

Solution: Move the Lua source file above any sources that reference its functions:

# CORRECT order
sources:
  - name: functions
    path: config.lua
  - name: ${functions:get_env}-oas
    path: openapi.yaml

Error: Expression in operation key

Problem: Trying to use expression in operation name.

# WRONG
operations:
  ${env:OPERATION_NAME}:
    target: api-spec:operation

Solution: Use static operation names, but dynamic descriptions:

# CORRECT
operations:
  standardOperation:
    description: "Testing ${env:SCENARIO}"
    target: api-spec:operation

Expression not resolving in description

Problem: Using dataset or parameter references in description.

#  WRONG: dataset not yet loaded when description is resolved
operations:
  test:
    description: "Test product ${product:name}"
    dataset: product

Solution: Use only env vars and functions in descriptions:

# CORRECT
operations:
  test:
    description: "Test product in ${env:ENVIRONMENT}"
    dataset: product

Best Practices

Use environment variables for secrets - Never commit tokens or passwords
Prefix environment variables - Use namespacing like DRIFT_, API_, TEST_ to avoid conflicts
Document required environment variables - Add a README with all required env vars

Provide defaults in Lua - Lua functions can provide fallback values:

local function get_api_url()
  return os.getenv("API_URL") or "http://localhost:8080"
end

Load Lua files first - Always place Lua sources at the top of the sources list
Use expressions sparingly - Only use them when you need dynamic behavior
Test locally first - Verify expressions resolve correctly before using in CI/CD

How Expressions DRY Up Your Tests

Expressions help you follow the DRY (Don't Repeat Yourself) principle by centralizing configuration and reusing values across multiple test operations.

Without Expressions: Repetition and Maintenance Burden

┌─────────────────────────────────────┐
│   drift.yaml (Hard-coded values)    │
├─────────────────────────────────────┤
│                                     │
│  operations:                        │
│    test1:                           │
│      parameters:                    │
│        headers:                     │
│          authorization: "token123"  │ ← Repeated
│          x-api-version: "v2"        │ ← Repeated
│                                     │
│    test2:                           │
│      parameters:                    │
│        headers:                     │
│          authorization: "token123"  │ ← Repeated
│          x-api-version: "v2"        │ ← Repeated
│                                     │
│    test3:                           │
│      parameters:                    │
│        headers:                     │
│          authorization: "token123"  │ ← Repeated
│          x-api-version: "v2"        │ ← Repeated
└─────────────────────────────────────┘

Problems:

Multiple updates needed to change API token
Copy-paste errors
No environment-specific configuration
Secrets in source control

With Expressions: Single Source of Truth

┌──────────────────────┐     ┌──────────────────────┐     ┌──────────────────────┐
│  Environment Vars    │     │   Lua Functions      │     │     Datasets         │
│ ──────────────────── │     │ ──────────────────── │     │ ──────────────────── │
│                      │     │                      │     │                      │
│ API_TOKEN=xyz789     │     │ function bearer()    │     │ testUsers:           │
│ ENVIRONMENT=staging  │     │   return token()     │     │   adminUser:         │
│ API_VERSION=v2       │     │ end                  │     │     id: 42           │
│                      │     │                      │     │     role: admin      │
└──────────┬───────────┘     └──────────┬───────────┘     └──────────┬───────────┘
           │                            │                            │
           │                            │                            │
           └────────────────┬───────────┴────────────────────────────┘
                            │
                            ▼
           ┌────────────────────────────────────────┐
           │         drift.yaml (DRY)               │
           ├────────────────────────────────────────┤
           │                                        │
           │ global:                                │
           │   auth:                                │
           │     apply: true                        │
           │     parameters:                        │
           │       headers:                         │
           │         authorization:                 │
           │           ${env:API_TOKEN}         ◄───┼─── ONE reference
           │         x-api-version:                 │
           │           ${env:API_VERSION}       ◄───┼─── ONE reference
           │                                        │
           │ operations:                            │
           │   getUserById:                         │
           │     parameters:                        │
           │       path:                            │
           │         id: ${testUsers:adminUser.id}◄─┼─── ONE reference
           │                                        │
           │   updateUser:                          │
           │     # Inherits global auth             │
           │     parameters:                        │
           │       path:                            │
           │         id: ${testUsers:adminUser.id}◄─┼─── Reused
           │                                        │
           │   deleteUser:                          │
           │     # Inherits global auth             │
           │     parameters:                        │
           │       path:                            │
           │         id: ${testUsers:adminUser.id}◄─┼─── Reused
           └────────────────────────────────────────┘

Benefits:

One place to change API token
Environment-specific via ${env:ENVIRONMENT}
Type-safe data from datasets
Dynamic values from Lua functions

Real-World Impact

Before Expressions:

operations:
  test1:
    parameters:
      headers:
        authorization: "Bearer prod-token-12345"
      path:
        userId: "100"
  test2:
    parameters:
      headers:
        authorization: "Bearer prod-token-12345"  # Copy-paste
      path:
        userId: "100"  # Hardcoded
  test3:
    parameters:
      headers:
        authorization: "Bearer prod-token-12345"  # Copy-paste
      path:
        userId: "100"  # Hardcoded

To test in staging, you'd need to update 50+ places manually.

After Expressions:

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        token: ${env:API_TOKEN}

operations:
  test1:
    dataset: users
    parameters:
      path:
        userId: ${users:testUser.id}
  test2:
    dataset: users
    parameters:
      path:
        userId: ${users:testUser.id}
  test3:
    dataset: users
    parameters:
      path:
        userId: ${users:testUser.id}

Change environment with ONE variable: ENVIRONMENT=staging drift verifier ...

Using Lua Expressions for Custom Validation

Expressions can also be used for custom validation in your test assertions, not just for generating values. By referencing exported Lua functions in your expected block, you can perform dynamic, field-level validation on response data.

Example: Field-Level Validation with Lua Functions

drift.yaml:

operations:
  getProductByID:
    target: product-oas:getProductByID
    description: "Get a product by ID"
    dataset: "product"
    parameters:
      path:
        id: ${dataset:products.product10.id}
    expected:
      response:
        statusCode: 200
        body:
          price: ${functions:validate_price}
          version: ${functions:validate_version}
    tags: get

functions.lua:

local exports = {
  exported_functions = {
    validate_price = function(price)
      if price == nil then
        return error("Price is required")
      end
      if price < 0 then
        return error("Price must be non-negative")
      end
      return price
    end,
    validate_version = function(version)
      if version == nil then
        return "1.0.0"
      end
      if version ~= "2.0.0" then
        return error("Version must be 2.0.0")
      end
      return version
    end
  }
}
return exports

How it works

The ${functions:validate_price} and ${functions:validate_version} expressions call your Lua functions with the actual value from the response.
If the function throws an error, the test fails with your message.
If the function returns a value, validation passes and the value is used for further checks.

This pattern is ideal for custom business rules, dynamic value checks, or conditional assertions.

In this section:

Advanced Data Expressions

Basic Data Injection

Using Built-in Expressions

The notIn Expression

Expression Types

Dataset Expressions

Environment Variable Expressions

Lua Function Expressions

Using Environment Variable Expressions

Basic Environment Variable Usage

Common Use Cases for Environment Variables

Using Lua Function Expressions

Basic Function Call

Dynamic Source Names

Where Expressions Can Be Used

Supported Locations

Unsupported Locations

Example: Valid Expression Usage

Execution Order and Dependencies

Resolution Order

Dependency Rules

Practical Use Cases

Use Case 1: Environment-Specific Testing

Use Case 2: Secure CI/CD Integration

Use Case 3: Dynamic Test Data Generation

Use Case 4: Team-Specific Configurations

Use Case 5: Feature Flag Testing

Troubleshooting

Error: "Cannot resolve expression"

Error: "Source not found" when using Lua function in source name

Error: Expression in operation key

Expression not resolving in description

Best Practices

How Expressions DRY Up Your Tests

Without Expressions: Repetition and Maintenance Burden

With Expressions: Single Source of Truth

Real-World Impact

Using Lua Expressions for Custom Validation

Example: Field-Level Validation with Lua Functions

How it works

Search results

The `notIn` Expression