Debugging Test Cases
When a test suite fails, or when you are developing complex new scenarios, running the entire suite can be slow. Drift provides several mechanisms to isolate and debug specific operations.
Tag-Based Filtering
Tags allow you to execute specific subsets of your test suite. You can include or exclude tags using the --tags flag.
Important
Important Logic Note: Drift uses OR logic between tags. If you provide multiple tags, Drift will run any operation that matches at least one of the specified criteria. It does not currently support AND logic.
Examples
Run operations tagged with "post" OR "error-response":
drift verifier -u http://localhost:8080 -f product.testcases.yaml --tags 'post,error-response'
Run operations tagged with "post" but EXCLUDE those tagged with "error-response": Use the ! prefix to exclude specific tags.
drift verifier -u http://localhost:8080 -f product.testcases.yaml --tags 'post,!error-response'
Running a Single Operation
To debug one specific failure, use the --operation flag to execute a single test case by its ID.
drift verifier -u http://localhost:8080 -f drift.yaml --operation getProductByID_Success
Re-running Failed Tests
If you have a large suite with multiple failures, the --failed flag allows you to focus exclusively on the tests that did not pass during the previous run.
drift verifier -u http://localhost:8080 -f drift.yaml --failed
Generating an AI Fix Prompt
When tests fail and the cause is not immediately obvious, use the --generate-fix-prompt flag to produce an AI prompt file. Drift writes this file to the output directory alongside the other result files. You can then pass it to an AI agent to get targeted suggestions for fixing the failing tests.
drift verify -u http://localhost:8080 -f drift.yaml --generate-fix-prompt
The prompt file captures the failure details in a format designed to give an AI agent the context it needs to diagnose root causes and suggest corrections to your test cases or API implementation.
Using the prompt file with an AI agent
Run drift verify with --generate-fix-prompt.
Locate the generated prompt file in the output directory (default: the same directory as your test case files).
Pass the file to your AI agent - for example, by attaching it to a chat session or referencing it in a prompt.
If you have the Drift AI agent skill installed, you can ask it to read the prompt file directly and suggest fixes.
Reading Test Output
Drift displays test results in a formatted table. This helps you quickly identify which operations passed or failed.
Successful Run
─[ Summary ]─────────────────────────────────────────────────────────────────────────────────────────── Executed 1 test case (1 passed, 0 failed) Executed 9 operations (9 passed, 0 failed, 0 skipped) Execution time 1.288865209s Setup time 72.192458ms ┌────────────────────────────────┬──────────────────────────────────┬───────────────────────────┬────────┐ │ Testcase ┆ Operation ┆ Target ┆ Result │ ╞════════════════════════════════╪══════════════════════════════════╪═══════════════════════════╪════════╡ │ Product API Tests ┆ createProduct_Success ┆ source-oas:createProduct ┆ OK │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ┆ getAllProducts_Success ┆ source-oas:getAllProducts ┆ OK │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ┆ getProductByID_Success ┆ source-oas:getProductByID ┆ OK │ └────────────────────────────────┴──────────────────────────────────┴───────────────────────────┴────────┘
Each row shows:
Testcase: The name of your test suite
Operation: The specific operation that was executed
Target: Where the operation came from (e.g.,
source-oas:operationId)Result:
OK(passed) orFAILED
Failed Run with Error Details
When tests fail, Drift displays them in the table with a FAILED status, followed by a detailed failure section:
─[ Summary ]─────────────────────────────────────────────────────────────────────────────────────────── Executed 1 test case (0 passed, 1 failed) Executed 9 operations (7 passed, 2 failed, 0 skipped) ┌───────────────────┬──────────────────────────────────┬───────────────────────────┬────────┐ │ Testcase ┆ Operation ┆ Target ┆ Result │ ╞═══════════════════╪══════════════════════════════════╪═══════════════════════════╪════════╡ │ Product API Tests ┆ createProduct_Success ┆ source-oas:createProduct ┆ FAILED │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤ │ ┆ getAllProducts_Success ┆ source-oas:getAllProducts ┆ FAILED │ └───────────────────┴──────────────────────────────────┴───────────────────────────┴────────┘ ─[ Failures ]────────────────────────────────────────────────────────────────────────────────────────── ┌─ Testcase 'Product API Tests' │ │ Operation 'createProduct_Success': │ │ status_code: Expected response status Client Error (404) but got 201 │ │ Operation 'getAllProducts_Success': │ │ status_code: Expected response status Success (201) but got 200 │
The Failures section provides:
Which operation failed
The specific assertion that didn't match (e.g.,
status_code)What was expected vs. what was received
Use this information to quickly identify and fix issues in your test cases or API implementation.
Increasing Log Verbosity
When a test fails and the reason is unclear, increase the logging level to debug or trace to see the full request/response exchange.
# Set level to 'debug' for detailed internal logs drift verifier -u http://localhost:8080 -f drift.yaml --log-level debug
Script-Level Debugging
Within your Lua scripts, use the built-in dbg() function to print the structure of event data directly to the console.
-- drift.lua ["operation:started"] = function(event, data) -- Prints a readable table of the operation metadata, including tags and ID print(dbg(data)) end
Debugging AsyncAPI Operations
Message-based operations fail differently from HTTP operations. The most common outcome is a capture timeout, which reports that no matching message arrived rather than pointing at the underlying cause.
Capture timeouts
Drift waits timeout-ms for a message matching your correlation ID. When the window closes, work through these in order:
Check | How |
|---|---|
Did the trigger run? | Trigger output appears in the Drift log. Hook paths are relative to the test case file. |
Did the trigger succeed? | A non-zero exit fails the operation. Write diagnostics to standard error so they appear in the output. |
Same correlation ID? | The published message must carry the value Drift is filtering on. Pass it through with |
Same channel address? | The message must go to the channel address in the AsyncAPI document, not a similar one. |
Does the channel exist? | Drift does not create topics or queues. Create them before the run. |
Is the window long enough? | Increase |
Broker connection failures
A connection error means Drift could not reach the broker at all. Check the host in your AsyncAPI servers block, and confirm the broker is running and reachable from where Drift runs. In CI, confirm the service container is ready before the Drift step starts.
For SNS and SQS, a signature or authorization error usually means the wrong region. See AWS Messaging Plugin.
Probe failures
Symptom | Cause |
|---|---|
Probe output rejected | The probe wrote non-JSON to standard output. Send logs to standard error. |
Probe passes on a broken service | State from an earlier run was not cleared. Reset it in an |
Probe fails intermittently | The probe ran before processing finished. Poll inside the probe rather than lengthening the operation timeout. |
Unexpected execution mode
If Drift ran a different mode than you expected, check the AsyncAPI action first. An operation with action: send means your service publishes, so Drift captures. An operation with action: receive means your service consumes, so Drift publishes. Drift's role is always the opposite of your service's.
Set execution-mode explicitly to confirm. Drift validates mode preconditions before contacting the broker, so a mismatch fails immediately with a clear message.