Kafka Plugin

The Kafka plugin gives Drift the ability to publish to and consume from Apache Kafka topics. It is used alongside the AsyncAPI plugin to verify message-based APIs whose AsyncAPI document declares the kafka protocol.

Drift connects to your broker and exchanges real messages. You must have a running Kafka broker, and the topics in your document must already exist.

Property

Value

Plugin name

kafka

Purpose

Kafka transport for AsyncAPI operations

Protocol

kafka

Used with

.asyncapi, .json

Supports

Asynchronous and request/reply interactions

Enabling the Plugin

plugins:
  - name: asyncapi
  - name: kafka
  - name: json

The plugin activates for AsyncAPI documents whose server declares protocol: kafka.

Plugin

Role

asyncapi

Reads the document and plans the interaction

kafka or aws-messaging

Connects to the broker and moves messages

json

Builds and validates message payloads

The plugin activates for AsyncAPI documents whose server declares protocol: kafka.

Supported Execution Modes

afka is the most complete transport. All four execution modes are available.

Mode

Supported

What Drift does

async-observe

Yes

Subscribes to the topic and captures the message your service publishes

async-inject

Yes

Publishes to the topic, then runs your probe

async-inject-capture

Yes

Subscribes to a probe topic, publishes, then captures the harness result

async-request-reply

Yes

Subscribes to the reply topic, publishes the request, captures the correlated reply

See AsyncAPI Execution Modes.

Connection Details

Drift reads the broker address from your AsyncAPI servers block:

servers:
  local:
    host: localhost:9092
    protocol: kafka
    description: Local Kafka broker

Override it per environment through plugin configuration, or per test through the test case. Test case values win, then plugin configuration, then the document.

Channel Addresses

The AsyncAPI channel address is the Kafka topic name:

channels:
  orderCreated:
    address: drift.examples.order-created

Correlation

Drift matches captured messages against a correlation identifier carried in a Kafka record header. Declare its location in your message:

components:
  messages:
    orderCreated:
      correlationId:
        location: "$message.header#/correlation-id"

Declaring the location is required. The value is optional: Drift takes it from the AsyncAPI document, then from the correlation-id parameter in your test case, and generates a UUID when neither supplies one. Drift filters captured records to the one whose header matches, and ignores everything else on the topic.

Set the value yourself when a trigger hook or an expectation needs the same identifier. Where you do set it, give each operation a distinct value - a shared one lets a message from one test satisfy another.

Capture Behavior

When Drift subscribes, it records the current end of the topic log and reads forward from there. Messages published before the subscription are not captured, which is what makes the subscribe-then-trigger ordering reliable.

Drift polls for new records until a matching message arrives or the capture window closes.

Setting

Where

Default

Purpose

timeout-ms

Test case parameter

30000

How long Drift waits for a matching message

poll-interval-ms

Plugin configuration

500

How often Drift polls the broker between fetches

parameters:
  timeout-ms: 5000

Lower poll-interval-ms reduces capture latency at the cost of more requests to the broker. The default suits local development and CI.

Creating Topics

Drift does not create topics. Create every topic in your AsyncAPI document, and any probe topic, before the run.

Kafka's automatic topic creation fires only when a producer or consumer first touches a topic. That is too late for async-inject-capture, where Drift publishes to the input topic before any consumer has subscribed.

docker exec <kafka-container> kafka-topics \
  --bootstrap-server localhost:9092 \
  --create --if-not-exists \
  --topic drift.examples.order-created \
  --partitions 1 --replication-factor 1

Running a Local Broker

A local broker is the recommended setup for development and CI. It is fast, repeatable, and nothing you publish affects anyone else.

services:
  kafka:
    image: confluentinc/cp-kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ENABLE_KRAFT: "yes"
      KAFKA_NODE_ID: "1"
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_LISTENERS: "PLAINTEXT://:9092,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1"
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1"
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0"

Running against a shared cluster is supported, but that cluster is yours to manage. Messages Drift publishes are real messages, and downstream consumers will process them.

Transport Security

The Kafka plugin speaks the Kafka wire protocol over raw TCP sockets. It does not support authentication or encryption in transit:

  • No SASL authentication of any mechanism

  • No TLS, and no mutual TLS

  • Every connection is plaintext

Point Drift at a broker that accepts plaintext connections, and treat contract verification as something you run against a local, disposable, or otherwise non-sensitive broker rather than a secured production cluster.

Limitations

  • Drift captures the first record matching the correlation ID. It does not verify ordering across a stream.

  • Drift does not create topics.

  • Drift does not verify consumer group behavior, partition assignment, rebalancing, or offset management.

  • Drift does not verify delivery guarantees, retry behavior, or dead-letter handling.

  • Authentication and TLS are not supported. Connections are plaintext.

Troubleshooting

Symptom

Likely cause

Connection refused

The broker is not running, or host in the servers block is wrong

Connection fails against a secured broker

The plugin connects over plaintext TCP only. It cannot reach a broker that requires SASL or TLS.

Capture timeout, trigger succeeded

The published record carries a different correlation ID, or went to a different topic

Capture timeout on async-inject-capture

The probe topic did not exist before the run

Topic not found

Create the topic in advance. Automatic creation is too late for Drift.

Works locally, times out in CI

The broker service container was not ready. Add a readiness check before the Drift step.

See Debugging Test Cases.

See Also

Publication date: