Skip to main content
Swagger
All documentation pages

Kafka Plugin

The Kafka plugin gives Drift the ability to publish to and consume from Apache Kafka topics. It is used alongside the AsyncAPI plugin to verify message-based APIs whose AsyncAPI document declares the kafka protocol.

Drift connects to your broker and exchanges real messages. You must have a running Kafka broker, and the topics in your document must already exist.

Property Value
Plugin name kafka
Purpose Kafka transport for AsyncAPI operations
Protocol kafka
Used with .asyncapi, .json
Supports Asynchronous and request/reply interactions

Enabling the Plugin

plugins:
  - name: asyncapi
  - name: kafka
  - name: json

The plugin activates for AsyncAPI documents whose server declares protocol: kafka.

Plugin Role
asyncapi Reads the document and plans the interaction
kafka or aws-messaging Connects to the broker and moves messages
json Builds and validates message payloads

The plugin activates for AsyncAPI documents whose server declares protocol: kafka.

Supported Execution Modes

afka is the most complete transport. All four execution modes are available.

Mode Supported What Drift does
async-observe Yes Subscribes to the topic and captures the message your service publishes
async-inject Yes Publishes to the topic, then runs your probe
async-inject-capture Yes Subscribes to a probe topic, publishes, then captures the harness result
async-request-reply Yes Subscribes to the reply topic, publishes the request, captures the correlated reply

See AsyncAPI Execution Modes.

Connection Details

Drift reads the broker address from your AsyncAPI servers block:

servers:
  local:
    host: localhost:9092
    protocol: kafka
    description: Local Kafka broker

Override it per environment through plugin configuration, or per test through the test case. Test case values win, then plugin configuration, then the document.

Channel Addresses

The AsyncAPI channel address is the Kafka topic name:

channels:
  orderCreated:
    address: drift.examples.order-created

Correlation

Drift matches captured messages against a correlation identifier carried in a Kafka record header. Declare its location in your message:

components:
  messages:
    orderCreated:
      correlationId:
        location: "$message.header#/correlation-id"

Declaring the location is required. The value is optional: Drift takes it from the AsyncAPI document, then from the correlation-id parameter in your test case, and generates a UUID when neither supplies one. Drift filters captured records to the one whose header matches, and ignores everything else on the topic.

Set the value yourself when a trigger hook or an expectation needs the same identifier. Where you do set it, give each operation a distinct value - a shared one lets a message from one test satisfy another.

Capture Behavior

When Drift subscribes, it records the current end of the topic log and reads forward from there. Messages published before the subscription are not captured, which is what makes the subscribe-then-trigger ordering reliable.

Drift polls for new records until a matching message arrives or the capture window closes.

Setting Where Default Purpose
timeout-ms Test case parameter 30000 How long Drift waits for a matching message
poll-interval-ms Plugin configuration 500 How often Drift polls the broker between fetches
parameters:
  timeout-ms: 5000

Lower poll-interval-ms reduces capture latency at the cost of more requests to the broker. The default suits local development and CI.

Creating Topics

Drift does not create topics. Create every topic in your AsyncAPI document, and any probe topic, before the run.

Kafka’s automatic topic creation fires only when a producer or consumer first touches a topic. That is too late for async-inject-capture, where Drift publishes to the input topic before any consumer has subscribed.

docker exec <kafka-container> kafka-topics \
  --bootstrap-server localhost:9092 \
  --create --if-not-exists \
  --topic drift.examples.order-created \
  --partitions 1 --replication-factor 1

Running a Local Broker

A local broker is the recommended setup for development and CI. It is fast, repeatable, and nothing you publish affects anyone else.

services:
  kafka:
    image: confluentinc/cp-kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ENABLE_KRAFT: "yes"
      KAFKA_NODE_ID: "1"
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_LISTENERS: "PLAINTEXT://:9092,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1"
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1"
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0"

Running against a shared cluster is supported, but that cluster is yours to manage. Messages Drift publishes are real messages, and downstream consumers will process them.

Transport Security

The Kafka plugin speaks the Kafka wire protocol over raw TCP sockets. It does not support authentication or encryption in transit:

  • No SASL authentication of any mechanism

  • No TLS, and no mutual TLS

  • Every connection is plaintext

Point Drift at a broker that accepts plaintext connections, and treat contract verification as something you run against a local, disposable, or otherwise non-sensitive broker rather than a secured production cluster.

Limitations

  • Drift captures the first record matching the correlation ID. It does not verify ordering across a stream.

  • Drift does not create topics.

  • Drift does not verify consumer group behavior, partition assignment, rebalancing, or offset management.

  • Drift does not verify delivery guarantees, retry behavior, or dead-letter handling.

  • Authentication and TLS are not supported. Connections are plaintext.

Troubleshooting

Symptom Likely cause
Connection refused The broker is not running, or host in the servers block is wrong
Connection fails against a secured broker The plugin connects over plaintext TCP only. It cannot reach a broker that requires SASL or TLS.
Capture timeout, trigger succeeded The published record carries a different correlation ID, or went to a different topic
Capture timeout on async-inject-capture The probe topic did not exist before the run
Topic not found Create the topic in advance. Automatic creation is too late for Drift.
Works locally, times out in CI The broker service container was not ready. Add a readiness check before the Drift step.

See Debugging Test Cases.

See Also

Was this page helpful?