Kafka REST

Kafka speaks a binary protocol, but Confluent Cloud, Redpanda and the Kafka REST Proxy expose an HTTPS v3 API. Hooks uses the generic HTTPS destination for that API: a bearer or basic credential, a record endpoint URL, and a body template that wraps the event as a Kafka record. No separate destination adapter or broker connection is needed.

Create the records endpoint

The REST v3 record endpoint has three path parts:

https://<rest-endpoint>/kafka/v3/clusters/<cluster_id>/topics/<topic>/records

• Confluent Cloud: copy the cluster REST endpoint and cluster id from the cluster overview.
• Redpanda: enable the HTTP proxy, take its address (for example https://<cluster>.redpanda.com, or http://localhost:8082 locally), and read the cluster id from GET /kafka/v3/clusters.
• Self-hosted REST proxy: use the proxy address and its cluster id.

The credential is an API key and secret sent with basic auth. Create the key with write access to the topic only.

Apply a template

Console → Quick start offers two templates:

Template	Endpoint placeholder
Any event → Confluent Cloud (Kafka REST)	https://pkc-….confluent.cloud
Any event → Redpanda (Kafka REST)	https://<cluster>.redpanda.com

Both ask for the REST endpoint, cluster id, topic, API key, API secret and a webhook signing secret. The key and secret are stored as write-only project secrets.

The template sets:

• POST to /kafka/v3/clusters/<cluster_id>/topics/<topic>/records.
• Authorization: Basic … derived from the key and secret.
• This JSON body:

{
  "key": { "type": "STRING", "data": "{{event.id}}" },
  "value": { "type": "JSON", "data": {{event.payload}} }
}

data under value receives the raw inbound payload, so consumers read the original webhook body. The delivery id is the record key, which makes ordering and deduplication predictable per event.

Configure it manually

curl -X POST "https://api.hooks.events/v1/endpoints/ep_7a3k2/destinations" \
  -H "Authorization: Bearer $HOOKS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://pkc-xxxx.us-east-1.aws.confluent.cloud/kafka/v3/clusters/lkc-abc123/topics/hooks-events/records",
    "method": "POST",
    "bodyMode": "json",
    "bodyTemplate": "{\"key\":{\"type\":\"STRING\",\"data\":\"{{event.id}}\"},\"value\":{\"type\":\"JSON\",\"data\":{{event.payload}}}}",
    "auth": { "mode": "basic", "username": "{{secret.KAFKA_API_KEY}}", "password": "{{secret.KAFKA_API_SECRET}}" }
  }'

Create the two secrets first. The payload is inserted verbatim, so a non-JSON source (form-encoded GitHub deliveries, for example) needs a body template that re-encodes it as a string, or a transform that produces JSON.

Partitions and message keys

REST v3 records accept an optional partition_id. Add a destination variable or a transform when events must land on a specific partition:

{
  "key": { "type": "STRING", "data": "{{event.subject.id}}" },
  "partition_id": 3,
  "value": { "type": "JSON", "data": {{event.payload}} }
}

Change the key template when downstream consumers group by a business id instead of the delivery id. Reusing an existing id keeps Kafka’s per-key ordering aligned with your own entity.

Verify a delivery

Deliveries → newest attempt shows the REST response. When the topic, cluster id, or credentials are wrong, REST v3 returns a 4xx JSON body with error_code and message; that body is stored in the attempt record and shown in the console. 429 and 5xx retry with the standard retry schedule; other 4xx responses fail immediately.

To receive messages from Kafka instead, push them to a Hooks receive URL with Confluent’s HTTP Sink connector or a small forwarder that posts to /in; see Kafka → Hooks.