Kafka Gen 2

4 min

The Kafka Gen 2 integration enables Belden Horizon Data Operations to connect with the Kafka broker via TCP or SSL.

Supported communication

Outbound (Publish data from Edge to Kafka broker) - Yes

Inbound (Subscribe data from Kafka broker to Edge) - Yes

Configurations

Field	Details	Required
Name	Connector name.	Yes
Connection Type	Choose tcp for an unencrypted connection or ssl for a TLS-encrypted connection to the Kafka broker.	Yes
Brokers list separated by comma	Comma-separated list of Kafka broker addresses in host:port format. Example: broker1:9092,broker2:9092.	Yes
SASL mechanism	Authentication mechanism: NONE for no auth, SASL_PLAINTEXT for plain credentials over an unencrypted connection, SCRAM-SHA-256 or SCRAM-SHA-512 for secure salted-hash authentication.	Yes
Username	Username for SASL authentication.	No
Password	Password for SASL authentication.	No
CA Certificate	CA certificate used to verify the server, in PEM format. Can also be installed in the system-wide certificate store.	No
Certificate	Client certificate for mutual TLS authentication with the Kafka broker. Paste the PEM-encoded certificate here.	No
Private key	Private key corresponding to the client certificate. Must match the Certificate field above.	No
Insecure Skip Verify	Insecure Skip Verify controls whether a client verifies the server's certificate chain and host name.	No
Topic	Kafka topic to publish messages to. Topic must exist on the broker unless Auto Create Topic is enabled.	Yes
Auto create topic	Enable automatic topic creation when the topic is not found. This also requires enabling the auto create topic setting on the Kafka broker.	No
Client ID	Client identifier sent to Kafka brokers.	No
Throttling limit	Set a limit on the number of outbound messages per second. If sending data to a server that has restrictions on the rate of incoming messages, this ensures that no messages are lost.	No

Advanced configurations

Field	Details	Required
Session Timeout (ms)	Consumer group session timeout.	No
Heartbeat Interval (ms)	Consumer group heartbeat interval.	No
Rebalance Timeout (ms)	Consumer group rebalance timeout.	No
Dial Timeout (ms)	Connection establishment timeout.	No
Request Timeout (ms)	Maximum time to wait for requests to complete.	No
Connection Idle Timeout (ms)	Maximum idle time before closing connections.	No
Metadata Max Age (ms)	Maximum age of metadata before refresh.	No
Metadata Min Refresh (ms)	Minimum time between metadata refreshes.	No
Required Acknowledgments	Level of acknowledgment reliability. Note: When idempotent write is enabled, this must be set to 'all'.	No
Disable Idempotent Write	Disable idempotent producer writes. When disabled, duplicate messages may be produced during retries. Note: When idempotent write is enabled (unchecked), Required Acknowledgments must be set to 'all'.	No
Producer Timeout (ms)	Maximum time to wait for producer response.	No
Producer Batch Max Bytes	Maximum size of producer batches in bytes.	No
Producer Linger (ms)	Time to wait for additional messages before sending batch.	No
Compression Type	Compression algorithm for message payloads.	No
Max Message Bytes	Maximum message size in bytes.	No
Fetch Min Bytes	Minimum bytes to fetch in each request.	No
Fetch Max Bytes	Maximum bytes to fetch in each request.	No
Fetch Max Wait (ms)	Maximum time to wait for fetch requests.	No
Max Partition Bytes	Maximum bytes to fetch per partition.	No
Retry Backoff (ms)	Backoff time between retry attempts.	No
Produce Retries	Maximum number of retry attempts for failed requests.	No
Record Delivery Timeout (s)	Maximum time a record may be retried before it fails terminally. Bounds retries so undeliverable fire-and-forget messages are eventually reported as dropped instead of retried forever.	No
Write Timeout (ms)	Write operation timeout.	No
Read Timeout (ms)	Read operation timeout.	No
Parallel Publish Count	Number of messages published in parallel. Higher values increase throughput and resource usage.	Yes
Verbose connection info	Log detailed connection events (connect, disconnect, reconnect) to assist with troubleshooting.	No

Persistent storage configurations

Field	Details	Required
Persistent storage	If enabled, messages in the outbound queue are stored on disk. If there are interruptions in connection or power loss, messages are retained and eventually sent when the connection is restored.	No
Persistent storage size limit	Set the size limit for storing persisted data in GB.	No
Queue mode	Select the sequence of data transfer: FIFO or LIFO. Selecting FIFO (First In First Out) sends the earliest message first. Selecting LIFO (Last In First Out) sends the latest message first.	No