DevOps Articles

Kafka Connect on Kubernetes The Easy Way!

This is a tutorial that shows how to set up and use Kafka Connect on Kubernetes using Strimzi, with the help of an example.

Although it's not too hard to deploy a Kafka Connect cluster on Kubernetes (just "DIY"!), I love the fact that Strimzi enables a Kubernetes-native way of doing this using the Operator pattern with the help of Custom Resource Definitions.

Before you proceed, clone the GitHub projectKafka Connect will need to reference an existing Kafka cluster (which in this case is Azure Event Hubs).

In this case its a single Kafka endpoint for Azure Event Hubs (yes, that's all you need!) config is just good old Kafka Connect configuration similar to what you would use in connect-distributed.propertiesThe authentication section simply refers to a Kubernetes Secret.

Here the definition of our connector: Just like we did before, let's understand what each component means: This is a KafkaConnector resource (specified by kind) whose name is my-source-connectorThis is where we refer to the Kafka Connect cluster - remember this annotation in the Kafka Connect definition shown above?