DevOps Articles

Introducing Amazon MSK Connect – Stream Data to and from Your Apache Kafka

At re:Invent 2018, we announced Amazon Managed Streaming for Apache Kafka, a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data. When you use Apache Kafka, you capture real-time data from sources such as IoT devices, database change events, and website clickstreams, and deliver it to destinations such as databases and persistent storage.

To test MSK Connect, I want to use it to stream data change events from one of my databases.

Because MSK Connect accepts custom plugins in ZIP or JAR format, I convert the downloaded archive to ZIP format and keep the JARs files in the main directory: Then, I use the AWS Command Line Interface (CLI) to upload the custom plugin to an Amazon Simple Storage Service (Amazon S3) bucket in the same AWS Region I am using for MSK Connect: On the Amazon MSK console there is a new MSK Connect section.

In the third terminal connection, I install a MySQL client using the MariaDB package and connect to the Aurora database: From this connection, I create the ecommerce database and a table for my orders: These database changes are captured by the Debezium connector managed by MSK Connect and are streamed to the MSK cluster.