7 months ago

https://aws.amazon.com/polly/ I’m excited to announce today a new capability of https://aws.amazon.com/msk/ (Amazon MSK) that allows you to continuously load data from an https://kafka.apache.org/ cluster to https://aws.amazon.com/s3/. We use https://aws.amazon.com/kinesis/data-firehose/—an extract, transform, and load (ETL) service—to read data from a Kafka topic, transform the records, and write them to an Amazon S3 destination.

Today, we announce the availability of a fully managed solution to deliver data from Amazon MSK to Amazon S3 using https://aws.amazon.com/kinesis/data-firehose/.

The Data Firehose delivery stream reads data from your MSK cluster, buffers the data for a configurable threshold size and time, and then writes the buffered data to Amazon S3 as a single file. MSK and Data Firehose must be in the same AWS Region, but Data Firehose can deliver data to Amazon S3 buckets in other Regions.