Source: dzone.com
Processing Large Messages with Apache KafkaCategory: Software, Database, Data, Infrastructure, Architecture, machine-learning, artificial-intelligence
Nevertheless, more and more projects send and process 1Mb, 10Mb, and even much bigger files and other large payloads via Kafka. One reason is that Kafka was designed for large volume/throughput - which is required for large messages.
It is valid to evaluate Kafka for processing large messages instead of using another tool for that (often in conjunction with Kafka).
The following three available alternatives exist to handle large messages with Kafka: Here are the characteristics and pros/cons of each approach (this is an extension from a LinkedIn presentation in 2016): Also, don't underestimate the power of compression for large messages.
As you learned in this post, plenty of use cases exist for handling large messages files with Apache Kafka and its ecosystem.
It is valid to evaluate Kafka for processing large messages instead of using another tool for that (often in conjunction with Kafka).
The following three available alternatives exist to handle large messages with Kafka: Here are the characteristics and pros/cons of each approach (this is an extension from a LinkedIn presentation in 2016): Also, don't underestimate the power of compression for large messages.
As you learned in this post, plenty of use cases exist for handling large messages files with Apache Kafka and its ecosystem.
Related Articles
Community Partners
DevOps Careers