Source: thenewstack.io
LinkedIn Layered Architecture Minimizes Kafka Scaling Issues
Fan-in and fan-out are both problems that occur when producers and consumers all fight for resources from the same https://thenewstack.io/how-kafka-and-redis-solve-stream-processing-challenges/. LinkedIn’s architecture separate producer and consumer groups for purposes of geo-replication.
Fan-out is a throughput multiplier (IO ∝ Fin + Fin × Fout).1 MBps per producer is moving through 100 producers and 10 consumers.
LinkedIn’s topology solves the issue of fan-in and fan-out because it separates the producers from consumers.
Just by putting a pipeline between the two layers of clusters and separating the producers from consumers, LinkedIn avoids any fan-in, fan-out issues without actually targeting that problem specifically.
Fan-out is a throughput multiplier (IO ∝ Fin + Fin × Fout).1 MBps per producer is moving through 100 producers and 10 consumers.
LinkedIn’s topology solves the issue of fan-in and fan-out because it separates the producers from consumers.
Just by putting a pipeline between the two layers of clusters and separating the producers from consumers, LinkedIn avoids any fan-in, fan-out issues without actually targeting that problem specifically.
Related Articles
Community Partners
DevOps Careers