ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved.
Tuning Kafka for Fun and Profit
ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 2
Zookeeper
5-node vs. 3-node Ensembles
Solid State Disks– Use good SSDs– Transaction logs only– Significant improvement in latency and outstanding requests
ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 3
Kafka Broker Disks
Disk Layout
JBOD vs. RAID– JBOD and RAID-0 are similar– RAID-5/6 has significant performance overhead– RAID-10 still offers the best performance and protection
Filesystem– New testing shows XFS has a clear benefit– No tuning required– Will be continuing testing with more production traffic
ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 4
Scaling Kafka Clusters
Disk Capacity
Network Capacity
Partition Counts– Per-Cluster– Per-Broker
Limitations– Topic list length
ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 5
Topic Configuration
Retention Settings
Partition Counts– Balance over consumers– Balance over brokers– Partition size on disk– Application-specific requirements
ORGANIZATION NAME©2013 LinkedIn Corporation. All Rights Reserved. 6
Mirror Maker
Network Locality
Consumer Tuning– Number of streams– Partition assignment strategy
Producer Tuning– Number of streams– In flight requests– Linger time