Kafka Cluster Sizing

Bhupendra patni
Jul 12, 2015
2 min read

The post is provide high level guideline on Kafka cluster sizing for the given use case.

The most accurate way to perform Kafka cluster sizing is to simulate the load you expect on your own hardware, and you can accomplish this by using the tools that comes along with Kafka.

However you can still perform high level sizing without true simulation, for which you need to look at multiple variables that go into determining the correct hardware size. The very basic estimation can be performed by looking at the total disk space required to store the data for the required retention period.

An advanced estimation can be performed based on disk & network throughput requirements to support number of messages per second, message size, replication factor, number of consumer.

Let us understand this in detail:

N - Number of messages to be processed per second

S - Maximum message size

F - Replication Factor

C - Number of Consumer

Writes

The write throughput is calculated with the total size of messages published by all producers. For write throughput 25,000 messages/sec with the max message size of 4KB, will require write throughput of 100MB/sec, 6 GB/minute, 360 GB/hour or 3.6 TB/hour, so at the very minimal you need to have ~1.2TB of total available disk space required to retain data for an hour with the replication factor of 3.

Write Throughput: WT = N * S * F

Reads

All the consumers and replicas other than the master will be reading each message written in the queue. The read volume for the replicas & consumers will be calculated as WT * (F-1) and WT * C simultaneously. The total read volume would be:

Read Throughput: RT = WT * (F + C - 1)

However, the actual disk I/O will not be required as the message for reads will actually be cached. The requirements for the reads can easily be modeled with the available GB of memory on the server.

Server Configuration

The key server configuration to consider for the server(s) in the cluster are the total storage, RAM and network card.

The server with 1 Gigabit ethernet card with full duplex will give us 125MB/sec read and 125MB/sec write.
One 7200 SATA drive might give us upto 50MB/sec read-write throughput so if you have 8 drives then expect upto 400MB/sec read-write throughput.

Once we know the read-write requirements, then we can divide it across number of servers we need in the cluster.