Even Data distribution in Cassandra

1
votes

I'm new to Cassandra, and I'm stuck at one point.

Consider I have a 5 node cluster with an RF=1 (for simplicity)

Token Ranges 
==============
N1 : 1-100
N2 : 101-200
N3 : 201-300
N4 : 301-400
N5 : 401-500

I have a keyspace with 10 partition keys:

ID (PartitionKey) | Name
------------------------
1                 Joe
2                 Sarah
3                 Eric
4                 Lisa
5                 Kate
6                 Agnus
7                 Lily
8                 Angela
9                 Rodger
10                Chris

10 partition keys ==> implies ==> 10 hash values

partitionkey ==> token generated
=================================
1                 289 (goes on N3)
2                 56 (goes on N1)
3                 78 (goes on N1)
4                 499 (goes on N5)
5                 376 (goes on N4)
6                 276 (goes on N3)
7                 2 (goes on N1)
8                 34 (goes on N1)
9                 190 (goes on N2)
10                68 (goes on N1)

If this is the case, then:

N1 has the partition keys : 2,3,7,8,10
N2 has the partition keys : 9
N3 has the partition keys : 1,6
N4 has the partition keys : 5
N5 has the partition keys : 4

So we see that N1 is loaded compared to others, the other nodes (as per my understanding).

Please help me understand how data is evenly distributed in Cassandra, w.r.t Partitioners and consistent hashing.

3
cassandra

3 Answers

1
votes

Selecting the partition key is very important in having even distribution of data among all the nodes. The partition key is supposed to be something that has very high cardinality.

For example, in a 10 node cluster, selecting state of a specific country as partition key may not be very ideal since there’s very high chance of creating hotspots, especially when the number of records itself may not be even across states. Whereas choosing something like zip code may be better or even better than that would be something like customer name or ordernumber. You can explore having a composite partition key if it helps your use case.

1
votes

There is some truth to what you're posting here, mainly because data distribution via hashing is tough with smaller numbers. But let's add one assumption... Let's say we use vNodes, with num_tokens: 4* set in the cassandra.yaml.

So with this new assumption, token range distribution likely looks more like this:

  Token Ranges
  ==============
  N1 :    1-25, 126-150, 251-275, 376-400
  N2 :   26-50, 151-175, 276-300, 401-425
  N3 :   51-75, 176-200, 301-325, 426-450
  N4 :  76-100, 201-225, 326-350, 451-475
  N5 : 101-125, 226-250, 351-375, 476-500

Given this distribution, your keys are now placed like this:

  N1 has the partition keys : 5, 7
  N2 has the partition keys : 1, 6, 8
  N3 has the partition keys : 2, 9, 10
  N4 has the partition keys : 3
  N5 has the partition keys : 4

Now figure-in that there is a random component to the range allocation algorithm, and the actual distribution could be even better.

As with all data sets, the numbers get better as the amount of data increases. I'm sure that you'd see better distribution with 1000 partition keys vs. 10.

Also, as the size of your data set increases, data distribution will benefit from new nodes being added with setting allocate_tokens_per_keyspace. This will allow the token allocation algorithm to make smart decisions (less random) about token range assignment based on your keyspace's replication factor.

*Note: Using vNodes with num_tokens: 4 is considered by many Cassandra experts to be an optimal production setting. With the new algorithm, the default of 256 tokens is quite high.

0
votes

In Cassandra data is distributing based on partition and hashing algorithm. We have many other parameters to configure for data distribution and replication such as replication factor, Replication strategy, Snitch etc. Below is the standard recommended document. https://docs.datastax.com/en/cassandra-oss/2.2/cassandra/architecture/archDataDistributeAbout.html