What are the correct Cassandra collection limits?

3
votes

I'm testing an application where the size of the collections is bound to grow in the future and 64k is a limit that may be reached in some cases.

This question is about the collection size limit as there seems to be a contradiction in the official documentation.

As per this document:

If you insert more than 64K items into a collection, only 64K of them will be queryable, resulting in data loss.

But if you click through to the CQL Limits link on that very page you see this:

  • Collection (List): collection size: 2B (2^31); values size: 65535 (2^16-1) (Cassandra 2.1 and later, using native protocol v3)

  • Collection (Set): collection size: 2B (2^31); values size: 65535 (2^16-1) (Cassandra 2.1 and later, using native protocol v3)

  • Collection (Map): collection size: 2B (2^31); number of keys: 65535 (2^16-1); values size: 65535 (2^16-1) (Cassandra 2.1 and later, using native protocol v3)

So which one is it? 64k items per collection, or 2 billion items per collection? Or are 2 billion writeable but not readable beyond 64k?

Thanks in advance.

2
cassandra
Although this is slightly off topic do be aware that updating collections you need to be careful of how you implement it to avoid excessive tombstones, see this tech note: support.datastax.com/hc/en-us/articles/205368125 - markc
Thanks for the observation @markc - I only use updates on all my set/list/map related queries. It just seems more natural. - Jose Fonseca

2 Answers

4
votes

which version of cassandra you are using ?

that documentaion is for 2.0 and 2.1 . and in that case there is a limitation of how many elements you can put in a collection. which is 64k. however each element can have a size of 2b if you are using native protocol v3. check this https://issues.apache.org/jira/browse/CASSANDRA-5428

but if you are using cassandra 2.2 and later you can insert 2billion items into collection. here is the link. http://docs.datastax.com/en/cql/3.3/cql/cql_using/useCollections.html

having said that you should not insert that many items into the collection. you will hit performance issues way before you hit the max elements insertion limit.

Collections cannot be "sliced"; Cassandra reads a collection in its entirety, impacting performance. Thus, collections should be much smaller than the maximum limits listed. The collection is not paged internally.

If you have to have that much of item then in that case collections are not appropriate anymore and a specific table (with clustering columns) should be used.

I hope this helps.

2
votes

You can't add more than 64k itens in cassandra collection if you are using cassandra 2.1. In this version, cassandra uses 16 bits to index elements. Otherwise, a single item can be 64k (in size) or 2B (depending on the protocol).