Cassandra experiencing OutOfMemory issues (Java Heap Space) on long runs

2
votes

We are experimenting a bit with Cassandra by trying out some of the long running test cases (stress test) and we are experiencing some memory issues on one node of the cluster at any given time (It could be any machine on the cluster!)

We am running DataStax Community with Cassandra 1.1.6 on a machine with Windows Server 2008 and 8 GB of RAM. Also, we have configured the Heap size to be 2GB as against the default value of 1GB.

A snippet from the logs:

java.lang.OutOfMemoryError: Java heap space

Dumping heap to java_pid2440.hprof ...

Heap dump file created [1117876234 bytes in 11.713 secs]

ERROR 22:16:56,756 Exception in thread Thread[CompactionExecutor:399,1,main]

java.lang.OutOfMemoryError: Java heap space

at org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)

at org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:220)

at java.io.DataOutputStream.write(Unknown Source)

Any pointers/help to investigate and fix this.??

1
cassandradatastax-enterprise
could you give more details of what you are inserting and where you are insterting it (key, column names, etc...)? what settings have you used for declaring your cluster and keyspaces? any details might be relevant. – le-doude
I am creating the keyspace with the following command:CREATE KEYSPACE T_V_0 with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options = {replication_factor:1}; – vinay sudhakar
The columns basically have a DateType, UUIDType, 3 IntegerType, 2 LongType and 1 DoubleType with WITH comparator = UTF8Type AND key_validation_class = UTF8Type AND default_validation_class = UTF8Type – vinay sudhakar
Are you inserting always at the same address (key + column name)? Don't you get write errors on some writes since UTF8Type is not compatible with other validation (you should probably leave as bytes ... which is a default that allows you to have multiple types of columns in one row). – le-doude
Nope, I do not insert at the same address. The column names are UTF8Type and the unique key that I am using is a combination of UUID and TimeStamp, so there is never a overlap. I am able to insert data sucessfully. Only when I insert over 1 Billion records continuously, one of the nodes in the cluster fails with OutOfMemory exception. This could be any node but I believe its mostly the node that accepts maximum connection from the client. – vinay sudhakar

1 Answers

3
votes

You're doing the right thing by long-running your load tests but in a production use case you wouldn't be writing data like this.

Your rows are probably growing too big to fit in RAM when it comes time to compact them. A compaction requires the entire row to fit in RAM.

There's also a hard limit of 2 billion columns per row but in reality you shouldn't ever let rows grow that wide. Bucket them by adding a day or server name or some other value common across your dataset to your row keys.

For a "write-often read-almost-never" workload you can have very wide rows but you shouldn't come close to the 2 billion column mark. Keep it in millions with bucketing.

For a write/read mixed workload where you're reading entire rows frequently even hundreds of columns may be too much.

If you treat Cassandra right you'll easily handle thousands of reads and writes per second per node. I'm seeing about 2.5k reads and writes concurrently per node on my main cluster.