Severe degradation in Cassandra Write performance with continuous streaming data over time

1
votes

I notice a severe degradation in Cassandra write performance with continuous writes over time.

I am inserting time series data with time stamp (T) as the column name in a wide column that stores 24 hours worth of data in a single row. Streaming data is written from data generator (4 instances, each with 256 threads) inserting data into multiple rows in parallel. Additionally, data is also inserted into a column family that has indexes over DateType and UUIDType.

CF1:

        Col1    |   Col2    |   Col3(DateType)  |   Col(UUIDType4)  |

RowKey1

RowKey2

:

:

CF2 (Wide column family):

RowKey1 (T1, V1) (T2, V3) (T4, V4) ......

RowKey2 (T1, V1) (T3, V3) .....

:

:

The no. of data points inserted/sec decreases over time until no further inserts are possible. The initial performance is of the order of 60000 ops/sec for ~6-8 hours and then it gradually tapers down to 0 ops/sec. Restarting the DataStax_Cassandra_Community_Server on all nodes helps restore the original throughput, but the behaviour is observed again after a few hours.

OS: Windows Server 2008 No.of nodes: 5 Cassandra version: DataStax Community 1.2.3 RAM: 8GB HeapSize: 3GB Garbage collector: default settings [ParNewGC]

I also notice a phenomenal increase in the no. of Pending write requests as reported by the OpsCenter (~of magnitude 200,000) when the performance begins to degrade.

I fail to understand what is preventing the write operations to be completed and why do they pile up over time? I do not see anything suspicious in the Cassandra logs.

Has the OS settings got anything to do with this? Any suggestions to probe this issue further?

2
cassandradatastax-enterprise
I would upgrade to the latest 1.2.X release, 1.2.12, and then try your test again. There have been a lot of bug fixes since 1.2.3 - Zanson

2 Answers

2
votes

Bringing your write timeout in line with the new default in 2.0 (of 2s instead of 10s) will help with your write backlog by allowing load shedding to kick in faster: https://issues.apache.org/jira/browse/CASSANDRA-6059

3
votes

Do you see an increase in pending compactions (nodetool compactionstats)? Or are you seeing blocked flush writers (nodetool tpstats)? I'm guessing you're writing data to Cassandra faster than it can be consumed.

Cassandra won't block on writes, but that doesn't mean that you won't see an increase in the amount of heap used. Pending writes have overhead, as do blocked memtables. In addition, each SSTable has some memory overhead. If compactions fall behind this is magnified. At some point you probably don't have enough headroom in your heap to allocate the objects required for a single write, and you end up spending all your time waiting for an allocation that the GC can't provide.

With increased total capacity, or more IO on the machines consuming the data you would be able to sustain this write rate, but everything indicates you don't have enough capacity to sustain that load over time.