Apache Cassandra crashes under heavy read load

2
votes

I've been working on an application which requires regular writes and massive reads at once.

The application is storing a few text columns which are not very big in size and a map of which is the biggest column in the table.

Working with Phantom-DSL in Scala (Datastax Java driver underneath), my application crashes when the data size increases.

Here is a log from my application.

[error] - com.websudos.phantom - All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.OperationTimedOutException: [/127.0.0.1:9042] Operation timed out))

And here are the cassandra logs.

I have posted Cassandra logs in a pastebin because they were pretty large to be embedded in the answers.

I can't seem to understand the reason for this crash. I have tried increasing the timeout and turning off row cache.

From what I understand, this is a basic problem and can be resolved by tuning cassandra for this special case.

My cassandra usage is coming from different data sources. So writes are not very frequent. But reads are big in size in that over 300K rows may be required at once which then need to be transferred over HTTP.

2
javascalacassandraphantom-dsl
Do you have paging enabled on the java driver? If not, enable it. If so, try lowering the page size (I think the default is 5k rows). Your problem is that the JVM garbage collector can't keep up, potentially because you're pulling too much data onto the heap at once. - Tyler Hobbs

2 Answers

1
votes

The logs show significant GC pressure (ParNew of 5 seconds).

When you say "reads are big in size in that over 300K rows may be required at once", do you mean you're pulling 300k rows in a single query? The Datastax driver supports native paging - set the fetch size significantly lower (500 or 1000), and allow it to page through those queries instead of trying to load all 300k rows in a single pass?

0
votes

Maps (and collections in general) can be very demanding for Cassandra heapspace. Changing your data model to replace the map with another table may solve your gc issues. But that's a lot of speculation due to the lack of further details on your Cassandra usage.