Cassandra very slow doing sliceRange read, TimedOutException

Question

Cassandra 1.0.12

We have a column family with name transactionRecords, its rowkey is "ApplicationName". The column family is used to record transactions, with namevalue pair like (UUID, transInfo).

At the heavy load about 1.5K TPS, after 1 hour, the SliceRange query is extremely slow. For the sliceRage, we only set limit, such as 10000. The start and end is empty. We want to use the sliceRange to read out transaction information, and do some business logic, and then delete from Cassandra.

We use org.scale7.cassandra.pelops.Mutator API to insert into DB: Mutator.writeColumn(columnFamily, rowkey, Mutator.newColumn(uuid, string));

Using nodetool, the latency is more than 10 seconds at the time of SliceRange Reading.

But now, it TimedOutException when read.

Does someone have similar problem? How to reduce the latency? Does our data structure well for our purpose? Any help or suggestion?

Answer 1

I think it is my programming error.

When I did the range scan, I always started from the beginning (I have 5G MEMTable), and with more and more entries in DB, the scan speed drops, finally it was larger than 30 seconds, and caused TimedOutException.

By changing to scanning from previous stopped location (on the sorted key), it worked well to me now.