Unable scale up cassandra using latest datastax python driver

Question

We are using multiple cassandra datastax cluster instances(6) to connect to cassandra using python. We are pooling these multiple connections to do some operations. Each operation is independent of other.

It works fine on a small number of operations, but once I try to scale up I get the following errors :

NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: 127.ption('Pool is shutdown',)})

and sometimes the following warning:

WARNING  Heartbeat failed for connection (140414695068880) to 127.0.0.1

I tried changing some cluster object parameters but it did not help.

Following is the configuration of key space in cassandra I am using :

'class': 'SimpleStrategy', 'replication_factor': '1'

I am using lastest versions of cassandra and datastax driver for python. There is only one node is cassandra.

EDIT: More details:

The multiple cluster instances are in different processes (processes are created using the Python multiprocessing module) - one cluster instance per process. Lets call the proccesses Cassandra-Processes (CP). There are a bunch of other process that do some computation and need to look up a Cassandra DB, and write to it, occassionally. The current design is that each of these processes is mapped to one CP, and all DB reads/writes to be done by the process is done via this mapped CP. 'what' exactly is to be read/written is passed into a queue (again from the multiprocessing library) which the mapped CP reads.

We observe that this setup runs for quite sometime - and then suddenly Cassandra begins erroring out.

Answer 1

It's unclear why you're using six cluster instances against a single Cassandra node. Generally, you should use one Cluster instance per application (per remote cluster). You can read about general design considerations for Cassandra drivers here

If you're looking to "scale" with regards to throughput, you might consider using multiprocessing. I discuss this in a blog post here.

---

Follow-on:

Two things can be inferred from the information we have so far:

1. The application is pushing more concurrent requests than your connection pool is configured to handle. I say this because the "Pool is shutdown" only occurs when a request is waiting for a connection/stream to become available. You can tune connection pooling to make more available initially using cluster settings. However, if your "cluster" (server node) is overwhelmed, you won't gain much there.
2. Your connection is being shutdown. This exception only happens when the node is suddenly marked down. In a single node setup this is most likely because of a connection error. Look for clues in the server log, or driver debug log if you're capturing that.

We probably need to know more about your execution model to help more. Is it possible you're running unfettered async requests without occasionally waiting for them to complete?

Answer 2

Remote diagnosis is hard to do without knowing anything on your specific topology, setup and system configuration. This however looks much like a configuration problem or even the python driver. If you google your error message you will find multiple topics on Datastax's Jira describing this or similar problems, I would check that the Python Driver is up to date. What would help in the first place is to see in detail what you try to do, how your cluster is configured aso.