Cassandra Loading Options

Question

I have deployed a 9 node DataStax Cluster in Google Cloud. I am new to Cassandra and not sure how generally people push the data to Cassandra.

My requirement is read the data from flatfiles and RDBMs table and load into Cassandra which is deployed in Google Cloud.

These are the options I see.

   1. Use Spark and Kafka
   2. SStables
   3. Copy Command
   4. Java Batch
   5. Data Flow ( Google product )

Is there any other options and which one is best.

Thanks,

Answer 1

For flat files you have 2 most effective options:

1. Use Spark - it will load data in parallel, but requires some coding.
2. Use DSBulk for batch loading of data from command line. It supports loading from CSV and JSON, and very effective. DataStax's Academy blog just started a series of the blog posts on DSBulk, and first post will provide you enough information to start with it. Also, if you have big files, consider to split them into smaller ones, as it will allow DSBulk to perform parallel load using all available threads.

For loading data from RDBMS, it depends on what you want to do - load data once, or need to update data as they change in the DB. For first option you can use Spark with JDBC source (but it has some limitations too), and then saving data into DSE. For 2nd, you may need to use something like Debezium, that supports streaming of change data from some databases into Kafka. And then from Kafka you can use DataStax Kafka Connector for submitting data into DSE.

CQLSH's COPY command isn't as effective/flexible as DSBulk, so I won't recommend to use it.

And never use CQL Batch for data loading, until you know how it works - it's very different from RDBMS world, and if it's used incorrectly it will really make loading less effective than executing separate statements asynchronously. (DSBulk uses batches under the hood, but it's different story).