Google Cloud Pub/Sub to Datastax Cassandra( In Google Cloud)

Question

I have deployed a 9-node Datastax Cluster on Google Cloud.

Now I've been given a requirement to read the data from QUEUE's and load into Datastax on Google Cloud( Continues Streaming).

I know Pub/Sub can read the data from QUEUE but not sure whether it can directly load into Datastax.

Is it possible for Pub/Sub to load the data into Datastax on Google Cloud ?

I am new to Pub/Sub so not sure where to start or how to start.

Thanks,

Answer 1

There is no automatic way to load the data from Google Cloud Pub/Sub into Datastax. You would have to create a subscription and write the subscriber code that gets the messages from this subscription and then writes them into Datastax. The quickstart guide is probably a good place to start with the steps necessary.

Answer 2

One way would be to have a Dataflow pipeline (running in streaming mode) read from Pub/Sub and write to Cassandra. There is no built-in Cassandra sink yet for Dataflow (though one is in the works as part of Apache Beam) but it might be sufficient for your use case to write to Cassandra in a Dataflow ParDo.