0
votes

I have an RDD(test_rdd) as below

[
{'user_lname': u'TEst1', 'user_id': u'2aa8ae30-c0e5-48bb-ab16-a2ed2e78c8c3', 'user_phone': u'1234567890', 'user_fname': u'TestingTesting2', 'amount': 1222,’event_timestamp': u’2016-09-29T07:49:50.866+00:00’}, 

{'user_lname': u'TEst2', 'user_id': u'2aa8ae30-c0e5-48bb-ac16-a2ed2e78c8c3', 'user_phone': u'1234567891', 'user_fname': u'TestingTesting', 'amount': 12,’event_timestamp': u’2016-10-27T07:49:50.866+00:00’},

{'user_lname': u'TEst3', 'user_id': u'2aa8ae30-c1e5-48bb-ab16-a2ed2e78c8c3', 'user_phone': u'1234567892', 'user_fname': u'TestingTesting3', 'amount': 122,’event_timestamp': u’2016-09-27T07:49:50.866+00:00'}
]

I want to save the above RDD to a cassandra table.
I get the below error when I use

test_rdd.saveToCassandra("keyspace1","table1")  

Traceback (most recent call last): File "/var/spark/test/k.py", line 179, in
parsed_data.saveToCassandra("keyspace1","table1") AttributeError: 'PipelinedRDD' object has no attribute 'saveToCassandra'

1

1 Answers

0
votes

Either

or

  • follow the instructions for the official spark-cassandra-connector
  • convert to DataFrame (toDF)
  • write Dataframe

    df.write.format("org.apache.spark.sql.cassandra").options(
      table=table, keyspace=keyspace
    ).save()