spark streaming write data to Hbase with python blocked on saveAsNewAPIHadoopDataset

0
votes

I’m using spark-streaming python read kafka and write to hbase, I found the job on stage of saveAsNewAPIHadoopDataset very easily get blocked. As the below picture: You will find the duration is 8 hours on this stage. Does the spark write data by Hbase api or directly write the data via HDFS api please?enter image description here

1
apache-sparkhbasespark-streaming

1 Answers

0
votes

A bit late , but here is a similar example To save an RDD to hbase :

Consider an RDD containing a single line :

{"id":3,"name":"Moony","color":"grey","description":"Monochrome kitty"}

Transform the RDD
We neet to transform the RDD into a (key,value) pair having the following contents:

( rowkey , [ row key , column family , column name , value ] )

datamap = rdd.map(lambda x: (str(json.loads(x)["id"]),[str(json.loads(x)["id"]),"cfamily","cats_json",x]))

Save to HBase
We can make use of the RDD.saveAsNewAPIHadoopDataset function as used in this example: PySpark Hbase example to save the RDD to HBase ?

datamap.saveAsNewAPIHadoopDataset(conf=conf,keyConverter=keyConv,valueConverter=valueConv)

You can refer to my blog :pyspark-sparkstreaming hbase for the complete code of the working example.