Which is the best HBase connector to use for batch loading data into HBase from Spark?

Question

As mentioned also in Which HBase connector for Spark 2.0 should I use? mainly there are two options:

RDD based https://github.com/apache/hbase/tree/master/hbase-spark
DataFrame based https://github.com/hortonworks-spark/shc

I do understand the optimizations and the differences with regard to READING from HBase.

However it's not clear for me which should I use for BATCH inserting into HBase.

I am not interested in one by one records, but by high throughput.

After digging through code, it seems that both resort to TableOutputFormat, http://hbase.apache.org/1.2/book.html#arch.bulk.load

The project uses Scala 2.11, Spark 2, HBase 1.2

Does the DataFrame library provide any performance improvements over the RDD lib specifically for BULK LOAD ?

The RDD code can easily be rewritten to a Dataset based API. I highly doubt that affects the throughput. Just different libraries doing similar things in a different way — OneCricketeer
for the RDD code there is an example of BulkPut. The client code aggregates PUT requests and sends them in one batch to the HBaser server. However, for the DF code, it's not clear how the batch is working. There is no clear example with a difference between Batch Insertion or inserting a single element. — cipri.l

Kumar Kumar · Accepted Answer · 2019-08-31T01:46:38

Lately, hbase-spark connector has been released to a new maven central repository with 1.0.0 version and supports Spark version 2.4.0 and Scala 2.11.12

  <dependency>
     <groupId>org.apache.hbase.connectors.spark</groupId>
     <artifactId>hbase-spark</artifactId>
     <version>1.0.0</version>
   </dependency>

This supports both RDD and DataFrames. Please refer spark-hbase-connectors for more details

Happy Learning !!

Which is the best HBase connector to use for batch loading data into HBase from Spark?

2 Answers