HBase Spark - Connectivity with Spark 2.0

3
votes

Have been trying to write data to HBase using Spark & Scala. Requirement is that - I have to set the TTL for HBase records based on the Timestamp of a Column which comes from the business.

I am not able to compile my code & the details are as below.

  1. I am using HBase Spark Alpha 4 Connector (hbase-spark-2.0.0-alpha4.jar) for this purpose.
  2. Converted a column's timestamp value in to Epoch Timestamp and trying to set the TTL based on that.
  3. When I am trying to execute the same code in Spark Shell - it is seamlessly executing and I am able to see the expected output in HBase.
  4. But when I try to compile it in Eclipse - it fails with the below error.
[ERROR] error: missing or invalid dependency detected while loading class file 'HBaseContext.class'.    
[INFO] Could not access type Logging in package org.apache.spark,    
[INFO] because it (or its dependencies) are missing.

Could someone please provide inputs on resolving this ?

1
scalaapache-sparkhbase

1 Answers

0
votes

In my case, I recompiled the connector: https://github.com/apache/hbase-connectors.git with the params based on my project. Just like:

mvn -Dspark.version=2.2.2 -Dscala.version=2.11.7 -Dscala.binary.version=2.11 clean install

The there is no error.

Tips: The groupId is different with the groupId in the maven centre repo