hbase-spark load data raise NullPointerException Error (scala)

Question

I want to load data from HBase by Spark SQL, I use the hbase-spark official example and it raise NullPointerException

My build.sbt file is:

name := "proj_1"
version := "0.1"
scalaVersion := "2.11.12"
libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-core_2.11" % "2.3.1",
    "org.apache.spark" % "spark-sql_2.11" % "2.3.1",
    "org.apache.spark" % "spark-mllib_2.11" % "2.3.1",
    "org.apache.spark" % "spark-streaming_2.11" % "2.3.1",
    "org.apache.spark" % "spark-hive_2.11" % "2.3.1",
    "org.elasticsearch" % "elasticsearch-hadoop" % "6.4.0",
    "org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.15.1",
    "org.apache.hbase" % "hbase" % "2.1.0",
    "org.apache.hbase" % "hbase-server" % "2.1.0",
    "org.apache.hbase" % "hbase-common" % "2.1.0",
    "org.apache.hbase" % "hbase-client" % "2.1.0",
    "org.apache.hbase" % "hbase-spark" % "2.1.0-cdh6.x-SNAPSHOT"
)
resolvers += "Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
resolvers += "clojars" at "https://clojars.org/repo"
resolvers += "conjars" at "http://conjars.org/repo"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"

The wrong code is:

def withCatalog(cat: String): DataFrame = {
    sqlContext
        .read
        .options(Map(HBaseTableCatalog.tableCatalog->cat))
        .format("org.apache.hadoop.hbase.spark")
        .option("zkUrl", "127.0.0.1:2181/chen_test")
        .load()
}
val df = withCatalog(catalog)

Exception info is:

Exception in thread "main" java.lang.NullPointerException
    at org.apache.hadoop.hbase.spark.HBaseRelation.<init> (DefaultSource.scala:139)
    at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:70)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
    at hbase_test$.withCatalog$1(hbase_test.scala:57)
    at hbase_test$.main(hbase_test.scala:59)
    at hbase_test.main(hbase_test.scala)

How do I fix it? Can you help me?

Raymond Tay Raymond Tay · Accepted Answer · 2018-09-19T01:48:51

Ran into this problem recently. Suggest you try this:

    import org.apache.hadoop.fs.Path
    val conf = HBaseConfiguration.create()
    conf.addResource(new Path("/path/to/hbase-site.xml"))
    new HBaseContext(sc, conf) // "sc" is the SparkContext you created earlier.

The last expression is introducing a stable value into the environment; found this quite accidentally by scanning Hbase's codebase. Hope it helps.

hbase-spark load data raise NullPointerException Error (scala)

1 Answers