Use HBaseStorage to load from Bigtable via Pig on Dataproc

Question

Does anybody have experience or succeeded on loading data from Bigtable via Pig on Dataproc using HBaseStorage?

Here's a very simple Pig script I'm trying to run. It fails with an error indicating it can't find the BigtableConnection class and I'm wondering what setup I may be missing to successfully load data from Bigtable.

raw = LOAD 'hbase://my_hbase_table'
       USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
       'cf:*', '-minTimestamp 1490104800000 -maxTimestamp 1490105100000 -loadKey true -limit 5')
       AS (key:chararray, data);

DUMP raw;

Steps I followed to setup my cluster:

Launched Bigtable cluster (my_bt); created and populated my_hbase_table
Launched Dataproc cluster (my_dp) via cloud.google.com Cloud Dataproc Console
Installed HBase shell on Dataproc master (/opt/hbase-1.2.1) following instructions on https://cloud.google.com/bigtable/docs/installing-hbase-shell
Added properties to hbase-site.xml for my_bt and BigtableConnection class
Created file t.pig with contents listed above
Invoked Pig via command: gcloud beta dataproc jobs submit pig --cluster my_dp --file t.pig --jars /opt/hbase-1.2.1/lib/bigtable/bigtable-hbase-1.2-0.9.5.1.jar
Got the following error indicating BigtableConnection class not found:

2017-03-21 15:30:48,029 [JobControl] ERROR org.apache.hadoop.hbase.mapreduce.TableInputFormat - java.io.IOException: java.lang.ClassNotFoundException: com.google.cloud.bigtable.hbase1_2.BigtableConnection

I would suggest using the shaded bigtable mapreduce jar, which has all of the dependencies you'll need. Go to search.maven.org/#search%7Cga%7C1%7Cbigtable%20mapreduce,and download "shaded.jar" . — Solomon Duskis
Looks like ,and was auto-appended to @SolomonDuskis' URL due to lack of space to separate them; you want to visit search.maven.org/#search%7Cga%7C1%7Cbigtable%20mapreduce to download the artifact. — Misha Brukman
I downloaded the shaded.jar and get the same error when submitting the pig job. I can upload the output I get when running the test if that helps. — EduBoom
Can you try to add netty-tcnative-boringssl-static? See ttp://search.maven.org/#search%7Cga%7C1%7Cg%3A%22io.netty%22%20AND%20a%3A%22netty-tcnative-boringssl-static%22%20AND%20v%3A%221.1.33.Fork26%22 and download "jar" — Solomon Duskis

Angus Davis Angus Davis · Accepted Answer · 2017-03-23T01:33:33

The trick is getting all dependencies on pig's classpath. Using the jars pointed to by Solomon, I've created the following initialization action that downloads two jars, the bigtable mapreduce jar and netty-tcnative-boringssl, and sets up the pig classpath.

#!/bin/bash
# Initialization action to set up pig for use with cloud bigtable
mkdir -p /opt/pig/lib/

curl http://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork19/netty-tcnative-boringssl-static-1.1.33.Fork19.jar \
    -f -o /opt/pig/lib/netty-tcnative-boringssl-static-1.1.33.Fork19.jar

curl http://repo1.maven.org/maven2/com/google/cloud/bigtable/bigtable-hbase-mapreduce/0.9.5.1/bigtable-hbase-mapreduce-0.9.5.1-shaded.jar \
    -f -o /opt/pig/lib/bigtable-hbase-mapreduce-0.9.5.1-shaded.jar

cat >>/etc/pig/conf/pig-env.sh <<EOF
#!/bin/bash

for f in /opt/pig/lib/*.jar; do
  if [ -z "\${PIG_CLASSPATH}" ]; then
    export PIG_CLASSPATH="\${f}"
  else
    export PIG_CLASSPATH="\${PIG_CLASSPATH}:\${f}"
  fi  
done
EOF

You can then pass in bigtable configuration in the usual ways:

Via hbase-site.xml

Specifying properties when submitting a job:

PROPERTIES='hbase.client.connection.impl='
PROPERTIES+='com.google.cloud.bigtable.hbase1_2.BigtableConnection'
PROPERTIES+=',google.bigtable.instance.id=MY_INSTANCE'
PROPERTIES+=',google.bigtable.project.id=MY_PROJECT'

gcloud dataproc jobs submit pig --cluster MY_DATAPROC_CLUSTER \
    --properties="${PROPERTIES}"  \
    -e "f =  LOAD 'hbase://MY_TABLE' 
         USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:*','-loadKey true') 
         AS (key:chararray, data); 
    DUMP f;"

Use HBaseStorage to load from Bigtable via Pig on Dataproc

1 Answers