Issue with Hive and Google Cloud Storage

0
votes

Please advice:

I have installed a hadoop 2.6.5 version cluster in GCP using VM's instances. Used GCP connector and pointed by hdfs to use gs bucket. Added the below 2 entries in coresite.xml: 

google.cloud.auth.service.account.json.keyfile=<Path-to-the-JSON-file> 
fs.gs.working.dir=/

When using hadoop gs -ls / works fine , but when I am creating a hive tables 

CREATE EXTERNAL TABLE test1256(name string,id  int)   LOCATION   'gs://bucket/';

I get the following error:

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.security.AccessControlException: Permission denied: user=hdpuser1, path="gs://bucket/":hive:hive:drwx------) (state=08S01,code=1)

2
hivegoogle-cloud-storage

2 Answers

0
votes

As the error message implies, you encountered a permissions issue. First, check your Google Cloud Console to make sure you have Cloud Storage IAM permissions to perform that action. Next, make sure you have the right permissions for the "hdpuser1" user in hdfs:

  • Login as the hdfs user: su hdfs
  • You can also change the permissions: hdfs dfs -chown -R <username_of_new_owner> /user

Then perform the following:

hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -mkdir /tmp
hadoop fs -chmod -R 777 /user/hive
hadoop fs -chmod 777 /tmp

Then try creating the Hive table again. I hope that helps.

Find a similar answer here. And a complete guide in GitHub for performing a test with Hive

0
votes

I had just this error today and was able to resolve it by adding two following properties:

fs.gs.reported.permissions=777
fs.gs.path.encoding=uri-path

both into core-site.xml and hive-site.xml (by going to advanced configs of HDFS and Hive services in ambari).

If you configure it just in core-site.xml, then hive external table creation will fail.