How to create an EMR cluster AWS Glue Data Catalog settings?

Question

AWS EMR MetaException, message:

Insufficient Lake Formation permission(s) on global_temp.

I am trying to use a spark session to query data using the hive metastore. The tables are cataloged in glue. I can read the s3 parquet files directly using spark.read.parquet. However, I would like to use the spark.sql method to query the catalog from the EMR.

When I run the line:

spark.sql("show tables from gluedatabase").show()

I get an error like this:

'org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Insufficient Lake Formation permission(s) on global_temp (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException));'
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 767, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[2], stackTrace)
pyspark.sql.utils.AnalysisException: 'org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Insufficient Lake Formation permission(s) on global_temp (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException));'

It seems you need to grant permissions to your IAM user for Lake Formation — blackbishop
Can you try this github.com/aws-samples/aws-glue-samples/tree/master/utilities/… ? It should help — Prabhakar Reddy

Petr Švec Petr Švec · Accepted Answer · 2021-02-11T08:58:54

I run into similar issue when setting up a Glue developer endpoint. Try adding the role of your EMR cluster to the list of database creators in Lake Formation. Lake Formation -> Admins and database creators

That should resolve it. Best of luck

How to create an EMR cluster AWS Glue Data Catalog settings?

1 Answers