Hive table lock info using spark.sql in Spark 2.4.0 throws error

Question

Not able to query hive table lock info using below SQL query in Spark 2.4.0 shell.

spark.sql("show locks dbName.tableName")

Getting below error

scala> spark.sql("show locks dbName.tableName")
org.apache.spark.sql.catalyst.parser.ParseException:
Operation not allowed: show locks(line 1, pos 0)

== SQL ==
show locks dbName.tableName
^^^
at 
org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:39)
  at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitFailNativeCommand$1.apply(SparkSqlParser.scala:1001)
  at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitFailNativeCommand$1.apply(SparkSqlParser.scala:992)
  at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
  at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitFailNativeCommand(SparkSqlParser.scala:992)
  at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitFailNativeCommand(SparkSqlParser.scala:55)
  at org.apache.spark.sql.catalyst.parser.SqlBaseParser$FailNativeCommandContext.accept(SqlBaseParser.java:723)
  at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
  at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
  at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:71)
  at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:99)
  at org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:70)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:69)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:68)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:97)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:68)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:637)
  ... 48 elided

But the same query works in Spark 1.6.0 with hiveContext

My use case is to get the lock info about the hive table inside a spark job in Spark 2.4.0

If there is any way that I can get the table lock info in spark job?

Neeraj Vyas Neeraj Vyas · Accepted Answer · 2020-08-12T20:45:39

See the below Link. Locks are not supported by spark 2.4.0

https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L195

unsupportedHiveNativeCommands | kw1=SHOW kw2=LOCKS

https://dbdb.io/db/spark-sql On the above blog also. Concurrency control section says, It doesn't support locks.

Hive table lock info using spark.sql in Spark 2.4.0 throws error

1 Answers