CDH5.4.2 spark can use HiveContent in spark-shell but can't open spark-sql

0
votes

I am using spark(standalone) of CDH5.4.2

After copying hive-site.xml to $SPARK_HOME/conf,i can query from hive in spark-shell,such as below:

scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc); hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@6c6f3a15 scala> hiveContext.sql("show tables").show();

But when i open spark-sql ,it show wrong:

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver Failed to load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.

You need to build Spark with -Phive and -Phive-thriftserver.

What is different between spark-shell and spark-sql? If spark of cdh don't support hive,why can i use HiveConext?

1
apache-sparkcloudera-cdhhivecontext
What exactly do you think the spark-sql command is supposed to do?!? - Samson Scharfrichter
Spoiler: Cloudera explicitly states that the Spark JDBC Thrift server is not supported in CDH, so it's not surprising that you cannot start it. - Samson Scharfrichter
Thank you so much ,I will carefully to read the official document. - 陈志明

1 Answers

0
votes

Cloudera has a list of unsupported features here:

https://docs.cloudera.com/runtime/7.2.6/spark-overview/topics/spark-unsupported-features.html

The Thrift server is not supported.

This is a copy of the list for 7.2.6:

  • Apache Spark experimental features/APIs are not supported unless stated otherwise.
  • Using the JDBC Datasource API to access Hive or Impala is not supported
  • ADLS not supported for All Spark Components. Microsoft Azure Data Lake Store (ADLS) is a cloud-based filesystem that you can access through Spark applications. Spark with Kudu is not currently supported for ADLS data. (Hive on Spark is available for ADLS.)
  • IPython / Jupyter notebooks is not supported. The IPython notebook system (renamed to Jupyter as of IPython 4.0) is not supported.
  • Certain Spark Streaming features, such as the mapWithState method, are not supported.
  • Thrift JDBC/ODBC server is not supported
  • Spark SQL CLI is not supported
  • GraphX is not supported
  • SparkR is not supported
  • Structured Streaming is supported, but the following features of it are not:
  • Continuous processing, which is still experimental, is not supported.
  • Stream static joins with HBase have not been tested and therefore are not supported.
  • Spark cost-based optimizer (CBO) not supported.