Spark-submit Sql Context Create Statement does not work

Question

My below code does not work with Spark-submit.

sqlContext.sql(s"""
create external table if not exists landing (
date string,
referrer string)
partitioned by (partnerid string,dt string)
row format delimited fields terminated by '\t' lines terminated by '\n'
STORED AS TEXTFILE LOCATION 's3n://....'
      """)

It gives error: Exception in thread "main" java.lang.RuntimeException: [1.2] failure: ``with'' expected but identifier create found

This code works in Spark-shell but not in Spark-submit. What can be the reason?

class Analysis extends Serializable{ val sc = new SparkContext() val sqlContext = SQLContextSingleton.getInstance(sc) import sqlContext.implicits._ => I run it like this but no way :( — Yusuf Can Gürkan

中国人威中国人威 · Accepted Answer · 2015-12-04T08:37:16

"sqlContext" in spark-shell is 'HiveContext' by default. Maybe you need to new a HiveContext instead of sqlContext in your script.

You may new it like that:

import SparkContext._
import org.apache.spark.sql.hive._ 
val sc = new SparkContext()
val sqlContext = new HiveContext(sc)

Spark-submit Sql Context Create Statement does not work

1 Answers