Spark program cannot open SQL Server context?

0
votes

In the file main.scala, 

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import java.sql._

object ConnTest extends App {
  val conf = new SparkConf()
  val sc = new SparkContext(conf.setAppName("Test").setMaster("local[*]"))
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)

  sc.stop()
}

However, the sbt run got the following error.

[info] Compiling 1 Scala source to C:\Users\user1\IdeaProjects\sqlServer\target\scala-2.11\classes...
[error] C:\Users\user1\IdeaProjects\sqlServer\src\main\scala\main.scala:9: type SQLContext is not a member of package org.apache.spark.sql
[error]   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
[error]                                             ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 1 s, completed May 11, 2017 6:11:47 PM
2
sql-serverscalaapache-spark
First, what are your dependencies - sql must be explicitly set? Also you shouldn't use sbt run for spark, but spark-submit instead. – Justin Pihony
I should use SBT compile the code first. Maybe I shouldn't use "run"? – ca9163d9

2 Answers

1
votes

You should use Spark 2 which has single entry point SparkSession. you can create SQLContext and SparkContext as 

val sparkSession = SparkSession.builder().master("local[*]").getOrCreate()

val sc = sparkSession.sparkContext
val sqlC = sparkSession.sqlContext

Include the dependency for spark core and spark sql

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1"

libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.1"
0
votes

Your class path is correct, please carefully check you maven or sbt configure

In maven, you should add this specific dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.1.1</version>
    <scope>compile</scope>
</dependency>

sbt is similar too, if you want to add this dependency at compile time and package an uber jar with it.

But sometime the spark-core jar is provided by the cluster at runtime, if it is in this case, you can adjust the scope value at you convenient, maybe provided.

What must to be said is that sbt or maven is just dependency control tool, not relevant to spark's execution, you should get your jar package upload on the cluster, and use the spark-submit program to run it!

Please see the community document of the spark example, and have a try:

http://spark.apache.org/examples.html

Good Luck!