creating sparksession using builder?

1
votes

How to create a sparkSession using builder in Java ? I have tried this :

sparkSession = SparkSession.builder()
    .master("local[*]")
    .config(conf)
    .appName(appName)
    .getOrCreate();

But I'm getting exception in "getOrCreate". Is there any way to create a spark session using JavaSparkContext/ SparkContext ?

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.UserGroupInformation.getCurrentUser()Lorg/apache/hadoop/security/UserGroupInformation; at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2373) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2373) at org.apache.spark.SparkContext.(SparkContext.scala:295) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)

3
javaapache-spark-sqlapache-spark-mllib

3 Answers

2
votes

You can create spark session object in Java as

import org.apache.spark.sql.SparkSession;
// Inside class
 SparkSession spark = SparkSession
      .builder()
      .appName("Application Name")
      .config("some-config", "some-value")
      .getOrCreate();

This should work. If this didn't work can you paste whole code so that we can figure out what is going wrong.

1
votes
    SparkConf conf= new SparkConf().setAppName("Java Spark").setMaster("local[*]");

    SparkSession spark = SparkSession
            .builder()
            .config(conf)
            .getOrCreate();

    JavaSparkContext sc = new JavaSparkContext();
    JavaRDD<String> lines=sc.textFile("E://Car.txt");
    JavaRDD<String> filteredLines = lines.filter(new Function<String, Boolean>() {

        @Override
        public Boolean call(String arg0) throws Exception {
            // TODO Auto-generated method stub
            return null;
        }
    });
-2
votes

Check the Java API for the SparkSession, the following code should work.

SparkSession.builder()
     .master("local")
     .appName("App Name")
     .config("spark.some.config.option", "some-value").
     .getOrCreate()

https://spark.apache.org/docs/2.0.1/api/java/org/apache/spark/sql/SparkSession.html