Error connecting Spark local cluster

Question

I am trying to run the following code in my local mac where a spark cluster with master and slaves are running

  public void run(String inputFilePath) {

    String master = "spark://192.168.1.199:7077";

    SparkConf conf = new SparkConf()
        .setAppName(WordCountTask.class.getName())
        .setMaster(master);
    JavaSparkContext context = new JavaSparkContext(conf);


    context.textFile(inputFilePath)
        .flatMap(text -> Arrays.asList(text.split(" ")).iterator())
        .mapToPair(word -> new Tuple2<>(word, 1))
        .reduceByKey((a, b) -> a + b)
        .foreach(result -> LOGGER.info(
            String.format("Word [%s] count [%d].", result._1(), result._2)));
  }
}

However I get the following exception both in the master console and

Error while invoking RpcHandler#receive() on RPC id 5655526795459682754 java.io.EOFException

and in the program console

18/07/01 22:35:19 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.1.199:7077 org.apache.spark.SparkException: Exception thrown in awaitResult

This runs well when I set the master as "local[*]" as given in this example.

I have seen examples where the jar is submited with spark-submit command but I am trying to run it programatically.

Hi. By talking about "my local mac where a spark cluster with master and slaves are running" do you mean Spark standalone cluster or a Dockerized cluster ? If you're using Standalone, please try spark://localhost:7077. If you're using a Docker image, ensure that all required ports are exposed by the container. — Bartosz Konieczny
Its a standalone cluster i.e. I open diffrent terminal window and start the mater and multiple slave nodes. 192.168.1.199 is the local ip of the mac which was displayed in the master console output. So localhost and 192.168.1.199 had the same effect. — Neil

Neil Neil · Accepted Answer · 2018-07-02T10:33:27

Just realised the version of Spark was different in the master/slave and the POM file of the code. Bumped up the version in the pom.xml to match the spark cluster and it worked.

Error connecting Spark local cluster

1 Answers