Error on opening the Apache Spark source code in IntelliJ IDEA

0
votes

I’m trying to open the Apache Spark source code in IntelliJ IDEA.

I opened pom.xml on the Spark source code root directory.
Project tree is displayed in the Project tool window.

But, when I open a source file, say org.apache.spark.deploy.yarn.ClientBase.scala, a lot of red marks shows on the editor scroll bar. It is the ‘Cannot resolve symbol’ error. Even it cannot resolve StringOps.format.

How can I fix it?

On File | Project Structure window, the following error message is displayed with pink background:

Library ‘Maven: org.scala-lang:scala-compiler-bundle:2.10.4’ is not used

Can it be a hint?

The versions I’m using are as follows:

  • OS: Windows 7
  • IntelliJ IDEA: 13.1.6
  • Scala plugin: 0.41.2
  • Spark source code: 1.1.1 (with a few file modified by me)

I’ve tried to fix this and error state changed somewhat, but eventually I gave up fixing it on my own (with googling) and deleted .idea folder and started over. So now I’m seeing the errors described above.

UPDATE:

I noticed thw following popup:

Maven projects need to be imported: Import Changes Enable Auto-Import

And enabled auto-import according to the articles IntelliJ: Maven projects need to be imported: Import Changes Enable Auto-Import and http://javafortesters.blogspot.kr/2013/09/do-enable-auto-import-in-intellij-for.html . Now IntelliJ resolves base Scala symbols.

But still it cannot resolve a few symbols.

The notable file is yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala. In this file, ClientArguments class is not resolved. IntelliJ suggests importing org.apache.spark.deploy.ClientBase, but in fact ClientArgument class is in the same package with ClientBase - that is, org.apache.spark.deploy.yarn.ClientArgument.

Why IntelliJ confuses this?

Thank you.

2
scalamavenintellij-ideaapache-spark
Can you try and see if, in the 'Maven projects' tool window, under 'Profiles', there are any overlapping profiles active? If, for example, both 'scala-2.10' and 'scala-2.11' are checked. – pzecevic
@pzecevic Thanks. But there are no overlapping profiles active. Active ones are as follows: hadoop-2.4, maven-3, yarn. – zeodtr

2 Answers

1
votes

You need to change the Scala compiler from IntelliJ to “sbt incremental compiler” (see the screenshot below).

You can access this by going to “preferences” -> “scala”.

NOTE: This is supported only for certain version of IntelliJ scala plugin. See this link for details. http://blog.jetbrains.com/scala/2014/01/30/try-faster-scala-compiler-in-intellij-idea-13-0-2/

1
votes

Seems your maven cannot download jars according to your pom dependencies setting.

Two possible factors:

  1. It could be due to your network, so you need check with proxy setting: (Ctrl+Shift+A in IntelliJ, enter "proxy", to check it connection).

  2. It could also because your maven home has not been set in IntelliJ. to set it, you need (Ctrl+Shift+A in IntelliJ, enter "maven setting", to set maven home to point to the place where you have installed your maven.