0
votes

I am using Scala IDE for creating a maven project with spark

1.I have created a maven project with skipped the archetype and added the following pom file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.nivi</groupId>
  <artifactId>testSpark</artifactId>
  <version>0.0.1-SNAPSHOT</version>

    <repositories>
    <repository>
      <id>cloudera</id>
      <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </repository>

    <repository>
      <id>general</id>
      <url>https://repo1.maven.org/maven2</url>
    </repository>

  </repositories> 


  <dependencies>
              <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.10.6</version>
              </dependency>
              <dependency>
                     <groupId>org.apache.spark</groupId>
                     <artifactId>spark-core_2.10</artifactId>
                     <version>1.6.0-cdh5.8.2</version>
                     <scope>provided</scope>
              </dependency>
              <dependency>
                     <groupId>org.apache.spark</groupId>
                     <artifactId>spark-mllib_2.10</artifactId>
                     <version>1.6.0-cdh5.8.2</version>
                     <scope>provided</scope>
              </dependency>

              <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-hive_2.10</artifactId>
                    <version>1.6.0-cdh5.8.2</version>
                   <scope>provided</scope>
              </dependency>

              <dependency>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
                   <scope>provided</scope>
              </dependency>
</dependencies>
        <build>
              <plugins>

                     <!-- mixed scala/java compile -->
                     <plugin>
                           <groupId>org.scala-tools</groupId>
                           <artifactId>maven-scala-plugin</artifactId>
                           <version>2.15.2</version>
                           <executions>
                                  <execution>
                                         <id>compile</id>
                                         <goals>
                                                <goal>compile</goal>
                                         </goals>
                                         <phase>compile</phase>
                                  </execution>
                                  <execution>
                                         <id>test-compile</id>
                                         <goals>
                                                <goal>testCompile</goal>
                                         </goals>
                                         <phase>test-compile</phase>
                                  </execution>
                                  <execution>
                                         <phase>process-resources</phase>
                                         <goals>
                                                <goal>compile</goal>
                                         </goals>
                                  </execution>
                           </executions>
                           <configuration> 
           <scalaVersion>2.10.6</scalaVersion> 
         </configuration> 
                     </plugin>
                     <plugin>
                           <artifactId>maven-compiler-plugin</artifactId>
                           <version>2.4</version>
                           <configuration>
                                  <source>1.8</source>
                                  <target>1.8</target>
                           </configuration>
                     </plugin>
                     <!-- for fatjar -->
                     <plugin>
                           <groupId>org.apache.maven.plugins</groupId>
                           <artifactId>maven-assembly-plugin</artifactId>
                           <version>2.4</version>
                           <configuration>
                                  <descriptorRefs>
                                         <descriptorRef>jar-with-dependencies</descriptorRef>
                                  </descriptorRefs>
                           </configuration>
                           <executions>
                                  <execution>
                                         <id>assemble-all</id>
                                         <phase>package</phase>
                                         <goals>
                                                <goal>single</goal>
                                         </goals>
                                  </execution>
                           </executions>
                     </plugin>
                     <plugin>
                           <groupId>org.apache.maven.plugins</groupId>
                           <artifactId>maven-jar-plugin</artifactId>
                           <version>2.4</version>
                           <configuration>
                                  <archive>
                                         <manifest>
                                                <addClasspath>true</addClasspath>
                                                <mainClass>fully.qualified.MainClass</mainClass>
                                         </manifest>
                                  </archive>
                           </configuration>
                     </plugin>
              </plugins>
              <pluginManagement>
                     <plugins>
                           <!--This plugin's configuration is used to store Eclipse m2e settings 
                                  only. It has no influence on the Maven build itself. -->
                           <plugin>
                                  <groupId>org.eclipse.m2e</groupId>
                                  <artifactId>lifecycle-mapping</artifactId>
                                  <version>1.0.0</version>
                                  <configuration>
                                         <lifecycleMappingMetadata>
                                                <pluginExecutions>
                                                       <pluginExecution>
                                                              <pluginExecutionFilter>
                                                                     <groupId>org.scala-tools</groupId>
                                                                     <artifactId>
                                                                           maven-scala-plugin
                                                                     </artifactId>
                                                                     <versionRange>
                                                                           [2.15.2,)
                                                                     </versionRange>
                                                                     <goals>
                                                                           <goal>compile</goal>
                                                                           <goal>testCompile</goal>
                                                                     </goals>
                                                              </pluginExecutionFilter>
                                                              <action>
                                                                     <execute></execute>
                                                              </action>
                                                       </pluginExecution>
                                                </pluginExecutions>
                                         </lifecycleMappingMetadata>
                                  </configuration>
                           </plugin>
                     </plugins>
              </pluginManagement>

  </build>
</project>
  1. I have added scala nature in Configure.

  2. I have added src/main/scala folder in the Properties-->Build path-->Source

  3. I have set java compiler 1.8 and scala compiler 2.10.6 with jvm 1.8 in my project properties.

But i am getting the following error on maven clean install :

enter image description here

2
delete .m2 repo and try againAfgan

2 Answers

0
votes

Go to this location c:\users\Nivedita.m2\repository\org\apache\hadoop\hadoop-common\2.6.0-cdh5.8.2 and try deleting hadoop-common-2.6.0-cdh5.8.2.jar file and do maven build again.

0
votes
  1. Can you try with an recent version of scala plugin ? (it's the successor of the one you used, I'm the author of both)

    <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.3.2</version>
    </plugin>
    

(see http://davidb.github.io/scala-maven-plugin/example_java.html)

  1. if you use a jdk 9+ to compile (regardless of jvm 1.8 in the config) you can have this kind of issue. So check that the jdk is 1.8 .