Spark streaming reading azure event hub - error compiling

0
votes

I am trying to read azure event hub using spark but I get the following error when compiled in intellij

Error:scalac: bad symbolic reference to org.apache.http.annotation.Experimental encountered in class file 'EventHubsConf.class'.
Cannot access type Experimental in package org.apache.http.annotation. The current classpath may be
missing a definition for org.apache.http.annotation.Experimental, or EventHubsConf.class may have been compiled against a version that's
incompatible with the one found on the current classpath.

I made sure the compiler in Intellij is pointing to the SCALA 11.0 libraries as I have seen scala 12.7 libraries in the Global SDK

enter image description here

Following is my set up :

name := "xxxxx"

version := "0.1"

scalaVersion := "2.11.0"


libraryDependencies ++= Seq(

  // Spark
  "org.apache.spark" %% "spark-core" % "2.4.0",
  "org.apache.spark" %% "spark-sql" % "2.4.0",
  "org.apache.spark" %% "spark-streaming" % "2.4.0",

  "com.microsoft.azure" % "azure-eventhubs-spark_2.11" % "2.3.10"
)

Any help is appreciated.

Thanks!

1
apache-sparkintellij-ideaazure-eventhub

1 Answers

0
votes

It is just the corresponding libraries did not get updated in the external libraries. I had to manually add it and it started working.

enter image description here