upgrade Twitter4J within Spark

0
votes

so as the spark streaming documentation says:

Twitter: Spark Streaming’s TwitterUtils uses Twitter4j 3.0.3 to get the public stream of tweets using Twitter’s Streaming API.

Since i need to work with some functions that aren't supported in this version like getLang(), i want to upgrade this version.

Until now i've imported twitter4J version 4.0.4 jars within my eclipse project but it still recognizing only the 3.0.3 version.

Help!

1
apache-sparkspark-streamingtwitter4j

1 Answers

0
votes

You will need to exclude the twitter4j dependency shipped with spark-streaming-twitter and explicitly add your own twitter4j version.

In sbt would be something like:

libraryDependencies +=  "org.apache.spark" % "spark-streaming-twitter_2.10" % "1.6.1" exclude("org.twitter4j", "twitter4j")

libraryDependencies += "org.twitter4j" % "twitter4j" % "4.0.4"

This might cause trouble in general as API may change between major versions, but I recall doing something similar for the same reason (language support).