Spark: java.io.InvalidClassException

0
votes

Since moving from a local spark deployment to a standalone cluster, I have been getting the following exception:

java.io.InvalidClassException: org.apache.spark.sql.cassandra.CassandraSQLRow; local class incompatible: stream classdesc serialVersionUID = -6648071136076699828, local class serialVersionUID = -1718118159864345582

I know exactly when it happens. It happens when I redeploy the shaded jar from the target directory of build to the Spark master WITHOUT actually recompiling the jar. (This was just a little convenience script for development that clearly doesn't work in a cluster)

However, I don't actually understand the reason for this Exception under these circumstances. How has the serialVersionUID been changed if the class has NOT been recompiled?

1
apache-sparkcassandra
What Scala version is target for local development & what on server? Can you show content of the script? – Alex Ott
exactly the same spark-2.2.1-bin-hadoop2.7 – Jake
I'm assuming it si related to the streaming of the jars to the slaves...but beyond that have no conceptual understanding of the issue – Jake

1 Answers

0
votes

I had different versions of the spark-cassandra-connector The --packages version was different to the version in my pom.xml. The latter dependency probably being unnecessary outside the test scope of maven