Spark Avro throws exception on file write: NoSuchMethodError

Question

Any file write attempt of Avro format fails with the stack trace below.

We are using Spark 2.4.3 (with user provided Hadoop), Scala 2.12, and we load the Avro package at runtime with either spark-shell:

spark-shell --packages org.apache.spark:spark-avro_2.12:2.4.3

or spark-submit:

spark-submit --packages org.apache.spark:spark-avro_2.12:2.4.3 ...

The spark Session reports loading the Avro package successfully.

... in either case, the moment we attempt to write any data to an avro format, like:

df.write.format("avro").save("hdfs:///path/to/outputfile.avro")

or with a select:

df.select("recordidstring").write.format("avro").save("hdfs:///path/to/outputfile.avro")

... produces the same stacktrace error (this copy from spark-shell):

java.lang.NoSuchMethodError: org.apache.avro.Schema.createUnion([Lorg/apache/avro/Schema;)Lorg/apache/avro/Schema;
  at org.apache.spark.sql.avro.SchemaConverters$.toAvroType(SchemaConverters.scala:185)
  at org.apache.spark.sql.avro.SchemaConverters$.$anonfun$toAvroType$1(SchemaConverters.scala:176)
  at scala.collection.Iterator.foreach(Iterator.scala:941)
  at scala.collection.Iterator.foreach$(Iterator.scala:941)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
  at scala.collection.IterableLike.foreach(IterableLike.scala:74)
  at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  at org.apache.spark.sql.types.StructType.foreach(StructType.scala:99)
  at org.apache.spark.sql.avro.SchemaConverters$.toAvroType(SchemaConverters.scala:174)
  at org.apache.spark.sql.avro.AvroFileFormat.$anonfun$prepareWrite$2(AvroFileFormat.scala:119)
  at scala.Option.getOrElse(Option.scala:138)
  at org.apache.spark.sql.avro.AvroFileFormat.prepareWrite(AvroFileFormat.scala:118)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:103)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:170)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:131)
  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:155)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
  at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:676)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:78)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
  at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:290)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)

We are able to write other formats (text-delimited, json, ORC, parquet) without any trouble.

We are using HDFS (Hadoop v3.1.2) as the filestore.

I have experimented with different package versions of Avro (e.g. 2.11, lower) which either raises the same error or fails to load entirely due to incompatibility. This error occurs with all of Python, Scala (using shell or spark-submit) and Java (using spark-submit).

There appears to be an Open Issue on apache.org JIRA for this, but this is a year old now without any resolution. I've bumped that issue, but also wondering if the community had a fix? Any help much appreciated.

Does this answer your question? Resolving dependency problems in Apache Spark — user10938362
Open the Spark UI and look at the job classpath. Is the jar correctly added? — OneCricketeer
@user10938362 have tried the suggestions from that post, thanks, but still getting same error. — Thomas Humphries
@cricket_007 yes the avro jars are reported as added successfully, and my app.jar launches and outputs fine - up until avro write command... — Thomas Humphries
Did you manage to create an uber jar? Also, is there a specific reason you want to use Avro rather than Parquet or ORC? — OneCricketeer

Gábor Bakos Gábor Bakos · Accepted Answer · 2020-02-19T06:19:41

Based on a comment in the linked error, you should specify avro with at least 1.8.0 version, something like this:

spark-submit --packages org.apache.spark:spark-avro_2.12:2.4.3,org.apache.avro:avro:1.9.2 ...

(You might want to try with the other order too.)

Spark Avro throws exception on file write: NoSuchMethodError

3 Answers