Spark Scala Write dataframe to MongoDB

Question

I am attempting to write my transformed data frame into MongoDB using this as a guide

https://docs.mongodb.com/spark-connector/master/scala/streaming/

So far, my reading of data frame from MongoDB works perfectly. As shown below.

  val mongoURI = "mongodb://000.000.000.000:27017"
  val Conf = makeMongoURI(mongoURI,"blog","articles")
  val readConfigintegra: ReadConfig = ReadConfig(Map("uri" -> Conf))


  val sparkSess = SparkSession.builder()
    .master("local")
    .appName("MongoSparkConnectorIntro")
    .config("spark.mongodb.output.uri", "mongodb://000.000.000.000:27017/blog.vectors")
    .getOrCreate()



  // Uses the ReadConfig
  val df3 = sparkSess.sqlContext.loadFromMongoDB(ReadConfig(Map("uri" -> "mongodb://000.000.000.000:27017/blog.articles")))

However, writing this data frame to MongoDB seems to prove more difficult.

 //reads data from mongo and does some transformations
    val data = read_mongo()
    data.show(20,false)
    data.write.mode("append").mongo()

For the last line, I receive the following error.

Exception in thread "main" java.lang.IllegalArgumentException: Missing database name. Set via the 'spark.mongodb.output.uri' or 'spark.mongodb.output.database' property

This seems confusing to me as I set this within my spark Session in the code blocks above.

 val sparkSess = SparkSession.builder()
    .master("local")
    .appName("MongoSparkConnectorIntro")
    .config("spark.mongodb.output.uri", "mongodb://000.000.000.000:27017/blog.vectors")
    .getOrCreate()

Can you spot anything I'm doing wrong?

Sajeed Sajeed · Accepted Answer · 2020-08-08T21:53:39

My answer is pretty much parallels how I read it but uses writeConfig instead.

data.saveToMongoDB(WriteConfig(Map("uri" -> "mongodb://000.000.000.000:27017/blog.vectors")))

Spark Scala Write dataframe to MongoDB

1 Answers