Issue with write from Databricks to Azure cosmos DB

Question

I am trying to write data from Spark (using Databricks) to Azure Cosmos DB(Mongo DB). There are no errors when executing notebook but i am getting below error when querying the collection.

I have used jar from databricks website azure-cosmosdb-spark_2.4.0_2.11-2.1.2-uber.jar. My versions is 6.5 (includes Apache Spark 2.4.5, Scala 2.11)

import org.joda.time.format._
import com.microsoft.azure.cosmosdb.spark.schema._
import com.microsoft.azure.cosmosdb.spark.CosmosDBSpark
import com.microsoft.azure.cosmosdb.spark.config.Config
import org.apache.spark.sql.functions._
val configMap = Map(
  "Endpoint" -> "https://******.documents.azure.com:***/",
  "Masterkey" -> "****==",
  "Database" -> "dbname",
  "Collection" -> "collectionname"
  )
val config = Config(configMap)
val df = spark.sql("select id,name,address from emp_db.employee")
CosmosDBSpark.save(df, config)

when i query the collection i get below response

Error: error: {
    "_t" : "OKMongoResponse",
    "ok" : 0,
    "code" : 1,
    "errmsg" : "Unknown server error occurred when processing this request.",
    "$err" : "Unknown server error occurred when processing this request."
}

Any help would be much appreciated. Thank you!!!

simon_dmorias simon_dmorias · Accepted Answer · 2020-06-14T20:50:44

That error suggests you are using CosmosDB with the MongoDB api.

The spark connector for CosmosDB only supports it when using the SQL api.

Instead you should use the MongoDB connector.

https://docs.microsoft.com/en-us/azure/cosmos-db/spark-connector

Use this instead https://docs.mongodb.com/spark-connector/master/

Issue with write from Databricks to Azure cosmos DB

1 Answers