Accessing Azure Data Lake Storage gen2 from Scala

Question

I am able to connect to ADLS gen2 from a notebook running on Azure Databricks but am unable to connect from a job using a jar. I used the same settings as I did in the notebook, save for the use of dbutils.

I used the same setting for Spark conf from the notebook in the Scala code.

Notebook:

spark.conf.set(
"fs.azure.account.key.xxxx.dfs.core.windows.net",
dbutils.secrets.get(scope = "kv-secrets", key = "xxxxxx"))

spark.conf.set
("fs.azure.createRemoteFileSystemDuringInitialization", "true")

spark.conf.set
("fs.azure.createRemoteFileSystemDuringInitialization", "false")

val rdd = sqlContext.read.format
("csv").option("header", 
"true").load(
"abfss://[email protected]/test/sample.csv")
// Convert rdd to data frame using toDF; the following import is 
//required to use toDF function.
val df: DataFrame = rdd.toDF()
// Write file to parquet
df.write.parquet
("abfss://[email protected]/test/Sales.parquet")

Scala code:

val sc = SparkContext.getOrCreate()
val spark = SparkSession.builder().getOrCreate()
sc.getConf.setAppName("Test")

sc.getConf.set("fs.azure.account.key.xxxx.dfs.core.windows.net",
"<actual key>")

sc.getConf.set("fs.azure.account.auth.type", "OAuth")

sc.getConf.set("fs.azure.account.oauth.provider.type",
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

sc.getConf.set("fs.azure.account.oauth2.client.id", "<app id>")

sc.getConf.set("fs.azure.account.oauth2.client.secret", "<app password>")

sc.getConf.set("fs.azure.account.oauth2.client.endpoint",
  "https://login.microsoftonline.com/<tenant id>/oauth2/token")

sc.getConf.set
("fs.azure.createRemoteFileSystemDuringInitialization", "false")

val sqlContext = spark.sqlContext
val rdd = sqlContext.read.format
("csv").option("header", 
"true").load
("abfss://[email protected]/test/sample.csv")
// Convert rdd to data frame using toDF; the following import is 
//required to use toDF function.
val df: DataFrame = rdd.toDF()
println(df.count())
// Write file to parquet

df.write.parquet
("abfss://[email protected]/test/Sales.parquet")

I expected the parquet file to get written. Instead I get the following error: 19/04/20 13:58:40 ERROR Uncaught throwable from user code: Configuration property xxxx.dfs.core.windows.net not found. at shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:385) at shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:802) at shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:133) at shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:103) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)

user2130332 user2130332 · Accepted Answer · 2019-04-21T00:31:56

Never mind, silly mistake. It should be:

val sc = SparkContext.getOrCreate()
val spark = SparkSession.builder().getOrCreate()
sc.getConf.setAppName("Test")

spark.conf.set("fs.azure.account.key.xxxx.dfs.core.windows.net",
"<actual key>")

spark.conf.set("fs.azure.account.auth.type", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type",
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id", "<app id>")

spark.conf.set("fs.azure.account.oauth2.client.secret", "<app password>")

spark.conf.set("fs.azure.account.oauth2.client.endpoint",
  "https://login.microsoftonline.com/<tenant id>/oauth2/token")

spark.conf.set
("fs.azure.createRemoteFileSystemDuringInitialization", "false")

Accessing Azure Data Lake Storage gen2 from Scala

1 Answers