Spark Streaming checkpoint to remote hdfs

Question

I am trying to checkpoint my spark streaming context to hdfs to handle a failure at some point of my application. I have my HDFS setup on a separate cluster and spark running on a separate standalone server. To do this, I am using :

ssc.checkpoint(directory: String)

This gives me : org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE when I try with directory as "hdfs://hostname:port/pathToFolder"

How can I checkpoint to a remote hdfs path? Is it possible to add credentials to the string uri? I tried googling, but no help so far.

Thanks and appreciate any help!

Answer 1

You can provide the credentials by using:

hdfs://username:password@hostname:port/pathToFolder