Data Factory V2 Error Code 2200 on Data Lake Sink

Question

Im using the following as a workshop https://www.youtube.com/watch? v=IAqJ6nCDtGc

I have some sales data in an Azure SQL Database (Rather than an on premise database)

And I have a data lake Gen Storage 1 I've successfully set up the Connectors and Data sets. Each Connector has tested ok

Do create the Azure Data Lake created an app registration to get the information I needed for the principal ID and the Principal Key

Ive created the pipeline with a copy activity as per the above video

When I run it I get the following

{ "errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The request to 'Unknown' failed and the status code is 'BadRequest', request id is ''. \r\nBad Request\r\n\r\n

Bad Request - Invalid URL

\r\n

HTTP Error 400. The request URL is invalid.

\r\n\r\n ,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (400) Bad Request.,Source=System,'", "failureType": "UserError", "target": "Copy Sales data to data lake" }

Ive checked the URL in the data lake connector and this seems fine

"dataLakeStoreUri": "https://.azuredatalakestore.net/webhdfs/v1",

The only other URL I can think of is the one set up when registering the app on sign on URL

https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

If anyone can help it would be greatly appreciated.

Debbie

400 error code means that there is an authentication issue, that means that it is either the target path or the credentials used to connect to ADLS (Sink), are you sure that your app has sufficient rights to access the target path? — dim_user
I've gone through all the set up (Registering an app with the correct permissions) and when I test the connection for both the Source and destination it says successful. I don't know what else to do with this? — Debbie Edwards
Hi Debbie, can you post the JSON code for the CopyActivity and Pipeline and try to reproduce this from my end. — dim_user

MarkD MarkD · Accepted Answer · 2018-12-19T19:39:39

A couple of thoughts.

Can you check you have the account uri set to "https://<your ADL endpoint name>.azuredatalakestore.net/webhdfs/v1"? Above you have ".azuredatalakestore.net/webhdfs/v1" but understand you may have deleted the endpoint in this post for privacy.

On the ADL permissions, have you assigned permissions through the data explorer in ADL? The service principal I believe needs execute permissions on the root folder and all the folders in the required path plus read/write on any subfolder that will be written to. i.e. if I am writing to \foo\bar.txt

\       permissions needed = x
foo     permissions needed = x
bar.txt permissions needed = rw

See here for details.

Finally, folders are case sensitive so check that they are being referenced correctly.

Hope this helps. Mark.

Data Factory V2 Error Code 2200 on Data Lake Sink

Bad Request - Invalid URL

2 Answers