Data Factory Pipeline failed with various errors

Question

I set up an Azure Data Factory with a MongoDB as a source and a Delta Lake Storage Gen2. All connections (to the source and the target destination) were successfully checked. But the execution of the pipeline failed with the following errors:

*ADLS Gen2 operation failed for: Operation returned an invalid status code 'Forbidden'. Account: 'xxxx'. FileSystem: 'marketing'. Path: 'output/users.txt'. ErrorCode: 'AuthorizationPermissionMismatch'

"errorCode": "2200", "message": "Failure happened on 'Source' side. ErrorCode=UserErrorTypeInSchemaTableNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to get the type from schema table. This could be caused by missing Sql Server System CLR

"errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorSchemaMappingCannotInferSinkColumnType,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Data type of column 'xx' can't be inferred from 1st row of data, please specify its data type in mappings of copy activity or structure of DataSet *

Any ideas how I can fix those errors?

Answer 1

The error "*ADLS Gen2 operation failed for: Operation returned an invalid status code 'Forbidden'. Account: 'xxxx'. FileSystem: 'marketing'. Path: 'output/users.txt'. ErrorCode: 'AuthorizationPermissionMismatch'"

points to the fact the pipeline does not have the required permission on the ADLS Gen2 . Please go through this doc : https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control.

Please beware that in GEN2 the permissions are more granular and so when to check connections on the ADF UI that mey not catch that .

Let me know how it goes .

Thanks Himanshu