Azure ADF Custom Activity With Existing ADLS Linked Service

Question

I'm trying to write data to ADLS using a custom ADF activity authored in visual studio using the azure SDKs. The pipeline I create will be deployed into an existing ADF that already has an ADLS and batch linked service defined. My question is this: In the custom activity method, how do I use the existing ADLS linked service to write data?

This question is similar to this: ADF - C # Custom Activity but I don't think I can simply lift a connection string because the linked service is using my SPN and as a developer I'm not permitted to know the SPN key.

Here's the link service.

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "hubName": "****-d-****-adf_hub",
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "adl://***************.azuredatalakestore.net/",
            "servicePrincipalId": "*****-**-****-****-************",
            "servicePrincipalKey": "**********",
            "tenant": "********.com"
        }
    }
}

In the custom activity I get hold of the linked service:

var output = activity.Outputs[0];
var inputLinkedService = linkedServices.First(linkedService => linkedService.Name == _ADLSLinkedServiceName).Properties.TypeProperties as AzureDataLakeStoreLinkedService;

-- What now?

Paul Andrew Paul Andrew · Accepted Answer · 2017-03-01T16:15:07

You can only get information like the file path and linked service account name using the method calls above.

The problem you'll have here is that the ADF custom activity runs in the Azure Batch Compute service as a separate application. It doesn't understand the linked service authentication (session token) already set up in ADF and you won't be able to get to it in the C#.

The solution is to register the custom activity as a Azure App within the context of the Azure Active Directory domain!.... Yes, seriously!

Once done you can grant the app access to read/write from the data lake store.

Check out my blog. I wrote a post on exactly this:

https://www.purplefrogsystems.com/paul/2016/12/azure-data-lake-authentication-from-azure-data-factory/

Warning: the PowerShell cmdlets have be superceeded because you can now actually register the app in the portal blades.

Azure ADF Custom Activity With Existing ADLS Linked Service

1 Answers