Azure Datalake Analytics U-SQL with Azure Datalake Storage Gen 2

Question

Question : what is the path forward for using ADLA (U-SQL) with ADLS(Gen2) ?

I have been running Azure Data lake Analytics (U-SQL) jobs via Azure Data factory (ADF v2) with Azure Data lake Store Generation 1 for quite a while now in East US2

I was planning to have another instance deployed to cater Canadian clients and wanted to setup Azure Data lake Store Generation 1

What I tried :

I was not able to create an Azure Datalake Storage Gen 1 account in Central Canada (or any Canadian region for that matter)
I tried to move to Azure Datalake Storage Gen2 but then ran into an issue where Azure Data Factory - U-SQL activity could not be linked with Gen2 Storage linked service to pick up U-SQL script

I stumbled upon multiple links about this topic :

which essentially say that U-SQL / ADLA won't be supporting ADLS Gen2

I am a bit confused since there is no official documentation on ADLA's direction

As I understand it, there is no forward direction for ADLA. Microsoft will continue to support it (at least for the time being), but I wouldn't expect any new product development. Unfortunately, this includes U-SQL as well, so the path forward most likely involves a migration to Synapse or some other Spark environment. — Joel Cochran

1_1 1_1 · Accepted Answer · 2020-08-03T09:16:36

Update:

This is the structure of my u-sql activity. It can work and process successfully:(You can try to create a new json of u-sql activity to replace your u-sql activity.)

{
    "name": "pipeline4",
    "properties": {
        "activities": [
            {
                "name": "U-SQL1",
                "type": "DataLakeAnalyticsU-SQL",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "scriptPath": "test1/u-sql.txt",
                    "scriptLinkedService": {
                        "referenceName": "LinkTo0730",
                        "type": "LinkedServiceReference"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureDataLakeAnalytics1",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": []
    }
}

Original Answer:

I was not able to create an Azure Datalake Storage Gen 1 account in Central Canada (or any Canadian region for that matter)

On my side, I also cannot create datalake gen1 on region Central Canada. This is the limit of my subscription. But you can have a check of the resource manager on your side, maybe you can.(Azure data lake gen1 is 'Microsoft.DataLakeStore')

Resource Manager is supported in all regions, but the resources you deploy might not be supported in all regions. In addition, there may be limitations on your subscription that prevent you from using some regions that support the resource. The resource explorer displays valid locations for the resource type.

Please have a check of this document:

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types

I tried to move to Azure Datalake Storage Gen2 but then ran into an issue where Azure Data Factory - U-SQL activity could not be linked with Gen2 Storage linked service to pick up U-SQL script

On my side, seems it is reading the u-sql script in gen2, did you get some error?

Azure Datalake Analytics U-SQL with Azure Datalake Storage Gen 2

What I tried :

1 Answers