Deploying USQL project

2
votes

I am new to data lake analytics and using USQL.

I am currently setting up data factory pipeline which would replace an existing SSIS workflow. The data factory pipeline would essentially

  1. Extract data transactional database into ADLS
  2. Transform raw entities using USQL
  3. Load the data into SSAS using custom activity

Question

I have a USQL project set up and wanted if there was a standard way of deploying them to ADLA other than just uploading the scripts to a folder in the store.

1
azure-data-lakeazure-data-factoryu-sql

1 Answers

2
votes

Great question!

I'm not sure about a standard way, or even a way that might be considered best practice yet. But I use all of the tools you mention to perform very similar tasks.

To try and answer your question: What I do is create the U-SQL scripts as stored procedures within the logical ADLA database. In the VS USQL project I have 1 script per stored proc. The ADF activities then call the proc name. This gives you the right level of disconnection between services and also means you don't need additional blob storage for USQL files.

In my VS solution I often also have a PowerShell project to help manage things. Specifically one what takes all my 'usp_' U-SQL scripts to create one big DDL style thing that can be deployed to the logical ADLA database.

The PowerShell then does the deployment for me using the submit job cmdlet. Example below.

Submit-AzureRmDataLakeAnalyticsJob `
        -Name $JobName `
        -AccountName $DLAnalytics `
        –Script $USQLProcDeployAll `
        -DegreeOfParallelism $DLAnalyticsDoP

Hope this gives you a steer. I also accept that these tools are still fairly new. So open to other suggestions.

Cheers