U-SQL User Defined Function in Azure Data Factory

0
votes

I'm currently using Azure Data Factory for an ETL job and in the end I want to start a U-SQL job. I've created my datalake.usql script and the UDF's the script uses are in the datalake.usql.cs file, the same structure a U-SQL project has in Visual Studio (which is where I developed the U-SQL job and succesfully ran it).

After that, I uploaded them both to Azure Blob Storage and set up the U-SQL step in Azure Data Factory to use the U-SQL script, but it doesn't see the datalake.usql.cs with my UDF's.

How can I do this?

1
azureazure-data-factoryazure-data-lakeu-sql
hi,what do you mean it doesn't see the datalake.usql.cs with my UDF's? You can't configure the U-SQL activity in pipeline? – Jay Gong
I can configure it. I ran some tests with basic stuff and it worked. But when I try to run the script with my actual code, the compiler doesn't find the .cs file which contains my UDF's. I get the error "E_CSC_USER_INVALIDCSHARP_0103, Error Message: C# error CS0103: The name 'MyNamespace' does not exist in the current context. Error Id: E_CSC_USER_INVALIDCSHARP_0103, Error Message: C# error CS0103: The name 'MyNamespace' does not exist in the current context". Is there some parameter I should configure? – Paulo Eduardo Dias da Silva

1 Answers

0
votes

I figured this one out so I'll post the answer here if someone else has this problem in the future.

When you're developing locally, you have a script.usql and a script.usql.cs file. When you run it, Visual Studio does all the heavy lifting, compiles it all in a usable way and your script runs. But when you're trying to run a script from Azure Data Factory, you can't perform that compilation on the fly, like Visual Studio does.

The solution to this problem is to make an Assembly, a .dll file, from your script.usql.cs, upload it to the data storage you're using and register it in the U-SQL Catalog. Then, you can reference this assembly and use it normally, as you would on your local machine.

All the steps needed for this are presented in these short guides:

https://saveenr.gitbooks.io/usql-tutorial/content/usql-catalog/intro.html

https://saveenr.gitbooks.io/usql-tutorial/content/usql-catalog/usql-databases.html

https://saveenr.gitbooks.io/usql-tutorial/content/usql-catalog/assemblies.html

https://www.c-sharpcorner.com/UploadFile/1e050f/creating-and-using-dll-class-library-in-C-Sharp/