How to trigger a message from Azure Service bus queue to Azure Data Lake?

1
votes

I have written C# program that sends a message to Azure service bus queue and this is working Fine. Now I need to pass the messages received at service bus queue to the data lake through Azure functions but as far as i have studied Azure function doesn't support bindings/triggers for data lake. Is there any way to trigger Service bus queue message to Azure Data Lake? Thanks in advance.

2
azureazure-webjobsazure-functionsazure-data-lakeazure-servicebus-queues
Azure Logic Apps currently has the Azure Data Lake connector in preview. Did you already have a look at that? – Steven Van Eycken
@StevenVanEycken Nope I just checked it but generally asking does it provide support for Service bus queue? Is there any useful documentation or tutorial you know related to it then kindly share its link. – Nasima Akhtar
Logic Apps also provide a connector to perform actions on Azure Service Bus. Documentation for these connectors can be found at docs.microsoft.com/en-us/azure/connectors/…. You can create a Logic App which is triggered by new messages on your queue. Documentation for the Azure Data Lake Connector does not yet seem to be available. – Steven Van Eycken
Is there any way to trigger Service bus queue message to Azure Data Lake? Does it means that according to service bus queue message then do action for Azure data lake such as uploading or downloading file. If it that case, we can do that with azure function service bus trigger. If we want to refernce other SDK we could #r. – Tom Sun - MSFT

2 Answers

3
votes

You can setup a Azure Functions ServiceBusTrigger and use Azure Data Lake .Net SDK in the function code. Follow instructions here for using external Nuget Packages. It might be easier to use logic app connector if it's available as that would take care of authenticating with AAD instead of using SDK directly.

1
votes

I have successfully submitted the data from service bus queue to data lake, the idea is to use webjobs which has a Function class that got triggered whenever our service bus queue recieves any data. Function's main method contain the following code:

 JobHostConfiguration config = new JobHostConfiguration();
 config.UseServiceBus();
 JobHost host = new JobHost(config);
 DataLakeClass.Authentication(); // this is basically a function where my client got authenticated and further able to create csv file at data lake and append data to it.
 host.RunAndBlock();

Note: For webjobs you need to create webapp on Azure and then create webjobs and then simply publish your code. For detail help you can ask more questions.