Azure Data Factory - MQ data flow

0
votes

I have been tasked to create an Azure Data Factory pipeline that will process messages being generated from an MQ Farm and that are stored in Data Storage in .xml format and then ingest them in a SharePoint Table.

The question is how would your approach be in that scenario to slice the .xml files in smaller pieces? The .xml files are nesting a lot of records in one file (with a valid separator on each record) and I wish to discard some while process the valid ones.

P.S.: For receiving and storing the MQ Farm messages I am using a logic app before Azure Data Factory

1
azureazure-data-factoryazure-logic-appsmqazure-data-factory-pipeline
You are using Logic Apps, MQ, XML files, Sharepoint and now you want to bring ADF into the mix. The technology complexity alarm is going off. - Nick.McDermaid
Well point A is MQ on prem and point B Sharepoint Online. Any suggestions are welcomed - Eos
I assume this is an IBM MQ? What is the sharepoint list being used for? I guess the MQ is sending messages that are XML and that can't be changed. That's quite an impedance mismatch there. Digging around, there doesn't seem to be a simple way to implement this. ADF can't read directly from MQ, and is not very good at shredding XML. It sounds like you have some nasty XML parsing requirements there. - Nick.McDermaid
I think you could go two ways on this. You could probably use event hubs to capture the XML and dump it to blob storage, then use Azure functions to batch apply that to a sharepoint list. Or you might be able to use Azure functions to read straight out of event hubs. It's probably just going to be easier to write some custom C# inside an Azure function to do the hard work rather of shredding the XML and loading it to the SP list, rather than trying to get ADF to do it. - Nick.McDermaid
ADF is great but quickly runs out of steam when you need to do complex stuff like shred XML - Nick.McDermaid

1 Answers

0
votes

OK the solution was more obvious than previously thought... Solved from the logic app designer and saving to blob

enter image description here