Azure data factory query

Question

Case:- there is one URL in which I need to select one input from the dropdown, another input from the dropdown to convert some data and one more input to select data in CSV format and then click submit button, the output is generated in form of CSV so I need to automate this process by ADF.

What I have done:-I am using web activity with the copy activity to load URL data using the POST method into the blob. But the output I am getting is full HTTP response of page. However, I have created also parameters for mapping but it didn't work.what would be the solution of getting only these parameters to select data in URL and get only selected values output?

Answer 1

Data Factory is not the best for web scraping, that's not its function in the technology stack.

What I usually do when I have to get data from a website, is an Azure Function activity. You can create an Azure Function in a language of your liking (python, .net, java, powershell, node, etc), do the web scraping and store whatever data you want in a blob storage. Then, call the function from data factory with the Azure Function Activity.

Hope this helped!