Preserve filename on sink when doing binary file copy with Azure Data Factory v2

0
votes

This might be really simple but I can't figure out how to do it. I'm using Azure Data Factory v2 to copy one Excel file from SharePoint Online to Azure Blob Storage using HTTP connector and binary file format. I am following this tutorial https://docs.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list#copy-file-from-sharepoint-online to get the bearer token and pull the file to be copied from SharePoint Online. I'm using two activities:

  1. GetBearerToken - gets bearer token for SharePoint Online
  2. CopyFromSPOToAzureStorage - using the bearer token from 1) copy an Excel file in binary format from a SharePoint Online folder into Azure Storage in binary format

Data Factory activities

I'm able to read and copy a file Dummy.xlsx but I'm having a gripe with that - the file ends up stripped of its original name and extension onto Azure Storage:

Copied file on sink

I am able to open the file in Excel, see the columns and rows and it looks fine in general. However, I would like to preserve the filename and extension and copy it as-is on the sink.

1
azure-data-factoryazure-data-factory-2
Hey @Georgi, Can you please show the properties of your input and output dataset? Because the output dataset is what decides what file name that should be copied. We too use a similar technique of transferring file wherein we have parameterized both the input dataset and the output dataset thereby controlling and preserving the filenames as needed. – Nandan
Hey @Georgi, I have provided the sample inputs. Happy to provide the detailed scope if needed :) – Nandan
Hello @Nandan. I haven't parameterized the input and output datasets as I don't know how. This is my input binary dataset whereby I have specified the following as Base URL: mysharepointsite.sharepoint.com/sites/mysite/_api/web/… Documents/Dummy.xlsx')/$value: ibb.co/qCxdRTZ And this is my output binary dataset to Azure Storage: ibb.co/NyN2qnr ibb.co/gz0VLR6 Please let me know in case you need any other information. Have a great day! – MrG
Hey @Georgi, Were you able to achieve it? docs.microsoft.com/en-us/answers/questions/221580/… A similar thread with some additional explanation if needed :) – Nandan
Hello @Nandan. I wasn't able to achieve it. I asked a colleague who is a data engineer for help. It turned out setting up the Data Factory pipeline was his responsibility and he made quick work of it. Thank you for the help though, I very much appreciated the detailed instructions! All the best! – MrG

1 Answers

1
votes

Just a sample scenario :

  1. Get all the file path and file name details :

enter image description here

  1. Parameterize the data set : a)Input/source dataset:

enter image description here

b) Output dataset :

enter image description here

So the filename is preserved as everything is parameterized