Why does Azure Data Factory use AppendFile instead of PutBlob to write files to blob storage container?

0
votes

I've got a Log Analytics Workspace stood up and created some (blob) Diagnostic Settings for some Azure Storage Accounts. Now I'm analyzing the blob traffic.

Seems that the various methods of getting blobs into blob storage (Azure Data Factory (ADF), Azure Storage Explorer (ASE), Python SDK, etc.) use different API methods out-of-the-box.

Example:

  • I'm seeing that all files uploaded using ADF are using the AppendFile OperationName

enter image description here

  • Whereas all files uploaded using ASE use the PutBlob OperationName

Question:

  • Is AppendFile a method that was explicitly set on the ADF side?
  • Or is it the default when writing blobs from ADF to blob storage?

Also, I don't see AppendFile listed as a method in the Blob Service REST API docs.

  • Is this just an ADF thing or is the method missing from the docs?
2
azure-data-factoryazure-blob-storagerestapi
Which activity did you use in ADF? – Joy Wang-MSFT
Its a different team which writes to the container with ADF. I have no idea. – ericOnline
Looking through the logs a bit more, I think ADF is using AzureDataFactoryCopy. In the StorageBlobLogs table, I'm seeing UserAgentHeader: AzureDataFactoryCopy. Is that what you're looking for? – ericOnline

2 Answers

2
votes

I can reproduce your issue on my side, I suppose your storage account is a Data Lake Storage Gen2 account i.e. Hierarchical namespace was enabled like below.

enter image description here

When you use the copy activity in ADF to copy blobs between containers(also named filesystem in datalake gen2), it will call the Data Lake Storage Gen2 REST API instead of the normal Storage REST API - Path - Update, if you look into the Uri parameter in the log, you will find its format is like below.

enter image description here

It is the same as the REST API sample, because it essentially calls this API.

enter image description here

Even if it is a Data Lake Storage Gen2 account, the normal Storage REST API also works for it, so if you use something like Azure Storage Explorer, it essentially calls the normal Storage REST API directly i.e. Put blob.

0
votes

If you are using the Azure Data Factory Flow and the primary id is the same, it usually does the AppendFile operation