Data Factory Copy Activity met storage failure at source side - AppendBlob was not found

Question

Error: Copy activity met storage operation failure at 'Source' side. Error message from storage execution : Requested value 'AppendBlob' was not found..

I was originally trying to copy a blob to on prem sql and I received the above error. Just for testing I am trying to copy blob to blob and I still received the same error.

I can see the blob in my container. But it seems like azure data factory copy activity cannot access it. Do AppendBlobs work with azure data factory copy activity? Any one else run into this issue, any ideas on how to resolve?

Thanks.

Azure Data Factory JSON definition files:

InputBlob:

{
"name": "InputBlobTest",
"properties": {
    "published": false,
    "type": "AzureBlob",
    "linkedServiceName": "AzureStorageLinkedService",
    "typeProperties": {
        "fileName": "ContractHeader.csv",
        "folderPath": "testcontainer/",
        "format": {
            "type": "TextFormat",
            "rowDelimiter": "\n",
            "columnDelimiter": ","
        }
    },
    "availability": {
        "frequency": "Hour",
        "interval": 1
    },
    "external": true,
    "policy": {
        "externalData": {
            "retryInterval": "00:01:00",
            "retryTimeout": "00:10:00",
            "maximumRetry": 3
        }
    }
}

OuputBlob:

{
"name": "OutputBlobTest",
"properties": {
    "published": false,
    "type": "AzureBlob",
    "linkedServiceName": "AzureStorageLinkedService",
    "typeProperties": {
        "fileName": "Sample.csv",
        "folderPath": "testcontainer/",
        "format": {
            "type": "TextFormat",
            "rowDelimiter": "\n",
            "columnDelimiter": ","
        }
    },
    "availability": {
        "frequency": "Hour",
        "interval": 1
    }
}

Pipeline:

{
"name": "PipelineBlobToBlob",
"properties": {
    "description": "Copy data from a blob to Azure SQL table",
    "activities": [
        {
            "type": "Copy",
            "typeProperties": {
                "source": {
                    "type": "BlobSource",
                    "skipHeaderLineCount": 1
                },
                "sink": {
                    "type": "BlobSink",
                    "writeBatchSize": 0,
                    "writeBatchTimeout": "00:00:00"
                }
            },
            "inputs": [
                {
                    "name": "InputBlobTest"
                }
            ],
            "outputs": [
                {
                    "name": "OutputBlobTest"
                }
            ],
            "policy": {
                "timeout": "01:00:00",
                "concurrency": 1,
                "executionPriorityOrder": "NewestFirst"
            },
            "scheduler": {
                "frequency": "Hour",
                "interval": 1
            },
            "name": "CopyFromBlobToBlob",
            "description": ""
        }
    ],
    "start": "2015-07-12T00:00:00Z",
    "end": "2015-07-12T01:00:00Z",
    "isPaused": false
}

I see that the writeBatchSize and writeBatchTimeout both are 0. Can you try increasing writeBatchSize it to say 1000 and writeBatchTimeout to 00:10:00 — Sandesh
That's interesting because that was the default when I saved because I never defined those properties. Now I am retrying and I can no longer get the append blob input to validate, not sure why that is. It just says "conflict" for the validation error. Also interesting if I change my input data set to point to a BlockBlob it validates and runs fine even with the batch size set at 0 and timeout set to 00:00:00. — JustLogic
Try removing the file name property from your datasets and just define the blob folder path excluding the file name. See if the copy works as expected. — Sandesh
First time I got this error but seems to confirm what I suspected: Copy activity met storage operation failure at 'Source' side. Error message from storage execution : The remote server returned an error: (409) Conflict. The type of a blob in the container is unrecognized by this version. - So I am going to assume this means appendblobs are not supported yet... — JustLogic

JustLogic JustLogic · Accepted Answer · 2016-05-31T17:49:52

Received confirmation, Append blobs are currently not supported with azure data factory copy activity.

Data Factory Copy Activity met storage failure at source side - AppendBlob was not found

1 Answers