Error importing Avro file generated by Event Hubs Archive using Copy Data tool in Azure Data Factory

Question

I'm seeing the following error in the Azure Data Factory 'Copy Data' tool (currently in Preview) in the 'file format settings' step:

Error when processing request: Column: SystemProperties,Location: Source,Format: Avro,The data type 'System.Collections.Generic.Dictionary`2[System.String,System.Object]' is currently not supported by Avro format. activityId: 0aa778ad-b496-46c1-8de4-97b4f4169be2

The source files are in Avro format, generated by Azure Event Hubs Archive. When I select the Avro format choose any of the avro files in the blob I get the above error when the tool attempts to determine the schema. I don't know whether this is a problem with the copy data tool or the format of the file as outputted by Event Hubs Archive.

Has anyone had this problem or any ideas on how i might investigate further?

Peter Pan Peter Pan · Accepted Answer · 2017-03-21T03:34:18

According to the section Exploring the archive and working with Avro of the offical document Azure Event Hubs Archive, the Avro file schema of Event Hubs Archive like below is an Avro complex type named records.

{
    "type":"record",
    "name":"EventData",
    "namespace":"Microsoft.ServiceBus.Messaging",
    "fields":[
                 {"name":"SequenceNumber","type":"long"},
                 {"name":"Offset","type":"string"},
                 {"name":"EnqueuedTimeUtc","type":"string"},
                 {"name":"SystemProperties","type":{"type":"map","values":["long","double","string","bytes"]}},
                 {"name":"Properties","type":{"type":"map","values":["long","double","string","bytes"]}},
                 {"name":"Body","type":["null","bytes"]}
             ]
}

But the records complex type of Avro is not supported by Azure Data Factory, you can refer to the note of the section Specifying AvroFormat of the offical document Move data to and from Azure Blob using Azure Data Factory to know it.

Note the following points:

Complex data types are not supported (records, enums, arrays, maps, unions and fixed).

So you got the error information. If you just want to copy or move the EventHubs Archive from the current Storage acount to another or other destination data stores like FileSystem, you can choose the option Binary Copy to do it, as below.

Or you want to transfer these data come from Event Hubs, per my experience, the way is that using Azure Stream Analytics to read data from Event Hubs to write to data store like Blob Storage, then you can copy data using Data Factory.

Error importing Avro file generated by Event Hubs Archive using Copy Data tool in Azure Data Factory

1 Answers