With Azure Data Factory Copy Activity, is it possible to output the current SliceStart as a property on the destination Table Storage entities?

3
votes

I'm using Azure Data Factory's Copy Activity to pick-up delimited Text files in Blob storage from daily folders. The columns are mapped and copied to Table Storage entities. This all works great.

I would like each set of entities to be partitioned with the correct data slice (SliceStart), ideally as part of a composite Partition Key (SliceStart plus some other property appended) or failing that, a property on the Table Entity itself.

This is so the next step in the pipeline (a .NET Custom Activity) can grab the correct set of daily data. I know I can pass the current SliceStart as an extended property to the .NET Custom Activity, but how would this Activity then use the SliceStart value to query the Table Storage data to isolate just the matching day's entities?

I can't use the TimeStamp reliably due to the possibility of slices getting queued or delayed and then multiple slices being processed together on the same day. In this case, they would all have the same TimeStamp and so I can't differentiate the various days.

1
azureazure-data-factory
If I understand you correctly! ... I'm not sure that you can control the definition of the Partition Key during the Copy. You could raise this against Azure Feedback to get it considered. – richaux
See related answer here: stackoverflow.com/questions/35324658/… – rawsharklives

1 Answers

0
votes

With ADF V2, you could be possible to achieve this. Use expressions for azureTableDefaultPartitionKeyValue properties. And pass trigger().scheduledTime to it if you are using schedule trigger. You can also use tumbling window trigger.