Azure Table storage to Azure Table storage using Azure Data Factory using DateTime Parameterized query

1
votes

I have an Azure Table storage where a few records are added every day (usually 3-5). There are days when no records can be added, so the volume is very low. Here is the structure of table with the pre-defined PartitionKey, RowKey, and Timestamp columns:

Screenshot showing Data in Azure Table

I need to query this table from the Azure Data Factory for the previous day's records. So for the example data shown below, I could be querying for 2019-04-25 data on 2019-04-26. The reason being, one day's staleness does not make a difference and that way, I don't have to worry about 'Watermarks' etc. I simply query for the data for the previous day and copy it to an identical Azure Table in Azure Data Lake Storage Gen 2.

I know that I need to specify a parameterized query based on the 'Timestamp' column for the previous day but do not know how to specify it.

Please advise.

1
parameter-passingazure-table-storageazure-data-factory

1 Answers

0
votes

You could set query sql in the copy activity table storage source.For your needs,it should be like:

time gt datetime'2019-04-25T00:00:00' and time le datetime'2019-04-2T00:00:00'

My sample data as below:

enter image description here

Preview data as below:

enter image description here

Pls see some examples in this doc: https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-table-storage#azuretablesourcequery-examples.