Azure Data Factory Copy Activity to Copy to Azure Data Lake Table

2
votes

I need to Copy data incrementally from On-Prem SQL server into Table in Azure Data Lake Store.

But when creating Copy Activity using Azure Portal, in the Destination I only see the folders(No option for Tables).

How can I do scheduled On-prem table to Data Lake Table Syncs?

2
azure-data-lakeazure-data-factory

2 Answers

2
votes

Data Lake Store does not have a notion of tables. It is file storage system (like HDFS). You can however use capabilities such as Hive or Data Lake Analytics on top of your data stored in Data Lake Store to conform your data to a schema. In hive, you can do that using external tables, while in Data Lake Analytics you can run a simple extract script.

I hope this helps!

1
votes

Azure Data Lake Analytics (ADLA) does have the concept of databases which have tables. However they are not currently exposed as a target in Data Factory. I believe it's on the backlog, although I can't find the reference right now.

What you could do is use Data Factory to copy data into Data Lake Store then run a U-SQL script which imports it into the ADLA database.

If you do feel this is an important feature, you can create a request here and vote for it:

https://feedback.azure.com/forums/327234-data-lake

ADLA Databases and tables:

ADLA Databases and tables