I have a simple pipeline , with copydata. I am trying to pull data from sql to data lake . I have created two datasets one for sql and other for datalake.
Data lake schema are defined in dataset are overridden when we run the pipeline and columns have string data type.
I tried to change the schema data type here and after running the pipelines they are overridden in datasets.
When we copy data from SQL to Azure Data Lake Storage, no matter what the column data type is in SQL Source dataset, it all will change to String in Data Lake Storage Sink dataset. Sink dataset will create a flat file(txt, json and so on) to store the data.
For example, when you copy data from a csv file as Source dataset, import the schema from source file, all the column data types are String. We only can change these String column data type during SQL Sink dataset schema mapping.
We can not not change the data types for datesets when using sink as Data lake Storage.
Hope this helps.
