I'm new to Azure Data Lake and big data in general and I apologize if my question seems stupid.
I've been looking at ADL and ADLA to develop a cold path data storage. I've an Azure Stream Analytics query that outputs to Power Bi for real time visualizations and another query which stores data in .CSV format in the data lake.
I've created a VS project where I've created a database, schema and tables corresponding to the csv files and one script extracts the data from the CSV file and copies it in the table to give my data some structure.
My question is that if the data keeps on storing in the csv files where the folder structure defines when the data has arrived, how do I update my tables with the new data. Do I drop the table and begin again, this is not a viable solution I believe?
I've scripts which I've to run in order to create a DB, schema, extracting the data and populate the tables. Surely, I can't run all the scripts whenever new data arrives.
Note: I want to point out that the databases and tables are within ADLA U-SQL Databases.
