Only refresh new files in Excel Power Query for data from folder?

Question

We receive new data pretty much everyday, which is saved in a local folder. I've created a data model in Power Query from these data to process and create visualization, which whenever refreshed will run through the whole folder again. As you might expect, this can be painfully slow, especially since my model handle quite a bit of data processing.

Is there anyway I can configure Excel Power Query so whenever refrshed, it only loads the file not previously loaded before, leaving everything else as-is?

Thank you in advance,

horseyride horseyride · Accepted Answer · 2019-01-08T16:29:48

One way would be to just pull the most recent file

let Source = Folder.Files("C:\Temp"),
#"Sorted Rows" = Table.Sort(Source,{{"Date modified", Order.Ascending}}),
Path = #"Sorted Rows"{0}[Folder Path] & #"Sorted Rows"{0}[Name]
in Path

Another way is to keep a cumulative table of file names you are processing and filter those out of the directory file list using a merge. For tips on creating an excel history table see Excel history table for one of the better explanations of this

Only refresh new files in Excel Power Query for data from folder?

1 Answers