R:Check existence of "today's" file and if it doesn't exist, download it

Question

I'm trying to check to see if a user has an up-to-date file version of London's Covid prevalence data in their working directory and if not, download it from here:

fileURL <-"https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=region;areaName=London&structure=%7B%22areaType%22:%22areaType%22,%22areaName%22:%22areaName%22,%22areaCode%22:%22areaCode%22,%22date%22:%22date%22,%22newCasesBySpecimenDateRollingSum%22:%22newCasesBySpecimenDateRollingSum%22,%22newCasesBySpecimenDateRollingRate%22:%22newCasesBySpecimenDateRollingRate%22%7D&format=csv"

Pasting that URL into the browser downloads the csv file. using download.file(URL, "data_.csv") creates junk. Why?

So far I have:

library(data.table)

#Look for COVID file starting with "data_"
destfile <- list.files(pattern = "data_") 
#Find file date
fileDate<-file.info(destfile)$ctime %>%as.Date()

   if(!file.exists(destfile) | fileDate != today()){
     res <- tryCatch(download.file(url = fileURL,
                               destfile = paste0("data_",today(),".csv"),
                               method = "auto"),
                 error=function(e) 1)
     if(res!=1) COVIDdata<-data.table::fread(destfile) #This doesn't read the file
   }

The function always downloads a file regardless of the date on it but it saves it an unreadable format. I've resorted to downloading the file every time as follows.

COVIDdata <- data.table::fread(fileURL)

The junk file that gets downloaded is this:

The condition in your if-clause obviously yields always TRUE - I guess, the reason is, that fileDate is a vector of dates in which you seem to have always older dates. max(fileDate) != today() might work. — Wolfgang Arnold
Thanks for such a quick reply. I think the biggest problem is why the download.file function doesn't save it as a csv file. I've attached a screenshot of what it looks like when I open it. — HCAI
@jangorecki it's a download.file issue, maybe the response headers aren't read correctly.... — Abdessabour Mtk

Abdessabour Mtk Abdessabour Mtk · Accepted Answer · 2020-11-06T14:23:28

I think this an issue with encoding the result of download.file, one way could be to use fread to get the data then write it with fwrite:

#Look for COVID file starting with "data_"
destfile <- list.files(pattern = "data_") 
#Find file date
fileDate <- file.info(destfile)$ctime %>% as.Date()
#>[1] "2020-11-06" "2020-11-06" "2020-11-06"

if(!length(destfile) | max(fileDate) != today()){
  COVIDdata <- fread(fileURL)
  fwrite(COVIDdata, file = paste0("data_",today(),".csv"))
}

R:Check existence of "today's" file and if it doesn't exist, download it

1 Answers