Loess scatterplot timeseries errors

Question

I am attempting to create a loess curve for a subset of time series. All subsets appear to have similar issues when applying loess so the problem may be in my df but I am not sure how to address it.

Data is available here: https://dl.dropbox.com/s/zy6b5mjcu7uteof/data_all_PAR_max.csv?dl=0

This function is part of a larger function so some values are passed on and are define here to help reproduce the errors:

sumfile <- read.csv('https://dl.dropbox.com/s/zy6b5mjcu7uteof/data_all_PAR_max.csv')
codename = "EXEM"
descriptor = "max"
radtype = "PAR"
totYrs = c(1997:2015)
ylbl = expression("PAR " ~ (mu ~ mol ~ photons ~ m^{-2} ~ s^{-1}))
clr = "blue"

group <- melt(sumfile,  id.vars = 'date', variable.name = 'series')
setnames(group, old = c('date','series','value'), new = c('Date','Year',radtype))
group$Date <- as.Date(group$Date)
# group <- na.omit(group) # uncommenting resolves error!
o <- order(group$Date)
lo <- loess(PAR ~ as.numeric(Date), span = 0.25, data=group)

plot(group$Date,group$PAR,pch=19,cex=0.1, col=clr,
     xlab ="Date",
     ylab = ylbl,
    main = paste('Loess curve for', descriptor, radtype, 'from', min(totYrs), 'to',
              max(totYrs), '\nmeasured at', codename, 'met',sep=' '))
lines(group$Date[o], lo$fitted[o], col='red', lwd=1)

Replacing lines for points gives a better picture of the error

points(group$Date[o], lo$fitted[o], col='red', lwd=1)

The plot should look something like this:

loessErrorPlot

The phantom points seem to be an artifact of NA's in the data set.

I created a loop to examine each individual year which uncovered more errors.

for (i in totYrs) {
  tryCatch({         
    yval <- paste(radtype, i, descriptor,sep='_')

    sumfile$date <- as.Date(sumfile$date)
    lo_ <- eval(parse(text = paste("loess(", yval, "~ as.numeric(date),
                                 span = 0.25, data=sumfile)")))
    oo <- order(sumfile$date)
    plot(sumfile$date, eval(parse(text = paste("sumfile$",yval))),
         pch=19,cex=0.1, col=clr,
         xlab ="Date",
         ylab = ylbl,
         main = paste('Loess curve for', descriptor, radtype, 'measured at\n',
                      codename, 'met during', i, '/', i+1, 'field season',sep=' '))
    lines(sumfile$date[oo], lo_$fitted[oo], col='red', lwd=1)  
  }, error=function(e){print("One or more years was not plotted because there was no data")})
}

The loop creates a plot for each year and illustrates how the curve smoothing appears to work for some years but not others.

Setting loess(y ~ x, na.action=na.exclude) does not appear to have any effect on the final results. Adding group <- na.omit(group) to the melted df prior to loess() resolves the errors for that data frame but the problem seems to persist when examining individual years. Here is one example:

sumfile$date <- as.Date(sumfile$date)
no_na <- na.omit(subset(sumfile, select=c(date,PAR_2013_max)))
lo13 <- loess(PAR_2013_max ~ as.numeric(date), span = 0.25, data=no_na)
oo <- order(sumfile$date)
plot(sumfile$date, sumfile$PAR_2013_max)
lines(sumfile$date[oo], lo13$fitted[oo], col='red', lwd=1)

Any help identifying a solution to plotting the yearly curves is greatly appreciated.

rosscova rosscova · Accepted Answer · 2016-05-26T03:58:38

I think you're right about where those "phantom" points are coming from. You should be able to omit any NA values, here's an example using ggplot2:

plot <- ggplot( data = group, aes( x = Date, y = PAR ) ) +
    geom_point( na.rm = TRUE ) +
    geom_smooth( method = "loess", span = 0.2, na.rm = TRUE )
plot

That gives a nice curve for me. Note the "span" variable determines the level of smoothing by the loess function, so adjust it to taste, or leave it out to accept a default smooth.

This does also work with each year individually:

plot <- ggplot( data = sumfile, aes( x = as.Date( date ), y = PAR_2013_max ) ) +
    geom_point( na.rm = TRUE ) +
    geom_smooth( method = "loess", span = 0.2, na.rm = TRUE )
plot

Loess scatterplot timeseries errors

1 Answers