Using a tibble to apply linear models by group to new data

Question

Let's say I have two datasets for the same group of irises over two years:

# Create data for reproducible results.
iris.2007 <- iris
iris.2008 <- iris
iris.2008[1:4] <- 2*iris.2008[1:4]  # let's make the 2008 data different

I would like to fit a separate linear model for each species in the 2007 data, which I can do like this:

# First nest by Species.
iris.2007.nested <- iris.2007 %>%
                    group_by(Species) %>%
                    nest()
# Now apply the linear model call by group using the data.
iris.2007.nested <- iris.2007.nested %>%
                    mutate(models = map(data,
                    ~ lm(Petal.Length ~ Petal.Width, data = .)))

When we look at the results, they make sense as a nicely-organized tibble.

head(iris.2007.nested)
# A tibble: 3 × 3
     Species              data   models
      <fctr>            <list>   <list>
1     setosa <tibble [50 × 4]> <S3: lm>
2 versicolor <tibble [50 × 4]> <S3: lm>
3  virginica <tibble [50 × 4]> <S3: lm>

Now let's do the same thing to the 2008 data.

# First nest by species.
iris.2008.nested <- iris.2008 %>%
                    group_by(Species) %>%
                    nest()
# Now apply the linear model call by species using the data.
iris.2008.nested <- iris.2008.nested %>%
                    mutate(models = map(data,
                    ~ lm(Petal.Length ~ Petal.Width, data = .)))

Again, we end up with a nice tibble.

head(iris.2008.nested)
# A tibble: 3 × 3
     Species              data   models
      <fctr>            <list>   <list>
1     setosa <tibble [50 × 4]> <S3: lm>
2 versicolor <tibble [50 × 4]> <S3: lm>
3  virginica <tibble [50 × 4]> <S3: lm>

Now what I would like to do is use the linear models from the 2008 data to predict results using the 2007 data. Thinking that the best way to do that would be to combine the two datasets (retaining the group structure), here is what happens when I try to merge the two nested tibbles:

iris.both.nested <- merge(iris.2007.nested, iris.2008.nested, by='Species')

As you can see below, the tibble no longer seems to have the same format as the individual tibbles above. Specifically, the organization is hard to discern (note that I am not including the full output in this chunk, but you get the idea).

head(iris.both.nested)
     Species
1     setosa
2 versicolor
3  virginica

data.x
1 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, ...
... <truncated>
1 1.327563, 0.5464903, -0.03686145, -0.03686145, -0.1368614, 0.06313855,
...

And although I can still apparently use the models fitted to the 2008 data (as models.y) to the data from 2007 (as data.x):

iris.both.nested.pred <- iris.both.nested %>%
                         mutate( pred = map2(models.y, 
                         data.x, predict))

The result is again not a nicely-organized tibble: (again not showing full output)

head(iris.both.nested.pred)
     Species
1     setosa
2 versicolor
3  virginica

data.x
1 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, ...
... <truncated>
1 1.327563, 0.5464903, -0.03686145, -0.03686145, -0.1368614,
...

So my question is -- is this process working even though the tibbles become strangely organized after the merge? Or am I missing something? Thanks!

When I try joining two nested tibbles together it seems to work fine, what exactly did you try? — Marijn Stevering
2007data and 2007data.nested aren't legal variable names- is there any chance you could share a reproducible example (e.g. a version on a built-in dataset) that reflects your issue? — David Robinson
Yes good point -- let me edit and add reproducible data/results. — MH765
It looks like it's working to me. If you want to get output that looks more like the tibble you expect, use dplyr joins (e.g., inner_join) rather than merge. — aosmith
Thanks @aosmith -- inner_join does seem to do the trick, though I'm not sure why. — MH765

RBA RBA · Accepted Answer · 2017-04-09T10:15:16

I would double nest it first and apply the models later

# Data
iris.2007 <- iris
iris.2008 <- iris
iris.2008[1:4] <- 2*iris.2008[1:4] 

joined<-bind_rows(
cbind(dset=rep("iris.2007",length(iris.2007$Species)),iris.2007)
,cbind(dset=rep("iris.2008",length(iris.2008$Species)),iris.2008)
)

# Double nesting
joined_nested<-
  joined %>% group_by(dset) %>% nest(.key=data1) %>%
  mutate(data1 = map(data1, ~.x %>% group_by(Species) %>% nest))

# Now apply the linear model call by group using the data.
joined_nested_models<-
joined_nested %>% mutate(data1 = map(data1, ~.x %>%
             mutate(models = map(data,
                                 ~ lm(Petal.Length ~ Petal.Width, data = .)))                                       
                                       ))
joined_nested_models %>% unnest
# # A tibble: 6 × 4
#        dset    Species              data   models
#       <chr>     <fctr>            <list>   <list>
# 1 iris.2007     setosa <tibble [50 × 4]> <S3: lm>
# 2 iris.2007 versicolor <tibble [50 × 4]> <S3: lm>
# 3 iris.2007  virginica <tibble [50 × 4]> <S3: lm>
# 4 iris.2008     setosa <tibble [50 × 4]> <S3: lm>
# 5 iris.2008 versicolor <tibble [50 × 4]> <S3: lm>
# 6 iris.2008  virginica <tibble [50 × 4]> <S3: lm>

Which is a Tidier version of what you get with inner_join

Using a tibble to apply linear models by group to new data

2 Answers