R stacking output from a loop

Question

I have some trouble saving the outputs of a loop in a dataframe. As my script is too complicated to be explained in this question, here is a simplified version of my code:

edited

df<-data.frame(matrix(NA, nrow = 100, ncol = 9)) #empty dataframe set for 100 max rows
df<-c("YEAR","RN","DATE","NOM","SITE","LONG","SP","SUMNB","NB100" )
do.call(rbind, by(DF, DF[c("YEAR","SITE")], FUN = function(x) {

 df<-dataframe filling code

})

So basically it is subsetting my DF by VAR1 and VAR2, and building a new df for each subset. How can I build a new dataframe (TOT) with all my "df" in top of each others ?

Notes: I do not know how many rows each "df" will have, that's why i set 100 rows max, and then remove NA rows right after. But DF and df have the same columns.

Thanks for the help!

EDIT:

My point is to subset every YEAR/SITE combination, apply a function to it (building a df of dummy data with altered dates based on each subset, in the same format than the input DF) and return the function output in a data frame. This dataframe, build by stacking all the df i got for each subset, will have the exact same format than my input DF. I need for further applications to merge my input with that output.

The use of do.call(rbind) is doing the job (subsetting) and applying the function (creating the df). At this point, i cant make ddply to work for some reason (seems like the function is not applied) and i dont think it will fit to my request: ddply applies the function to the input and returns the modified input. My input is not modified, and i want to return another dataframe.

Its seems i cant apply any of the answers i found to my function because i dont know the number of rows the output should have.

EDIT 2:

As my question seems to be fully developped to be understood, here is my full problem:

the first 15 rows of my input dataframe (AGG100):

    YEAR    RN  DATE    NOM SITE    LONG    SP  SUMNB   NB100
1   2011    RNN027  15056   ESTAGNOL    RNN027-Estagnol 02  310 Anthocharis cardamines (Linnaeus, 1758) 1   0.3225806
2   2011    RNN027  15075   ESTAGNOL    RNN027-Estagnol 02  310 Anthocharis cardamines (Linnaeus, 1758) 1   0.3225806
3   2003    RNN027  12166   ESTAGNOL    RNN027-Estagnol 03  330 Anthocharis cardamines (Linnaeus, 1758) 2   0.6060606
4   2006    RNN027  13252   ESTAGNOL    RNN027-Estagnol 03  330 Anthocharis cardamines (Linnaeus, 1758) 2   0.6060606
5   2006    RNN027  13257   ESTAGNOL    RNN027-Estagnol 03  330 Anthocharis cardamines (Linnaeus, 1758) 2   0.6060606
6   2005    RNN027  12895   ESTAGNOL    RNN027-Estagnol 01  540 Anthocharis cardamines (Linnaeus, 1758) 2   0.3703704
7   2005    RNN027  12910   ESTAGNOL    RNN027-Estagnol 01  540 Anthocharis cardamines (Linnaeus, 1758) 2   0.3703704
8   2011    RNN027  15075   ESTAGNOL    RNN027-Estagnol 01  540 Anthocharis cardamines (Linnaeus, 1758) 1   0.1851852
9   2008    RNN027  14120   ESTAGNOL    RNN027-Estagnol 05  960 Anthocharis cardamines (Linnaeus, 1758) 2   0.2083333
10  2011    RNN027  15065   ESTAGNOL    RNN027-Estagnol 05  960 Anthocharis cardamines (Linnaeus, 1758) 1   0.1041667
11  2011    RNN027  15075   ESTAGNOL    RNN027-Estagnol 05  960 Anthocharis cardamines (Linnaeus, 1758) 1   0.1041667
12  2007    RNN027  13679   ESTAGNOL    RNN027-Estagnol 05  960 Apatura ilia (Denis & Schifferm�ller, 1775) 2   0.2083333
13  2004    RNN027  12573   ESTAGNOL    RNN027-Estagnol 06  260 Aporia crataegi (Linnaeus, 1758)    2   0.7692308
14  2005    RNN027  12917   ESTAGNOL    RNN027-Estagnol 06  260 Aporia crataegi (Linnaeus, 1758)    2   0.7692308
15  2006    RNN027  13301   ESTAGNOL    RNN027-Estagnol 06  260 Aporia crataegi (Linnaeus, 1758)    2   0.7692308

This dataframe is filled with species(66 in this example) observations in differents sites (ESTAGNOL 01 to 06) at 9 years.

here is my full code:

MIN<-data.frame(matrix(NA, nrow = 100, ncol = 9)) #reglé pour 100 lignes max
colnames(MIN)<-c("YEAR","RN","DATE","NOM","SITE","LONG","SP","SUMNB","NB100" )
MAX<-data.frame(matrix(NA, nrow = 100, ncol = 9))
colnames(MAX)<-c("YEAR","RN","DATE","NOM","SITE","LONG","SP","SUMNB","NB100" )
do.call(rbind, by(AGG100, AGG100[c("YEAR","SITE")], FUN = function(x) {
  splist<-unique(x$SP)
  lsp<-length(splist)  
  for (i in 1:lsp){
    MIN$SP[i]<-as.character(splist[i])
    MIN$SITE[i]<-as.character(unique(x$SITE))
    MIN$DATE[i]<-as.character(min(x$DATE) - 7)
    MIN$RN[i]<-as.character(unique(x$RN))
    MIN$YEAR[i]<-as.character(unique(x$YEAR))
    MIN$NOM[i]<-as.character(unique(x$NOM))
    MIN$LONG[i]<-as.numeric(unique(x$LONG))
    MIN$SUMNB[i]<-0
    MIN$NB100[i]<-0
    MAX$SP[i]<-as.character(splist[i])
    MAX$SITE[i]<-as.character(unique(x$SITE))
    MAX$DATE[i]<-as.character(min(x$DATE) + 7)
    MAX$RN[i]<-as.character(unique(x$RN))
    MAX$YEAR[i]<-as.character(unique(x$YEAR))
    MAX$LONG[i]<-as.numeric(unique(x$LONG))
    MAX$SUMNB[i]<-0
    MAX$NB100[i]<-0

  MINMAX<- rbind(MIN,MAX)
  MINMAX<-MINMAX[complete.cases(MINMAX),]   
}

      return(MINMAX)
}))

What I am able to do:

-Subset every YEAR/SITE combination to isolate the first and last date of observation (limits of temporal range) for the sum of all species. Done with do.call(rbind) or ddply().

-Create a row filled with all the respective data of AGG100 (every 9 columns) with zero observation and a date seven days before the first observation for every species (66 rows). Compile the rows in a df named MIN. -create another row with a date seven days after the last observation (66 rows). Compile these rows in a df called MAX -Both rows type created with a for loop based on the number of species i at each subset.

-Merge MIN and MAX in a new data frame (MINMAX), which has the same format than my input data frame.

What I need to do:

-Be able to return MINMAX for each subset. I currently only got the last iteration (last YEAR and last SITE). Build a dataframe out of these MINMAX (MINMAXTOT).

-Merge MINMAXTOT with AGG100 and sort the result by date.

I hope its clear enough now. I just want to add a line to my script that dont overwrite the output at every iteration.

EDIT 3:

 summary(AGG100)
      YEAR            RN            DATE                                      NOM      
 2009   : 775   RNN027 :5360   Min.   :2003-04-17   ESTAGNOL                    :5360  
 2003   : 717   RNN037 :   0   1st Qu.:2005-05-27   ANCIENNES CARRIERES D'ORIVAL:   0  
 2006   : 689   RNN044 :   0   Median :2007-07-13   BAIE DE L'AIGUILLON (VENDEE):   0  
 2008   : 598   RNN046 :   0   Mean   :2007-06-17   CHERINE                     :   0  
 2011   : 557   RNN060 :   0   3rd Qu.:2009-07-16   COMBE LAVAUX-JEAN ROLAND    :   0  
 2004   : 524   RNN066 :   0   Max.   :2011-10-06   COTE DE MANCY               :   0  
 (Other):1500   (Other):   0                        (Other)                     :   0  
                 SITE           LONG                                           SP               SUMNB        
 RNN027-Estagnol 01:1135   Min.   :260.0   Pieris PC (rapae / mannii) #complexe : 651   Min.   :  1.000  
 RNN027-Estagnol 05: 957   1st Qu.:310.0   Polyommatus icarus (Rottemburg, 1775): 482   1st Qu.:  2.000  
 RNN027-Estagnol 04: 951   Median :540.0   Maniola jurtina (Linnaeus, 1758)     : 456   Median :  2.000  
 RNN027-Estagnol 03: 915   Mean   :543.5   Brintesia circe (Fabricius, 1775)    : 446   Mean   :  6.215  
 RNN027-Estagnol 02: 801   3rd Qu.:710.0   Lasiommata megera (Linnaeus, 1767)   : 341   3rd Qu.:  6.000  
 RNN027-Estagnol 06: 601   Max.   :960.0   Pieris brassicae (Linnaeus, 1758)    : 313   Max.   :156.000  
 (Other)           :   0                   (Other)                              :2671                    
 NB100        
 Min.   : 0.1042  
 1st Qu.: 0.3226  
 Median : 0.6452  
 Mean   : 1.2876  
 3rd Qu.: 1.2903  
 Max.   :28.8889

At the moment, your example isn't reproducible (df doesn't have columns named VAR1 and SVAR2) so its hard to understand what you want. Isn't splitting by those variables and then calling do.call(rbind) just the same as sorting by those variables? While you have a think about what you want, I'm resorting to the standard advice: look at ddply in the plyr package. — Richie Cotton
Usually, when i give the full code, the discussion focuses on the part i dont want to discuss (here, the dataframe filling code, mostly because its a for loop). I voluntarily dont show it because i just want to know a way to automatically concatenate the dataframes obtained in my function in a global dataframe. I did mentionned that my input DF and output df have the same variables ( i just skipped the line where i names the nine columns of df). However I updated my post. — user2542995
The issue is that all your code seems to do is split a df and then rbind it back together, so its hard to see the point of what you're trying to do. — Thomas
That is why i think i cant make any of the answers i found working: i split my DF to apply a function to every subset, but i do not want to modify my input DF. The dataframe output i seek is independant of my input dataframe. — user2542995
It looks like you are trying to calculate values by species as well as by site and year, so split by species too. by(AGG100, AGG100[c("YEAR","SITE", "SP")] — Richie Cotton

Richie Cotton Richie Cotton · Accepted Answer · 2013-12-16T14:10:09

A dataset that is reproducible:

n <- 50
DF <- data.frame(
  VAR1 = sample(letters[1:3], n, replace = TRUE),
  VAR2 = sample(LETTERS[1:5], n, replace = TRUE)
  Y    = runif(n)
)

An example of calculating statistics split by VAR1 and VAR2.

library(plyr)

ddply(
  DF, 
  .(VAR1, VAR2), 
  summarise, 
  MeanOfY = mean(Y), 
  SumOfY  = sum(Y)
)

For future reference, try searching for the phrase "split apply combine".

Update:

The latest description of the problem still isn't terribly clear, but it seems like you just want to add new variables that are functions of the year and site. You don't need to do any subsetting or recombining things.

Here's your dataset:

n <- 50
DF <- data.frame(
  YEAR = sample(2000:2013, n, replace = TRUE),
  SITE = sample(LETTERS[1:5], n, replace = TRUE)
  #some other variables
)

Here's a dummy function of year and site. Change it to whatever you need.

do_something <- function(year, site)
{
  ifelse(
    year < 2005,  
    ifelse(
      site %in% c("A", "C", "E"),
      1,
      2
    ),
    3
  )
}

This is one way of adding the new variable:

DF$NewVar <- with(DF, do_something(YEAR, SITE))

R stacking output from a loop

1 Answers