Not Enough Columns Slots

Question

When try to create 12 lags for each variable in a data.table (108 variables) i get an error that says not enough column slots. This operation should create approx 1200 variables or columns.

Data_A = as.data.table (Datos_A)
Varnames = names(Datos_A)
Lagnumber= seq_len(12)

for(y in Varnames) { 
  for (z in Lagnumber)  set(Datos_GO, j = eval(paste0(y,"_lag_",z)), value = shift(Data_A [[y]],n = z, type = "lag"))
}

Error in set(Data_A, j = eval(paste0(y, "lag", z)), value = shift(Datos_A[[y]], : Internal logical error. DT passed to assign has not been allocated enough column slots. l=1132, tl=1132, adding 1

I tried to use the following, but it didn´t work.

alloc.col(Data_A,3500)

It displays now on the GLobal Enviroment that Data_A has 3132 variables, but when using the following code, I´ve got the same error

for(y in Varnames) {
  for (z in Lagnumber) set(Datos_GO, j = eval(paste0(y,"_lag_",z)), value = shift(Data_A [[y]],n = z, type = "lag"))
}

Error in set(Datos_GO, j = eval(paste0(y, "lag", z)), value = shift(Datos_GO[[y]], : Internal logical error. DT passed to assign has not been allocated enough column slots. l=6632, tl=6632, adding 1

That code worked properly with just about 80 variables in the original data.table but I don´t understand why this do not work with 30 more.

Sample data: https://ufile.io/3ka6g

ERROR and CODE on SAMPLE DATA

library(readxl)
> Data_A <- read_excel("C:/Example_data_source_108variables.xlsx")
> library(data.table)
data.table 1.10.4.3
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
> Data_A = as.data.table (Data_A)
> Varnames = names(Data_A)
> Lagnumber= seq_len(12)
> for(y in Varnames){for (z in Lagnumber) set(Data_A, j = eval(paste0(y,"_lag_",z)), value = shift(Data_A [[y]],n = z, type = "lag"))}
Error in set(Data_A, j = eval(paste0(y, "_lag_", z)), value = shift(Data_A[[y]],  : 
  Internal logical error. DT passed to assign has not been allocated enough column slots. l=1132, tl=1132, adding 1

do you have some sample data? it might make sense to create a long data.table and then dcast it into your desired output. or stay in a long format — chinsoon12
Cannot reproduce it. I used 125 variables and up to 15 lags and I get a beautifullyy 20x2000 data.table. — Martin Schmelzer
I dont´understand, cause I have already tried the sample data, and get the same Error — Luis Carmona Martinez
I have uploaded all the code with the sample data, I don´t understand why you don´t get the error, is there something we are making different?¿ — Luis Carmona Martinez

chinsoon12 chinsoon12 · Accepted Answer · 2018-03-30T01:04:20

I am also encountering the same error message when i run OP's code snippet. Data is too lengthy to be posted as dput here.

Here is a workaround using the long format as mentioned in comment:

Varnames <- copy(names(Data_A))
Data_A[, (names(Data_A)) := lapply(.SD, as.numeric)][,
    rn := .I]
melted <- melt(Data_A, id.vars="rn")[,
    (paste0("lag_", seq_len(12))) := shift(value, seq_len(12), type="lag"),
    by=variable][, 
        value:=NULL][]
res <- dcast.data.table(melt(melted, id.vars=c("rn", "variable"), variable.name="lag"),
    rn ~ variable + lag, sum)

#view results
res[, ncol(res), with=FALSE]

sessionInfo()

R version 3.3.0 (2016-05-03)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] openxlsx_4.0.17   readxl_1.0.0      data.table_1.10.4 zoo_1.8-1        

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.13           rstudioapi_0.5-1       knitr_1.18             magrittr_1.5          
 [5] lattice_0.20-33        rlang_0.1.4            stringr_1.2.0          plyr_1.8.4            
 [9] tools_3.3.0            grid_3.3.0             htmltools_0.3.6        yaml_2.1.16           
[13] rprojroot_1.3-2        digest_0.6.9           tibble_1.3.4           bookdown_0.5          
[17] reshape2_1.4.2         RStudioShortKeys_0.1.0 evaluate_0.10.1        rmarkdown_1.8         
[21] stringi_1.1.1          cellranger_1.1.0       backports_1.1.2

Not Enough Columns Slots

1 Answers