using for loop to extract vectors from multiple data frames

Question

I have 50 data frames, each of which is identically structured (each with half a dozen variables and no more 300 rows). I need to extract and transform two vectors from each data frame that correspond to some but not all of the rows.

So from each data set I'm extracting two columns and then I'm extracting a couple of rows from those columns.

All 50 of these vectors are bound into a matrix to be subsequently used for network analysis (a citation matrix, for what it's worth--so a a directed graph).

The following code gets at this extraction and transformation.

library(tidyverse)

# read the original .csv file and extract the relevant
# vectors

SOME_JOURNAL <- read_csv("SOME_JOURNAL.csv") %>%
 select(X3, X4) %>% 
 rename("journal" = X3,
        "citations" = X4) %>%
 mutate(citations = as.integer(citations)) %>% 
 na.omit() %>% 
 tail(-3)

# identify the specfiic rows I want to extract
extract_list <- sort(c("SOME_JOURNAL",
                   "ANOTHER_JOURNAL",
                  "YET_ANOTHER_JOURNAL",
                  "ONE_MORE_JOURNAL"))

# extract the rows
SOME_JOURNAL <- SOME_JOURNAL %>% 
 filter(!!sym(names(.)[1]) %in% extract_list) %>% 
 # filters out the items I want  
 add_row(journal = setdiff(extract_list, SOME_JOURNALL$journal), citations = 0) %>% 
 # adds rows for which there is no data and assigns them zeros
 arrange(journal) %>%
 # need things in alphabetical order to manage things later on
 pivot_wider(names_from = journal, values_from = citations) 
# transposes the vector so that I can bind it with other vectors as a matrix
# for a directed graph

# make another adjustment to help transforming the matrix into a graph
SOME_JOURNAL <- data.frame(SOME_JOURNAL, row.names = "SOME_JOURNAL")

# create thee matrix by binding extracted vectors
matrix <- as.matrix(rbind(SOME_JOURNAL,
                          ANOTHER_JOURNAL,
                          YET_ANOTHER_JOURNAL,
                          ONE_MORE_JOURNAL))

^{Created on 2021-02-26 by the reprex package (v0.3.0)}

Given that I have 50 of these data frames, I'd like to automate this. And I'm running into a snag (mostly because I'm a newb). The code below leads to a "$ operator is invalid for atomic vectors" error. I've tried to use [ and [[ but I don't know enough to know if that's the kind of solution that might help.

Any help would be appreciated.

library(tidyverse)

# get a list of all the filenames
filenames <- list.files(path="data/",
                        pattern=".*csv")

# for loop to read files and extract vectors 
for(i in filenames){
 filepath <- file.path("data/", paste(i))
 short_name <-str_replace_all(str_remove_all(i,
                                             "#.*"), "-", "_")
 # the data frames have very long names; this just shortens them
 assign(short_name, read_csv(filepath) %>% 
         select(X3, X4) %>% 
         rename("journal" = X3,
                "citations" = X4) %>%
         mutate(citations = as.integer(citations)) %>% 
         na.omit() %>% 
         tail(-2) %>%
         filter(!!sym(names(.)[1]) %in% extract_list) %>% 
         # everything works fine to this point; the code after produces 
         # the "$ operator is invalid for atomic vectors" error
         add_row(journal = setdiff(extract_list, SOME_JOURNAL$journal), 
                 citations = 0) 
        
 )  
}

^{Created on 2021-02-26 by the reprex package (v0.3.0)}

You have SOME_JOURNALL in your second code extract and SOME_JOURNAL in your first. — Limey
@Limey Thank you. That was a typo from the reprex; it is corrected now. It is/was not responsible for the error. — avgoustisw
I cannot reproduce the error with made up data. Without seeing your inputs, I can't investigate further. I suspect it's an unexpected feature of your input data. If you can identify which iteration of the loop is causing the error, that might help. Also, please see this post for advice on creating a good MWE: help us to help you.. — Limey

avgoustisw avgoustisw · Accepted Answer · 2021-02-26T21:55:04

I can't take credit for this solution, but the person who solved it was good with me posting it for posterity.

To quote the solver, the "general idea is to split the long pipe into steps such that the setdiff() step gets access to the journal vector in the dataframe".

There might be other ways of doing this, but this worked for my purposes.

for(i in filenames){
        filepath <- file.path("data/", paste(i))
        short_name <-str_replace_all(str_remove_all(i,
                                                    "#.*"), "-", "_")
        # the data frames have very long names; this just shortens them
        df <- read_csv(filepath) %>% 
                select(X3, X4) %>% 
                rename("journal" = X3,
                       "citations" = X4) %>%
                mutate(citations = as.integer(citations)) %>% 
                na.omit() %>% 
                tail(-2) %>%
                filter(!!sym(names(.)[1]) %in% extract_list)
        df_all_rows <- df %>%
                add_row(journal = setdiff(extract_list, df$journal), 
                        citations = 0) %>% 
                arrange(journal) %>% 
                pivot_wider(names_from = journal, values_from = citations)
        df_all_rows <- data.frame(df_all_rows, row.names = short_name)
        assign(short_name, df_all_rows)
}
#> Error in eval(expr, envir, enclos): object 'filenames' not found

^{Created on 2021-02-26 by the reprex package (v0.3.0)}

using for loop to extract vectors from multiple data frames

1 Answers