16
votes

I am trying to create a pandas DataFrame and it works fine for a single file. If I need to build it for multiple files which have the same data structure. So instead of single file name I have a list of file names from which I would like to create the DataFrame.

Not sure what's the way to append to current DataFrame in pandas or is there a way for pandas to suck a list of files into a DataFrame.

5

5 Answers

34
votes

The pandas concat command is your friend here. Lets say you have all you files in a directory, targetdir. You can:

  1. make a list of the files
  2. load them as pandas dataframes
  3. and concatenate them together

`

import os
import pandas as pd

#list the files
filelist = os.listdir(targetdir) 
#read them into pandas
df_list = [pd.read_table(file) for file in filelist]
#concatenate them together
big_df = pd.concat(df_list)
3
votes

Potentially horribly inefficient but...

Why not use read_csv, to build two (or more) dataframes, then use join to put them together?

That said, it would be easier to answer your question if you provide some data or some of the code you've used thus far.

1
votes

I might try to concatenate the files before feeding them to pandas. If you're in Linux or Mac you could use cat, otherwise a very simple Python function could do the job for you.

0
votes

Are these files in a csv format. You could use the read_csv. http://pandas.sourceforge.net/io.html

Once you have read the files and save it in two dataframes, you could merge the two dataframes or add additional columns to one of the two dataframes( assuming common index). Pandas should be able to fill in missing rows.

0
votes
import os
import pandas as pd
data = []

thisdir = os.getcwd()

for r, d, f in os.walk(thisdir):
    for file in f:
        if ".docx" in file:
            data.append(file)

df = pd.DataFrame(data)