python - Is it possible to add several columns at once to a pandas DataFrame?

23
votes

If I want to create a new DataFrame with several columns, I can add all the columns at once -- for example, as follows:

data = {'col_1': [0, 1, 2, 3],
        'col_2': [4, 5, 6, 7]}
df = pd.DataFrame(data)

But now suppose farther down the road I want to add a set of additional columns to this DataFrame. Is there a way to add them all simultaneously, as in

additional_data = {'col_3': [8, 9, 10, 11],
                   'col_4': [12, 13, 14, 15]}
#Below is a made-up function of the kind I desire.
df.add_data(additional_data)

I'm aware I could do this:

for key, value in additional_data.iteritems():
    df[key] = value

Or this:

df2 = pd.DataFrame(additional_data, index=df.index)
df = pd.merge(df, df2, on=df.index)

I was just hoping for something cleaner. If I'm stuck with these two options, which is preferred?

4
pythonpandas
Tbh I think the for loop could be the best/most efficient way from a dict, although join feels "cleaner". - Andy Hayden
@AndyHayden may be it's possible to create new expand method. Can't say for sure, but if the number of columns is large, loop should be less efficient than one-time expand of array, or I'm wrong? - Roman Pekar
@RomanPekar if it's already a python object (a dict) I don't think it's going to get any faster. Hmmm have to test - Andy Hayden
This question seems very similar: stackoverflow.com/questions/39050539/… - j sad

4 Answers

19
votes

Pandas has assign method since 0.16.0. You could use it on dataframes like

In [1506]: df1.assign(**df2)
Out[1506]:
   col_1  col_2  col_3  col_4
0      0      4      8     12
1      1      5      9     13
2      2      6     10     14
3      3      7     11     15

or, you could directly use the dictionary like

In [1507]: df1.assign(**additional_data)
Out[1507]:
   col_1  col_2  col_3  col_4
0      0      4      8     12
1      1      5      9     13
2      2      6     10     14
3      3      7     11     15
9
votes

What you need is the join function:

df1.join(df2, how='outer')
#or
df1.join(df2) # this works also

Example:

data = {'col_1': [0, 1, 2, 3],
    'col_2': [4, 5, 6, 7]}
df1 = pd.DataFrame(data)

additional_data = {'col_3': [8, 9, 10, 11],
               'col_4': [12, 13, 14, 15]}
df2 = pd.DataFrame(additional_data)

df1.join(df2, how='outer')

output:

   col_1  col_2  col_3  col_4
0      0      4      8     12
1      1      5      9     13
2      2      6     10     14
3      3      7     11     15
8
votes

If you don't want to create new DataFrame from additional_data, you can use something like this:

>>> additional_data = [[8, 9, 10, 11], [12, 13, 14, 15]]
>>> df['col3'], df['col4'] = additional_data
>>> df
   col_1  col_2  col3  col4
0      0      4     8    12
1      1      5     9    13
2      2      6    10    14
3      3      7    11    15

It's also possible to do something like this, but it would be new DataFrame, not inplace modification of existing DataFrame:

>>> additional_header = ['col_3', 'col_4']
>>> additional_data = [[8, 9, 10, 11], [12, 13, 14, 15]]
>>> df = pd.DataFrame(data=np.concatenate((df.values.T, additional_data)).T, columns=np.concatenate((df.columns, additional_header)))
>>> df
   col_1  col_2  col_3  col_4
0      0      4      8     12
1      1      5      9     13
2      2      6     10     14
3      3      7     11     15
0
votes

All you need to do is create the new columns with data from the additional dataframe.

data =            {'col_1': [0, 1, 2, 3],
                   'col_2': [4, 5, 6, 7]}
additional_data = {'col_3': [8, 9, 10, 11],
                   'col_4': [12, 13, 14, 15]}
df = pd.DataFrame(data)
df2 = pd.DataFrame(additional_data)

df[df2.columns] = df2

df now looks like:

   col_1  col_2  col_3  col_4
0      0      4      8     12
1      1      5      9     13
2      2      6     10     14
3      3      7     11     15

Indices from the original dataframe will be used as if you had performed an in-place left join. Data from the original dataframe in columns with a matching name in the additional dataframe will be overwritten. For example:

data =            {'col_1': [0, 1, 2, 3],
                   'col_2': [4, 5, 6, 7]}
additional_data = {'col_2': [8, 9, 10, 11],
                   'col_3': [12, 13, 14, 15]}
df = pd.DataFrame(data)
df2 = pd.DataFrame(additional_data, index=[0,1,2,4])

df[df2.columns] = df2

df now looks like:

   col_1  col_2  col_3
0      0      8     12
1      1      9     13
2      2     10     14
3      3    NaN    NaN