python - Panda's Write CSV - Append vs. Write

Question

I would like to use pd.write_csv to write "filename" (with headers) if "filename" doesn't exist, otherwise to append to "filename" if it exists. If I simply use command:

     df.to_csv('filename.csv',mode = 'a',header ='column_names')

The write or append succeeds, but it seems like the header is written every time an append takes place.

How can I only add the header if the file doesn't exist, and append without header if the file does exist?

Answer 1

Not sure there is a way in pandas but checking if the file exists would be a simple approach:

import os
# if file does not exist write header 
if not os.path.isfile('filename.csv'):
   df.to_csv('filename.csv', header='column_names')
else: # else it exists so append without writing the header
   df.to_csv('filename.csv', mode='a', header=False)

Answer 2

with open(filename, 'a') as f:
    df.to_csv(f, mode='a', header=f.tell()==0)

it will add header when writes to the file first time

Answer 3

In Pandas dataframe "to_csv" function, use header=False if csv file exists & append to existing file.

    import os

    hdr = False  if os.path.isfile('filename.csv') else True
    df.to_csv('filename.csv', mode='a', header=hdr)

Answer 4

The above solutions are great, but I have a moral obligation to include the pathlib solution here:

from pathlib import Path

file_path = Path(filename)
if file_path.exists():
   df.to_csv(file_path, header=False, mode='a')
else:
   df.to_csv(file_path, header=True, mode='w')

Alternatively (depending on your inlining preferences):

file_exists = file_path.exists()
df.to_csv(file_path, header=not file_exists, mode='a' if file_exists else 'w')

Answer 5

Apart from file exist check, you can also check for non zero file size. Since it will make sense to add header if file exists but file size is zero i.e file without content. I find it helpful in some exceptional cases

import os.path
header_flag = False if (os.path.exists(fpath) and (os.path.getsize(fpath) > 0)) else True
df.to_csv(fpath, mode='a', index=False, header=header_flag)