How to merge csv files and precise their encoding in Pandas

0
votes

I'm trying to merge four csv files using Pandas but I get this error :

ERROR

UnicodeDecodeError Traceback (most recent call last) Input In [10], in <cell line: 9>() ----> 9 df = pd.concat(map(pd.read_csv, merged_csv), ignore_index=True) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 1867: invalid continuation byte

Here is the code I use :

CODE

import os
import glob
import pandas as pd

csv_files = os.path.join(r"C:\Users\_M92\Desktop\Projects\Ext\files\csv", "STATS-*.csv")

merged_csv = glob.glob(csv_files)

df = pd.concat(map(pd.read_csv, merged_csv), ignore_index=True)

display(df)

When I opened every single csv file in Notepad++, I figured out that their encodings are set to Unix(lf).

enter image description here

Where should I put the encoding argument in my code ? And what it the keyword corresponding to Unix(lf) ?

pythonpandasencoding