Python read csv - BOM embedded into the first key

Question

I'm using Python 2.7.12. With this code snippet I'm saving a utf-8 csv file. I wrote the BOM (byte order mark) at the beginning of the file.

import codecs
import csv

outputFile = open("test.csv", "wb")
outputFile.write(codecs.BOM_UTF8)
fieldnames = ["a", "b"]
writer = csv.DictWriter(outputFile, fieldnames, delimiter=";")
writer.writeheader()
row = dict([])
for i in range(10):
    row["a"] = str(i).encode("utf-8")
    row["b"] = str(i*2).encode("utf-8")
    writer.writerow(row)
outputFile.close()

I want to load that csv file:

import codecs
import csv
inputFile = open("test.csv", "rb")
reader = csv.DictReader(inputFile, delimiter=";")
for row in reader:
    print row["a"]
inputFile.close()

The above code is going to fail: KeyError: 'a' If I print the row keys this is how they look: [u'\ufeffa', u'b']. The BOM has been embedded into the key a. What am I doing wrong?

Answer 1

You have to tell open that this is UTF-8 with BOM. I know that works with io.open:

import io

.
.
.
inputFile = io.open("test.csv", "r", encoding='utf-8-sig')
.
.
.

And you have to open the file in text mode, "r" instead of "rb".

Answer 2

In Python 3, the built-in open function is an alias for io.open.

All you need to open a file encoded as UTF-8 with BOM:

open(path, newline='', encoding='utf-8-sig')

Example

import csv

...

with open(path, newline='', encoding='utf-8-sig') as csv_file:
    reader = csv.DictReader(csv_file, dialect='excel')
    for row in reader:
        print(row['first_name'], row['last_name'])