Python reading from a file and saving to utf-8

Question

I'm having problems reading from a file, processing its string and saving to an UTF-8 File.

Here is the code:

try:
    filehandle = open(filename,"r")
except:
    print("Could not open file " + filename)
    quit() 

text = filehandle.read()
filehandle.close()

I then do some processing on the variable text.

And then

try:
    writer = open(output,"w")
except:
    print("Could not open file " + output)
    quit() 

#data = text.decode("iso 8859-15")    
#writer.write(data.encode("UTF-8"))
writer.write(text)
writer.close()

This output the file perfectly but it does so in iso 8859-15 according to my editor. Since the same editor recognizes the input file (in the variable filename) as UTF-8 I don't know why this happened. As far as my reasearch has shown the commented lines should solve the problem. However when I use those lines the resulting file has gibberish in special character mainly, words with tilde as the text is in spanish. I would really appreciate any help as I am stumped....

Which editor is this? Which python version? From here this code seems to be completely valid and should work as expected … — filmor
Kate is the editor. The output of python --version is Python 2.7.5+ — aarelovich
I've tested your code with 2.6.8, 2.7.5+ and 3.3.2+ everything works fine. Could you provide some example input? — zero323
Since the text was processed in raw bytes the unseen processing code probably messed up the UTF8 encoding. — Mark Tolonen
Ok. I've solved it. It was mostly my fault so sorry everyone. Here is what happened. The code provided by @MarkTolonen worked if I change iso-8859-15 instead of utf-8 when opening the file. However as my editor updated the file from memory having already loaded the old encoding it showed me the gibberish. When I opened the file again it showed it to me fine. Thank you all and sorry for the bother!!! — aarelovich

Mark Tolonen Mark Tolonen · Accepted Answer · 2013-10-25T13:55:41

Process text to and from Unicode at the I/O boundaries of your program using open with the encoding parameter. Make sure to use the (hopefully documented) encoding of the file being read. The default encoding varies by OS (specifically, locale.getpreferredencoding(False) is the encoding used), so I recommend always explicitly using the encoding parameter for portability and clarity (Python 3 syntax below):

with open(filename, 'r', encoding='utf8') as f:
    text = f.read()

# process Unicode text

with open(filename, 'w', encoding='utf8') as f:
    f.write(text)

If still using Python 2 or for Python 2/3 compatibility, the io module implements open with the same semantics as Python 3's open and exists in both versions:

import io
with io.open(filename, 'r', encoding='utf8') as f:
    text = f.read()

# process Unicode text

with io.open(filename, 'w', encoding='utf8') as f:
    f.write(text)

Python reading from a file and saving to utf-8

3 Answers