Can the python string be changed by string finding?

0
votes
doc = open("1.html").read().strip()
doc = doc.decode("utf-8","ignore")

this example is OK. I can get the right unicode string doc.

doc = open("1.html").read().strip()
if u"charset=utf" in doc or u"charset=\"utf" in doc:
    doc = doc.decode("utf-8","ignore")

there is error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 289: ordinal not in range(128)" Anyone can explain this? the string doc can be changed by string find? forgot to say,the 1.html contain the chinese word.

1
pythonstringunicodeencodingutf-8

1 Answers

3
votes

The problem is that you are comparing the byte string you read from the file with your unicode literal strings u"charset=utf" and u"charset=\"utf". In order to compare them, Python must convert the byte string to unicode at that point - before you call decode manually - which it does using the default ASCII codec.

The solution is to always compare byte strings with byte strings:

if "charset=utf" in doc or "charset=\"utf" in doc: