UnicodeEncodeError : 'ascii' codec can't encode character '\xf1' in position 61: ordinal not in range(128)

Question

I am calling the google adwords api in Python and then I am logging that data to a CSV File and I am dealing with the UnicodeDecode/EncodeError and I have tried everything to understand it without avail now.

with open('adgroups.csv', 'w', newline='') as csvfile:
    campaign_name = seed_keyword.title().encode('utf-8','ignore').decode()
    kw_writer = csv.writer(csvfile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
    kw_writer.writerow(
        ["CAMPAIGN", "ADGROUP", "MAX BID", "KEYWORD", "MATCH TYPE", "AVERAGE CPC", "SEARCH VOLUME", "COMPETITION"])
    for ad_group in ad_group_list:
        ad_group_name = ad_group['keyword'].title().encode('utf-8','ignore').decode()
        try:
            for keyword_data in ad_group['keyword_data_list']:
                kw_writer.writerow(
                    [campaign_name, ad_group_name, str(ad_group['rpc'] * 0.5).encode('utf-8','ignore').decode(), keyword_data['kw'].encode('utf-8','ignore').decode(), "BROAD",
                     str(keyword_data['cpc']).encode('utf-8','ignore').decode(),
                     str(keyword_data['search_volume']).encode('utf-8','ignore').decode(), str(keyword_data['competition']).encode('utf-8','ignore').decode()])
                kw_writer.writerow(
                    [campaign_name, ad_group_name, str(ad_group['rpc'] * 0.5).encode('utf-8','ignore').decode(), keyword_data['kw'].encode('utf-8','ignore').decode(), "PHRASE",
                     "",
                     "", ""])
                kw_writer.writerow(
                    [campaign_name, ad_group_name, str(ad_group['rpc'] * 0.5).encode('utf-8','ignore').decode(), keyword_data['kw'].encode('utf-8','ignore').decode(), "EXACT",
                     "",
                     "", ""])
        except UnicodeError as e:
            print(e)
            print(str(ad_group))
            raise

File "/var/www/html/ARB-Automation/MultiProcessController.py", line 158, in perform_automation
CSVCampaignSetup.get_in_kw_and_make_adgroups(seed_keyword, ad_group_bid_mapping, cluster_bid_dict)
File "/var/www/html/ARB-Automation/CSVCampaignSetup.py", line 163, in get_in_kw_and_make_adgroups
make_single_adgroup_csv(best_ad_group_data)
File "/var/www/html/ARB-Automation/CSVCampaignSetup.py", line 222, in make_single_adgroup_csv
str(keyword_data['search_volume']).encode('utf-8','ignore').decode(), str(keyword_data['competition']).encode('utf-8','ignore').decode()])

UnicodeEncodeError: 'ascii' codec can't encode character '\xf1' in position 61: ordinal not in range(128)

Why do I get an error in the first place when I have mentioned 'ignore'?
How to correctly handle this?

Weird thing - The stacktrace notes error on line 222, which is an "encode" error, as I understand it tries to create a byte representation of the unicode string. The data on line 222 that needs to be encoded is a number - int or float! How does that not find representation under 127 bits - hence the 'ascii' coded problem?

Also, I got one solution in mind - instead of encode('utf-8','ignore') if I do encode('ascii','ignore'), shouldn't it solve the problem? The question remains whether that is the ideal solution?

I even printed the raw data and I ran this code step by step in the interpreter, I did not get any error then. Please help me.

Hard to be sure without more code, but this error can happen when you try to print a unicode string to an ascii file or terminal, or if you managed to set the default encoding to ascii instead of utf8. — Serge Ballesta
What happens if you try str(keyword_data['search_volume']).encode('utf-8') (and also for keyword_data['competition']) without any 'ignore' or .decode()? — James Lemieux
@JamesLemieux, 'ignore' basically says if you can't encode a part of string, rather than raising an error, ignore the string in the output. So ideally in this case there should be no error... — Chintan Shah

csaszizoltan csaszizoltan · Accepted Answer · 2017-01-13T07:58:41

Please change python source filename's (path) non english characters (in this case ñ) to english chars [a-z0-9] and valid OS filename chars. I use pyscripter and I had this like problem.

UnicodeEncodeError : 'ascii' codec can't encode character '\xf1' in position 61: ordinal not in range(128)

1 Answers