How to encode character '\u0107' in python

Question

I'm trying to scrape data from Wikipedia page (it is a table of top 100 singles of certain years), while saving output to csv it got from 1951-1959 then it gave an error:

line 43, in writer.writerow(songs) File "C:\Python36_64\lib\encodings\cp1252.py",

line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0]

UnicodeEncodeError: 'charmap' codec can't encode character '\u0107' in position 29: character maps to <undefined>

code:


from bs4 import BeautifulSoup
import requests
import csv

data = []


def scrape_data(search_year):
    year_data = []
    url = f'https://en.wikipedia.org/wiki/Billboard_Year-End_Hot_100_singles_of_{str(search_year)}'
    # Get a source code from url
    r = requests.get(url).text
    soup = BeautifulSoup(r, 'html.parser')
    # Isolate the table part from the source code
    table = soup.find('table', attrs={'class': 'wikitable'})
    # Extract every row of the table
    rows = table.find_all('tr')

    # Iterate through every row
    for row in rows[1:]:
        # Extract cols (with tags td and th)
        cols = row.find_all(['td', 'th'])
        # List comprehension (create a list of lists, list of rows, in which every row is a list of table text)
        year_data.append([col.text.replace('\n', '') for col in cols])

    # Add the year, this data is from to the beginning of the list
    for n in year_data:
        n.insert(0, search_year)
    return year_data


for year in range(1951, 2019):
    try:
        data.append(scrape_data(year))
        print(f'Year {str(year)} Scrapped')
    except AttributeError as e:
        print(f'Year {str(year)} is not aviable')

writer = csv.writer(open('songs.csv', 'w'), delimiter=',', lineterminator='\n', quotechar='"')
for year_data in data:
    for songs in year_data:
        writer.writerow(songs)
        print(songs)

Why is your system set up for cp1252? Set it up for UTF-8 if you want to be able to manipulate Unicode. — tripleee
@tripleee This is the first time I'm doing this, how can I set it to UTF-8 than? — user10051040
Not sure how to do that on Windows, and it also apparently depends on which version of Windows. I hear Windows 10 has some fixes in these areas. Not using Windows is always an attractive option in my book. — tripleee
@tripleee I'm using Win10, but Jonah Bishop's answer solved it. — user10051040

Jonah Bishop Jonah Bishop · Accepted Answer · 2019-01-09T19:33:54

I think you can correct this by using the correct unicode encoding when writing your output:

writer = csv.writer(open('songs.csv', 'w', encoding='utf-8'),
                    delimiter=',', lineterminator='\n', quotechar='"')

How to encode character '\u0107' in python

1 Answers