Python code scraping data from kickstarter does not work after some iteration

Question

I try to scrap data from kickstarter, the code is working but it gives the following error in page 15 (you might get get error in different page since webpage is dynamic):

Traceback (most recent call last): File "C:\Users\lenovo\kick.py", line 30, in csvwriter.writerow(row) File "C:\Users\lenovo\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\uff5c' in position 27: character maps to

What might be the issue? Any suggestion?

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json
import csv
KICKSTARTER_SEARCH_URL = "https://www.kickstarter.com/discover/advanced?category_id=16&sort=newest&seed=2502593&page={}"
DATA_FILE = "kickstarter.csv"
csvfile = open(DATA_FILE, 'w')
csvwriter = csv.writer(csvfile, delimiter=',')
page_start = 0
while True:
    url = KICKSTARTER_SEARCH_URL.format(page_start)
    print(url)
    response = urlopen(url)
    html = response.read()
    soup = BeautifulSoup(html, 'html.parser')
    project_details_divs = soup.findAll('div', {"class":"js-react-proj-card"})

    if len(project_details_divs) == 0:
        break;

    for div in project_details_divs:
        project = json.loads(div['data-project'])
        row = [project["id"],project["name"],project["goal"],project["pledged"]]
        csvwriter.writerow(row)

    page_start +=1

csvfile.close()

If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so. — keepAlive

keepAlive keepAlive · Accepted Answer · 2019-01-04T00:50:39

Add the argument encoding to your file-opener. I mean, change

csvfile = open(DATA_FILE, 'w')

into

csvfile = open(DATA_FILE, 'w', encoding='utf-8')

Python code scraping data from kickstarter does not work after some iteration

1 Answers