Python shell not running crawler

Question

i had to try this code

import scrapy
from scrapy.cmdline import execute
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from metacritic.items import MetacriticItem
class MetacriticSpider(scrapy.spider):
name = "metacritic" # Name of the spider, to be used when crawling
    allowed_domains = ["metacritic.com"] # Where the spider is allowed to go
start_urls = [
    "http://www.metacritic.com/browse/games/title/pc?page=0"
]
def parse(self, response):
    hxs = HtmlXPathSelector(response) # The XPath selector
    sites = hxs.select('//li[contains(@class, "product game_product")]/div[@class="product_wrap"]')
    items = []
    for site in sites:
        item = MetacriticItem()
        item['title'] = site.select('div[@class="basic_stat product_title"]/a/text()').extract()
        item['link'] = site.select('div[@class="basic_stat product_title"]/a/@href').extract()
        item['cscore'] = site.select('div[@class="basic_stat product_score brief_metascore"]/div/div/span[contains(@class, "data metascore score")]/text()').extract()
        item['uscore'] = site.select('div[@class="more_stats condensed_stats"]/ul/li/span[contains(@class, "data textscore textscore")]/text()').extract()
        item['date'] = site.select('div[@class="more_stats condensed_stats"]/ul/li/span[@class="data"]/text()').extract()
        items.append(item)
    return items

i already try some way to fix this code, but i always get an error /home/kautsar/metacritic 2/metacritic/spiders/metacritic_spider.py:3: ScrapyDeprecationWarning: Module scrapy.spider is deprecated, use scrapy.spiders instead from scrapy.spider import BaseSpider Traceback (most recent call last): File "/home/kautsar/metacritic 2/metacritic/spiders/metacritic_spider.py", line 6, in class MetacriticSpider(scrapy.spider): TypeError: Error when calling the metaclass bases module.init() takes at most 2 arguments (3 given) anybody know how to fix this?

Paul Becotte Paul Becotte · Accepted Answer · 2016-06-22T04:46:49

from scrapy.spider import BaseSpider
...
class MetacriticSpider(scrapy.spider):

As you can see, scrapy.spider is a module name. Your class tries to inherit from it. Typically, your class should inherit from another class- probably BaseSpider in this case.

Python shell not running crawler

1 Answers