Scrapy request does not callback

Question

I am trying to create a spider that takes data from a csv (two links and a name per row), and scrapes a simple element (price) from each of those links, returning an item for each row, with the item's name being the name in the csv, and two scraped prices (one from each link).

Everything works as expected except the fact that instead of returning the prices, that would be returned from the callback function of each request, I get a request object like this :

< GET https://link.com>..

The callback functions don't get called at all, why is that?

Here is the spider:

f = open('data.csv')
f_reader = csv.reader(f)
f_data = list(f_reader)

parsed_data = []

for product in f_data:
    product = product[0].split(';')
    parsed_data.append(product)

f.close()

class ProductSpider(scrapy.Spider):
    name = 'products'
    allowed_domains = ['domain1', 'domain2']

    start_urls = ["domain1_but_its_fairly_useless"]

    def parse(self, response):
        global parsed_data
        for product in parsed_data:

            item = Product()

            item['name'] = product[0]
            item['first_price'] = scrapy.Request(product[1], callback=self.parse_first)
            item['second_price'] = scrapy.Request(product[2], callback=self.parse_second)
            yield item


    def parse_first(self, response):
        digits = response.css('.price_info .price span').extract()
        decimals = response.css('.price_info .price .price_demicals').extract()
        yield float(str(digits)+'.'+str(decimals))

    def parse_second(self, response):
        digits = response.css('.lr-prod-pricebox-price .lr-prod-pricebox-price-primary span[itemprop="price"]').extract()
        yield digits

Thanks in advance for your help!

a bit off topic but you shouldn't use globals like that and writing to file like this is a bad idea as well. Scrapy can automatically produce csv with scrapy crawl spider -o output.csv. — Granitosaurus
How can I do without the global ? I know it's bad habit but I don't know how to do otherwise.. The CSV is only to read an existing file. I export using openpyxl to excel — AimiHat
Ah, I see what you want to do here. Why not use simple instance variable? It's really hard to show code in a comment but what you should look into is how __init__ works in python and what are instance and class variables. — Granitosaurus

Granitosaurus Granitosaurus · Accepted Answer · 2016-07-18T17:05:22

TL;DR: You are yielding an item with Request objects inside of it when you should yield either Item or Request.

Long version:
Parse methods in your spider should either return a scrapy.Item - in which case the chain for that crawl will stop and scrapy will put out an item or a scrapy.Requests in which case scrapy will schedule a request to continue the chain.

Scrapy is asynchronious so to create an item from multiple requests means you need to chain all of those requests while carrying your item to every one of item and fill it up little by little. Request object has meta attribute where you can store anything you want to (well pretty much) and it will be carried to your callback function. It's very common to use it to chain requests for items that require multiple requests to form a single item.

Your spider should look something like this:

class ProductSpider(scrapy.Spider):
    # <...>
    def parse(self, response):
        for product in parsed_data:
            item = Product()
            item['name'] = product[0]
            # carry next url you want to crawl in meta
            # and carry your item in meta
            yield Request(product[1], self.parse_first,
                          meta={"product3": product[2], "item":item})  


    def parse_first(self, response):
        # retrieve your item that you made in parse() func
        item = response.meta['item']
        # fill it up
        digits = response.css('.price_info .price span').extract()
        decimals = response.css('.price_info .price .price_demicals').extract()
        item['first_price'] = float(str(digits)+'.'+str(decimals))
        # retrieve next url from meta
        # carry over your item to the next url
        yield Request(response.meta['product3'], self.parse_second,
                      meta={"item":item})


    def parse_second(self, response):
        # again, retrieve your item
        item = response.meta['item']
        # fill it up
        digits = response.css('.lr-prod-pricebox-price .lr-prod-pricebox-price-primary 
                              span[itemprop="price"]').extract()
        item['secodn_price'] = digits
        # and finally return the item after 3 requests! 
        yield item

Scrapy request does not callback

1 Answers