Scrapy Spider not following Request callback using yield

Question

I'm new to scrapy and I cant get my spider to enter parse_votes in code bellow, even though I set it as callback. The others parse methods are working fine, I don't get any ERROR and checked the 'link' variable which has the correct info. HELP?

EDIT - Full code

class DeputadosSpider(scrapy.Spider):
    name = "deputies"

    allowed_domains = ["camara.leg.br"]
    start_urls = ["http://www2.camara.leg.br/deputados/pesquisa"]

    def parse(self, response):
        sel = Selector(response)
        sel_options = sel.xpath('//*[@id="deputado"]/option[position()>1]')
        iteration = 1
        # get deputies pages
        for sel_option in sel_options:
            item = DeputiesInfo()           
            item["war_name"] = sel_option.xpath("text()").extract()
            item["link_id"] = sel_option.extract().partition('?')[-1].rpartition('"')[0]
            item["page_link"] = 'http://www.camara.leg.br/internet/Deputado/dep_Detalhe.asp?id=' + item["link_id"]
            item["id"] = iteration
            iteration += 1
            # go scrap their page
            yield scrapy.Request(item["page_link"], callback=self.parse_deputy, meta={'item': item})

    def parse_deputy(self, response):
        item = response.meta['item']
        sel = Selector(response)
        info = sel.xpath('//div[@id="content"]/div/div[1]/ul/li')
        # end to fill the data
        item["full_name"] = info.xpath("text()").extract_first()
        item["party"] = info.xpath("text()").extract()[2].partition('/')[0]
        item["uf"] = info.xpath("text()").extract()[2].partition('/')[-1].rpartition('/')[0]
        item["legislatures"] = info.xpath("text()").extract()[5]
        item["picture"] = sel.xpath('//div[@id="content"]/div/div[1]//img[1]/@src').extract()
        # save data to json file 
        file = open('deputies_info.json', 'a')
        line = json.dumps(dict(item)) + ",\n"
        file.write(line)
        # colect votes info
        get_years = sel.xpath('//*[@id="my-informations"]/div[3]/div/ul/li[1]/a[position()<4]')
        for get_year in get_years:
            vote = VotesInfo()
            vote["deputy_id"] = item["id"]
            vote["year"] = get_year.xpath("text()").extract_first()
            link = get_year.xpath("@href").extract_first()
            print(vote["year"])
            print(link)
            # go to voting pages
            yield scrapy.Request(link, callback=self.parse_votes, meta={'vote': vote})

    def parse_votes(self, response):
        #vote = response.meta['vote']
        print('YYYYYYYYYYYYYUHUL IM IN!!')

How did you check the link variable that it has the correct value? Isn't it a relative link in which case you would have to use response.urljoin method? — Tomáš Linhart
@TomášLinhart the xpath("@href") get a complete http:// link, that goes to a different page (in the same domain). I checked the noob way, by printing it on terminal and it was correct. I did the same thing on parse(self, response) and it worked fine. But when I tried to do the same on parse_dep(self, response) it doesn't work. It seems like it doesn't process the Request, because I can't get in parse_votes(self, response). — Ninousw
Could you edit the question so that whole spider code is shown? Problems like this are sometimes caused by using too restrictive allowed_domains etc. — Tomáš Linhart

Jgaldos Jgaldos · Accepted Answer · 2017-08-09T15:08:12

Your problem is allowed_domains, because the link you are trying to request in parse_deputy is for example: http://www.camara.gov.br/internet/deputado/RelVotacoes.asp?nuLegislatura=55&nuMatricula=410&dtInicio=01/01/2016&dtFim=30/12/2016 and its domain is camara.gov.br so add it to allowed_domains.

allowed_domains = ["camara.leg.br", "camara.gov.br"]

PS: I ran your code commentingallowed_domains, and parse_votes works perfectly.

Scrapy Spider not following Request callback using yield

2 Answers