Scrapy Craigslist script

Question

I want to create a Scrapy script to scrape all of the results for computer gigs in any craigslist subdomain: for example here: http://losangeles.craigslist.org/search/cpg/ This query returns a list of many articles and I've tried to scrape the title and href of each of this results (not only the ones on the first page) to no avail using CrawlSpider and linkExtractor, but the Script returns nothing. I'll paste my script here, thanks

    import scrapy
    from scrapy.spiders import Rule,CrawlSpider
    from scrapy.linkextractors import LinkExtractor

    class CraigspiderSpider(CrawlSpider):
        name = "CraigSpider"
        allowed_domains = ["http://losangeles.craigslist.org"]
        start_urls = (
                    'http://losangeles.craigslist.org/search/cpg/',
        )

        rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_page", follow= True),)

        def parse_page(self, response):
            items = response.selector.xpath("//p[@class='row']")
        for i in items:
            link = i.xpath("./span[@class='txt']/span[@class='pl']/a/@href").extract()
            title = i.xpath("./span[@class='txt']/span[@class='pl']/a/span[@id='titletextonly']/text()").extract()
            print link,title

Steven Almeroth Steven Almeroth · Accepted Answer · 2016-03-12T16:47:30

According to the code you pasted, parse_page:

does not return/yield anything, and
only contains one line: "items = response.selector..."

The reason for #2 above is that the for loop is not properly indented.

Try to indent the for loop:

class CraigspiderSpider(CrawlSpider):
    name = "CraigSpider"
    allowed_domains = ["http://losangeles.craigslist.org"]
    start_urls = ('http://losangeles.craigslist.org/search/cpg/',)

    rules = (Rule(
        LinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)),
        callback="parse_page", follow= True))

    def parse_page(self, response):
        items = response.selector.xpath("//p[@class='row']")

        for i in items:
            link = i.xpath("./span[@class='txt']/span[@class='pl']/a/@href").extract()
            title = i.xpath("./span[@class='txt']/span[@class='pl']/a/span[@id='titletextonly']/text()").extract()
            print link, title
            yield dict(link=link, title=title)

Scrapy Craigslist script

1 Answers