Using scrapy with Cherrypy?

0
votes

I have got two classes CountriesSpider class and Countries class. I want to call the parse() function from CountriesSpider inside the countries class but I get this error

TypeError: parse() missing 1 required positional argument: 'response'

Pls Note: def parse(self, response) is an inbuilt function of class scrapy.Spider which takes the reponse parameter.

How do I call the this function self.countriesSpider.parse() from countries class ?

import scrapy
import cherrypy

class CountriesSpider(scrapy.Spider):
    name = 'countries'
    allowed_domains = ['www.worldometers.info/']
    start_urls = ['https://www.worldometers.info/world-population/population-by-country/']

    def parse(self, response):        
        title = response.xpath("//h1/text()").get()        
        countries = response.xpath("//td/a/text()").getall()

        yield {
            'title': title,
            'countries': countries
        }

        return countries


class Countries(object):

    def __init__(self):
        # Create an instance of Snacks class
        self.countriesSpider = CountriesSpider()

    @cherrypy.expose    
    def index(self):
        return self.countriesSpider.parse()

if __name__ == '__main__':
    cherrypy.quickstart(Countries())
1
pythonpython-3.xoopscrapy
What should it parse? - po.pe
The parse function you've defined takes a response parameter. You need to give it that parameter, or you need to rewrite the function to not require it. It's not going to appear from thin air. - Samwise
It returns a list like this {'title': 'Countries in the world by population (2020)', 'countries': ['China', 'India', 'United States', 'Indonesia', 'Pakistan', 'Brazil', 'Nigeria', 'Bangladesh', 'Russia']} - jeril
If you want to start your spider from another script you need to use from scrapy.crawler import CrawlerProcess - gangabass

1 Answers

-3
votes

You function requires a parameter called response. You didn't enter in the parameter.