Run parallel parse function python scrapy

Question

i'm using scrapy mixing with selenium. I want to run my parse function in many task in parallelism. I want to open many url simultaneously. So i use Pool.map function to map my parse() function to an array(1,2,3). But the embarrassing thing is my function is passed by an HttpResponse object by scrapy, so it couldn't pass i argument by Pool.map() function. I put my map() function in my spider class or out of it. but i don't know how to complete.

class FacebookSpider(scrapy.Spider):
    name = 'facebk'
    start_urls = ['https://www.facebook.com']
    options = Options()
    options.add_argument("--disable-notifications")
    client = MongoClient()
    db = client.test
    original_usr = db.user 

    def parse(self,i):
      driver = webdriver.Chrome(chrome_options=self.options)
      self.logger.info("executing")
      pickle.dump(self.driver.get_cookies() , open("cookies.pkl","wb"))   
      try:
        for target in self.original_usr.find().limit(3).skip(i*3):
            self.logger.info("email:"+target['email'])
            url = "https://www.facebook.com/search/people/?        q="+target['email']
            self.driver.get(url)
            cookies = pickle.load(open("cookies.pkl", "rb"))
            for cookie in cookies:
                self.driver.add_cookie(cookie)   
            self.parse_pages(url)
      finally:
        driver.close()

if __name__ == '__main__':        
  target_nums= range(2)
  pool = ThreadPool(2)
  fs = FacebookSpider()
  pool.map(fs.parse,target_nums)
  pool.close()
  pool.join()

It got error:

Traceback (most recent call last):
File "C:\Users\warrior\AppData\Local\Enthought\Canopy\User\lib\site-   packages\twisted\internet\defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "D:\mega folder\webcrawler\tutorial\tutorial\spiders\facebook_spider -        Copy (3).py", line 56, in parse
for target in self.original_usr.find().limit(3).skip(i*3):
TypeError: unsupported operand type(s) for *: 'HtmlResponse' and 'int'

If i use pool in my spider class instead of create new Facebook spider instance, it got an error TypeError: parse() takes exactly 2 arguments (1 given). Please help me. thanks!

the problem is parse function auto receives htmlresponse object as argument so here i variable is htmlresponse type not int . How to avoid it?. — Linh Dinh

Mangu Singh Rajpurohit Mangu Singh Rajpurohit · Accepted Answer · 2016-11-02T03:43:46

One issue is with your creation of instance of FacebookSpider. Actually, you are not creating FacebookSpider instance and trying to access parse member method.

if __name__ == '__main__':        
  target_nums= range(2)
  pool = ThreadPool(2)
  fs = FacebookSpider()
  pool.map(fs.parse,target_nums)
  pool.close()
  pool.join()

Run parallel parse function python scrapy

1 Answers