Scrapy Error: TypeError: __init__() got an unexpected keyword argument 'deny'

1
votes

I've put together a spider and it was running as intended until I've added the keyword deny into the rules.

This is my spider :

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy.selector import Selector
from bhg.items import BhgItem

class BhgSpider (CrawlSpider):
    name = 'bhg'
    start_urls = ['http://www.bhg.com/holidays/st-patricks-day/']
    rules = (Rule(LinkExtractor(allow=[r'/*'], ),
                  deny=('blogs/*', 'videos/*', ),
                  callback='parse_html'), )

def parse_html(self, response):
    hxs = Selector(response)
    item = BhgItem()

    item['title'] = hxs.xpath('//title/text()').extract()
    item['h1'] = hxs.xpath('//h1/text()').extract()
    item['canonical'] = hxs.xpath('//link[@rel = \"canonical\"]/@href').extract()
    item['meta_desc'] = hxs.xpath('//meta[@name=\"description"]/@content').extract()
    item['url'] = response.request.url
    item['status_code'] = response.status
    return item

When I run this code I get: 

deny=('blogs/', 'videos/', ),), )
TypeError: __init__() got an unexpected keyword argument 'deny'

What am i doing wrong? Well, I guess a function or something was not expecting the extra argument (deny) but which function? parse_html()?

I did not define any other spiders and there is no __init__()

2
pythonscrapy

2 Answers

2
votes

deny is supposed to be passed as an argument to LinkExtractor, but you put it outside those parentheses and passed it to Rule. Move it inside, so you have:

rules = (Rule(LinkExtractor(allow=[r'/*'], deny=('blogs/*', 'videos/*', )),
                  callback='parse_html'), )

__init__ is the method that is called when you pass arguments when instantiating a class, as you did here with the Rule and LinkExtractor classes.

1
votes

Er, wouldn't it be the function you actually passed deny to? That would be Rule. The __init__ is the internal method of the Rule class that is called when you construct the object.

The examples I found online with this pass deny= to the LinkExtractor, not the Rule. So you want:

rules = (Rule(LinkExtractor(allow=('/*',),
                            deny=('blogs/*', 'videos/*', )),
              callback='parse_html'), )