Inconsistent Scrapy Xpath Behavior (OS X + Linux)

Question

I'm having trouble with an Xpath selector in Scrapy. Whenever I run the spider on OS X, the selector works as expected; however when I run the same script in Ubuntu 12.04 and Raspbian, the selector doesn't work. I'm at a loss as to why this is happening.

The selector is:

sel.xpath('//table[@class="times"]')

which returns 2 selectors of the tables to me, but when I take it a step further and add in an index to select the 2nd table returned:

sel.xpath('//table[@class="times"][2]')

OS X returns the 2nd table to me no problem whereas Ubuntu/Raspbian returns nothing: []

I'm at a complete loss as to why this is happening. I've tried my best to replicate the environment (scrapy version 0.20, python 2.7) but I still havent figured it out. For reference, the page I am trying to scrape via shell is

scrapy shell "http://washington.dc.gegov.com/webadmin/dhd_431/lib/mod/inspection/paper/_paper_food_inspection_report.cfm?inspectionID=200651&wgdmn=431&wguid=1367&wgunm=sysact"

Appreciate any help on the matter.

Does sel.xpath('//table[@class="times"]') return the same thing in both environments? — Robin
Yes, both environments return 2 selection objects. Its only when I try and call it by the [2] index I get different results — Zach Cowell
Did you check if the content of the selection object is also the same in both environments (ie sel.xpath('//table[@class="times"]') is not [] in the second case)? Or could it be that the index starts at 0 in the second case, therefore you exceed your selection's length? — Robin

paul trmbrth paul trmbrth · Accepted Answer · 2014-01-23T23:08:14

Be cautious when using //table[@class="times"][2]: it will select all descendant 2nd children table elements with class "times" of their parent element. So if a node has only 1 child table with class "times", there's not match for this node.

It's not the same as (//table[@class="times"])[2] which will give you the 2nd table with class "times" under the root node (which is probably what you want). You could also use /descendant::table[@class="times"][2].

And it's still different from //table[2][@class="times"] that selects all descendant table elements that are 2nd child of their parent, AND that have class "times".

See XPath 1.0 specifications on abbreviated syntax //:

NOTE: The location path //para1 does not mean the same as the location path /descendant::para1. The latter selects the first descendant para element; the former selects all descendant para elements that are the first para children of their parents.

Let's illustrate that with a scrapy shell session

paul@wheezy:~$ scrapy shell "http://washington.dc.gegov.com/webadmin/dhd_431/lib/mod/inspection/paper/_paper_food_inspection_report.cfm?inspectionID=200651&wgdmn=431&wguid=1367&wgunm=sysact"

In [1]: sel.xpath('//table[@class="times"][2]')
Out[1]: []

In [2]: sel.xpath('(//table[@class="times"])[2]')
Out[2]: [<Selector xpath='(//table[@class="times"])[2]' data=u'<table class="times" style="font-size:9p'>]

In [3]: sel.xpath('/descendant-or-self::table[@class="times"]')
Out[3]: 
[<Selector xpath='/descendant-or-self::table[@class="times"]' data=u'<table class="times" style="margin-top:1'>,
 <Selector xpath='/descendant-or-self::table[@class="times"]' data=u'<table class="times" style="font-size:9p'>]

In [4]: sel.xpath('/descendant-or-self::table[@class="times"][2]')
Out[4]: [<Selector xpath='/descendant-or-self::table[@class="times"][2]' data=u'<table class="times" style="font-size:9p'>]

In [5]: sel.xpath('/descendant::table[@class="times"][2]')
Out[5]: [<Selector xpath='/descendant::table[@class="times"][2]' data=u'<table class="times" style="font-size:9p'>]

In [6]:

Another thing, in your sample page, I see 4 tables that contain "times" in their class attribute:

<table class="times" style="margin-top:1...
<table class="times" style="font-size:9p...
<table class="times pt6" cellpadding="0"...
<table class="times fs_10px" style="bord...

so remember that a [@class="times"] predicate means class attribute being exactly "times", not simply containing "times" (you could use [contains(@class, "times")] for that)

Inconsistent Scrapy Xpath Behavior (OS X + Linux)

1 Answers