Scrapy not finding table css

Question

Just started using Scrapy recently and I've been having good luck with it so far until this issue. I can't seem to 'find' the standings table here;

http://www.baseball-reference.com/leagues/MLB/2016-standings.shtml#all_expanded_standings_overall

It has the id = '#expanded_standings_overall' but I can't find it with my spider or in the shell. I was able to get a result for #all_expanded_standings_overall because there is a div with that ID. Extracting this in the shell shows me the table I want but even within that I can not find it with 'tbody' or 'tr' or anything else I've tried.

can you post your attempts so we can see where you went wrong? — Hannah Schipper
@Hannah, I don't know what to even show you? if I do scrapy shell thatsite.com and then enter response.css('#expanded_standings_overall') returns []. I'm completely lost as to it why it couldn't find that ID? that's the same way I've located several other similar tables across this domain already. — Jordan Wayne Crabb

rfelten rfelten · Accepted Answer · 2017-03-11T18:22:07

If you have a look on the page source, you see that the id in question (expanded_standings_overall)

<div class="placeholder"></div>
<!--
    <div class="table_outer_container">
        <div class="overthrow table_container" id="div_expanded_standings_overall">
            <table class="sortable stats_table" id="expanded_standings_overall" data-cols-to-freeze=2>
                <caption>MLB Detailed Standings</caption>
                    ... sweet data here ..
                </table>
        </div>
    </div>
-->
</div>

The HTML comments seems to be a trick to hide the content to our innocent scraper ;)

It is interesting that Firebug don't show this comments ...?

One approach to overcome the issue is to extract the comments, remove them and proceed with the data in the comments. For instance:

$ scrapy shell www.baseball-reference.com/leagues/MLB/2016-standings.shtml
>>> view(response)
>>> from scrapy.selector import Selector
>>> sel = Selector(response)
>>> sel.xpath('//table[@id="expanded_standings_overall"]')
[]
>>> import re
>>> regex = re.compile(r'<!--(.*)-->', re.DOTALL)
>>> for comment in sel.xpath('//comment()').re(regex):
>>>     table = Selector(text=comment).xpath('//table[@id="expanded_standings_overall"]')
>>>     print(table)
...
[]
[]
[<Selector xpath='//table[@id="expanded_standings_overall"]' data='<table class="sortable stats_table" id="'>]
[]
[]

As you see I prefer XPATH selectors over CSS, but they are in principle the same, see https://doc.scrapy.org/en/latest/topics/selectors.html.

Scrapy not finding table css

1 Answers