Empty Div return with Xpath or Css Selector Using Scrapy

Question

I'm using Scrapy to crawl a web page which contains a specific article.

I'm trying to get the informations stored inside the div with the class "return". The big problem that the div return always empty when i use Scrapy Xpath or Css selectors.

The Div that i'm trying to extract:

<div class="return">  

                        <p><strong>Conditionnement : </strong></p>
                        <p class="one-product-detail">2 colis :<br>
                        L178xl106xH80&nbsp;72kg<br>L178xl112xH80&nbsp;60kg<br>
                        <span itemprop="weight" alt="3fin" class="hidden" hidden="">132kg</span></p>   

 </div>

My Spider Code:

import scrapy
from alinea.items import AlineaItem

class AlineaSpider(scrapy.Spider):
    name = "alinea"
    start_urls = [
        "http://www.alinea.fr/",
    ]
    def parse(self, response):
        # ref = input("Enter Item Reference ?\n")
        # 25321050
        # link = "http://www.alinea.fr/alinea_fredhopper/catalogSearch_result/products/search/" + str(ref)
        link = "http://www.alinea.fr/alinea_fredhopper/catalogSearch_result/products/search/" + str(25321050)
        print(link)
        return scrapy.Request(link,
                              callback=self.parse_page2)

    def parse_page2(self, response):
        self.logger.info("Visited %s", response.url)

        for sel in response.xpath('//li[contains(@itemprop,"title")]/text()'):
            print("**************")
            print("Description")
            print(sel.extract())
            print("**************")

        # print("------------------------------------------------------------------")
        #
        # for sel in response.xpath('//*[@class="delivery"]'):
        #
        #     print("**************")
        #     print("Details")
        #     print(sel.extract())
        #     print("**************")

        print("------------------------------------------------------------------")

        for sel in response.css('[class="return"]'):

            print("**************")
            print("Details")
            print(sel.extract())
            print("**************")

My Terminal Log:

2016-07-28 12:57:21 [alinea] INFO: Visited http://www.alinea.fr/orca-canape-angle-gauche-droit-convertible-gris.html
**************
Description

                    Orca - Canapé CONVERTIBLE d'angle gauche ou droit gris                
**************
------------------------------------------------------------------
**************
Details
<div class="return">



</div>
**************

That div is empty in the HTML source. It's also empty in my Chrome browser, until you select an option for the product. I tested with alinea.fr/orca-canape-angle-gauche-droit-convertible-gris.html. The data you're after is most probably loaded through JavaScript. Scrapy itself only downloads HTML, and does not intepret JavaScript. You'll need to either reproduce the XHR call or use a JavaScript rendering tool, like Selenium or Splash. — paul trmbrth

Harrison Harrison · Accepted Answer · 2016-07-29T00:33:41

The page you visited has no content for that div at all. So you are supposed to get what you got.

If you change to other pages, for example http://www.alinea.fr/orca-canape-angle-droit-gris-fonce.html, you will see the div is there and not empty.

Output from the shell: scrapy shell 'http://www.alinea.fr/orca-canape-angle-droit-gris-fonce.html'

In [1]: response.xpath('//div[@class="return"]').extract()
Out[1]: [u'<div class="return">\n\n            \n<p><strong>Conditionnement : </strong></p>\n<p class="one-product-detail">\n\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t2 colis :<br>\n\t\t\t\t\t\t\t\t\t L178xl106xH80\xa055kg<br>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t L178xl112xH80\xa053kg<br>\t\t\t\t\t\t<span itemprop="weight" alt="3fin" hidden class="hidden">108kg</span></p>\n        \n</div>']

If you want the text, you use //text() instead, as /text() only gives you text directly under div, in your case whitespace.

In [2]: response.xpath('//div[@class="return"]/text()').extract()
Out[2]: [u'\n\n            \n', u'\n', u'\n        \n']

In [3]: [x.strip() for x in response.xpath('//div[@class="return"]//text()').extract()]                                                                                                                                                             
Out[3]: 
[u'',
 u'Conditionnement :',
 u'',
 u'2 colis :',
 u'L178xl106xH80\xa055kg',
 u'L178xl112xH80\xa053kg',
 u'',
 u'108kg',
 u'']

Empty Div return with Xpath or Css Selector Using Scrapy

1 Answers