extracting text xpath scrapy

Question

Hi all I would like to extract all the text from an html block using xpath in scrapy

Let's say we have a block like this:

<div>
   <p>Blahblah</p>
   <p><a>Bluhbluh</a></p>
   <p><a><span>Bliblih</span></a></p> 
</div>

I want to extract the text as ["Blahblah","Bluhbluh","Blihblih"]. I want xpath to recursively look for text in the div node. I have heard tried: //div/p[descendant-or-self::*]/text() but it does not extract nested elements.

Cheers! Seb

Answer 1

You can use XPath's string() function on each p element:

>>> import scrapy
>>> selector = scrapy.Selector(text="""<div>
...    <p>Blahblah</p>
...    <p><a>Bluhbluh</a></p>
...    <p><a><span>Bliblih</span></a></p> 
... </div>""")
>>> [p.xpath("string()").extract() for p in selector.xpath('//div/p')]
[[u'Blahblah'], [u'Bluhbluh'], [u'Bliblih']]
>>> import operator
>>> map(operator.itemgetter(0), [p.xpath("string()").extract() for p in selector.xpath('//div/p')])
[u'Blahblah', u'Bluhbluh', u'Bliblih']
>>> 

Answer 2

>>> selector.xpath('//div/p/descendant-or-self::*/text()').extract()
[u'Blahblah', u'Bluhbluh', u'Bliblih']

You were close! All you had to do is to regard the text of the descendant or self, and not put it as an attribute. [] are used for "talking" to attributes, in your case the attributes of p, which are non-existent.

Answer 3

If you want to get ALL text nodes from a given element you need this XPath:

//div/p//text()

So you code will look like this:

text_array = selector.xpath('//div/p//text()').extract()

Answer 4

I like @Elvira Gandelman's solution.

But, there is another clumsy but more intuitive way:

response.xpath('(//div/p | //div/p/a | //div/p/a/span)/text()').extract()
['Blahblah', 'Bluhbluh', 'Bliblih']

Answer 5

If you're willing to tolerate additional dependencies, html-text has a method extract_text" which normalizes whitespace and removes inline styles. This is helpful if you simply want to return a single string with all child text rather than a list of strings.

>>> from html_text import extract_text
>>> extract_text(response.xpath("//div").extract())
'Blahblah\n\nBluhbluh\n\nBliblih'