XPath/Python - How to get different html tags and text inside a <div>

Question

I'm trying to scrape the html content at this url: http://www.dlib.org/dlib/november14/beel/11beel.html with this Python sintax:

    s="http://www.dlib.org/dlib/november14/beel/11beel.html"
    content = requests.get(s)
    tree = html.fromstring(content.text)
    titoli = tree.xpath('/html/body/form/table[3]/tr/td/table[5]/tr/td/table[1]/tr/td[2]/h3/text()')
    par = tree.xpath('/html/body/form/table[3]/tr/td/table[5]/tr/td/table[1]/tr/td[2]/p/text()')
    articoli = json.dumps({'titoli':titoli,'contenuti':par})
    print ("Content-type: json")
    print
    print (articoli)

The main request is to find a XPath query for return every tags, tags content and text inside the most useful div of the page, you can find it with this path /html/body/form/table[3]/tr/td/table[5] or using a web inspector under the commented line: !-- CONTENT TABLE --. With the code i've posted before is not possible to get the entire content of the div but only titles and text inside the p div, now i can't find another way.

of course you're only going to get all the text in the p div you specified it on your par variable — Zion

gtlambert gtlambert · Accepted Answer · 2015-08-31T10:35:50

To get the actual HTML content of a certain section of a website using Python/XPath, it is easier to use from lxml import etree instead of from lxml import html. When you set up the element tree, there is a function which allows you to return the HTML content of an element, rather than just returning the text content (as you mentioned). Your code would be as follows:

from lxml import etree
import requests

s = "http://www.dlib.org/dlib/november14/beel/11beel.html"
page = requests.get(s)
tree = etree.HTML(page.text)
element = tree.xpath('./body/form/table[3]/tr/td/table[5]')
content = etree.tostring(element[0])

tree.xpath returns a list of selected elements. In this case, because you are using a specific XPath, it returns a list containing just one element. We therefore have to use etree.tostring(element[0]) to access the first element of the list and return the HTML content of the element as a string.

XPath/Python - How to get different html tags and text inside a <div>

1 Answers