href attribute empty using xpath (python3)

Question

Using chrome and xpath in python3, I try to extract the value of an "href" attribute on this web page. "href" attributes contains the link to the movie's trailer ("bande-annonce" in french) I am interested in.

First thing, using xpath, it appears that the "a" tag is a "span" tag. In fact, using this code:

response_main=urllib.request.urlopen("http://www.allocine.fr/film/fichefilm_gen_cfilm=231874.html")
htmlparser = etree.HTMLParser()
tree_main = etree.parse(response_main, htmlparser)
tree_main.xpath('//*[@id=\"content-start\"]/article/section[3]/div[2]/div/div/div/div[1]/*')

I get this result:

[<Element span at 0x111f70c08>]

So the "div" tag contains no "a" tag but just a "span" tag. I've read that html visualization in browsers doesn't always reflects the "real" html sent by the server. Thus I tried to use this command to extract the href:

    response_main=urllib.request.urlopen("http://www.allocine.fr/film/fichefilm_gen_cfilm=231874.html")
htmlparser = etree.HTMLParser()
tree_main = etree.parse(response_main, htmlparser)
tree_main.xpath('//*[@id=\"content-start\"]/article/section[3]/div[2]/div/div/div/div[1]/span/@href')

Unfortunately, this returns nothing... And when I check the attributes within the "span" tag with this command:

tree_main.xpath('//*[@id=\"content-start\"]/article/section[3]/div[2]/div/div/div/div[1]/span/@*')

I got the value of the "class" attribute, but nothing about "href"... :

['ACrL3ZACrpZGVvL3BsYXllcl9nZW5fY21lZGlhPTE5NTYwMDcyJmNmaWxtPTIzMTg3NC5odG1s meta-title-link']

I'd like some help to understand what's happening here. Why the "a" tag is a "span" tag? And the most important question to me, how can I extract the value of the "href" attribute?

Thanks a lot for your help!

Andersson Andersson · Accepted Answer · 2017-03-20T11:56:33

Required link generated dynamically with JavaScript. With urllib.request you can get only initial HTML page source while you need HTML after all JavaScript been executed.

You might use selenium + chromedriver to get dynamically generated content:

from selenium import webdriver as web
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait 

driver = web.Chrome("/path/to/chromedriver")
driver.get("http://www.allocine.fr/film/fichefilm_gen_cfilm=231874.html")
link = wait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[@class='meta-title']/a[@class='xXx meta-title-link']")))
print(link.get_attribute('href'))

href attribute empty using xpath (python3)

1 Answers