Python Scrapy Xpath?

Question

For a non-profit college assignment I'm trying to scrape data from the website www.rateyourmusic.com using the scrapy framework in python, I have had a small amount of success as I have been able to scrape the Name of an artist from an artist page but the xpath for the other info (birth date, nationality) is proving difficult for me to scrape. do any of you know what the correct xpath for these objects would be? here is my parsing method which has at least worked for the artist name.

def parse_dir_contents(self, response):
    item = rateyourmusicartist()

    for sel in response.xpath('//div/div/div/div/table/tbody/tr/td'):  
        item['dateofbirth'] = sel.xpath('td/text()').extract() #these two selectors aren't working
        item['nationality'] = sel.xpath('td/a/text()').extract()

    for sel in response.xpath('//div/div/div/div/div/h1'): 
        item['name'] = sel.xpath('text()').extract() #this is the one that works

    yield item

here is a sample URL of an artist page I'm scraping http://rateyourmusic.com/artist/kanye_west

Remove the td/ from the two XPaths that don't currently work. They should then work. — gtlambert
Thanks for noticing, unfortunately I have tried doing that already and it didn't work, I added the td/ to see if if would make a difference, would having the parsing in two separate while loops make a difference? I assumed I would have to since they are in different parts of the page source — user3545370
your issue is you relay on virtual DOM (I guess you look in inspector to get HTML structure). You must check real source on a page. F.x. there is no tbody tag on a page but only in virtual DOM. — Dmytro Pastovenskyi

Dmytro Pastovenskyi Dmytro Pastovenskyi · Accepted Answer · 2015-10-22T13:01:27

Here is real snippet of HTML you have on a page (you can see it if you open page as a source).

<table class="artist_info">
<tr><td><div class="info_hdr">Born</div> June 8, 1977, <a class="location" href="/location/Atlanta/GA/United States">Atlanta, GA, United States</a></td></tr>
<tr><td><div class="info_hdr">Currently</div><a class="location" href="/location/Hidden Hills/CA/United States">Hidden Hills, CA, United States</a></td></tr>
</table>

In order to get birthday run suhc xPage (content of first row in table)

//table[@class='artist_info']/tr[1]/td/text()

result

'June 8, 1977,'

In order to get currently run suhc xPage (content of 2-nd row in table)

//table[@class='artist_info']/tr[2]/td/a/text()

result

'Hidden Hills, CA, United States'

Python Scrapy Xpath?

1 Answers