Scrapy Xpath output empty

Question

I would like extract data on this website : http://www.pokepedia.fr/Pikachu I'm learning the python and how to use Scrapy and my problem is : Why I can't retrieve the data with Xpath ?

My Xpath look good when i test xpath in my browser, it return me the correct value. (Google Chrome)

import re
from scrapy import Spider
from scrapy.selector import Selector

from stack.items import StackItem


class StackSpider(Spider):
name = "stack"
allowed_domains = ["pokepedia.fr"]
start_urls = [
    "http://www.pokepedia.fr/Pikachu",
]

def unicodize(seg):
    if re.match(r'\\u[0-9a-f]{4}', seg):
        return seg.decode('unicode-escape')

    return seg.decode('utf-8')

def parse(self, response):
    pokemon = Selector(response).xpath('//*[@id="mw-content-text"]/table[2]')

    for question in pokemon:
        item = StackItem()
        item['title'] = question.xpath(
            '//*[@id="mw-content-text"]/table[2]/tbody/tr[1]/th[2]/text()').extract()[0]
        yield item

I want to extract the name of the pokemon in the page but when I use :

scrapy crawl stack -o items.json -t json

My Json output :

In my console i've this error :

IndexError : list index out of range

I've followed this tuto : https://realpython.com/blog/python/web-scraping-with-scrapy-and-mongodb/

as outlined in a provided answer, be careful with trusting any web browsers dev console/xpath viewer, because the document they display is not always the exact HTML that the page produces. Often it will add tags, and fix any broken html. it is often better to download the html of the page directly (simple python script can do) and word from that. Web Scraping is a great tool to learn but keep this tip in mind always, it has bitten me a few times. — Kaden

Noah Bishop Noah Bishop · Accepted Answer · 2016-06-28T13:17:00

Try this

question.xpath( '//*[@id="mw-content-text"]/table[2]/tr[1]/th[2]/text()').extract()[0]

The browser adds the tbody tags. They are not in the original HTML, that's why scrapy returns an empty file.

PS: you might want to consider using

scrapy shell URL

and then using

response.xpath('...YOUR SELECTOR..')

for debugging and testing.

Scrapy Xpath output empty

1 Answers