Using Scrapy response, XPath element doesn't exist, although Google Chrome Inspect Element shows that it does exist

Question

I am encountering a problem in which elements that I am trying to select using their XPath do not exist according to Scrapy response. However, the when I inspect the same page on Google Chrome, the element DOES exist.

This problem is occurring on a LinkedIn scrape after using LinkedIn advanced search and getting to a results page. I want to scrape links in the results container.

For example: On the results page for a search on "John," there should be a div element with id="results-container" according to an Inspect Element on Google Chrome. When I use Scrapy response.xpath('//div[@id="results-container]'), there are no selectors returned.

url of result page: https://www.linkedin.com/vsearch/p?firstName=John&openAdvancedForm=true&locationType=Y&rsid=4319659841436374935558&orig=ADVS

Answer 1

Did you try to look up the URL you provided in a private session window of your browser (sometimes called incognito mode)?

If you do this you see that you get a registration form for LinkedIn.

As alecxe suggests in his comment try using the LinkedIn API (it is REST) and you can get XML responses which you can parse along to gather the information needed.

Alternatively you could try to log-in with Scrapy and store the authentication credentials and repeat your request (but I would use the API anyway).