XPath Query Problem using HTML Agility Pack

2
votes

I'm trying to scrape the price field from this website using the HTML Agility Pack.

My code is as follows;

var web = new HtmlWeb();
var doc = web.Load(String.Format(overClockersURL, componentID));
var priceContent = doc.DocumentNode.SelectSingleNode("//*[@id=\"prodprice\"]");

I obtained the XPath query by using Firebug's "Copy as XPath" feature.

The problem I'm having is that SelectSingleNode is returning null - it doesn't seem to find the element specified by the query. I'm a bit stumped as to why, but I don't have much experience with XPath, so would appreciate some pointers as to what I've done wrong.

2
c#xpathhtml-agility-pack
It is an irritation of mine that the HTML Agility Pack returns null when an empty collection would be better. Have you tried experimenting the the XPath and replacing the " * " with known element names, e.g. if you know that there is a div with an id of "prodprice" to replace the " * " with "div" to see if you get a different result? - Colin Mackay
Sorry, had to add spaces in the last comment to make the asterisks show up as it interpreted them as start/end of italic markers. - Colin Mackay
Hi @Colin Mackay: I tried "//span[@id='prodprice']", but same result - null is returned :S - Chris McAtackney
@Oscar Mederos: That's exactly what was happening - thanks for your suggestion. If you move your comment up to an answer then I can mark it as the accepted solution. - Chris McAtackney

2 Answers

3
votes

When that happens, you should check if the page is being loaded correctly (you said you're through a HTTP Proxy?)

Try writing the content of doc.DocumentNode.OuterHtml to a text file so you can see if the page is being loaded correctly. Maybe you're getting an error page instead of the original page.

1
votes

If I run this code:

    var web = new HtmlWeb();
    var doc = web.Load("http://www.overclockers.co.uk/showproduct.php?prodid=GX-033-HS");
    var priceContent = doc.DocumentNode.SelectSingleNode("//*[@id=\"prodprice\"]");
    Console.WriteLine("price=" + priceContent.InnerHtml);

It outputs:

price=529.99

So it seems to be working. You can also use //span[@id=\"prodprice\"]" which is better as it avoids all non SPAN tags.