0
votes

I have a problem that my xpath is not working.

I am trying to get the url from Google.com's search result list into a string list.

But i am unable to reach on url using Xpath.

Please help me in correcting my xpath. Also tell me what should be on the place of ??

HtmlWeb hw = new HtmlWeb();
List<string> urls = new List<string>();
HtmlAgilityPack.HtmlDocument doc = hw.Load("http://www.google.com/search?q=" +txtURL.Text.Replace(" " , "+"));
HtmlNodeCollection linkNodes = doc.DocumentNode.SelectNodes("//div[@class='f kv']");
foreach (HtmlNode linkNode in linkNodes)
{
    HtmlAttribute link = linkNode.Attributes["?????????"];
    urls.Add(link.Value);

}
for (int i = 0; i <= urls.Count - 1; i++)
{
    if (urls.ElementAt(i) != null)
    {
        if (IsValid(urls.ElementAt(i)) != true)
        {
            grid.Rows.Add(urls.ElementAt(i));

        }
    }
}
2
Why do you expect the URL to be in an attribute of a div? - Oded
i know this is incorrect. I want help on this - Suresh Sharma
I also pick xpath from Chrome but that is not working tooo. - Suresh Sharma
From the page source, I would say you need to select the cite elements and get their InnerText. - Oded
I did tried that too, But i think i am mistaken on xpath. Can you please help me in correcting xpath? - Suresh Sharma

2 Answers

1
votes

The URLs seem to live in the cite element under that selected divs, so the XPath to select those is //div[@class='f kv']/cite.

Now, since these contain markup but you only want the text, select the InnerText of the selected nodes. Note that these do not begin with http://.

HtmlNodeCollection linkNodes = 
                       doc.DocumentNode.SelectNodes("//div[@class='f kv']/cite");
foreach (HtmlNode linkNode in linkNodes)
{
    HtmlAttribute link = linkNode.InnerText;
    urls.Add(link.Value);
}
0
votes

The correct XPath is "//div[@class='kv']/cite". The f class you see in the browser element inspector is (probably) added after the page is rendered using javascript.

Also, the link text is not in an attribute, you can get it using the InnerText property of the <div> element(s) obtained at the earlier step.

I changed these lines and it works:

var linkNodes = doc.DocumentNode.SelectNodes("//div[@class='kv']/cite");

foreach (HtmlNode linkNode in linkNodes)
{
    urls.Add(linkNode.InnerText);
}

There's a caveat though: some links are trimmed (you'll see a ... in the middle)