I am using HTML-Agility-Pack to scrape data about domains and IPs from the Mcafee Threat Intelligence lab, but no matter what XPATH I provide to DocumentNode.SelectNodes it cannot find the node I am looking for (the image for threat level). It can find other nodes and images, but not this specific one as far as I can tell. The specific img has an id of "ctl00_mainContent_imgRisk" on this page: www.mcafee.com/threat-intelligence/domain/default.aspx?domain=facebook.com (facebook.com as an example). Here is the code I am using:
string XPATH = "foo"
string url = "http://www.mcafee.com/threat-intelligence/ip/default.aspx?";
if (y == true)
url = url + "ip=" + ipaddress;
if (y == false)
url = url + "domain=" + domainname;
HttpWebRequest mcafeeReq = (HttpWebRequest)WebRequest.Create(url);
mcafeeReq.UserAgent = @"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5";
HtmlDocument mcafeeWeb = new HtmlDocument();
HttpWebResponse resp = (HttpWebResponse)mcafeeReq.GetResponse();
var resultStream = resp.GetResponseStream();
mcafeeWeb.Load(resultStream);
HtmlNodeCollection nodes = mcafeeWeb.DocumentNode.SelectNodes(XPATH);
try
{
foreach (HtmlNode node in nodes)
{
try
{
HtmlAttribute att = node.Attributes["title"];
Console.WriteLine(att.Value);
}
catch
{
Console.WriteLine("No Value Found");
}
}
}
catch
{
Console.WriteLine("No Nodes Found.");
}
I have tried a number of different values for XPATH to find the img including
"//img[@id='ctl00_mainContent_imgRisk']", which returns no nodes
"//img", which returns some img's but not the correct one
the full XPATH to the img "/html/body[@id='ctl00_MasterBody']/div[@class='wrapper']/div[@class='page']/div[@class='container pageContainer']/div[@class='pageMiddle']/div[@class='pageTopGradient']/div[@class='pageBottomGradient']/div[@class='canvas pageContent pageBannerBolster']/div[@class='row pageBanner pageBannerTeal']/div[@class='area primaryContent']/div[@class='bannerContent bannerContentImage']/img[@id='ctl00_mainContent_imgRisk']", which returns no nodes.
Does anyone know what I am doing wrong?
