how to handle xpath with html-agility-pack

Question

I'm trying to scrape some data from aliexpress using c# and html-agility-pack.

Usually, the xpath of some element looks like this :

/html/body/div[7]/div[2]/div[4]/div/div/div[2]/div[1]/div[2]/div/div[1]/a

But when i try to copy the xpath of an element in aliexpress it looks like this :

//*[@id="node-gallery"]/div[4]/div/div/ul/li[1]/div[1]/div[1]/a

and then the list of nodes return null and the program can't make any progress.

         var html = @"https://best.aliexpress.com/?lan=en";
        HtmlWeb web = new HtmlWeb();

        var htmlDoc = web.Load(html);

        var nodes = htmlDoc.DocumentNode.SelectNodes("//*[@id]/div/div[2]/div/div[2]/dl//dd/div/div[2]/ul/li//a");
        if (nodes.Count <= 0)
        {
            Console.WriteLine("nothing found");

        }
        else
        {
            foreach (HtmlNode n in nodes)
            {
                Console.WriteLine(n.Attributes);
            }
        }
        Console.ReadKey();

Neither of those xpaths match for me. on th page. What is the expected output please? — QHarr
the output should be the attribute, but however, //*[@id="node-gallery"] does not show in the source code of the page, only in the xpath, i assume it's generated through javascript — EL-Mehdi Loukach
I opened the page in the browser and used that XPath - no much. Also searches by the id attribute - no match — QHarr
okay here's some steps : 1- go to best.aliexpress.com/?lan=en 2- inspect elements of the categories -> women's clothing -> list of brands 3- it's gonna give this one : //*[@id="home-firstscreen"]/div/div[2]/div/div[2]/dl[1]/dd/div/div[2]/ul/li[1]/a — EL-Mehdi Loukach
Thanks - will look. But where is [@id="node-gallery"] as per question please? — QHarr

QHarr QHarr · Accepted Answer · 2019-08-16T20:21:55

Indeed when you hover over those items an API request is made. You can probably find the detail in one of the source files however looking at the first 2 in the network tab they have the following pattern (url decoded):

https://best.aliexpress.com/api/load_ams_path.do?path=aliexpress.com/common/@langField/ru/c-women-content.htm https://best.aliexpress.com/api/load_ams_path.do?path=aliexpress.com/common/@langField/ru/c-men-content.htm

I suspect the others follow suit.

You can make requests to these endpoints to get the html you can then retrieve you desired content from. To get the href for the xpath element matched in browser by your xpath you could do the following:

fiddle

using System;
using HtmlAgilityPack;

public class Program
{

public static void Main()
    {
        string url = "https://best.aliexpress.com/api/load_ams_path.do?path=aliexpress.com/common/@langField/ru/c-women-content.htm";
        HtmlWeb web = new HtmlWeb();
        var htmlDoc = web.Load(url);
        var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("*//li[@class='sup-brand-item'][1]/a");  
        Console.WriteLine(nodetest1.Attributes["href"].Value); 
    }
}

how to handle xpath with html-agility-pack

1 Answers