webclient or httpwebrequest to retrieve hrefs and url

Question

How do I use either webclient or httpwebrequest to do two things:

1)Say after downloading the resource as a string using:

 var result = x.DownloadString("http://randomsite.com);

there's a relative url(also query string):

<a href="/q?name=john&age=50">Click here to get your name and age</a>

how do I click(follow) on that link using webclient? after initially loading the resource in result. i was able to use htmlagilitypack to isolate the href but I would now like to follow it in code.

2) If the httpwebrequest does not redirect but instead loads the same page with different parameters how would i use webclient to retrieve the new url that is generated? i.e if i call

var result = x.DownloadString("http://randomsite.com);

but this actually calls

http://randomsite.com/q?site=default

I then want to retrieve the second url

Thanks in advance

shriek shriek · Accepted Answer · 2013-05-30T12:59:09

You can construct the url from the link and the link that you just downloaded like this:

Uri baseUri = new Uri("http://randomsite.com");
Uri myUri = new Uri(baseUri, "/q?name=john&age=50");

Console.WriteLine(myUri.ToString()); // gives you http://randomsite.com/q?name=john&age=50

This also works if you base Url has url parameters.

As for the second question, i guess you meant that the request was redirected and you want that url instead? Then the easiest way to do so is to sub-class WebClient described here.

Uri baseUri = new Uri("http://randomsite.com");
using(var client=new WebClient())
{
  var result = client.DownloadString(myUri);
  //get href via HtmlAgilityPack...
  Uri myUri = new Uri(baseUri, "/q?name=john&age=50");
  result = client.DownloadString(myUri);
}

webclient or httpwebrequest to retrieve hrefs and url

1 Answers