Faking browser request in ASP.net C#

Question

I'm using the code below to pull one of our 3rd party developed pages in so I can parse it as XML for my random bits of work.

Irritatingly we stil have a browser detection level set on the server that only allows certain browsers on to the site; so the question is how would I fake it so that the server thinks its a browser request?

   static string GetHtmlPage(string strURL)
    {

        String strResult;
        System.Net.WebResponse objResponse;

        System.Net.WebRequest objRequest = System.Net.HttpWebRequest.Create(strURL);

        objResponse = objRequest.GetResponse();
        using (System.IO.StreamReader sr = new System.IO.StreamReader(objResponse.GetResponseStream()))
        {
            strResult = sr.ReadToEnd();
            sr.Close();
        }
        return strResult;
    }

Answer 1

Browser detection is done based on a header in the request to the server. All you need to do is set that header. However, with HttpWebRequest you don't set that through the headers collection but rather with the .UserAgent property.

...
System.Net.WebRequest objRequest = 
   System.Net.HttpWebRequest.Create(strURL);

//Pretend to be IE7
((System.Net.HttpWebRequest)objRequest).UserAgent = 
   "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)";

objResponse = objRequest.GetResponse();
...

Answer 2

You can use the ClientTarget attribute in the Page. E.g.

http://msdn.microsoft.com/en-us/library/system.web.ui.page.clienttarget.aspx

http://msdn.microsoft.com/en-sg/library/6379d90d(v=vs.85).aspx

Set the configuration as you wish...

E.g.

<configuration>
   <system.web>
      <clientTarget>
         <add alias="ie5" userAgent="Mozilla/4.0 (compatible;MSIE 5.5;Windows NT 4.0)"/>
         <add alias="ie4" userAgent="Mozilla/4.0 (compatible;MSIE 4.0;Windows NT 4.0)"/>
         <add alias="uplevel" userAgent="Mozilla/4.0 (compatible;MSIE 4.0;Windows NT 4.0)"/>
         <add alias="downlevel" userAgent="Unknown"/>
      </clientTarget>
   </system.web>
</configuration>

Then you can use it as follows.

<asp:Page ClientTarget="downlevel" />

This will fake the request!

Answer 3

I think most (if not all) browser detection is based on the User-Agent header, set by the HttpRequest.UserAgent property. I see there is a website for user-agent strings of various browsers: http://www.user-agents.org/

Answer 4

as with Waldens above but had to replace

objRequest.UserAgent = 
   "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)";

with

((System.Net.HttpWebRequest)objRequest).UserAgent = "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)";

Otherwise it fell over. (I changed the browser to googlebot to evade our cookie server)