HttpWebRequest with pages that have dynamically generated html from javascript?

Question

This is kind of a two part question. Is it possible to retrieve data from a page that loads content dynamically through javascript by using WebClient/HttpWebRequest?

I'd also like to how I would be able to use WebClient/HttpWebRequest to replicate a XMLHttpRequest like you would see executed via javascript.

Edit: I captured the headers of the request I am trying to replicate which looks like this: http://www.tagged.com/api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034

POST /api/?application_id=user&format=json&session_token=6thk20fhv7d727emgdhfka6034 HTTP/1.1 Host: www.tagged.com

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8 Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

X-Requested-With: XMLHttpRequest

Content-Type: application/x-www-form-urlencoded; charset=UTF-8

Content-Length: 88

connect_status=-1; __utmb=50703532.0.10.1303366930

DNT: 1

Connection: keep-alive

Pragma: no-cache

Cache-Control: no-cache

method=tagged.usermgmt.addFriend&uid_to_add=5402501977&api_signature=&track=1mJ0lY7-W3

I understand everything up until the method. As far as I know you can only supply GET or POST to HttpWebRequest.Method. Could someone maybe fill in the blanks for me?

Answer 1

If you are trying to talk to the server in the same manner that javascript does however you might have some options.

First, you have to determine how those pages communicate with the server. For example, JSON or SOAP. You can do this with a network sniffer such as wireshark.

Once you have done this, you can send a JSON request of your own to that server using and parse the JSON response using one of the available C# JSON parsers such as JSON.NET.

You can use a similar method in the case of SOAP. Be mindful they might not like you talking to their API's this way.

Answer 2

Is it possible to retrieve data from a page that loads content dynamically through javascript by using WebClient/HttpWebRequest?

Nope - you can retrieve all of the JavaScript files and other documents required to execute the JavaScript, but the HttpWebRequest class won't execute that JavaScript for you - this is an altogether more complex task.

If you wanted to do this then you could host a web browser inside your application, get the browser to open and "render" the page, then inspect the result. In general however is far easier just to come up with an alternative solution based on the situation, for example if the JavaScript populates the page from an AJAX request then just run the request yourself directly.

Would I be able to use WebClient/HttpWebRequest to replicate a XMLHttpRequest like you would see executed via javascript?

If you mean "Can I use HttpWebRequest to execute a HTTP request similar to an AJAX request executed by client JavaScript" then the answer is yes, however exactly how you do this will depend on the JavaScript and the parameters used to exeucte the AJAX request.

In this situation a web debugging tool (such as Fiddler) is useful as it allows you to inspect and compare the request being made.