Retrieve comments from website using disqus

5
votes

I would like to write a scraping script to retrieve comments from cnn articles. For example, this article: http://www.cnn.com/2012/01/19/politics/gop-debate/index.html?hpt=hp_t1

I realize that cnn uses disqus for their comment discussion. As the comment loading is not webpage-based (ie, prev page, next page) and is dynamic (ie, need to click "load next 25"), I have no idea how to retrieve all the 5000+ comments for this article.

Any idea or suggestion?

Thanks so much!

2
web-scrapingdisqus
Are you asking how to scrape? or do you know how to scrape using a certain tool and you need assistance with scraping a disqus-based comment system? if its the latter then retag you question to include the name of the tool - Boaz
Hi Boaz.Jan, I am asking how to scrape. For webpage-based newswire page such as guardian.co.uk/politics/2012/jan/10/…, we just need to replace commentpage by 2, 3, 4.. and we can do a http get to download the page and scrape it. With disqus, seems that there is no way to retrieve all comments with http get... - qwertyl

2 Answers

5
votes

I needed to get comments via scraping a page that had disqus comments via ajax. Because they were not rendered on the server, I had to call the disqus api. In the source code, you will need the identifier code: 

var identifier = "456643" // take note of this from the page source
// this is the ident url query param in the following js request

also,look in the js source code to get the pages public key, and forum name. Place these in the url where appropriate.

I used javascript nodejs to test this, ie :

var request = require("request");

var publicKey  = "pILMw27bsbJsdfsdQDh9Eh0MzAgFL6xx0hYdsdsdfaIfBHRvLGqFFQ09st";

var disqusUri = "https://disqus.com/api/3.0/threads/listPosts.json?&api_key=" + publicKey + "&thread:ident=456643&forum=nameOfForumFromSource";

request(disqusUri, function(res,status,err){
    console.log(res.body);

    if(err){
        console.log("ERR: " + err);
    }
});
2
votes

The option for scraping (other then getting the page), which might be less robust (depends on you're needs) but will offer a solution for the problem you have, is to use some kind of wrapper around a full fledged web browser and literally code the usage pattern and extract the relevant data. Since you didn't mention which programming language you know, I'll give 3 examples: 1) Watir - ruby, 2) Watin - IE & Firefox via .net, 3) Selenium - IE via C#/Java/Perl/PHP/Ruby/Python

I'll provide a little example using Watin & C#:

IE browser = new IE();
browser.GoTo(YOUR CNN URL);
List visibleComments = Browser.List(Find.ById("dsq-comments"));
//do your scraping thing
Link moreComments = Browser.Link(Find.ByClass("dsq-paginate-append-text");
moreComments.click();
//wait util ajax ended by searching for some indicator
Browser.WaitUntilContainsText(SOME TEXT);
//do your scraping thing

Notice: I'm not familiar with disqus, but it might be a better option to force all the comments to show by looping the Link & click parts of the code I posted until all the comments are visible and the scrape the List element dsq-comments