jsoup - extract text from wikipedia article

3
votes

I'm writing some Java code in order to realize NLP tasks upon texts using Wikipedia. How can I use JSoup to extract all the text of a Wikipedia article (for example all the text in http://en.wikipedia.org/wiki/Boston)?

3
javaparsingjsoupwikipedia
Is parsing the text with jsoup part of the interesting problem? Because if not, you should just use the action=raw parameter to get the source for each page. e.g. en.wikipedia.org/w/index.php?title=Elephant&action=raw - beerbajay
That returns the Wiki markup. - Hauke Ingmar Schmidt
use this, it's more robust and esier on the wikipedia servers too: trulymadlywordly.blogspot.com/2011/03/… - Maarten

3 Answers

3
votes
Document doc = Jsoup.connect("http://en.wikipedia.org/wiki/Boston").get();
Element contentDiv = doc.select("div[id=content]").first();
contentDiv.toString(); // The result

You retrieve formatted content this way, of course. If you want "raw" content you can filter the result with Jsoup.clean or use the call contentDiv.text().

3
votes
Document doc = Jsoup.connect(url).get();
    Elements paragraphs = doc.select(".mw-content-ltr p");

    Element firstParagraph = paragraphs.first();
    Element lastParagraph = paragraphs.last();
    Element p;
    int i=1;
    p=firstParagraph;
    System.out.println(p.text());
    while (p!=lastParagraph){
        p=paragraphs.get(i);
        System.out.println(p.text());
        i++;
    }
0
votes
Document doc = Jsoup.connect("http://en.wikipedia.org/wiki/Boston").timeout(5000);

Element iamcontaningIDofintendedTAG= doc.select("#iamID") ;

System.out.println(iamcontaningIDofintendedTAG.toString());

OR

Elements iamcontaningCLASSofintendedTAG= doc.select(".iamCLASS") ;

System.out.println(iamcontaningCLASSofintendedTAG.toString());