Reading InputStream as UTF-8

101
votes

I'm trying to read from a text/plain file over the internet, line-by-line. The code I have right now is:

URL url = new URL("http://kuehldesign.net/test.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
LinkedList<String> lines = new LinkedList();
String readLine;

while ((readLine = in.readLine()) != null) {
    lines.add(readLine);
}

for (String line : lines) {
    out.println("> " + line);
}

The file, test.txt, contains ¡Hélló!, which I am using in order to test the encoding.

When I review the OutputStream (out), I see it as > ¬°H√©ll√≥!. I don't believe this is a problem with the OutputStream since I can do out.println("é"); without problems.

Any ideas for reading form the InputStream as UTF-8? Thanks!

3
javautf-8inputstream
The HTTP protocol specifies the encoding. Why aren’t you using a library API that handles that for you? You should never have to guess the encoding like this. I don’t mean to be negative: you’re doing great! I just wonder whether there isn’t an easier way. - tchrist
I won't have access to the server which is serving the text/plain file, unfortunately, and it's not using a UTF-8 encoding. I wasn't aware of any good network libraries; any suggestions? - Chris Kuehl
Looking at the docs, I wouldn’t think you would have to specify the encoding at all. I am surprised they give you a byte stream! You do have access to underlying URLConnection, from which you can check the Content-Encoding, then open an InputStreamReader with the correct argument. A quick check of the source doesn’t turn up anything that seems to do that for you, which seems pretty darned lame and error prone, so I probably missed something. - tchrist

3 Answers

198
votes

Solved my own problem. This line:

BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));

needs to be:

BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream(), "UTF-8"));

or since Java 7:

BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream(), StandardCharsets.UTF_8));
18
votes
String file = "";

try {

    InputStream is = new FileInputStream(filename);
    String UTF8 = "utf8";
    int BUFFER_SIZE = 8192;

    BufferedReader br = new BufferedReader(new InputStreamReader(is,
            UTF8), BUFFER_SIZE);
    String str;
    while ((str = br.readLine()) != null) {
        file += str;
    }
} catch (Exception e) {

}

Try this,.. :-)

10
votes

I ran into the same problem every time it finds a special character marks it as ��. to solve this, I tried using the encoding: ISO-8859-1

BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("txtPath"),"ISO-8859-1"));

while ((line = br.readLine()) != null) {

}

I hope this can help anyone who sees this post.