Unsupported characters (Chinese Characters) in Java String

Question

Following is the function I used to crawl a website and to save the text/content inside a text file. But since the website is in Chinese the data I get from the website is unsupported. I did some research and noticed that String most likely has UTF-16 encoding, which theoretically should support Chinese characters. But in this case, it does not. I even tried to print out some Chinese sentences using print statement in Java and everything worked flawlessly. I just don't understand why it doesn't string object does not support Chinese characters. Can someone here help me please?

    void contentGetter() throws IOException {
//      the string is kind of messed up so all i see in the file is question marks
//        Writer writer = new OutputStreamWriter(
//                new FileOutputStream("a.txt"), "UTF-8");
        ArrayList content = new ArrayList();
//        for (int i = 0; i<urlList.size(); i++){
            URL url;
            InputStream is = null;
            BufferedReader br;
            String line;
            try {
                // the url used here is http://ds.eywedu.com/jinyong/tlbb/mydoc001.htm feel free to try it
                url = new URL((String)urlList.get(0));
                is = url.openStream();
                br = new BufferedReader(new InputStreamReader(is));

                while ((line = br.readLine()) != null) {
                    content.add(line);
                }

                boolean title = false;
                for (int m = 0; m<content.size(); m++){
                    String contents = (String) content.get(m);
                    if (contents.contains("script type=\"text") && !title){
                        title = true;
//                        writer.write(contents.substring(contents.indexOf("\"4\"")+4,contents.indexOf("</font>")));
                        titles.add(
                                contents.substring(contents.indexOf("\"4\"")+4,contents.indexOf("</font>")));
                        for (int j = 0; j<=+1; j++){
                            content.remove(0);
                        }
                    }else if (contents.contains("script type=\"text")){
                        int loc = contents.indexOf("</DIV>");
                        int fLoc = contents.indexOf("<BR>");
                        if (loc != -1 && fLoc != -1){
                            filtered.add(contents.substring(loc+6, fLoc));
//                            writer.write(contents.substring(loc+6, fLoc));
                        }
                    } else if (contents.contains("<BR>")){
//                        writer.write(contents.substring(0,contents.indexOf("<BR>")));
                        filtered.add(contents.substring(0,contents.indexOf("<BR>")));

                    }

                }

            } catch (MalformedURLException mue) {
                mue.printStackTrace();
            } catch (IOException ioe) {
                ioe.printStackTrace();
            } finally {
                try {
                    if (is != null) is.close();
                } catch (IOException ioe) {
                    //exception
                }
            }
        System.out.println(filtered);
//        writer.close();
        }
//    }
}

Lothar Lothar · Accepted Answer · 2017-10-02T15:15:27

br = new BufferedReader(new InputStreamReader(is));

You're opening an InputStreamReader with your system's charset/encoding. This most likely is not the charset/encoding of the HTML-page you receive. You should (always, not only here) use InputStreamReader's constructor that allows you to explicitly specify the charset:

url = new URL((String)urlList.get(0));
URLConnection uc = url.openConnection();
is = uc.getInputStream();
br = new BufferedReader(new InputStreamReader(is, uc.getContentEncoding()));

That might solve your problem but if chinese characters are specified as character entity references (like ) or as numeric entity references (like &#12345) you still have some decoding to do yourself.

Unsupported characters (Chinese Characters) in Java String

1 Answers