JSON character encoding in javascript different from java

Question

the java code below

    JSONObject obj = new JSONObject();
    try{
        obj.put("alert","•é");
        byte[] test = obj.toString().getBytes("UTF-8");
        logger.info("bytes are"+ test);
    } catch (JSONException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (UnsupportedEncodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    };

produces a JSONObject which escapes the bullet character, but not the latin letter e with grave, e.g ""\u2022é", the byte code is [123, 34, 97, 108, 101, 114, 116, 34, 58, 34, 92, 117, 50, 48, 50, 50, -61, -87, 34, 125]

How can get I the same exact output in Javascript (in terms of byte sequence)? I don't understand why JSONObject is only escaping one character but not the other. I don't know what rule it followed.

It seems in javascript I can only either escape everything other than the ASCII, (eg.\u007f-\uffff) or don't escape at all.

Thanks!

What is the purpose of creating a byte[] anyway? That's a different issue that the escaping shown. — user2864740
because the length of the byte array is used later in the backend, that is why the front end javascript code needs to calculate the exact length the final byte array in the java code — user3277841
The back-end should calculate the length then. The front-end can guess at the length, but it is the back-end which is responsible and the authoritative source (and it should be understood that the length itself is not necessary canonical, but merely the result of the current operation). — user2864740
Unfortunately the UI can't afford a backend call to do that, it needs to provide user feedback right away when the characters are typed in. — user3277841

user2864740 user2864740 · Accepted Answer · 2014-06-11T00:17:09

There are two different things happening: Unicode encoding and JSON string escaping.

Per 2.5 Strings of the JSON RFC:

.. All Unicode characters may be placed within the quotation marks except for the characters that must be escaped ..

Any character may be escaped. If the character is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence .. [and characters outside the BMP are escaped as UTF-16 encoded surrogate pairs]

That is, the JSON strings of "•é" and "\u2022é" are equivalent. It is entirely up to the serialization implementation on which (additional) characters to escape, and both forms are valid.

It is this JSON string (which is Unicode text) that can be encoded when converted to a byte-stream. In the example it's encoded via UTF-8 encoding. A JSON string may then be equivalent without being byte-equivalent at the stream level or character-equivalent at the JSON text level.

As far as the rules for JSONObject, it escapes according to

    c < ' '
|| (c >= '\u0080' && c < '\u00a0')
|| (c >= '\u2000' && c < '\u2100')

One reason these characters, in the range [\u2000, \u2100], may be escaped is to ensure the resulting JSON is also valid JavaScript. The article JSON: The JavaScript subset that isn't discusses the issue: the problem is the Unicode code-points \u2028 and \u2029 are treated as line terminators in JavaScript string literals, but not JSON. (There are other Unicode Separator characters in the range: might as well catch them in one go.)

JSON character encoding in javascript different from java

1 Answers