How does Facebook encode emoji in the json Graph API?

3
votes

Does anyone know how Facebook encodes emoji with high-surrogate pairs in the Graph API?

Low surrogate pairs seem fine. For example, ❤️ (HEAVY BLACK HEART, though it is red in iOS/OSX, link to image if you can't see the emoji) comes through as \u2764\ufe0f which appears to match the UTF-16 hex codes / "Formal Unicode Notation" shown here at iemoji.com.

And indeed, in Ruby when parsing the JSON output from the API:

ActiveSupport::JSON.decode('"\u2764\ufe0f"')

you correctly get:

"❤️"

However, to pick another emoji, ???? (SLEEPING SYMBOL, link to image here. Facebook returns \udbba\udf59. This seems to correspond with nothing I can find on any unicode resources, e.g., for example this one at iemoji.com.

And when I attempt to decode in Ruby using the same method above:

ActiveSupport::JSON.decode('"\udbba\udf59"')

I get:

"????"

Any idea what's going on here?

1
rubyfacebookfacebook-graph-apiunicodeemoji
\u2764\ufe0f isn't a surrogate pair, it's a normal Basic Multilingual Plane character followed by a variation selector. Using a variant to try to distinguish when emoji should be rendered as colour icons is an ugly new addition in Unicode 6.2. \udbba\udf59 does seem to be an error though... the corresponding codepoint U+FEB59 is Private Use character that you shouldn't be getting. - bobince
There are no "high-surrogate pairs" and no "low surrogate pairs". Valid surrogate pairs (in UTF-16) are composed of one low surrogate and one high surrogate (in that order). None of the characters in your first example is a surrogate. - R. Martinho Fernandes
I clearly don't understand this well enough to use the right language. Any emoji character where the unicode looks like U+2764 works. But one that looks like U+1F4A4 (note the 1) does not. - philoye
@bobince U+FEB59 is a clue. It is the "Google" encoding according to this page on unicode.org. Is that the answer then, for "proposed" encodings per that table, Facebook is using the "Google" version instead? - philoye
@philoye: Ahh! Good catch, I had completely forgotten about the temporary code points in that proposal... weird that anyone is actually using it, but I guess Facebook must just have been implementing emoji at that uncertain period before Unicode 6.0. You can grab the mapping data from emoji4unicode - bobince

1 Answers

2
votes

Answering my own question though most of the credit belongs to @bobince for showing me the way in the comments above.

The answer is that Facebook encodes emoji using the "Google" encoding as seen on this Unicode table.

I have created a ruby gem called emojivert that can convert from one encoding to another, including from "Google" to "Unified". It is based on another existing project called rails-emoji.

So the failing example above would be fixed by doing:

string = ActiveSupport::JSON.decode('"\udbba\udf59"')
> "󾭙"
fixed = Emojivert.google_to_unified(string)
> "💤"