How to convert Unicode escaped characters to utf8?

0
votes

I saw the other questions about the subject but all of them were missing important details: I want to convert \u00252F\u00252F\u05de\u05e8\u05db\u05d6 to utf8. I understand that you look through the stream for \u followed by four hex which you convert to bytes. The problems are as follows:

  1. I heard that sometimes you look for 4 bytes after and sometimes 6 bytes after, is this correct? If so, then how do you determine which it is? E.g. is \u00252F 4 or 6 bytes?
  2. In the case of \u0025 this maps to one byte instead of two (0x25), why? Is the four hex supposed to represent utf16 which i am supposed to convert to utf8?
  3. How do I know whether the text is supposed to be the literal characters \u0025 or the unicode sequence? Does that mean that all backslashes must be escaped in the stream?
  4. Lastly, am I being stupid in doing this by hand when I can use iconv to do this for me?
2
cencodingutf-8
"then how do you determine which it is?" That's what we should be asking you. Is \u00252F always one single character, or is it the character U+0025 followed by 2 (U+0032) followed by F (U+0046)? "Is the four hex supposed to represent utf16 which i am supposed to convert to utf8?" Again, how can we know? Where does this data come from? - R. Martinho Fernandes
In most escaping systems that use \u, exactly 4 hexadecimal digits must follow the \u and are considered to make up one UTF-16 code unit. - R.. GitHub STOP HELPING ICE
@R.MartinhoFernandes I am asking from a parsing standpoint, i see the data \u00252F how can i determine whether it is \u0025+2F or \u00252F? - chacham15
@chacham15 By knowing what it is you are parsing. - R. Martinho Fernandes
@chacham15: It depends on the specification governing the format of the input. Fundmanetally, \u00252F does not represent either; it represents the backslash character, followed by a lowercase u, followed by the digits 0, 0, 2, 5, 2, and the letter F. - R.. GitHub STOP HELPING ICE

2 Answers

2
votes

If you have the iconv interfaces at your disposal, you can simply convert the \u0123\uABCD etc. sequences to an array of bytes 01 23 AB CD ..., replacing any unescaped ASCII characters with a 00 byte followed by the ASCII byte, then run the array through iconv with a conversion descriptor obtained by iconv_open("UTF-8", "UTF-16-BE").

Of course you can also do it much more efficiently working directly with the input yourself, but that requires reading and understanding the Unicode specification of UTF-16 and UTF-8.

0
votes

In some conventions (like C++11 string literals), you parse a specific number of hex digits, like four after \u and eight after \U. That may or may not be the convention with the input you provided, but it seems a reasonable guess. Other styles, like C++'s \x you parse as many hex digits as you can find after the \x, which means that you have to jump through some hoops if you do want to put a literal hex digit immediately after one of these escaped characters.

Once you have all the values, you need to know what encoding they're in (e.g., UTF-16 or UTF-32) and what encoding you want (e.g., UTF-8). You then use a function to create a new string in the new encoding. You can write such a function (if you know enough about both encoding formats), or you can use a library. Some operating systems may provide such a function, but you might want to use a third-party library for portability.