C++14: Conversion between UTF-8/UTF-16 and native character encoding

Question

I have 4 closely related questions:

Does C++14 have a built-in mechanism for converting between UTF-8 and the systems native multibyte encoding, i.e., the multibyte encoding assumed by the std::codecvt<wchar_t, char> specialization (http://en.cppreference.com/w/cpp/locale/codecvt)?
Does C++14 have a built-in mechanism for converting between UTF-8 and the systems native wide character encoding, i.e., the wide character encoding assumed by the std::codecvt<wchar_t, char> specialization (http://en.cppreference.com/w/cpp/locale/codecvt)?
Same as question 1, but for UTF-16 instead of UTF-8.
Same as question 2, but for UTF-16 instead of UTF-8.

EDIT: I realize that a "yes" to any of these questions, effectively means "yes" to all 4, because C++14 clearly does provide ways of converting between UTF-8 and UTF-16 (std::codecvt<char16_t, char, std::mbstate_t>), as well as between native multibyte and native wide character encodings (std::codecvt<wchar_t, char>).

Yes, yes, yes, yes, but for the first I think you have to detour via the wide encoding. Also remember to set proper locale. The default, C locale, isn't usable. — Cheers and hth. - Alf
I always found C++'s character set conversions to be awkward and difficult to work with. libiconv is just as portable, and much easier to work with. — Sam Varshavchik
@KristianSpangsege: If you're looking for something generic, you shouldn't be using narrow or wide characters. You should be using strings with a specific encoding to them. — Nicol Bolas
See the table documented on cppreference.com for std::codecvt and std::wstring_convert(), it shows the available standard conversions. — Remy Lebeau

Cubbi Cubbi · Accepted Answer · 2017-02-27T18:20:23

the systems native multibyte encoding, i.e., the multibyte encoding assumed by the std::codecvt<wchar_t, char> specialization

There is confusion, possibly due to misleading wording on cppreference (my fault, now fixed to match the standard and reality); in the existing implementations (libc++ and libstdc++), the locale-independent codecvt specialization codecvt<wchar_t, char> does not deal with any multibyte encodings. The standard wording is "native character sets for narrow and wide characters.", and the existing implementations took that to mean 1:1 conversions only, like what btowc/wctob do in C:

template<class F> struct facet : F { using F::F; ~facet() {} };
facet<std::codecvt<wchar_t, char, std::mbstate_t>> fp;
std::cout << fp.max_length() << '\n'; // prints 1 in libc++ and libstdc++

in fact, here's the libc++ implementation

In every useful context, multibyte encoding is either an encoding specified by a locale-provided codecvt facet, or by a custom codecvt facet, or UTF-8 (provided by the std::codecvt_utf8* facets). Meaning,

between UTF-8 and the systems native multibyte encoding

means "between UTF-8 and a multibyte encoding specified by a locale"

#include <codecvt>
#include <locale>
#include <cassert>

template<class F> struct myFacet : F { using F::F; ~myFacet() {} };
int main()
{
    std::string in = u8"水"; // UTF-8
    // utf8 to wide (could've used en_US.utf8, but this one exists as-is)
    std::wstring ws = std::wstring_convert<std::codecvt_utf8<wchar_t>>{}.from_bytes(in);
    assert(ws == L"水");
    // wide to another mb (have to use a named locale now)
    typedef myFacet<std::codecvt_byname<wchar_t, char, std::mbstate_t>> F;
    std::string out = std::wstring_convert<F>{ new F("zh_CN.gb18030") }.to_bytes(ws);
    assert(out == "\xcb\xae");
}

Does C++14 have a built-in mechanism for converting between UTF-8 and the systems native wide character encoding

Native wide is effectively defined to be Unicode or (as on Windows) its arbitrary subset, and that's what you get from std::codecvt_utf8. A hostile implementation could possibly have a wchar_t holding values numerically different from the Unicode code points, as long as they map 1:1, but given that they must be equal to them for the basic charset, it is unrealistic.

C++14: Conversion between UTF-8/UTF-16 and native character encoding

1 Answers