Problem with decoding UTF-8 JSON in perl

Question

UTF-8 characters are destroyed when processed with the JSON library (maybe this is similar to Problem with decoding unicode JSON in perl, however setting binmode only creates another problem).

I have reduced the problem down to the following example:

(hlovdal) localhost:/tmp/my_test>cat my_test.pl
#!/usr/bin/perl -w
use strict;
use warnings;
use JSON;
use File::Slurp;
use Getopt::Long;
use Encode;

my $set_binmode = 0;
GetOptions("set-binmode" => \$set_binmode);

if ($set_binmode) {
        binmode(STDIN,  ":encoding(UTF-8)");
        binmode(STDOUT, ":encoding(UTF-8)");
        binmode(STDERR, ":encoding(UTF-8)");
}

sub check {
        my $text = shift;
        return "is_utf8(): " . (Encode::is_utf8($text) ? "1" : "0") . ", is_utf8(1): " . (Encode::is_utf8($text, 1) ? "1" : "0"). ". ";
}

my $my_test = "hei på deg";
my $json_text = read_file('my_test.json');
my $hash_ref = JSON->new->utf8->decode($json_text);

print check($my_test), "\$my_test = $my_test\n";
print check($json_text), "\$json_text = $json_text";
print check($$hash_ref{'my_test'}), "\$\$hash_ref{'my_test'} = " . $$hash_ref{'my_test'} .  "\n";

(hlovdal) localhost:/tmp/my_test>

When running testing the text is for some reason crippeled into iso-8859-1. Setting binmode sort of solves it but then causes double encoding of other strings.

(hlovdal) localhost:/tmp/my_test>cat my_test.json 
{ "my_test" : "hei på deg" }
(hlovdal) localhost:/tmp/my_test>file my_test.json 
my_test.json: UTF-8 Unicode text
(hlovdal) localhost:/tmp/my_test>hexdump -c my_test.json 
0000000   {       "   m   y   _   t   e   s   t   "       :       "   h
0000010   e   i       p 303 245       d   e   g   "       }  \n        
000001e
(hlovdal) localhost:/tmp/my_test>
(hlovdal) localhost:/tmp/my_test>perl my_test.pl
is_utf8(): 0, is_utf8(1): 0. $my_test = hei på deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei p� deg
(hlovdal) localhost:/tmp/my_test>perl my_test.pl --set-binmode
is_utf8(): 0, is_utf8(1): 0. $my_test = hei på deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei på deg
(hlovdal) localhost:/tmp/my_test>

What is causing this and how to solve?

---

This is on a newly installed and up to date Fedora 15 system.

(hlovdal) localhost:/tmp/my_test>perl --version | grep version
This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux-thread-multi
(hlovdal) localhost:/tmp/my_test>rpm -q perl-JSON
perl-JSON-2.51-1.fc15.noarch
(hlovdal) localhost:/tmp/my_test>locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
(hlovdal) localhost:/tmp/my_test>

---

Update: Adding use utf8 does not solve it, characters are still not processed right (although slightly different from before):

(hlovdal) localhost:/tmp/my_test>perl  my_test.pl
is_utf8(): 1, is_utf8(1): 1. $my_test = hei p� deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei p� deg
(hlovdal) localhost:/tmp/my_test>perl  my_test.pl --set-binmode
is_utf8(): 1, is_utf8(1): 1. $my_test = hei på deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei på deg
(hlovdal) localhost:/tmp/my_test>

As noted by perlunifaq

Can I use Unicode in my Perl sources?

Yes, you can! If your sources are UTF-8 encoded, you can indicate that with the use utf8 pragma.

use utf8;

This doesn't do anything to your input, or to your output. It only influences the way your sources are read. You can use Unicode in string literals, in identifiers (but they still have to be "word characters" according to \w ), and even in custom delimiters.

Answer 1

You saved your program in UTF-8, but forget to tell Perl. Add use utf8;.

Also, you are programming much too complicated. The JSON functions DWYM. To inspect stuff, use Devel::Peek.

use utf8; # for the following line
my $my_test = 'hei på deg';

use Devel::Peek qw(Dump);
use File::Slurp (read_file);
use JSON qw(decode_json);

my $hash_ref = decode_json(read_file('my_test.json'));

Dump $hash_ref; # Perl character strings
Dump $my_test;  # Perl character string

Answer 2

Is it only my impression, or does this perl library expect you to write UTF-8 byte code into a isoLatin1 string (utf-8 flag disabled on the string); And likewise, it returns to you UTF-8 byte code in an iso latin string:

#! /usr/bin/perl -w
use strict;
use Encode;
use Data::Dumper qw(Dumper);
use JSON; # imports encode_json, decode_json, to_json and from_json.
use utf8;

###############
## EXAMPLE 1:
################
my $json = JSON->new->allow_nonref;
my $exampleAJsonObj = { key1 => 'a'};
my $exampleAText = $json->utf8->encode( $exampleAJsonObj ); 
my $exampleAJsonObfUtf = { key1 => 'ä'};
my $exampleATextUtf = $json->utf8->encode( $exampleAJsonObfUtf); 


#binmode(STDOUT, ":utf8");
print "EXAMPLE1: ";
print "\n";
print encode 'UTF-8', "exampleAText: $exampleAText and as object: " . Dumper($exampleAJsonObj);
print "\n";
print encode 'UTF-8', "exampleATextUtf: $exampleATextUtf and as object: " . Dumper($exampleAJsonObfUtf) . " Key1 was: " . $exampleAJsonObfUtf->{key1};
print "\n";
print hexdump($exampleAText);
print "\n";
print hexdump($exampleATextUtf);
print "\n";

#############################
## SUB.
#############################
# For a given string parameter, returns a string which shows
# whether the utf8 flag is enabled and a byte-by-byte view
# of the internal representation.
#
sub hexdump
{
    my $str = shift;
    my $flag = Encode::is_utf8($str) ? 1 : 0;
    use bytes; # this tells unpack to deal with raw bytes
    my @internal_rep_bytes = unpack('C*', $str);
    return
        $flag
        . '('
        . join(' ', map { sprintf("%02x", $_) } @internal_rep_bytes)
        . ')';
}

Finally, The output is:

exampleAText: {"key1":"a"} and as object: $VAR1 = {
          'key1' => 'a'
        };

exampleATextUtf: {"key1":"ä"} and as object: $VAR1 = {
          'key1' => "\x{e4}"
        };
 Key1 was: ä
0(7b 22 6b 65 79 31 22 3a 22 61 22 7d)
0(7b 22 6b 65 79 31 22 3a 22 c3 a4 22 7d)

So, we see that at the end of this process neither of the outpu strings is an UTF-8 string, which is false. At the very least, the 0(7b 22 6b 65 79 31 22 3a 22 c3 a4 22 7d). Notice that c3 A4 is the correct byte code for ä http://www.utf8-chartable.de/

Therefore, library seems expect for one to lode into a non utf-8 string the utf-8 byte code, and as a result, it will do the same thing, it will output a NON utf-8 string with utf-8 byte code.

Am I wrong?

Further experimentation has lead me to the concluions that: perlObjects returned and consumed have strings flagged as UTF-8 (as i would have expected). perl strings consumed and returned from decode/encode must appear to perl as ISO latin 1 strings but have utf8 byte code. So, when opening a file containing an UTF8 json, do not use "<:encoding(UTF-8)".

Answer 3

The core of the problem was JSON's expectation of an octet array instead of character string (solved in this question). However I was also missing several things related to unicode, like "use utf8". Here is the diff required to make the code in the example work fully:

--- my_test.pl.orig 2011-08-03 15:44:44.217868886 +0200
+++ my_test.pl  2011-08-03 15:55:30.152379269 +0200
@@ -1,19 +1,14 @@
-#!/usr/bin/perl -w
+#!/usr/bin/perl -CSAD
 use strict;
 use warnings;
 use JSON;
 use File::Slurp;
 use Getopt::Long;
 use Encode;
-
-my $set_binmode = 0;
-GetOptions("set-binmode" => \$set_binmode);
-
-if ($set_binmode) {
-        binmode(STDIN,  ":encoding(UTF-8)");
-        binmode(STDOUT, ":encoding(UTF-8)");
-        binmode(STDERR, ":encoding(UTF-8)");
-}
+use utf8;
+use warnings qw< FATAL utf8 >;
+use open qw( :encoding(UTF-8) :std );
+use feature qw< unicode_strings >;

 sub check {
         my $text = shift;
@@ -21,8 +16,9 @@
 }

 my $my_test = "hei på deg";
-my $json_text = read_file('my_test.json');
-my $hash_ref = JSON->new->utf8->decode($json_text);
+my $json_text = read_file('my_test.json', binmode => ':encoding(UTF-8)');
+my $json_bytes = encode('UTF-8', $json_text);
+my $hash_ref = JSON->new->utf8->decode($json_bytes);

 print check($my_test), "\$my_test = $my_test\n";
 print check($json_text), "\$json_text = $json_text";