Properly displaying UTF-8 chars in Perl

Question

I am running perl 5, version 24, subversion 3 (v5.24.3) built for MSWin32-x64-multi-thread (with 1 registered patch, see perl -V for more detail) (Active State).

Trying to parse HTML page encoded in UTF-8:

$request = new HTTP::Request('GET', $url);
$response = $ua->request($request);
$content = $response->content();

I parse the $content as one giant string using INDEX and SUBSTR functions, that works fine. HTML page contains string with value ÖBB and I need to insert it in the database exactly as ÖBB When I print it and insert in the db, instead of Ö I get some ascii characters.

NOTE: this question is not database related; MySQL handles utf-8 just fine, so if I insert value "ÖBB" it will take it no problem.

I've looked at great number of similar questions/answers here and in other forums and I am none wiser.

use utf-8 and binmode(STDOUT, ":utf8") has not worked for me... Would greatly appreciate a code snippet that would solve the issue, thank you.

What database do you use? What driver? How do you connect to the database? — choroba
Consider using utf8::all which tries to make Perl as UTF8 friendly as possible. — Schwern
When you say "print", do you mean print to console? In that case, make sure you have a the Unicode codepage active. Type chcp on the command line. If it says "65001", you're good in that regard. If not, type chcp 65001 to activate the correct codepage. — Holli
Does your database configured to support UTF8? You did not indicate what database you use. If you use MYSQL DB then see the following instruction. In case if you in MS Windows do not forget to change code page in terminal window chcp 65001. — Polar Bear

ikegami ikegami · Accepted Answer · 2020-01-04T05:40:46

Decode inputs; encode outputs.

First of all, you don't decode your inputs.

$response->content returns the raw content that could be in any encoding. Use $response->decoded_content(); to get the decoded response if it's HTML.

Second of all, you might not be encoding your outputs.

You didn't specify which database driver you use. Most DBI drivers have an option you need to specify. For example, with MySQL, you want

my $dbh = DBI->connect(
   'dbi:mysql:...',
   $user, $password,
   {
      mysql_enable_utf8mb4 => 1,
      ...
   },
);

You mentioned use utf8;. That tells Perl that your source code is encoded using UTF-8 rather than ASCII. Do use it if your source code is encoded using UTF-8.

This is not directly related to your issue.

You mentioned binmode(STDOUT, ":utf8"). That's a very poor way of writing

use open ':std', ':encoding(UTF-8)';

The above handles that for STDIN, STDOUT and STDERR, and does so at compile time. It also sets the default for files open in scope of the pragma.

But that's assuming the terminal expects UTF-8. That would be the case if you used chcp 65001. For a version that handles whatever encoding the terminal expects, you can use the following:

BEGIN {
   require Win32;
   my $cie = "cp" . Win32::GetConsoleCP();
   my $coe = "cp" . Win32::GetConsoleOutputCP();
   my $ae  = "cp" . Win32::GetACP();

   binmode(STDIN,  ":encoding($cie)");
   binmode(STDOUT, ":encoding($coe)");
   binmode(STDERR, ":encoding($coe)");

   require open;
   "open"->import(":encoding($ae)");
}

This has a few more details.

This is not directly related to your issue.

Properly displaying UTF-8 chars in Perl

3 Answers