sqlite3 .read command not working with unicode script

0
votes

I used command .read script.sql (of command line tool sqlite3.exe that I've downloaded from https://www.sqlite.org/download.html) to import data to an existing database sqlite. File script.sql was saved with notepad++ in encoding UTF8 without BOM, has content like this:

INSERT INTO "table1" VALUES('Các con cáo nâu nhanh chóng nhảy chó lười biếng');
INSERT INTO "table1" VALUES('速い茶色のキツネは、のろまなイヌに飛びかかっ');
INSERT INTO "table1" VALUES('빠른 갈색 여우가 게으른 개 점프');

But after I ran above command, data was convert to ansii, like this:

Các con cáo nâu nhanh chóng nhảy chó lÆ°á»i biếng
速ã„茶色ã®ã‚­ãƒ„ãƒã¯ã€ã®ã‚ã¾ãªã‚¤ãƒŒã«é£›ã³ã‹ã‹ã£
빠른 갈색 여우가 게으른 ê°œ ì í”„

Does anybody know how to solve the problem?

1
sqlcmdsqlite
When running sqlite3.exe in the Windows console, UTF-8 keyboard input or screen output is not handled correctly. - CL.
@CL. I did not enter unicode character directly into console. I just ran .read script.sql. I checked data by tool SQLite Expert, not by output to console screen. - KevinBui
What is the output of SELECT hex(Column1) FROM ... for the first row? - CL.
@CL. the hex is 43C383C2A16320636F6E2063C383C2A16F206EC383C2A275206E68616E68206368C383C2B36E67206E68C3A1C2BAC2A379206368C383C2B3206CC386C2B0C3A1C2BBC29D69206269C3A1C2BAC2BF6E67. This is my sample, you can download check it mediafire.com/download/xc1kc744dbjcyf5 - KevinBui
I guess your database data are not converted; the problem lays in displaying them in Windows console. Change the codepage to UTF-8 by executing chcp 65001 before running sqlite3.exe. - JosefZ

1 Answers

1
votes

According to the source code, this is a bug in the sqlite3 tool. On Windows in interactive mode it unconditionally converts all input from ANSI to UTF-8, even from file.

To fix this issue line 488 of shell.c (an amalgamation, version 3.11.1) can be changed to: 

if( stdin_is_interactive && in == stdin){

Just checked it and fixed my similar issue.