Script to convert unicode characters in <U9999> format to their ASCII equivalents

2
votes

Im doing some changes in Linux locale files /usr/share/i18n/locales (like pt_BR), to change the default format of dates, time, numbers, etc. But since unicode chars are presented as strings in the <U9999> format, text is very hard to read.

Here is a snippet of it:

LC_TIME
abday   "<U0044><U006F><U006D>";"<U0053><U0065><U0067>";/
    "<U0054><U0065><U0072>";"<U0051><U0075><U0061>";/
    "<U0051><U0075><U0069>";"<U0053><U0065><U0078>";/
    "<U0053><U00E1><U0062>"

So, how to make a simple script (may be bash, python, pearl, whatever) to convert this text replacing the <Uxxxx> codes to their ASCII equivalents? (yes, they are all ASCI chars below 255, most even below 127)

If several answers are received, Ill accept the most elegant and/or the more detailed explained one (like options and flags used in comands)

As an example, the above text would be converted to:

LC_TIME
abday   "Dom";"Seg";/
    "Ter";"Qua";/
    "Qui";"Sex";/
    "Sáb"

Bonus points for another script that could do the opposite: convert all chars of a given string to <Uxxx> format.

Thanks!

2
pythonxmlbashunicode
I think there are tools in the XML pantheon of utilties that could handle this much better. I recommend you tag this post with XML (maybe XSLT too?). Good luck! - shellter
The Unicode notation is used because not all Unicode characters have an ASCII equivalent. So, what do you want done with Unicode sequences that don't have an ASCII equivalent (which is the majority of Unicode sequences - by the number of possible sequences; not necessarily the majority by the number of used sequences). - Jonathan Leffler
Characters U0080 to U00FF (i.e. decimal from 128 to 255) inclusive are NOT ASCII. They would have to be converted using an encoding of your choice, probably latin1 in your case. - John Machin
@shelter: Ok, ive added the tag. Are there any tools in simple linux scripting (bash, python, etc) that handle XML? - MestreLion
@Jonathan/John: true, but in that file (pt_BR) 99% of the unicode used is actually asci chars between U0020 and U0079. Very few is between 80 and FF, and none is above FF. So they would be perfectly printable in my system. I dont mind if a few characters go wrong, as long as 99% of the text becomes human readable. Its very time-consuming to check an ASCI table, char by char, just to decode strings like "%d %Y %z %HH:%MM", or "%d-%m-%Y", or "Saturday". I would like to change date, currency formats, but it would take hours to decode, and then re-encode. Hence the need for a script to help - MestreLion

2 Answers

2
votes

Using Fields

#!/bin/bash

awk -F'<U0+|>' '{
    for(i=1;i<=NF;i++)
        if($i ~ "^[0-9A-F]+$")
            $i=sprintf("%c", strtonum("0x"$i))
}1' OFS="" /path/to/infile

Explanation

  1. -F'<U0+|>': This is the magic that makes this script so short. We tell awk that the field separator is either <U0+ or a simple >. The benefit of doing this is that awk will auto-strip these characters for us so we don't have to do it manually with gsub() when it comes time to do the strtonum() conversion.

  2. for(i=1;i<=NF;i++): iterate over each field

  3. if($i ~ "^[0-9A-F]+$"): check if the current field is only composed of hex digits. Remember that due to #1 above something like <U006F> will be seen as 6F at this point
  4. $i=sprintf("%c", strtonum("0x"$i)): replace the hex digit with its corresponding ascii value. We must prefix the field $i with "0x" so awk knows its a hex value
  5. }1: shortcut for a mandatory print or always print each line
  6. OFS="": set the Output Field Separator to the null string. If we don't do this, we will get spaces in the output everywhere there was a <U0+ or >

Using match() [requires gawk]

#!/bin/bash

gawk '{
    while(match($0, /<U[0-9A-F]+>/)){
        pat = substr($0,RSTART,RLENGTH)
        gsub(/U0+|[<>]/,"",pat)
        asc = sprintf("%c", strtonum("0x"pat))
        $0 = substr($0, 1, RSTART-1) asc substr($0, RSTART+RLENGTH)
    }
}1' /path/to/infile
1
votes

Here's a script in Python that converts <U9999> strings to their ASCII (0-127) equivalent using unidecode module:

#!/usr/bin/env python
import fileinput, re, sys
from unidecode import unidecode # to install, run: $ pip install unidecode

for line in fileinput.input(inplace='--inplace' in sys.argv):
    print re.sub(r'<U([0-9A-F]{4})>',
                 lambda m: unidecode(unichr(int(m.group(1), 16))),
                 line),

It accepts input from stdin and/or files given at command-line.

$ u9999-to-ascii data.in
LC_TIME
abday   "Dom";"Seg";/
    "Ter";"Qua";/
    "Qui";"Sex";/
    "Sab"

Note, there is no á character due to ascii doesn't support it, so the script replaced it by its ascii analog a.

If you don't need ascii then:

#!/usr/bin/env python
from __future__ import print_function
import fileinput, re, sys

for line in fileinput.input(mode='rb', inplace='--inplace' in sys.argv):
    print(re.sub(br'<U([0-9A-F]{4})>', lambda m: br'\u'+m.group(1),
                 line).decode('raw-unicode-escape'), end='')

This script works in both Python2.6+ and Python3.x. Example:

$ u9999-to-unicode.py data.in
LC_TIME
abday   "Dom";"Seg";/
    "Ter";"Qua";/
    "Qui";"Sex";/
    "Sáb"

Note, there is á. This script might produce an error if your terminal encoding doesn't support all Unicode characters from data.in. You could use .encode() method in this case.