SAS session encoding LATIN1 failing for unicode characters

3
votes

My SAS session details are:-

ENCODING=LATIN1
LOCALE=EN_US

I am reading encrypted customer names, decrypting them using a java code that has been implemented in the sas code using javaobj.

data outlib.X1 ( ENCODING =  'asciiany' );
set inlib.INP1 (ENCODING = 'asciiany' )   end=eof;
length decryptedvalue  $100;
declare javaobj jObj("<jarname>");
jObj.callstringmethod("<method>", FIRST_NAME , decryptedvalue);
run;

The jObj.callstringmethod returns the decrypted first_name value in the string decryptedvalue. I do a proc export at the end of my SAS code and store all the decrypted names as csv files.

In the last run some of the names have special characters e.g. RÉAL.

This is causing SAS execution to fail with following error :-

ERROR: Transcoding failure at line 30 column 2.

ERROR: DATA STEP Component Object failure. Aborted during the EXECUTION phase.

Is there some way to make SAS session (LATIN1) accept these unicode characters? Can i set the encoding of the decryptedvalue variable? I dont want to run my entire SAS session in unicode using sas_u8. I only want to accept these characters Even reading these problematic values as blank is ok.

I have already tried following things :-

  1. Set inencoding= utf8 for the libname

  2. Make ENCODING = 'asciiany' for the input dataset.

Any inputs would be useful.

Thanks in advance!

1
utf-8sastranscoding
Have you tried running SAS with Unicode session encoding? I don't see why that is such a problem? – Joe
I tried running with sas_u8 and the decryption works fine. But my main code has redshift db query and they are failing with the following error ** "Error occurred while trying to execute a query: ERROR: Only ASCII characters are allowed in an identifier. Invalid ASCII char: ef bf bd" **. As of now I have split my sas code into 2 parts - one for redshift and the other one for decryption which I run using sas_u8. But this is not a workable solution. – NVK
Hmm, interesting. I would raise that latter issue with SAS technical support, there's no reason you shouldn't be able to connect to redshift (via the appropriate drivers) from a u8 session, the driver should handle the ascii down-conversion. – Joe
Thanks @joe , we have raised this issue with SAS Tech support. Meanwhile my analysis has identified that "Invalid ASCII char: ef bf bd" actually refers to BOM (en.wikipedia.org/wiki/Byte_order_mark). Is there some option to stop this when we run sas_u8 session? Thanks in advance. – NVK

1 Answers

0
votes

SAS Tech support suggested this :- Adding export EASYSOFT_UNICODE=YES in the sasenv_local file .

We are now able to run the entire SAS code (including SQL queries) in the sas u8 session.

Thanks to all for your support.