perl utf8 corruption

0
votes

I am using the perl module sapnwrfc to connect to SAP and retrieve reports. This module uses utf8 and when the data is returned some of the data has a pattern of utf8 character corruption. This appears to happen when a line in the SAP report is more than 4096 in length and my current thinking is that the read buffer of perl is splitting utf8 characters and causing the corruption.

$abap_lookup = $sap_rfc->function_lookup("REPORT");
$abap_program = $abap_lookup->create_function_call;

# set abap program input variables
$abap_program->REPORT($abap_program_name);
$abap_program->VARIANT($abap_variant_name);

# call the abap program
$abap_program->invoke;

$abap_program->DATA has the corruption in one place in each line that is more than 4Kb

This is the fragment with the corruption, the actual line is a byte or two more than 4Kb.

\x{f8fc}\x{2500}     \x{500}/\x{f8fc}\x{2500}

This is what is expected, so I am assuming something is splitting the line and causing the problem.

\x{f8fc}\x{2500}\x{f8fc}\x{2500}\x{f8fc}\x{2500}

I have tried all manner of open ':utf8' pragma and other settings (use utf8, binmode(STDIN, ":utf8"), binmode(STDOUT, ":utf8");). Also have tried to turn off buffering ($| = 1;). I cannot tell if this is a utf8 problem or a buffering problem. Does anyone know why this would be doing this and how to fix it?

1
perlutf-8
Ouch! That's as close to current as it gets...so often it turns out to Perl 5.6.1 or 5.8.2 or something similarly archaic where UTF8 support was known to be iffy. - Jonathan Leffler
I'm puzzled by the \x{f8fc} value...is that supposed to be a Unicode character U+F8FC (which is a character in the BMP Private Use Area - U+E000-U+F8FF), or the UTF8 encoding of a character? If the latter, it is bogus; no valid UTF8 character contains bytes 0xC0, 0xC1, or 0xF5..0xFF. - Jonathan Leffler
@tchrist recommends use open qw( :std IO :utf8 ). - Sinan Ünür
Well, it works for me, but I think this is something else. I don’t trust that the utf8_on bit is actually set on the stuff he’s pulling out of the database. He should use Devel::Peek and check. - tchrist

1 Answers

0
votes

was not able to figure out where the corruption is happening, but it is repeatable so I built a filter.