NiFi : Split Record processor UTF-16 CSV to JSON

Question

Trying to convert a csv tab delimited file to a JSON where the CSV file is a UTF-16.
All the Fields are string in CSV and JSON.

Using a SplitRecord processor in NiFi. The processor runs without any error but the data values converted as JSON looks like below

"\u0000R\u0000e\u0000t\u0000a\u0000i\u0000l......."

Should I mention that this is a UTF-16 in the avro schema? If so how do I do it?

The same works well with a UTF-8 file but not with a UTF-16.

UPDATE: Not able to find the character set Property in the reader.

Using the HDF Version 3.0.1.1. Image below:

daggett daggett · Accepted Answer · 2018-01-30T11:23:00

In the nifi 1.5.0 in the reader service there is a property Character Set. Set it to UTF-16

in case when you are using version without this parameter you can update your version or as a workaround before using SplitRecord you can convert encoding from UTF-16 to UTF-8 with the following execute groovy script:

def ff = session.get()
if(!ff)return
ff = session.write(ff, {rawIn, rawOut->
    // ## transform streams into reader and writer
    rawIn.withReader("UTF-16"){reader->
        rawOut.withWriter("UTF-8"){writer->
            //transfer reader UTF-16 to writer UTF-8
            writer << reader
        }
    }
} as StreamCallback)
session.transfer(ff, REL_SUCCESS)

NiFi : Split Record processor UTF-16 CSV to JSON

1 Answers