Nifi ExecuteStreamCommand returns UnicodeEncodeError

0
votes

I am executing a python script within Nifi via ExecuteStreamCommand. When I run the file within the command line, I do not get any errors, and the output is correct. When I run the same file within Nifi- I recieve a unicode error. Is there a workaround for this?

error log:

'unicodeencodeerror python 'charmap' codec can't encode charecter '\u221'

I am saving a pandas df that consists of multiple data types. The code that is envoking the error:

wo_new.to_csv(sys.stdout, encoding='utf-8')

Within Nifi ExecuteStreamCommand Properties:

command path: Python

command arguments: references location of .py file

Things I have already tried:

  • forcing utf-8 encoding
  • forcing windows encoding cp1252 (the type that the error log says anaconda is referencing)
  • enabling unicode via these two techniques https://vladikk.com/2017/03/30/nifi-unicode/
  • normalizing the df to ascii via unicodedata (this application didn't work- .str errors for aplying at a column level as well as to single string)
1
pythonpandasunicodeapache-nifiunicode-normalization

1 Answers

0
votes

Forcing utf-8 encoding on each column with text seemed to do the trick. I don't know why applying the encoding when saving the output didn't do the trick. I used the following code:

x.str.encode('utf-8')