Error- invalid char between token and delimiter using ConvertRecord in NiFi

1
votes

I am using a flow as follows(basically to fetch a file from s3 and then convert few records from the main CSV file n later push it to Elasticsearch) : GetSQS ->UpdateAtttribute->SplitJson->EvaluateJsonPath->UpdateAttribute->convertRecord-> other processor...

I am able to fetch the file from s3 correctly but the ConvertRecord processor thows error: Invalid char between encapsulated token a delimiter

Please find the ConvertRecord Configs below:

**CSVRecordReader** : Schema Access strategy as "Use 'Schema Text' Property

Schema Text: 


{
  "type": "record",
  "name": "AVLRecord0",
  "fields" : [
    {"name": "TimeOfDay","type": "string", "logicalType":"timestamp-millis"},
    {"name": "Field_0", "type": "double"},
    {"name": "Field_1", "type": "double"},
    {"name": "Field_2", "type": "double"},
    {"name": "Field_3", "type": "double"}}
]
}
**CSVRecordWritter**: 

Schema Write Strategy : Set 'Avro. schema' Attribute

Schema Access Strategy: Use Schema Text Property

Please tell me why am i not able to see the converted record after succesfully fetching from S3.

The desired output is CSV format only. Please find attached sample file uploaded on s3 and I want to convert only upto field_5.

enter image description here

Attached the contoller services screenshots:

enter image description here

enter image description here

enter image description here

Thank you!

2
apacheamazon-s3apache-nifi
@Shu I have updated the question. Thank You! Please help to fix this. - Shrads
it worked as expected for me xml is here: github.com/shureddy/NiFiTemplates/blob/master/…, Please check 1.The data in your input file and is there any extra quote/comma characters embedded(or not in valid csv format) 2.change ignore csv Header column names property to True in csv reader controller service (as you are not using csv file header). Please let us know what's your findings - notNull
@Shu I fixed it. It was an extra comma in the ConvertRecord processor. Thank You. - Shrads
@Shu I have posted my finding and result. Thank You for your help. - Shrads

2 Answers

0
votes

I have figured my error: 1. I forgot to add FetchS3Object Processor after EvaluateJsonPath 2. There was an extra comma in my Schema text Property.

-1
votes

Can you tell where exactly was that extra comma in your convert record processor? As I am facing the same issue. As per my understanding, issue is occurring because of size_dimension field Below is my csv data :

id,project,name,depth,parentid,description,createdtime,lastupdatedtime,metadata,path,source,sourceid
75125,abcd,P200184,4,74861,"WIRELINE RUNNING / RETRIEVING TOOL, SUPP",2002-06-04 00:00:00.0,2019-04-26 00:00:00.0,"{""material_group"":""group"",""weight_unit"":""LB"",""laboratory"":""PMC"",""object_type"":""material"",""pyspark_generated_time"":""2019-06-07, 13:32:20.287657"",""size_dimension"":""3'5\""L X 3'5\""W X 1'H"",""gross_weight"":""100.000"",""net_weight"":""100.000"",""valid_from_date"":""20031219""}","[59941,64249,74859,74861,75125]",RPA_SAA.MRA,P200184

And the avro schema which i have used is:

{
    "name":"abc",
    "namespace":"nifi",
    "type":"record",
    "fields": [
    {"name":"id", "type": ["long", "null"], "default": null},
    {"name":"project", "type": ["string", "null"], "default": null},
    {"name":"name", "type": ["string", "null"], "default": null},
    {"name":"depth", "type": ["int", "null"], "default": null},
    {"name":"parentid", "type": ["long", "null"], "default": null},
    {"name":"description", "type": ["string", "null"], "default": null},
    {"name":"createdtime","type": ["null",{ "type":"long", "logicalType":"timestamp-millis"}], "default":null},
    {"name":"lastupdatedtime","type": ["null",{ "type":"long", "logicalType":"timestamp-millis"}], "default":null},
    {"name":"metadata","type": ["string", "null"], "default": null},
    {"name":"path","type": ["string", "null"], "default": null},
    {"name":"source", "type": ["string", "null"], "default": null},
    {"name":"sourceid", "type": ["string", "null"], "default": null}
    ]
}