I'm able to get Apache NiFi to generate a schema via the CSVReader, and then I can write the schema out to an attribute using ConvertRecord. However I then need to add fields using UpdateRecord, but the fields are not being added to the flow file or the to the schema attribute. I believe this is because the fields are not part of the initially inferred schema. I can't create the schema in the registry because it's being inferred from the file. So how can I add fields to a record when the schema doesn't include the fields?
3
votes
3 Answers
5
votes
Are you using InferAvroSchema to not have to worry about generating the schema(s), or because you really will not know the schema of the CSV files? If the former, then send one CSV through, then copy the inferred schema into a CSVReader, and add the fields from UpdateRecord into the write schema.
I've written up NIFI-5524 to cover the automation of adding/updating fields in the outgoing schema based on UpdateRecord properties.
1
votes
Yes, that is because your writer controller service doesn't have the new fields defined in it.
If you are adding new fields then we need to define new avro schema with the additional fields included in the schema writer controller service.
Change the Schema Access Strategy to either
Use 'Schema Name' Property (or) Use 'Schema Text' Property
then Define your new schema including new fields in it so that Update Record processor will add the new fields to the output flowfile.
Please look into this article, as i have added ts_tz,current_ts..etc fields in it that doesn't exist in the input data and defined the writer controller service with the new avro schema that includes all the new/old fields in it.
0
votes
I achieved the same by adding columns to the CSV using replace text processor (this will add same values for header and values in the csv), use replacement mode "Line-By-Line" and then use update record to update values only of the new columns to something meaningful.
No need to know the schema beforehand using this approach.
