Nifi and Avro: Convert the data and metadata into avro file using specified avro schema?

Question

This is related to Apache Nifi.

I have a fixed schema, which I need to use.

AVSC file (schema) (Sample only):

​

{"name": "person","type": "record","fields": [{"name": "address","type": {"type" : "record","name" : "AddressUSRecord","fields" : [{"name": "streetaddress", "type": "string"},{"name": "city", "type":"string"}]}}]}

Basically this chema has 2 parts, i.e. one in metadata and other is actual Data.

I have metadata created in a csv format and actual data in another csv.

I can use,

GetFile-->InferAvroSchema-->ConvertCSVtoAvro

flow to convert them separately into avro.

But that will not be in the format defined in schema.

I am looking for some flow or processor, where I can give or use 2 different csv as input and convert into avro as per schema provided?

Answer 1

Not sure I understand your use case well enough, but you should be able to use LookupRecord with a CSVRecordLookupService to get the data from the "actual data" CSV into a record that has its metadata fields already in it. You just need to specify which field (via a RecordPath expression) in the metadata corresponds to which field in the data CSV file.