How to do BULK INSERT into PostgreSQL without specifying physical csv file in Apache NiFi

0
votes

I need to do BULK INSERT into PostgreSQL table in Apache NiFi without specifying the physical csv file in COPY command. I just cannot store the CSV files on the disk and would like to do BULK INSERT using a flow file that is coming from previous processors and is already in CSV format (or I can change it to json, that's not an issue).

Please advise, what is the best way to do this in Apache NiFi?

1
postgresqlcsvapache-nifibulkinsert
did you try PutSQL processor? - Vikramsinh Shinde
@VikramsinhShinde I am trying to use PutSQL but I don't know how to specify not the physical csv file but csv file that is coming from the previous processor. This command doesn't work. Please advise what I should change here? I can see that attribute ${filename} is coming from the previous processor. COPY position__c (Id, Name, Description__c) FROM ${filename} DELIMITER ';'; - user15092322

1 Answers

0
votes

PostgreSQL's COPY operation requires a path to a file. I'd recommend looking at PutDatabaseRecord which generates a PreparedStatement based on your CSV data and executes the statement in a single batch (unless Maximum Batch Size is set)