Nifi:making one flowfile from splited log flowfiles

Question

I want to make log files for nifi processors , i get them form tailFail and split text line by line then check if it is error , info or warn log and route to executescript processors but at this time i have 5 flowfiles and i want to unify this split flowfiles and write it in one flowfile, i tried to use merge content but i think it doesn't fit my task.

I also want to know if nifi custom log return log files for all processors i have added in my workflow and is it nessecary to add appenders inside logback.xml.
I want to know if it is possible to unify split log data? (p.s i tried routeonAttriute also but it doesn't work for me) my workflow looks like this:

I have severla flowfiles in queue and i want to write it in one flowfile — Sagitarius
i don't use any spesific processor, is it neseccary to use appenders in this case — Sagitarius
you are like asking different questions not linked with each other. how merging flow files is connected with logback.xml? — daggett

daggett daggett · Accepted Answer · 2017-09-01T08:01:16

After splitting lines you can use RouteOnContent to check that line matches regexp.

Then if you want to join lines you can use the following script.

That's just an example:

//get 1000 flow file list from incoming queue but not more then 1000
def ffList = session.get(1000)
if(!ffList)return

ffList.each{ff->
    session.read(ff, {rawIn ->
        //you can write here to a new output flowfile
        //but in this example i will just add content into a plain file on disk
        new File('./logs/warn.log') << rawIn << '\n'
    } as InputStreamCallback)

    session.remove(ff)
}

Nifi:making one flowfile from splited log flowfiles

1 Answers