Hadoop : Using Pig to add text at the end of every line of a hdfs file

Question

We have files in HDFS with raw logs, each individual log is a line as these logs are line separated.

Our requirement is that to add a text (' 12345' for e.g. ) by the end of every log in these files ... using pig / hadoop command / or any other map reduce based tool.

Please advice

Thanks AJ

What have you tried so far? Please post so we can try and help — Raquel Guimarães
You just need a mapper to do this. What have you tried so far? — user238607

VK_217 VK_217 · Accepted Answer · 2017-06-29T00:58:47

Load the files where each log entry is loaded into one field i.e. line:chararray and use CONCAT to add the text to each line.Store it into new log file.If you want the individual files then you will have to parameterize the script to load each file and store into a new file instead of wildcard load.

Log = LOAD '/path/wildcard/*.log' USING TextLoader(line:chararray);
Log_Text = FOREACH Log GENERATE CONCAT(line,'Your Text') as newline;
STORE Log_Text INTO /path/NewLog.log';

Hadoop : Using Pig to add text at the end of every line of a hdfs file

2 Answers