Concatenation of huge number of selective files from a directory in Shell

Question

I have more than 50000 files in a directory such as file1.txt, file2.txt, ....., file50000.txt. I would like to concatenate of some files whose file numbers are listed in the following text file (need.txt).

need.txt
1
4
35
45
71
.
.
.

I tried with the following. Though it is working, but I look for more simpler and short way.

n1=1
n2=$(wc -l < need.txt)
while [ $n1 -le $n2 ]
do
f1=$(awk 'NR=="$n1" {print $1}' need.txt)
cat file$f1.txt >> out.txt
(( n1++ ))
done

Etan Reisner Etan Reisner · Accepted Answer · 2015-07-21T12:31:51

Something like this should work for you:

sed -e 's/.*/file&.txt/' need.txt | xargs cat > out.txt

It uses sed to translate each line into the appropriate file name and then hands the filenames to xargs to hand them to cat.

Using awk it could be done this way:

awk 'NR==FNR{ARGV[ARGC]="file"$1".txt"; ARGC++; next} {print}' need.txt > out.txt

Which adds each file to the ARGV array of files to process and then prints every line it sees.

Concatenation of huge number of selective files from a directory in Shell

3 Answers