Unix(AIX): tail: unable to malloc memory Error, when using pipes

Question

My requirement is to chop off the header and trailer records from a large file, I'm using a file of size 2.5GB with 1.8 million records. For doing so, I'm executing:

head -n $((count-1)) largeFile | tail -n $((count-2)) > outputFile

Whenever I select count>=725,000 records (size=1,063,577,322), the prompt is returning an error:

tail:unable to malloc memory

I assumed that the pipe buffer went full and tried:

head -n 1000000 largeFile | tail -n 720000 > outputFile

which should also fail since i'm passing count> 725000 to head, but, it generated the output. Why it is so? As head is generating same amount of data (or more), both commands should fail, but the command is depending on tail count. Is it not the way where, first head writes into pipe and then tail uses pipe as input. If it is not, how parallelism is supported here, since tail works from end which is not known until head completes execution. Please correct me, I've assumed lot of things here.

PS: For the time being I've used grep to remove header and trailer. Also, ulimit on my machine returns:

pipe (512 byte) 64 {32 KB}

Thanks guys...

Answer 1

Just do this instead:

awk 'NR>2{print prev} {prev=$0}' largeFile > outputFile

it'll only store 1 line in memory at a time so no need to worry about memory issues.

Here's the result:

$ seq 5 | awk 'NR>2{print prev} {prev=$0}'
2
3
4

Answer 2

I did not test this with a large file, but it will avoid a pipe.

sed '1d;$d' largeFile > outputFile

Answer 3

Ed Morton and Walter A have already given workable alternatives; I'll take a stab at explaining why the original is failing. It's because of the way tail works: tail will read from the file (or pipe), starting at the beginning. It stores the last lines seen, and then when it reaches the end of the file, it outputs the stored lines. That means that when you use tail -n 725000, it needs to store the last 725,000 lines in memory, so it can print them when it reaches the end of the file. If 725,000 lines (most of a 2.5GB file) won't fit in memory, you get a malloc ("memory allocate") error.

Solution: use a process that doesn't have to buffer most of the file before outputting it, as both Ed and Walter's solutions do. As a bonus, they both trim the first line in the same process.