Average of multiple files having different row sizes

Question

I have few files with different row sizes, but number of columns in each file is same. e.g.

ifile1.txt

1       1001    ?       ?
2       1002    ?       ?
3       1003    ?       ?
4       1004    ?       ?
5       1005    ?       0
6       1006    ?       1
7       1007    ?       3
8       1008    5       4
9       1009    3       11
10      1010    2       9

ifile2.txt

1       2001    ?       ?
2       2002    ?       ?
3       2003    ?       ?
4       2004    ?       ?
5       2005    ?       0
6       2006    6       12
7       2007    6       5
8       2008    9       10
9       2009    3       12
10      2010    5       7
11      2011    2       ?
12      2012    9       ?

ifile3.txt

1       3001    ?       ?
2       3002    ?       6
3       3003    ?       ?
4       3004    ?       ?
5       3005    ?       0
6       3006    1       25
7       3007    2       3
8       3008    ?       ?

In each file 1st column represents the index number and 2nd column as ID. I would like to calculate the average for each index number from 3rd column onward.

The desired output:

1       ?       ?      ----  [Here ? is computed from ?, ?, ?] So answer is ?
2       ?       6.0    ----  [Here 6 is computed from ?, ?, 6] So answer is 6/1=6.0
3       ?       ?
4       ?       ?
5       ?       0.0
6       3.5     12.7 
7       4.0     3.7
8       7.0     7.0    ----- [Here 7 is computed from 5, 9, ? ] So answer is 14/2=7.0   
9       3.0     11.5
10      3.5     8.0
11      2.0     ?
12      9.0     ?

but it is taking all "?" as 0. So check it. If all elements in s[i] are equal to ? then output a ?. Ie. all_questions=1; for (i in s) if (s[i] != "?") all_questions=0 then later if (all_questions) print "?" else print <the avarage> — KamilCuk
some row is not "?" if c[i]>0. awk '{n[$1]++}$3!="?"{s[$1]+=$3;c[$1]++;} END{for(i in n)print i,(c[i]?sprintf("%.1f",s[i]/c[i]):"?");}' ifile* — jhnc

thanasisp thanasisp · Accepted Answer · 2020-07-07T13:38:12

You could parse the files and store sum and count of each position in some kind of 2-dimensional array, which doesn't really exist for awk, but can be implemented using the appropriate index string, see also: https://www.gnu.org/software/gawk/manual/html_node/Multidimensional.html

Here is a script tested with your sample input and output.

{
    c = NF
    if (r<FNR) r = FNR
    
    for (i=3;i<=NF;i++) {
        if ($i != "?") {
            s[FNR "," i] += $i
            n[FNR "," i] += 1
        }
    }
}

END {
    for (i=1;i<=r;i++) {
        printf("%s\t", i)
        for (j=3;j<=c;j++) {
            if (n[i "," j]) {
                printf("%.1f\t", s[i "," j]/n[i "," j])
            } else {
                printf("?\t")
            }
        }
        printf("\n")
    }
}

test

> awk -f test.awk file1 file2 file3
1       ?       ?
2       ?       6.0  
3       ?       ?
4       ?       ?
5       ?       0.0
6       3.5     12.7
7       4.0     3.7
8       7.0     7.0
9       3.0     11.5
10      3.5     8.0
11      2.0     ?
12      9.0     ?

Average of multiple files having different row sizes

1 Answers