Unix shell scripting - multiple file concatenate

1
votes

consider the scenario-

file1:

mike;john;552
mike;mike;555
john;mike;121

file2:

aks;raj;425
man;joe;895
mike;john;552

Assuming file1 and file2 contain the above two sets of data. I would like to put the data from these two files into another file , where the data is uniq ( meaning: file1 and file2 contain common data mike;john;552, but when conbining files i do not want to have duplicates.

I used the command:

cat file1 file2 | sort -u > file3

but this gave me only the common line ie, the duplicate into file3.

Also tried 

cat file1 file2 | uniq > file3

Didnt yield required result.

Expected output: file3:

mike;john;552
mike;mike;555
john;mike;121
aks;raj;425
man;joe;895

Note: the data in file3 can be in any order.

Please help on this.

1
shellunix
The cat file1 file2 | sort -u works well for me: all lines from file1 and file2 merged and duplicate lines are printed only once. - Lars Fischer
CNR. These tools work precisely in accordance with the described desired behavior. - Charles Duffy
Okay worked now, guess had the white space issue.Thanx! - Akshay

1 Answers

0
votes

Your first command works for me and gives the expected output:

$ cat -v file1
mike;john;552
mike;mike;555
john;mike;121

$ cat -v file2
aks;raj;425
man;joe;895
mike;john;552

$ cat file1 file2 | sort -u > file3
$ cat file3
aks;raj;425
john;mike;121
man;joe;895
mike;john;552
mike;mike;555

If this doesn't happen for you, use cat -vE to find out why. Here are two examples:

$ cat file1 file2 | sort -u
foo
foo   

$ cat file1 file2 | sort -u | cat -vE
foo$
foo   $

In this case, it looks like you get duplicates, but the lines are actually different because of trailing whitespace.

$ cat file1 file2 | sort -u 
foo
foo
$ cat file1 file2 | sort -u | cat -vE
foo$
foo^M$

In this case, it also looks like you get duplicates, but one file has carriage returns because it was saved in DOS/Windows mode instead of Unix mode.