*nix: perform set union/intersection/difference of lists

问题

I sometimes need to compare two text files. Obviously, diff shows the differences, it also hides the similarities, which is kind of the point.

Suppose I want to do other comparisons on these files: set union, intersection, and subtraction, treating each line as an element in the set.

Are there similarly simple common utilities or one-liners which can do this?

Examples:

a.txt

john
mary

b.txt

adam
john

$> set_union a.txt b.txt
john
mary
adam

$> set_intersection a.txt b.txt
john

$> set_difference a.txt b.txt
mary

回答1:

Union: sort -u files...

Intersection: sort files... | uniq -d

Difference: sort files... | uniq -u

回答2:

If you want to get the common lines between two files, you can use the comm utility.

A.txt :

A
B
C

B.txt

A
B
D

and then, using comm will give you :

$ comm <(sort A.txt) <(sort B.txt)
        A
        B
C
    D

In the first column, you have what is in the first file and not in the second.

In the second column, you have what is in the second file and not in the first.

In the third column, you have what is in the both files.

回答3:

If you don't mind using a bit of Perl, and if your file sizes are reasonable such that they can be written into a hash, you could collect the files into two hashes to do:

#...get common keys in an array...
my @both_things
for (keys %from_1) {
    push @both_things, $_ if exists $from_2{$_};
}

#...put unique things in an array...
my @once_only
for (keys %from_1) {
    push @once_only, $_ unless exists $from_2($_);
}

回答4:

I can't comment on Aaron Digulla's answer, which despite being accepted does not actually compute the set difference.

The set difference A\B with the given inputs should only return mary, but the accepted answer also incorrectly returns adam.

This answer has an awk one-liner that correctly computes the set difference:

awk 'FNR==NR {a[$0]++; next} !a[$0]' b.txt a.txt

来源：https://stackoverflow.com/questions/8520463/nix-perform-set-union-intersection-difference-of-lists

标签

perl

bash

unix

text

diff