How to use find command with sed and awk to remove duplicate IP from files

问题

Howdie do,

I'm writing a script that will remove duplicate IP's from two files. For example,

 grep -rw "123.234.567" /home/test/ips/
 /home/test/ips/codingte:123.234.567
 /home/test/ips/codingt2:123.234.567

Ok, so that IP is in two different files and so I need to remove the IP from the second file.

The grep gives me the file path and the IP address. My thinking: store the file path in a variable with awk and then use find to go to that file and use sed to remove the duplicate IP, so I changed my grep statement to:

 grep -rw "123.234.567" . | awk -F ':' '{print $1}'

which returns:

 ./codingte
 ./codingt2

I originally tried to use the fully pathname in the find command, but that didn't work either

 find -name /var/cpanel/dips/codingte -exec sed '/123.234.567/d' {} \;

So, I just did a CD in the directory and changed the find command to:

 find -name 'codingt2' -exec sed '/123.234.567/d' {} \;

Which runs, but doesn't delete the IP address:

 cat codingt2
 123.234.567

Now, I know the issue is with the dots in the IP address. They need to be escaped, but I'm not sure how to do this. I've been reading for hours on escaping the regex, but I'm not sure how to do this with sed

Any help would be appreciated. I'm just trying to learn more about regex and using them with other linux tools such as awk and find.

I haven't written the full script yet. I'm trying to break it into pieces and then bring it together in the script.

So you know what the output should look like:

 codingte
 123.234.567

 codingt2

The second file would just have the IP removed

回答1:

cat FILE1.txt | while read IP ; do sed -i "/^${IP}$/d" FILE2.txt ; done

The command does the following:

• There are two files: FILE1.txt and FILE2.txt
• It will remove in FILE2.txt lines (in your case, IP addresses) found in FILE1.txt

回答2:

You want grep -l which only print the filenames containing a match:

grep -lrw "123.234.567" /home/test/ips/

would print

/home/test/ips/codingte
/home/test/ips/codingt2

So, to skip the first file and work on the rest:

grep -l ... | sed 1d | while IFS= read -r filename; do
    whatever with "$filename"
done

回答3:

I think you're just missing the -i argument to sed to edit the files in place.

echo foo > test find -name test -exec sed -i 's/foo/bar/' {} \;

seems to do the trick.

来源：https://stackoverflow.com/questions/21057394/how-to-use-find-command-with-sed-and-awk-to-remove-duplicate-ip-from-files