Host file make a unique file for all servers

问题

I have many hosts files. I collect them from all servers and i put them together in host_files.txt and then I must make one hosts file for all servers.

I do this command to make a unique file, but some rows share the same ip address or hostname.

awk '!a[$0]++' host_files.txt

Here is my host_files.txt

#backup server IPs
95.23.23.56
95.23.23.57 

#ftp server IPs
45.89.67.5 
45.89.67.3 

#apache
12.56.35.36 
12.56.35.35 

#ftp server IPs
95.23.23.50

#apache
12.56.35.37

I want to output file, but I need to keep the comment line

#backup server IPs <= comment line, i need to keep them
95.23.23.56 
95.23.23.57 

#ftp server IPs <= comment line, i need to keep them
45.89.67.5 
45.89.67.3 
95.23.23.50

#apache <= comment line, i need to keep them
12.56.35.36
12.56.35.35 
12.56.35.37

i already try :

sort -ur host_files.txt

cat host_files.txt | uniq > ok_host.txt

I need the ip without # just need ip adresse please help me

Thanks in advance

回答1:

In GNU awk for using multidimensional arrays:

$ awk '
/^#/ { k=$0; next }          # group within identical comments, k is key to hash
/./  { a[k][$1]=$0 }         # remove empty records and hash ips
END  { for(k in a) {         # after everything, output
           print k
           for(i in a[k]) 
               print a[k][i]
     }
}' file*
#apache
12.56.35.35 #apacheprivate
12.56.35.36 #apachepub
12.56.35.37 #apachepub
#ftp server IPs
45.89.67.3 #ftpssh
45.89.67.5 #ftpmain
95.23.23.50 #ftp
#backup server IPs
95.23.23.56 #masterbasckup
95.23.23.57 #agentbasckup

The output is random order because of for(k in a), ie. comment groups and ips within groups are in no particular order.

回答2:

If awk is not a requirement.

#!/bin/ksh

cat host_files.txt | while read line ; do
    [[ $line =~ ^$ ]] && { continue; }              # skip empty lines
    [[ $line =~ ^# ]] && { group=$line; continue; } # remember the group name
    print "$group|$line"                            # print with group name in front
done | sort  \
    | while read line ; do
        if [[ ${line%\|*} != $last ]]; then         # if the group name changed
            print "\n${line%\|*}"                   #  print the group name
            last=${line%\|*}                        #  remember the new group name
        fi
        print "${line#*\|}"                         # print the entry without the group name
    done

put the group name in front of the line
sort
detect changing group name and print it
print entry without group name

Using the same concept with awk (avoiding the while loop in shell).

awk '
    /^#/ { k=$0; next }
    /./  { print k "|" $0 }
' host_files.txt |   sort   | awk -F '|' '{
        if ( k != $1 ) { print "\n" $1; k = $1; }
        print $2
    }' -

Because it does not use an array it would not loose lines due to duplicate keys.

And, thinking a bit more, the second awk can be avoided. Adding the key to each line. For the header without 'x'. So the header is sorted above the rest. In the output, just remove the added sort-key.

awk '
    /^#/ { k=$0; print k "|" $0; next; }
    /./  { print k "x|" $0 }
' t18.dat  |   sort -u   | cut -d '|' -f 2

来源：https://stackoverflow.com/questions/43721258/host-file-make-a-unique-file-for-all-servers

标签

sorting

awk