Remove line from file if containing word from another .txt file in python/bash

Question

I'm learning python, and then i have the following difficulties. The file i want to be cleaned is an .csv file. The file that contains the words that have to be removed from the .csv file is an .txt The .txt file is a list of domain names:

domain.com
domain2.com
domain3.com

The .csv file is a config file just like this:

domain.com;8;Started;C:\inetpub\wwwroot\d\domain.com;"http *:80:www.domain.com"

if the .txt file contains "domain.com" i want the complete line above to be removed. I would be realy gratefully if some python ninja could fix this.(or in bash?)

Answer 1

Well, since OP is learning python ...

$ python SCRIPT.py

TXT_file = 'TXT.txt'
CSV_file = 'CSV.csv'
OUT_file = 'OUTPUT.csv'

## From the TXT, create a list of domains you do not want to include in output
with open(TXT_file, 'r') as txt:
    domain_to_be_removed_list = []

    ## for each domain in the TXT
    ## remove the return character at the end of line
    ## and add the domain to list domains-to-be-removed list
    for domain in txt:
        domain = domain.rstrip()
        domain_to_be_removed_list.append(domain)


with open(OUT_file, 'w') as outfile:
    with open(CSV_file, 'r') as csv:

        ## for each line in csv
        ## extract the csv domain
        for line in csv:
            csv_domain = line.split(';')[0]

            ## if csv domain is not in domains-to-be-removed list,
            ## then write that to outfile
            if (not csv_domain in domain_to_be_removed_list):
                outfile.write(line)

Answer 2

Will this suffice ?

import sys

def main():
    with open(sys.argv[1]) as fh:
        fhDomains = fh.read().split(";")
    with open(sys.argv[2]) as fh:
        fhExcludes = fh.read().split("\n")

    for i, dom in enumerate(fhDomains):
        if dom in fhExcludes:
            del fhDomains[i]

    fh = open(sys.argv[1], "w")
    fh.write(";".join(fhDomains))





if __name__ == "__main__":
    main()

execute with:

script.py Domains.txt excludes.txt

Answer 3

try:

grep -vf <(sed 's/.*/^&;/' domains.txt) file.csv

@glenn jackman's suggetion - shorter.

grep -wFvf domains.txt file.csv

but, the foo.com in domains, will stll will match both lines (one unwanted), like:

foo.com;.....
other.foo.com;.....

soo...

my domains.txt

dom1.com
dom3.com

my file.csv (only the 1st column needed)

dom1.com;wedwedwe
dom2.com;wedwedwe 2222
dom3.com;wedwedwe 333
dom4.com;wedwedwe 444444

result:

dom2.com;wedwedwe 2222
dom4.com;wedwedwe 444444

if you have windows file - the lines ends with \r\n not only with \n, use:

grep -vf <(<domains.txt tr -d '\r' |sed -e 's/.*/^&;/') file.csv

Answer 4

This awk one-liner should do the trick:

awk -F';' 'NR == FNR {a[$1]++; next} !($1 in a)' txtfile csvfile