grep a large list against a large file

Question

I am currently trying to grep a large list of ids (~5000) against an even larger csv file (3.000.000 lines).

I want all the csv lines, that contain an i

Answer 1

Use grep -f for this:

grep -f the_ids.txt huge.csv > output_file

From man grep:

-f FILE, --file=FILE

Obtain patterns from FILE, one per line. The empty file contains zero patterns, and therefore matches nothing. (-f is specified by POSIX.)

If you provide some sample input maybe we can even improve the grep condition a little more.

Test

$ cat ids
11
23
55
$ cat huge.csv 
hello this is 11 but
nothing else here
and here 23
bye

$ grep -f ids huge.csv 
hello this is 11 but
and here 23