read multiple files using multiprocessing

Question

I need to read some very huge text files (100+ Mb), process every lines with regex and store the data into a structure. My structure inherits from defaultdict, it has a read

Answer 1

Multiprocessing is more suited to CPU- or memory-oriented processes since the seek time of rotational drives kills performance when switching between files. Either load your log files into a fast flash drive or some sort of memory disk (physical or virtual), or give up on multiprocessing.