how to improve grep efficiency in perl when the file number is huge

Question

I want to grep some log information from the log files located in the following directory structure using perl: $jobDir/jobXXXX/host.log where XXXX is

Answer 1

While it would be more elegant to use the matching built into perl (see the other answer), calling the grep command can be more efficient and faster, especially if there are lots of data but only few matches. But the way you call it is to first run grep and collect all data, and then scan through all the data. This will need more memory because you first collect all data, and you have to wait for the output until all data are collected. Better would be to output as soon as the first data are collected:

open( my $fh,'-|','grep',"information",'-r',$jobDir) or die $!;
while (<$fh>) {
    if(/\((\d+)\)(.*)\((\d+)\)/){
        Output(xxxxxxxx);
    }
    $Num=$Num+1; #number count      
}