how to improve grep efficiency in perl when the file number is huge

Question

I want to grep some log information from the log files located in the following directory structure using perl: $jobDir/jobXXXX/host.log where XXXX is

Answer 1

You should search those log file one by one, and scan each log file line by line, instead of reading the output of grep to memory (that could cost lots of memory, and slow your program, even your system):

# untested script

my $Num;
foreach my $log (<$jobDir/job*/host.log>) {
    open my $logfh, '<', "$log" or die "Cannot open $log: $!";
    while (<$logfh>) {
        if (m/information/) {
            if(m/\((\d+)\)(.*)\((\d+)\)/) {
                Output(xxx);
            }
            $Num++;
        }
    }
    close $logfh;
}