问题
I am on Linux and I am trying to find thousands of files from a directory (SOURCE_DIR) that contain millions of files. I have a list of file names that I need to find, stored in a single text file (FILE_LIST). Each line of this file contain a single name corresponding to a file in SOURCE_DIR and there are thousands of lines in the file.
## FILE_LIST contain single word file names, each per line
#Name0001
#Name0002
#..
#Name9999
I want to copy the files to another directory (DESTINATION_DIR). I wrote the below loop, with a loop inside to find one by one.
#!/bin/bash
FILE_LIST='file.list'
## FILE_LIST contain single word file names, each per line
#Name0001
#Name0002
#..
#Name9999
SOURCE_DIR='/path/to/source/files' # Contain millions of files in sub-directories
DESTINATION_DIR='/path/to/destination/files' # Files will be copied to here
while read FILE_NAME
do
echo $FILE_NAME
for FILE_NAME_WITH_PATH in `find SOURCE_DIR -maxdepth 3 -name "$FILE_NAME*" -type f -exec readlink -f {} \;`;
do
echo $FILE
cp -pv $FILE_NAME_WITH_PATH $DESTINATION_DIR;
done
done < $FILE_LIST
This loop is taking a lot of time and I was wondering whether there is a better way to achieve my goal. I searched, but did not find a solution to my problem. Please direct to me to a solution if already exist or kindly suggest any tweak in the above code. I am also fine if another approach or even a python/perl solution. Thanks for your time and help!
回答1:
Note Same names in different directories can appear, as confrimed. Code added below
The files to copy need to be found as they aren't given with a path (don't know in which directories they are), but searching anew for each is extremely wasteful, increasing complexity greatly.
Instead, build a hash with a full-path name for each filename first.
One way, with Perl, utilizing the fast core module File::Find
use warnings;
use strict;
use feature 'say';
use File::Find;
use File::Copy qw(copy);
my $source_dir = shift // '/path/to/source'; # give at invocation or default
my $copy_to_dir = '/path/to/destination';
my $file_list = 'file_list_to_copy.txt';
open my $fh, '<', $file_list or die "Can't open $file_list: $!";
my @files = <$fh>;
chomp @files;
my %fqn;
find( sub { $fqn{$_} = $File::Find::name unless -d }, $source_dir );
# Now copy the ones from the list to the given location
foreach my $fname (@files) {
copy $fqn{$fname}, $copy_to_dir
or do {
warn "Can't copy $fqn{$fname} to $copy_to_dir: $!";
next;
};
}
The remaining problem is about filenames that may exists in multiple directories, but we need to be given a rule for what to do then.†
I disregard that a maximal depth is used in the question, since it is unexplained and seemed to me to be a fix related to extreme runtimes (?). Also, files are copied into a "flat" structure (without restoring their orginal hierarchy), taking the cue from the question.
Finally, I skip only directories, while various other file types come with their own issues (copying links around needs care). To accept only plain files change to if -f.
† A clarification came that, indeed, there may be files with the same name in different directories. Those should be copied to same name suffixed with a sequential number before the extension.
For this we need to check whether a name exists already, and to keep track of duplicate ones, while building the hash, so this will take a little longer. There is a little conundrum of how to account for duplicate names then? I use another hash where only duped-names† are kept, in arrayrefs; this simplifies and speeds up both parts of the job.
my (%fqn, %dupe_names);
find( sub {
return if -d;
(exists $fqn{$_})
? push( @{ $dupe_names{$_} }, $File::Find::name )
: ( $fqn{$_} = $File::Find::name );
}, $source_dir );
To my surprise, this runs barely a little slower than the code with no concern for duplicate names (on a quarter million files spread over a sprawling hierarchy), even as a test runs for each item.
The parens around the assignment in the ternary operator are needed since the operator may be assigned to (if the last two arguments are valid "lvalues," as they are here) and so one need be careful with assignments inside the branches.
Then after copying %fqn as in the main part of the post, also copy other files with the same name. We need to break up filenames so to add enumeration before .ext; I use core File::Basename
use File::Basename qw(fileparse);
foreach my $fname (@files) {
next if not exists $dupe_names{$fname}; # no dupe (and copied already)
my $cnt = 1;
foreach my $fqn (@{$dupe_names{$fname}}) {
my ($name, $path, $ext) = fileparse($fqn, qr/\.[^.]*/);
copy $fqn, "$copy_to_dir/${name}_$cnt$ext";
or do {
warn "Can't copy $fqn to $copy_to_dir: $!";
next;
};
++$cnt;
}
}
(basic testing done but not much more)
I'd perhaps use undef instead of $path above, to indicate that the path is unused (while that also avoids allocating and populating a scalar), but I left it this way for clarity for those unfamiliar with what the module's sub returns.
Note. For files with duplicates there'll be copies fname.ext, fname_1.ext, etc. If you'd rather have them all indexed, then first rename fname.ext (in the destination, where it has already been copied via %fqn) to fname_1.ext, and change counter initialization to my $cnt = 2;.
† Note that these by no means need be same files.
回答2:
I suspect the speed issues are (at least partly) coming from your nested loops - for every FILE_NAME, you're running a find and looping over its results. The following Perl solution uses the technique of dynamically building a regular expression (which works for large lists, I've tested it on lists of 100k+ words to match), that way you only need to loop over the files once and let the regular expression engine handle the rest; it's quite fast.
Note I have made a couple of assumptions based on my reading of your script: That you want the patterns to match case-sensitively at the beginning of filenames, and that you want to recreate the same directory structure as the source in the destination (set $KEEP_DIR_STRUCT=0 if you do not want this). Also, I am using the not-exactly-best-practice solution of shelling out to find instead of using Perl's own File::Find because it makes it easier to implement the same options you're using (such as -maxdepth 3) - but it should work fine unless there are any files with newlines in their name.
This script uses only core modules so you should already have them installed.
#!/usr/bin/env perl
use warnings;
use strict;
use File::Basename qw/fileparse/;
use File::Spec::Functions qw/catfile abs2rel/;
use File::Path qw/make_path/;
use File::Copy qw/copy/;
# user settings
my $FILE_LIST='file.list';
my $SOURCE_DIR='/tmp/source';
my $DESTINATION_DIR='/tmp/dest';
my $KEEP_DIR_STRUCT=1;
my $DEBUG=1;
# read the file list
open my $fh, '<', $FILE_LIST or die "$FILE_LIST: $!";
chomp( my @files = <$fh> );
close $fh;
# build a regular expression from the list of filenames
# explained at: https://www.perlmonks.org/?node_id=1179840
my ($regex) = map { qr/^(?:$_)/ } join '|', map {quotemeta}
sort { length $b <=> length $a or $a cmp $b } @files;
# prep dest dir
make_path($DESTINATION_DIR, { verbose => $DEBUG } );
# use external "find"
my @cmd = ('find',$SOURCE_DIR,qw{ -maxdepth 3 -type f -exec readlink -f {} ; });
open my $cmd, '-|', @cmd or die $!;
while ( my $srcfile = <$cmd> ) {
chomp($srcfile);
my $basename = fileparse($srcfile);
# only interested in files that match the pattern
next unless $basename =~ /$regex/;
my $newname;
if ($KEEP_DIR_STRUCT) {
# get filename relative to the source directory
my $relname = abs2rel $srcfile, $SOURCE_DIR;
# build new filename in destination directory
$newname = catfile $DESTINATION_DIR, $relname;
# create the directories in the destination (if necessary)
my (undef, $dirs) = fileparse($newname);
make_path($dirs, { verbose => $DEBUG } );
}
else {
# flatten the directory structure
$newname = catfile $DESTINATION_DIR, $basename;
# warn about potential naming conflicts
warn "overwriting $newname with $srcfile\n" if -e $newname;
}
# copy the file
print STDERR "cp $srcfile $newname\n" if $DEBUG;
copy($srcfile, $newname) or die "copy('$srcfile', '$newname'): $!";
}
close $cmd or die "external command failed: ".($!||$?);
You may also want to consider possibly using hard links instead of copying the files.
回答3:
With rsync
I have no idea how fast this will be for millions of files but here's a method that uses rsync.
Format your file.list as below (ex: such as with $ cat file.list | awk '{print "+ *" $0}').
+ *Name0001
+ *Name0002
...
+ *Name9999
Call file.list with --include=from option in rsync command:
$ rsync -v -r --dry-run --filter="+ **/" --include-from=/tmp/file.list --filter="- *" /path/to/source/files /path/to/destination/files
Option explanations:
-v : Show verbose info.
-r : Traverse directories when searching for files to copy.
--dry-run : Remove this if preview looks okay
--filter="+ *./" : Pattern to include all directories in search
--include-from=/tmp/file.list : Include patterns from file.
--filter="- *" : Exclude everything that didn't match previous patterns.
Option order matters.
Remove --dry-run if the verbose info looks acceptable.
Tested with rsync version 3.1.3.
回答4:
here is bashv4+ solution with find, not sure about the speed though.
#!/usr/bin/env bash
files=file.list
sourcedir=/path/to/source/files
destination=/path/to/destination/files
mapfile -t lists < "$files"
total=${#lists[*]}
while IFS= read -rd '' files; do
counter=0
while ((counter < total)); do
if [[ $files == *"${lists[counter]}" ]]; then
echo cp -v "$files" "$destination" && unset 'lists[counter]' && break
fi
((counter++))
done
lists=("${lists[@]}")
total=${#lists[*]}
(( ! total )) && break ##: if the lists is already emtpy/zero, break.
done < <(find "$sourcedir" -type f -print0)
The inner
breakwill exit the inner loop if a match was found in the file.list and the files in the source_directory, so it will not process the file.list until the end, and it removes the entry in the"${lists[@]}"(which is an array) with theunset, so the next inner loop will skip the already matched files.File name collision should not be a problem, the
unsetand the innerbreakmakes sure of that. The down side is if you have multiple files to match in different sub directories.If speed is what you're looking for then use the general scripting languages like,
python,perland friends
An alternative to the (excruciating slow) pattern match inside the loop is grep
#!/usr/bin/env bash
files=file.list
source_dir=/path/to/source/files
destination_dir=/path/to/destination/files
while IFS= read -rd '' file; do
cp -v "$file" "$destination_dir"
done < <(find "$source_dir" -type f -print0 | grep -Fzwf "$files")
The
-zfromgrepbeing a GNU extension.Remove the
echoif you think the output is correct.
回答5:
Try locate with grep instead of find. I uses file index db and thus should be pretty fast. Remember to run sudo updatedb to update the db beforehand.
来源：https://stackoverflow.com/questions/61843060/find-thousands-of-files-efficiently-with-exact-match-from-a-directory-containing