Argument list too long - Unix

Question

This scripts will sort the files by date then move the first 2500 files to another directory.
When I run below scripts, system prompt out Argument list too long msg. Anyone can help me enhance the scripts ? Thanks

NUM_OF_FILES=2500
FROM_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in
DESTINATION_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in_load

if [ ! -d $DESTINATION_DIRECTORY ]  
        then  
                echo "unused_file directory does not exist!"  
        mkdir $DESTINATION_DIRECTORY   
        echo "$DESTINATION_DIRECTORY directory created!"  
else   
        echo "$DESTINATION_DIRECTORY exist!"    
fi  


echo "Moving $NUM_OF_FILES oldest files to $DESTINATION_DIRECTORY directory"  

ls -tr  $FROM_DIRECTORY/MSCERC*.Z|head -$NUM_OF_FILES |
    xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"  

Answer 1

You didn't say, but I assume this is where the problem occurs:

ls -tr  $FROM_DIRECTORY/MSCERC*.Z|head -2500 | \
    xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"  

(You can verify it by adding "set -x" to the top of your script.)

The problem is that the kernel has a fixed maximum size of the total length of the command line given to a new process, and your exceeding that in the ls command. You can work around it by not using globbing and instead using grep:

ls -tr  $FROM_DIRECTORY/ | grep '/MSCERC\*\.Z$' |head -2500 | \
    xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"  

(grep uses regular expressions instead of globs, so the pattern looks a little bit different.)

Answer 2

Change

ls -tr  $FROM_DIRECTORY/MSCERC*.Z|head -2500 | \
    xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"  

do something like the following:

find "$FROM_DIRECTORY" -maxdepth 1 -type f -name 'MSCERC*.Z' -printf '%p\t%T@\n' | sort -k2,2 -r | cut -f1 | head -$NUM_OF_FILES | xargs mv -t "$DESTINATION_DIRECTORY"

This uses find to create a list of files with modification timestamps, sorts by the timestamp, then removes the unneeded field before passing the output to head and xargs

EDIT

Another variant, should work with non GNU utils

find "$FROM_DIRECTORY" -type f -name 'MSCERC*.Z' -printf '%p\t%T@' |sort -k 2,2 -r | cut -f1 | head -$NUM_OF_FILES | xargs -i mv \{\} "$DESTINATION_DIRECTORY"

Answer 3

First of create a backup list of the files to be treated. Then read the backup file line-by-line and heal it. For example

 #!/bin/bash
 NUM_OF_FILES=2500
 FROM_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in
 DESTINATION_DIRECTORY=/apps/data01/RAID/RC/MD/IN_MSC/ERC/in_load

 if [ ! -d $DESTINATION_DIRECTORY ]  
    then  
            echo "unused_file directory does not exist!"  
    mkdir $DESTINATION_DIRECTORY   
    echo "$DESTINATION_DIRECTORY directory created!"  
  else   
    echo "$DESTINATION_DIRECTORY exist!"    
 fi  

 echo "Moving $NUM_OF_FILES oldest files to $DESTINATION_DIRECTORY directory" 

 ls -tr  $FROM_DIRECTORY/MSCERC*.Z|head -2500 > list
 exec 3<list

 while read file <&3
 do
    mv $file $DESTINATION_DIRECTORY
 done

Answer 4

A quick way to fix this would be to change to $FROM_DIRECTORY, so that you can refer the files using (shorter) relative paths.

cd $FROM_DIRECTORY && ls -tr MSCERC*.Z|head -2500 |xargs -i sh -c "mv {} $DESTINATION_DIRECTORY"

This is also not entirely fool-proof, if you have too many files that match.