xargs just working with built in functions

Question

I am trying to speed up the processing of a database. I migrated towards xargs. But I'm seriously stuck. Piping a list of arguments to xargs does not work if the command invoked by xargs isn't a built in. I can't figure out why. Here is my code:

#!/bin/bash

list='foo
bar'

test(){
    echo "$1" 
}

echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' 'test' {}

So there is no output at all. And test function never gets executed. But if I replace "test" in the "xargs" command with "echo" or "printf" it works fine.

Answer 1

xargs takes an executable as an argument (including custom scripts) rather than a function defined in the environment.

Either move your code to a script or use xargs to pass arguments to an external command.

Answer 2

You can't pass a shell function to xargs directly, but you can invoke a shell.

printf 'foo\0bar\0' |
xargs -r -0 sh -c 'for f; do echo "$f"; done' _

The stuff inside sh -c '...' can be arbitrarily complex; if you really wanted to, you could declare and then use your function. But since it's simple and nonrecursive, I just inlined the functionality.

The dummy underscore parameter is because the first argument after sh -c 'script' is used to populate $0.

Because your question seems to be about optimization, I imagine you don't want to spawn a separate shell for every item passed to xargs -- if you did, nothing would get faster. So I put in the for loop and took out the -I etc arguments to xargs.

Answer 3

Change from:

echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' 'test' {}

To:

export -f test
echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' sh -c 'test {}'

Answer 4

I've seen a solution from 'jac' on the bbs.archlinux.org web-site that uses a primary and secondary (slave) pair of scripts that are very efficients. Instead of an internal 'function' that normally would accept a single $1 parameter, the primary sends a list of parameters to its secondary where a while-loop handles each member of the list as consecutive $1 values. Here's a sample pair I'm using to apply the 'file' command to a bunch of executables, which in my case all begin with "em" in the filename. Make changes as necessary:

#!/bin/bash
# primary:  showfil
ls -l em* | grep  '^-rwx' | awk '{$1=$2=$3=$4=$5=$6=$7=$8=""; print $0}' | xargs -I% ~/showfilf "%"
~/showfilf fixmstr spisort trc
exit 0

#!/bin/bash
# secondary:  showfilf
myarch=$(uname -s | grep 'arwin')
while [[ -n "$1" ]]; do
  if [ -x "$1" ]; then
     if [ -n "$myarch" ]; then
        file "./$1"
     else
        myfile=$(file "./$1" | awk '{print $1" "$3" "$10" "$11" "$12}')
        myfile=${myfile%(uses}
        myfile=${myfile%for}
        echo "$myfile"
     fi
  fi
  shift
done
exit 0

This code works on Darwin (Mac) and Linux, and probably other systems. The 'grep' in the primary retains only executable files, not directories or symlinks. The 'awk' eliminates the first eight fields of 'ls' and retains just the filename,which is passed to 'xargs', which builds a list of quoted filenames to send to 'showfilf'. There's a separate invocation of 'showfilf' with three other filenames in the list. 'showfilf' has a while-loop which processes the list. Note that there is system-dependent code here, determined by 'uname -s' and 'grep'. Lastly, make these scripts executable, and place them on your $PATH, such as $HOME. If your $PATH doesn't include your $HOME, I recommend you modify it in your .bashrc or .bash_login something like this: export PATH=$PATH:$HOME