问题
cat a.txt | xargs -I % echo %
In the example above, xargs takes echo % as the command argument. But in some cases, I need multiple commands to process the argument instead of one. For example:
cat a.txt | xargs -I % {command1; command2; ... }
But xargs doesn\'t accept this form. One solution I know is that I can define a function to wrap the commands, but it\'s not a pipeline, I don\'t prefer it. Is there another solution?
回答1:
cat a.txt | xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _
...or, without a Useless Use Of cat:
<a.txt xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _
To explain some of the finer points:
The use of
"$arg"instead of%(and the absence of-Iin thexargscommand line) is for security reasons: Passing data onsh's command-line argument list instead of substituting it into code prevents content that data might contain (such as$(rm -rf ~), to take a particularly malicious example) from being executed as code.Similarly, the use of
-d $'\n'is a GNU extension which causesxargsto treat each line of the input file as a separate data item. Either this or-0(which expects NULs instead of newlines) is necessary to prevent xargs from trying to apply shell-like (but not quite shell-compatible) parsing to the stream it reads. (If you don't have GNU xargs, you can usetr '\n' '\0' <a.txt | xargs -0 ...to get line-oriented reading without-d).The
_is a placeholder for$0, such that other data values added byxargsbecome$1and onward, which happens to be the default set of values aforloop iterates over.
回答2:
With GNU Parallel you can do:
cat a.txt | parallel 'command1 {}; command2 {}; ...; '
Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
回答3:
This is just another approach without xargs nor cat:
while read stuff; do
command1 "$stuff"
command2 "$stuff"
...
done < a.txt
回答4:
You can use
cat file.txt | xargs -i sh -c 'command {} | command2 {} && command3 {}'
{} = variable for each line on the text file
回答5:
One thing I do is to add to .bashrc/.profile this function:
function each() {
while read line; do
for f in "$@"; do
$f $line
done
done
}
then you can do things like
... | each command1 command2 "command3 has spaces"
which is less verbose than xargs or -exec. You could also modify the function to insert the value from the read at an arbitrary location in the commands to each, if you needed that behavior also.
回答6:
A little late to the party.
I use format below for compressing my directories with thousands of tiny files before migrating. If you don't need single quotes inside commands, it should work.
With some modification, I'm sure it will be useful for someone. Tested in Cygwin (babun)
find . -maxdepth 1 ! -path . -type d -print0 | xargs -0 -I @@ bash -c '{ tar caf "@@.tar.lzop" "@@" && echo Completed compressing directory "@@" ; }'
find . Find here-maxdepth 1 Don't go into child directories! -path . Exclude . / Current directory path-type d match only directories-print0 Separate output by null bytes \0| xargs Pipe to xargs-0 Input is null separated bytes-I @@ Placeholder is @@. Replace @@ with input.bash -c '...' Run Bash command{...} Command grouping&& Execute next command only if previous command exited successfully (exit 0)
Final ; is important, otherwise it will fail.
Output:
Completed compressing directory ./Directory1 with meta characters in it
Completed compressing directory ./Directory2 with meta characters in it
Completed compressing directory ./Directory3 with meta characters in it
2018 July Update:
If you love hacks and playing around, here is something interesting:
echo "a b c" > a.txt
echo "123" >> a.txt
echo "###this is a comment" >> a.txt
cat a.txt
myCommandWithDifferentQuotes=$(cat <<'EOF'
echo "command 1: $@"; echo 'will you do the fandango?'; echo "command 2: $@"; echo
EOF
)
< a.txt xargs -I @@ bash -c "$myCommandWithDifferentQuotes" -- @@
Output:
command 1: a b c
will you do the fandango?
command 2: a b c
command 1: 123
will you do the fandango?
command 2: 123
command 1: ###this is a comment
will you do the fandango?
command 2: ###this is a comment
Explanation:
- Create a single liner script and store it in a variable
- xargs reads a.txt and executes it as bash script
- @@ makes sure every time an entire line is passed
- Putting @@ after -- makes sure @@ is taken as positional parameter input to bash command, not a bash start OPTION, i.e. like -c itself which means run command
-- is magical, it works with many other things, i.e. ssh, even kubectl
回答7:
I prefer style which allows dry run mode (without | sh) :
cat a.txt | xargs -I % echo "command1; command2; ... " | sh
Works with pipes too:
cat a.txt | xargs -I % echo "echo % | cat " | sh
回答8:
Another possible solution that works for me is something like -
cat a.txt | xargs bash -c 'command1 $@; command2 $@' bash
Note the 'bash' at the end - I assume it is passed as argv[0] to bash. Without it in this syntax the first parameter to each command is lost. It may be any word.
Example:
cat a.txt | xargs -n 5 bash -c 'echo -n `date +%Y%m%d-%H%M%S:` ; echo " data: " $@; echo "data again: " $@' bash
回答9:
This seems to be the safest version.
tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'command1 "$@"; command2 "$@";' ''
(-0 can be removed and the tr replaced with a redirect (or the file can be replaced with a null separated file instead). It is mainly in there since I mainly use xargs with find with -print0 output) (This might also be relevant on xargs versions without the -0 extension)
It is safe, since args will pass the parameters to the shell as an array when executing it. The shell (at least bash) would then pass them as an unaltered array to the other processes when all are obtained using ["$@"][1]
If you use ...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' '', the assignment will fail if the string contains double quotes. This is true for every variant using -i or -I. (Due to it being replaced into a string, you can always inject commands by inserting unexpected characters (like quotes, backticks or dollar signs) into the input data)
If the commands can only take one parameter at a time:
tr '[\n]' '[\0]' < a.txt | xargs -r0 -n1 /bin/bash -c 'command1 "$@"; command2 "$@";' ''
Or with somewhat less processes:
tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'for f in "$@"; do command1 "$f"; command2 "$f"; done;' ''
If you have GNU xargs or another with the -P extension and you want to run 32 processes in parallel, each with not more than 10 parameters for each command:
tr '[\n]' '[\0]' < a.txt | xargs -r0 -n10 -P32 /bin/bash -c 'command1 "$@"; command2 "$@";' ''
This should be robust against any special characters in the input. (If the input is null separated.) The tr version will get some invalid input if some of the lines contain newlines, but that is unavoidable with a newline separated file.
The blank first parameter for bash -c is due to this: (From the bash man page) (Thanks @clacke)
-c If the -c option is present, then commands are read from the first non-option argument com‐
mand_string. If there are arguments after the command_string, the first argument is assigned to $0
and any remaining arguments are assigned to the positional parameters. The assignment to $0 sets
the name of the shell, which is used in warning and error messages.
回答10:
My current BKM for this is
... | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
It is unfortunate that this uses perl, which is less likely to be installed than bash; but it handles more input that the accepted answer. (I welcome a ubiquitous version that does not rely on perl.)
@KeithThompson's suggestion of
... | xargs -I % sh -c 'command1; command2; ...'
is great - unless you have the shell comment character # in your input, in which case part of the first command and all of the second command will be truncated.
Hashes # can be quite common, if the input is derived from a filesystem listing, such as ls or find, and your editor creates temporary files with # in their name.
Example of the problem:
$ bash 1366 $> /bin/ls | cat
#Makefile#
#README#
Makefile
README
Oops, here is the problem:
$ bash 1367 $> ls | xargs -n1 -I % sh -i -c 'echo 1 %; echo 2 %'
1
1
1
1 Makefile
2 Makefile
1 README
2 README
Ahh, that's better:
$ bash 1368 $> ls | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
1 #Makefile#
2 #Makefile#
1 #README#
2 #README#
1 Makefile
2 Makefile
1 README
2 README
$ bash 1369 $>
来源：https://stackoverflow.com/questions/6958689/calling-multiple-commands-with-xargs