How to redirect stdout+stderr to one file while keeping streams separate?

Question

Redirecting stdout+stderr such that both get written to a file while still outputting to stdout is simple enough:

cmd 2>&1 | tee output_file

But then now both stdout/stderr from cmd are coming on stdout. I'd like to write stdout+stderr to the same file (so ordering is preserved assuming cmd is single threaded) but then still be able to also separately redirect them, something like this:

some_magic_tee_variant combined_output cmd > >(command-expecting-stdout) 2> >(command-expecting-stderr)

So combined_output contains the both with order preserved, but the command-expecting-stdout only gets stdout and command-expecting-stderr only gets stderr. Basically, I want to log stdout+stderr while still allowing stdout and stderr to be separately redirected and piped. The problem with the tee approach is it globs them together. Is there a way to do this in bash/zsh?

Answer 1

From what I unterstand this is what you are looking for. First I made a litte script to write on stdout and stderr. It looks like this:

$ cat foo.sh 
#!/bin/bash

echo foo 1>&2
echo bar

Then I ran it like this:

$ ./foo.sh 2> >(tee stderr | tee -a combined) 1> >(tee stdout | tee -a combined)
foo
bar

The results in my bash look like this:

$ cat stderr
foo
$ cat stdout 
bar
$ cat combined 
foo
bar

Note that the -a flag is required so the tees don't overwrite the other tee's content.

Answer 2

{ { cmd | tee out >&3; } 2>&1 | tee err >&2; } 3>&1

Or, to be pedantic:

{ { cmd 3>&- | tee out >&3 2> /dev/null; } 2>&1 | tee err >&2 3>&- 2> /dev/null; } 3>&1

Note that it's futile to try and preserve order. It is basically impossible. The only solution would be to modify "cmd" or use some LD_PRELOAD or gdb hack,

Answer 3

Order can indeed be preserved. Here's an example which captures the standard output and error, in the order in which they are generated, to a logfile, while displaying only the standard error on any terminal screen you like. Tweak to suit your needs.

1.Open two windows (shells)

2.Create some test files

touch /tmp/foo /tmp/foo1 /tmp/foo2

3.In window1:

mkfifo /tmp/fifo
</tmp/fifo cat - >/tmp/logfile

4.Then, in window2:

(ls -l /tmp/foo /tmp/nofile /tmp/foo1 /tmp/nofile /tmp/nofile; echo successful test; ls /tmp/nofile1111) 2>&1 1>/tmp/fifo | tee /tmp/fifo 1>/dev/pts/1

Where /dev/pts/1 can be whatever terminal display you want. The subshell runs some "ls" and "echo" commands in sequence, some succeed (providing stdout) and some fail (providing stderr) in order to generate a mingled stream of output and error messages, so that you can verify the correct ordering in the log file.

Answer 4

Here's how I do it:

exec 3>log ; example_command 2>&1 1>&3 | tee -a log ; exec 3>&-

Worked Example

bash$ exec 3>log ; { echo stdout ; echo stderr >&2 ; } 2>&1 1>&3 | \
      tee -a log ; exec 3>&-
stderr
bash$ cat log
stdout
stderr

Here's how that works:

exec 3>log sets up file descriptor 3 to redirect into the file called log, until further notice.

example_command to make this a working example, I used { echo stdout ; echo stderr >&2 ; }. Or you could use ls /tmp doesnotexist to provide output instead.

Need to jump ahead to the pipe | at this point because bash does it first. The pipe sets up a pipe and redirects the file descriptor 1 into this pipe. So now, STDOUT is going into the pipe.

Now we can go back to where we were next in our left-to-right interpretation: 2>&1 this says errors from the program are to go to where STDOUT currently points, i.e. into the pipe we just set up.

1>&3 means STDOUT is redirected into file descriptor 3, which we earlier set up to output to the log file. So STDOUT from the command just goes into the log file, not to the terminal's STDOUT.

tee -a log takes it's input from the pipe (which you'll remember is now the errors from the command), and outputs it to STDOUT and also appends it to the log file.

exec 3>&- closes the file descriptor 3.

Answer 5

Victor Sergienko's comment is what worked for me, adding exec to the front of it makes this work for the entire script (instead of having to put it after individual commands)

exec 2> >(tee -a output_file >&2) 1> >(tee -a output_file)