Can you fool isatty AND log stdout and stderr separately?

Question

Problem

So you want to log the stdout and stderr (separately) of a process or subprocess, without the output being different from what you'd see in the terminal if you weren't logging anything.

Seems pretty simple no? Well unfortunately, it appears that it may not be possible to write a general solution for this problem, that works on any given process...

Background

Pipe redirection is one method to separate stdout and stderr, allowing you to log them individually. Unfortunately, if you change the stdout/err to a pipe, the process may detect the pipe is not a tty (because it has no width/height, baud rate, etc) and may change its behaviour accordingly. Why change the behaviour? Well, some developers make use of features of a terminal which don't make sense if you are writing out to a file. For example, loading bars often require the terminal cursor to be moved back to the beginning of the line and the previous loading bar to be overwritten with a bar of a new length. Also colour and font weight can be displayed in a terminal, but in a flat ASCII file they can not. If you were to write such a program's stdout directly to a file, that output would contain all the terminal ANSI escape codes, rather than properly formatted output. The developer therefore implements some sort of "isatty" check before writing anything to the stdout/err, so it can give a simpler output for files if that check returns false.

The usual solution here is to trick such programs into thinking the pipes are actually ttys by using a pty - a bidirectional pipe that also has width, height, etc. You redirect all inputs/outputs of the process to this pty, and that tricks the process into thinking its talking to a real terminal (and you can log it directly to a file). The only problem is, that by using a single pty for stdout and stderr, we can now no longer differentiate between the two.

So you might want to try a different pty for each pipe - one for the stdin, one for the stdout, and one for the stderr. While this will work 50% of the time, many processes unfortunately do additional redirection checks that make sure that the output path of the stdout and stderr (/dev/tty000x) are the same. If they are not, there must be redirection, thus they give you the same behaviour as if you had piped the stderr and stdout without a pty.

You might think this over-the-top checking for redirection is uncommon, but unfortunately it is actually quite prevalent because a lot of programs re-use other code for checking, like this bit of code found in OSX:

http://src.gnu-darwin.org/src/bin/stty/util.c

Challenge

I think the best way to find a solution is in the form of a challenge. If anyone can run the following script (ideally via Python, but at this point I'll take anything) in such a way that the stdout and stderr is logged separately, AND you managed to fool it into thinking it was executed via a tty, you solve the problem :)

#!/usr/bin/python

import os
import sys

if sys.stdout.isatty() and sys.stderr.isatty() and os.ttyname(sys.stdout.fileno()) == os.ttyname(sys.stderr.fileno()):
    sys.stdout.write("This is a")
    sys.stderr.write("real tty :)")
else:
    sys.stdout.write("You cant fool me!")

sys.stdout.flush()
sys.stderr.flush()

Note that a solution should really work for any process, not just this code specifically. Overwriting the sys/os module and using LD_PRELOAD is very interesting ways to beat the challenge, but they don't solve the heart of the problem :)

Answer 1

Like this?

% ./challenge.py >stdout 2>stderr
% cat stdout 
This is a real tty :)
standard output data
% cat stderr 
standard error data

Because I cheated a little bit. ;-)

% echo $LD_PRELOAD
/home/karol/preload.so

Like so...

% gcc preload.c -shared -o preload.so -fPIC

I feel dirty now, but it was fun. :D

% cat preload.c
#include <stdlib.h>

int isatty(int fd) {
    if(fd == 2 || fd == 1) {
        return 1;
    }
    return 0;
}

char* ttyname(int fd) {
    static char* fake_name = "/dev/fake";
    if(fd == 2 || fd == 1) {
        return fake_name;
    }
    return NULL;
}

Answer 2

For a simpler use-case (e.g. development testing), use strace (linux) or dtruss (OSX). Of course that won't work in privileged process.

Here's a sample, you can distinguish stdout fd1 from stderr fd2:

$ strace -ewrite python2 test.py
[snip]
write(1, "This is a real tty :)\n", 22This is a real tty :)
) = 22
write(2, "standard error data", 19standard error data)     = 19
write(1, "standard output data", 20standard output data)    = 20
+++ exited with 0 +++

In the sample above you see each standard xxx data doubled, because you can't redirect stdout/stderr. You can, however ask strace to save its output to a file.

On a theoretical side, if stdout and stderr refer to the same terminal, you can only distinguish between the 2 while still in the context of your process, either in user mode (LD_PRELOAD), or kernel space (ptrace interface that strace tool uses). Once the data hits actual device, real of pseudo, the distinction is lost.

Answer 3

You can always allocate Pseudo-TTY, that's what screen does.

In Python you'd access it using pty.openpty()

This "master" code passes your test:

import subprocess, pty, os

m, s = pty.openpty()
fm = os.fdopen(m, "rw")
p = subprocess.Popen(["python2", "test.py"], stdin=s, stdout=s, stderr=s)
p.communicate()
os.close(s)
print fm.read()

Of course if you want to distinguish between stdin/out/err, your "slave" process will see different PYT names:

inp = pty.openpty()
oup = pty.openpty()
erp = pty.openpty()

subprocess.Popen([command, args], stdin=inp[1], stdout=uop[1], stderr=erp[1])

Answer 4

$ PYTHONPATH=/tmp/python:$PYTHONPATH ./challenge.py
$ cat stdout
This is a real tty :)
standard output data

$ cat stderr
standard error data

Because this script imports the os module, I've cheated by creating my own os module in /tmp/python and prepending /tmp/python to sys.path.

os.py

import sys

sys.path.remove('/tmp/python')
this_module = sys.modules['os']
del sys.modules['os']

import os
globals().update(vars(os))


class File(file):
    isatty = lambda self: True

sys.stdout = File('stdout', 'w')
sys.stderr = File('stderr', 'w')

isatty = lambda fd: True
ttyname = lambda fd: '/dev/fake'

sys.modules['os'] = this_module

Answer 5

When you can use the script command:

$ script --return -c "[executable string]" >stdout 2>stderr