Interaction between Python script and linux shell

Question

I have a Python script that needs to interact with the user via the command line, while logging whatever is output.

I currently have this:

# lots of code

popen = subprocess.Popen(
    args,
    shell=True,
    stdin=sys.stdin,
    stdout=sys.stdout,
    stderr=sys.stdout,
    executable='/bin/bash')

popen.communicate()

# more code

This executes a shell command (e.g. adduser newuser02) just as it would when typing it into a terminal, including interactive behavior. This is good.

Now, I want to log, from within the Python script, everything that appears on the screen. But I can't seem to make that part work.

I've tried various ways of using subprocess.PIPE, but this usually messes up the interactivity, like not outputting prompt strings.

I've also tried various ways to directly change the behavior of sys.stdout, but as subprocess writes to sys.stdout.fileno() directly, this was all to no avail.

Answer 1

Popen might not be very suitable for interactive programs due to buffering issues and due to the fact that some programs write/read directly from a terminal e.g., to retrieve a password. See Q: Why not just use a pipe (popen())?.

If you want to emulate script utility then you could use pty.spawn(), see the code example in Duplicating terminal output from a Python subprocess or in log syntax errors and uncaught exceptions for a python subprocess and print them to the terminal:

#!/usr/bin/env python
import os
import pty
import sys

with open('log', 'ab') as file:
    def read(fd):
        data = os.read(fd, 1024)
        file.write(data)
        file.flush()
        return data

    pty.spawn([sys.executable, "test.py"], read)

Or you could use pexpect for more flexibility:

import sys
import pexpect # $ pip install pexpect

with open('log', 'ab') as fout:
    p = pexpect.spawn("python test.py")
    p.logfile = fout # or .logfile_read
    p.interact()

If your child process doesn't buffer its output (or it doesn't interfere with the interactivity) and it prints its output to its stdout or stderr then you could try subprocess:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE, STDOUT

with open('log','ab') as file:
    p = Popen([sys.executable, '-u', 'test.py'],
              stdout=PIPE, stderr=STDOUT,
              close_fds=True,
              bufsize=0)
    for c in iter(lambda: p.stdout.read(1), ''):
        for f in [sys.stdout, file]:
            f.write(c)
            f.flush()
    p.stdout.close()
    rc = p.wait()

To read both stdout/stderr separately, you could use teed_call() from Python subprocess get children's output to file and terminal?

Answer 2

This should work

import subprocess
f = open('file.txt','w')
cmd = ['echo','hello','world']
subprocess.call(cmd, stdout=f)