How to process input and output streams in Steel Bank Common Lisp?

Question

I'm trying to figure out how to use the output stream of one program I start with RUN-PROGRAM so it can be used as the input of another program started with RUN-PROGRAM (i.e., the moral and perhaps literal equivalent of piping). I've tried using a number of combinations of the :INPUT, :OUTPUT and :WAIT keyword arguments, but nothing I've hit upon has been productive so far. Any tips would be helpful; for example, how would I go about doing something like ls | grep lisp from the shell?

One of my attempts is

(defun piping-test () 
  (let ((grep-process (run-program "/usr/bin/grep" '("lisp") 
                                  :input :stream 
                                  :output :stream))) 
    (unwind-protect 
        (with-open-stream (s (process-input grep-process)) 
          (let ((ls-process (run-program "/bin/ls" '() 
                                        :output s))) 
            (when ls-process 
              (unwind-protect 
                  (with-open-stream (o (process-output grep-process)) 
                   (loop 
                      :for line := (read-line o nil nil) 
                      :while line 
                      :collect line)) 
               (process-close ls-process))))) 
     (when grep-process (process-close grep-process))))) 

Running this in a SLIME REPL causes everything to hang until I break with C-c C-c, so it's pretty obviously not the right thing, but I'm not sure how to change it so it is the right thing.

EDIT: Adding :WAIT NIL to both RUN-PROGRAM invocations, or to only the invocation for grep, doesn't do the trick. In that case, the function will hang, and breaking with C-c C-c gets a stack trace indicating that there's a local function (defined via FLET) called SB-UNIX:SELECT that has hung.

Answer 1

I got a working answer from Raymond Toy on comp.lang.lisp. His solution was for CMUCL, but it worked with the essentially identical RUN-PROGRAM function on the closely related SBCL, and with minor changes it will work on CCL as well, because CCL's RUN-PROGRAM is basically a clone of the one from CMUCL/SBCL.

The secret, as it were, is to set up the ls process first, and then provide its output stream to the grep process as input, like so:

(defun piping-test2 () 
  (let ((ls-process (run-program "/bin/ls" '() 
                                 :wait nil 
                                 :output :stream))) 
    (unwind-protect 
        (with-open-stream (s (process-output ls-process)) 
          (let ((grep-process (run-program "/usr/bin/grep" '("lisp") 
                                          :input s 
                                          :output :stream))) 
            (when grep-process 
              (unwind-protect 
                  (with-open-stream (o (process-output grep-process)) 
                    (loop 
                       :for line := (read-line o nil nil) 
                       :while line 
                       :collect line)) 
                (process-close grep-process))))) 
      (when ls-process (process-close ls-process))))) 

I also experimented with omitting the :WAIT NIL argument from the RUN-PROGRAM call for ls, and it worked just as well.

Answer 2

Try adding :wait nil to your arguments to run-program. That should have both your grep and your ls running in the background. As is, you're starting the grep process, waiting for that to finish, then starting the ls you're intending to feed into the grep process. Alas, since you're waiting for the grep to finish, you never get that far.

Answer 3

Relatedly, but perhaps not spot on to your question, you could do:

(with-output-to-string (s)
      (ccl:run-program "sh" (list "-c" "ls a/directory/somewhere/*.lisp") :output s)
      s)

or

(with-output-to-string (s)
      (ccl:run-program "sh" (list "-c" "ls /a/directory/somewhere/*.lisp | wc -l") :output s)
      s)

or

(with-output-to-string (s)
      (ccl:run-program "ssh" (list "a-user@some-ip" "sh -c ls /a/directory/somewhere/on/remote/server/*.lisp | wc -l") :output s)
      s)

And, of course, you can use

(format nil "ls ~a" directory)

To get input, you can do something like:

(with-output-to-string (out)
      (format t "~%Enter your sudo password:~%")
      (with-input-from-string (s (read))
        (ccl:run-program "ssh" (list *remote* "sudo cat /etc/init.d/nginx")  :input s :output out))
      out)