Jenkins console output not in realtime

Question

Pretty new to Jenkins and I have simple yet annoying problem. When I run job (Build) on Jenkins I am triggering ruby command to execute my test script.

Problem is Je

Answer 1

Each of the other answers is specific to one program or another, but I found a more general solution here:

https://unix.stackexchange.com/a/25378

You can use stdbuf to alter the buffering behavior of any program.

In my case, I was piping output from a shell script through tee and grep to split lines into either the console or a file based on content. The console was hanging as described by OP. This solved it:

./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | stdbuf -oL -eL grep LOG:

Eventually I discovered I could just pass --line-buffered to grep for the same result:

./slowly_parse.py login.csv |tee >(grep -v LOG: > out.csv) | grep --line-buffered LOG:

Answer 2

To clarify some of the answers.

• ruby or python or any sensible scripting language will buffer the output; this is in order to minimize the IO; writing to disk is slow, writing to a console is slow...
• usually the data gets flush()'ed automatically after you have enough data in the buffer with special handling for newlines. e.g. writing a string without newline then sleep() would not write anything until after the sleep() is complete (I'm only using sleep as an example, feel free to substitute with any other expensive system call).

e.g. this would wait 8 seconds, print one line, wait 5 more seconds, print a second line.

from time import sleep

def test():
    print "ok",
    time.sleep(3)
    print "now",
    time.sleep(5)
    print "done"
    time.sleep(5)
    print "again"

test()

• for ruby, STDOUT.sync = true, turns the autoflush on; all writes to STDOUT are followed by flush(). This would solve your problem but result in more IO.

  STDOUT.sync = true
  

• for python, you can use python -u or the environment variable PYTHONUNBUFFERED to make stdin/stdout/stout not buffered, but there are other solutions that do not change stdin or stderr

  export PYTHONUNBUFFERED=1
  

• for perl, you have autoflush

  autoflush STDOUT 1;
  

Answer 3

Python buffered its output traces and print it at the end of script to minimize writing on console as writing to console is slow.

You can use following command after your traces. It will flush all traces to console, which are queued before that command.

sys.stdout.flush()