hadoop streaming: how to see application logs?

Question

I can see all hadoop logs on my /usr/local/hadoop/logs path

but where can I see application level logs? for example :

mapper.py

import logging

def main():
    logging.info("starting map task now")
    // -- do some task -- //
    print statement  

reducer.py

import logging
def main():
    for line in sys.stdin:
        logging.info("received input to reducer - " + line)  
        // -- do some task -- //
        print statement

Where I can see logging.info or related log statements of my application?
I am using Python and using hadoop-streaming

Thank you

Answer 1

Hadoop streaming uses STDIN/STDOUT for passing the key/value pairs between the mappers and reducers, so the log messages have to be written to a specific log file - check the sample code and the python logging documentation for more details. This Query might also help.

Answer 2

Hadoop gathers stderr, which can be viewed on hadoop map/reduce status site. So you can just write to stderr.