Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)

Question

I want to debug a mapreduce script, and without going into much trouble tried to put some print statements in my program. But I cant seem to find them in any of the logs.

Answer 1

Actually stdout only shows the System.out.println() of the non-map reduce classes.

The System.out.println() for map and reduce phases can be seen in the logs. Easy way to access the logs is

http://localhost:50030/jobtracker.jsp->click on the completed job->click on map or reduce task->click on tasknumber->task logs->stdout logs.

Hope this helps

Answer 2

Another way is through the terminal:

1) Go into your Hadoop_Installtion directory, then into "logs/userlogs".
2) Open your job_id directory.
3) Check directories with _m_ if you want the mapper output or _r_ if you're looking for reducers.

Example: In Hadoop-20.2.0:

> ls ~/hadoop-0.20.2/logs/userlogs/attempt_201209031127_0002_m_000000_0/
log.index   stderr      stdout      syslog

The above means:
Hadoop_Installation: ~/hadoop-0.20.2
job_id: job_201209031127_0002
_m_: map task , "map number": _000000_

4) open stdout if you used "system.out.println" or stderr if you used "system.err.append".

PS. other hadoop versions might have a sight different hierarchy but they're all should be under $Hadoop_Installtion/logs/userlogs.

Answer 3

to get your stdout and log message on the console you can use apache commons logging framework in to your mapper and reducer.

public class MyMapper extends Mapper<..,...,..,...> {

    public static final Log log = LogFactory.getLog(MyMapper.class)

    public void map() throws Exception{
        // Log to stdout file
        System.out.println("Map key "+ key);

        //log to the syslog file
        log.info("Map key "+ key);

        if(log.isDebugEanbled()){
            log.debug("Map key "+ key);
        }

        context.write(key,value);
    }
}

Answer 4

On a Hadoop cluster with yarn, you can fetch the logs, including stdout, with:

yarn logs -applicationId application_1383601692319_0008

For some reason, I've found this to be more complete than what I see in the webinterface. The webinterface did not list the output of System.out.println() for me.