override log4j.properties in hadoop

问题

How do I override the default log4j.properties in hadoop? If I set the hadoop.root.logger=WARN,console, it doesnot print the logs on the console, whereas what I want is that it shouldn't print the INFO in the logs file. I added a log4j.properties file in my jar, but I am unable to override the default one. In short, I want the log file to print only the errors and warnings.

# Define some default values that can be overridden by system properties
hadoop.root.logger=INFO,console
hadoop.log.dir=.
hadoop.log.file=hadoop.log

#
# Job Summary Appender 
#
# Use following logger to send summary to separate file defined by 
# hadoop.mapreduce.jobsummary.log.file rolled daily:
# hadoop.mapreduce.jobsummary.logger=INFO,JSA
# 
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log

# Define the root logger to the system property "hadoop.root.logger".
log4j.rootLogger=${hadoop.root.logger}, EventCounter

# Logging Threshold
log4j.threshold=ALL

#
# Daily Rolling File Appender
#

log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}

# Rollver at midnight
log4j.appender.DRFA.DatePattern=.yyyy-MM-dd

# 30-day backup
#log4j.appender.DRFA.MaxBackupIndex=30
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout

# Pattern format: Date LogLevel LoggerName LogMessage
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
# Debugging Pattern format
#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n


#
# console
# Add "console" to rootlogger above if you want to use this 
#

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n

#
# TaskLog Appender
#

#Default values
hadoop.tasklog.taskid=null
hadoop.tasklog.iscleanup=false
hadoop.tasklog.noKeepSplits=4
hadoop.tasklog.totalLogFileSize=100
hadoop.tasklog.purgeLogSplits=true
hadoop.tasklog.logsRetainHours=12

log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender
log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}
log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}

log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

#
#Security appender
#
hadoop.security.log.file=SecurityAuth.audit
log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender 
log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}

log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
#new logger
# Define some default values that can be overridden by system properties
hadoop.security.logger=INFO,console
log4j.category.SecurityLogger=${hadoop.security.logger}

#
# Rolling File Appender
#

#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
#log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}

# Logfile size and and 30-day backups
#log4j.appender.RFA.MaxFileSize=1MB
#log4j.appender.RFA.MaxBackupIndex=30

#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n

#
# FSNamesystem Audit logging
# All audit events are logged at INFO level
#
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=WARN

# Custom Logging levels

#log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG
#log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG
#log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=DEBUG

# Jets3t library
log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR

#
# Event Counter Appender
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
#
log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter

#
# Job Summary Appender
#
log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.JSA.File=${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file}
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd
log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=${hadoop.mapreduce.jobsummary.logger}
log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=false

#
# MapReduce Audit Log Appender
#

# Set the MapReduce audit log filename
#hadoop.mapreduce.audit.log.file=hadoop-mapreduce.audit.log

# Appender for AuditLogger.
# Requires the following system properties to be set
#    - hadoop.log.dir (Hadoop Log directory)
#    - hadoop.mapreduce.audit.log.file (MapReduce audit log filename)

#log4j.logger.org.apache.hadoop.mapred.AuditLogger=INFO,MRAUDIT
#log4j.additivity.org.apache.hadoop.mapred.AuditLogger=false
#log4j.appender.MRAUDIT=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.MRAUDIT.File=${hadoop.log.dir}/${hadoop.mapreduce.audit.log.file}
#log4j.appender.MRAUDIT.DatePattern=.yyyy-MM-dd
#log4j.appender.MRAUDIT.layout=org.apache.log4j.PatternLayout
#log4j.appender.MRAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n

回答1:

If you use the default Log4j.properties file the logging settings get overridden by environment variables from the startup script. If you want to use the default log4j and just simply want to change the logging level, use $HADOOP_CONF_DIR/hadoop-env.sh

For example, to change your logger to DEBUG log level and DRFA logger, use

export HADOOP_ROOT_LOGGER="DEBUG,DRFA"

回答2:

You could remove the log4j.properties from your hadoop jar
OR make sure that your jar/log4j.properties is first in the classpath (log4j picks the first log4j.properties from the classpath that it finds)
OR specify the system variable: -Dlog4j.configuration=PATH_TO_FILE

See the documentation to learn how log4j finds the configuration.

回答3:

Modify the log4j file inside HADOOP_CONF_DIR. Note that hadoop job wont consider the log4j file of your application. It will consider the one inside HADOOP_CONF_DIR.

If you want to force hadoop to use some other log4j file, try one of these:

You can try what @Patrice said. ie.

-Dlog4j.configuration=file:/path/to/user_specific/log4j.xml
Customize HADOOP_CONF_DIR/log4j.xml and set the logger level for "your" classes as per your wish. Other user(s) wont be affected due to this unless both are having classes with same package structure. This wont work for core hadoop classes as all users will get afftected.
Create your customized log4j file. Replicate the directory HADOOP_CONF_DIR and put your log4j file inside it. export HADOOP_CONF_DIR to your conf directory. Other users will point to the default one.

回答4:

Maven Packaging:

Once I realized I needed to add my custom debug-log.properties file to src/main/java/resources, Maven added it to the application.jar root directory and then it was just a matter of referring to it or not in -Dlog4j.configuration=debug-log.properties from the command line.

Oozie <java> Action:

In regard to Oozie, use <java-opts>-Dlog4j.configuration=${log4jConfig}</java-opts> in the workflow.xml actions and define the following in a job.properties file.

#one of the following log4j.config parameters must be defined 
#log4jConfig=log4j.properties
log4jConfig=debug-log.properties

Oozie <map-reduce> Action:

 <property>
      <name>mapred.child.java.opts</name>
      <value>-Dlog4j.configuration=${log4jConfig}</value>
 </property>

回答5:

I was faced with the same problem (CDH3U3, Hadoop 0.20.2). I finally found a solution with (note file: prefix in the path):

-Dlog4j.configuration=file:/path/to/user_specific/log4j.xml

回答6:

As mentioned by Sulpha, for hadoop 1.2.1, it is important to override the task-log4j.properties that is present inside hadoop-core.jar

For my pseudo distributed mode,

I was unable to print the debug messages of my pig UDFs and had to delete the task-log4j.properties from hadoop-core.jar and replace it with a copy of the $HADOOP_INSTALL/conf/log4j.properties.

Used the

zip -d hadoop-core-1.2.1.jar task-log4j.properties #to delete

and

zip -g hadoop-core-1.2.1.jar task-log4j.properties #to add back

回答7:

If there is already configured log4j properties file inside the jar file. you can override by simple putting -Dlog4j.configuration= before the -classpath

here is the sample:

java -Dlog4j.configuration=..\conf\log4j.properties -classpath %CLASSPATH%

回答8:

Put log4j.configuration option in the child java options.

I.e.

hadoop jar ... -Dmapred.child.java.opts=-Dlog4j.configuration=file:/...../log4j_debug.properties

You must put log4j_debug.properties file on all slave servers in a same directory path like /home/yourname/log4j_debug.properties or /tmp/log4j_debug.properties

This setting overwrites mapred.child.java.opts settings. If you want to use with another options like -Xmx32m, which means 32MB heap size, then do like following:

hadoop jar ... -Dmapred.child.java.opts='-Xmx32m -Dlog4j.configuration=file:/...../log4j_debug.properties'

回答9:

In the Hadoop 1.2.1 there are 2 config files: log4j.properties and task-log4j.properties So to make example above work the change have to be done in task-log4j.properties not in log4j.properties

you can add follwing line in your task-log4j.properties:

log4j.logger.org.xxx=WARN

来源：https://stackoverflow.com/questions/9081625/override-log4j-properties-in-hadoop

标签

Hadoop

log4j