How to delete yarn logs

问题

I'm pretty new to YARN. I ran my oozie jobs which creates logs.

I can see yarn logs with

yarn logs -applicationId application_123456789_12345678

I want to know how I can delete those logs? Can I just delete the file to remove the logs?

回答1:

There no yarn commands to delete yarn logs from CLI.

You can delete by using Linux rm by going to the yarn log directory yarn.nodemanager.log-dirs /application_${appid}. Individual containers log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.

回答2:

Once the job has completed the NodeManager will keep the log for each container for ${yarn.nodemanager.log.retain-seconds} which is 10800 seconds by default ( 3 hours ) and delete them once they have expired. But if ${yarn.log-aggregation-enable} is enabled then the NodeManager will immediately concatenate all of the containers logs into one file and upload them into HDFS in ${yarn.nodemanager.remote-app-log-dir}/${user.name}/logs/ and delete them from the local userlogs directory

Documentation indicates 'yarn.nodemanager.delete.debug-delay-sec':

Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below).

You must configure yarn-default.xml:

  <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory.
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>0</value>
  </property>

Maybe retain seconds can be useful:

<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>

回答3:

delete ${yarn.nodemanager.remote-app-log-dir}/${user}/${application_id}/...

来源：https://stackoverflow.com/questions/39195478/how-to-delete-yarn-logs

标签

Hadoop

yarn