yarn [hadoop 2.2] where is the location mapper or reducer log outputed?

I want to check the log mapper or reducer output ?I can not find it in syslog under container foler?So where is the log outputing?

        public class SkipStat {
private static Log log = LogFactory.getLog(SkipStat.class);

private static BlockWorkerRepository blockWorkerRepository;
static {
    blockWorkerRepository = new BlockWorkerRepositoryImpl();
}

private static class SkipInfoMapper extends Mapper<Object, BSONObject, Text, AssignmentWritable> {

    private final String invalidResult = "^";

    private static final Calendar currentCalendar = Calendar.getInstance();
    static {
        currentCalendar.add(Calendar.HOUR, -24);
    }

    protected void map(Object key, BSONObject value, Context context) throws IOException, InterruptedException {
        String result = (String) value.get("result");
        log.info("lol... get one result " + result); // LOG ...
        if (invalidResult.equals(result)) {

Answers


The historyserver is collecting those logs and keeping them. You can view them using the historyserver WebUI or using yarn logs CLI. See Simplifying user-logs management and access in YARN. Before they're uploaded the logs are:

Logs for all the containers belonging to a single Application and that ran on a given NM are aggregated and written out to a single (possibly compressed) log file at a configured location in the FS

The ApplicationMasterUI will show current executing application logs.


To enable log aggregation, which then allows you to run the "yarn logs" command to see the logs for an application id, set these in your yarn-site.xml:

yarn.log-aggregation-enable
yarn.log-aggregation.retain-seconds
yarn.log-aggregation.retain-check-interval-seconds

It seemed necessary to restart yarn after making this change.

Note the command that was referenced in a comment will work after aggregation has been enabled:

yarn logs -applicationId application_1397033985470_0006

See: http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

Descriptions for these parameters are available in the 2.3.0 documentation: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml


Need Your Help

How to create custom archive page on Tumblr

html css tumblr

I am creating custom Tumblr blog layout: http://thespektrdev.tumblr.com/