Hadoop pipes wordcount: Task Id : attempt Status : FAILED AttemptID:attempt Timed out after 600 secs

My Haddop is the lastest version 2.3.0. I can run Hadoop Java application well. But there is some wrong in Hadoop pipes wordcount application.

This is wordcount-simple.cc:

#include "Pipes.hh"
#include "TemplateFactory.hh"
#include "StringUtils.hh"

const std::string WORDCOUNT = "WORDCOUNT";
const std::string INPUT_WORDS = "INPUT_WORDS";
const std::string OUTPUT_WORDS = "OUTPUT_WORDS";

class WordCountMap: public HadoopPipes::Mapper {
  HadoopPipes::TaskContext::Counter* inputWords;

  WordCountMap(HadoopPipes::TaskContext& context) {
    inputWords = context.getCounter(WORDCOUNT, INPUT_WORDS);

  void map(HadoopPipes::MapContext& context) {
    std::vector<std::string> words =
      HadoopUtils::splitString(context.getInputValue(), " ");
      for(unsigned int i=0; i < words.size(); ++i) {
          context.emit(words[i], "1");
    context.incrementCounter(inputWords, words.size());

class WordCountReduce: public HadoopPipes::Reducer {
  HadoopPipes::TaskContext::Counter* outputWords;

  WordCountReduce(HadoopPipes::TaskContext& context) {
    outputWords = context.getCounter(WORDCOUNT, OUTPUT_WORDS);

  void reduce(HadoopPipes::ReduceContext& context) {
    int sum = 0;
    while (context.nextValue()) {
      sum += HadoopUtils::toInt(context.getInputValue());
    context.emit(context.getInputKey(), HadoopUtils::toString(sum));
    context.incrementCounter(outputWords, 1);

Following is Makefile:


CC = g++

wordcount :wordcount-simple.cc
        $(CC) $(CCFLAGS) $< -Wall -L$(HADOOP_INSTALL)/lib/native -lhadooppipes -lhadooputils -lpthread -lcrypto -g -O2 -o $@

It compiles OK. Then I upload some data for testing. My run command is :

hadoop pipes -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -D mapred.job.name=wordcount -input /data/wc_in -output /data/wc_out -program /bin/wordcount

but some error occurs:

14/04/03 23:59:48 INFO client.RMProxy: Connecting to ResourceManager at /
14/04/03 23:59:49 INFO client.RMProxy: Connecting to ResourceManager at /
14/04/03 23:59:50 WARN mapreduce.JobSubmitter: No job jar file set. User classes may not be found. See Job or Job#setJar(String). 
14/04/03 23:59:50 INFO mapred.FileInputFormat: Total input paths to process : 2
14/04/03 23:59:51 INFO mapreduce.JobSubmitter: number of splits:2
14/04/03 23:59:51 INFO Configuration.deprecation: hadoop.pipes.java.recordreader is deprecated. Instead, use mapreduce.pipes.isjavarecordreader
14/04/03 23:59:51 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/04/03 23:59:51 INFO Configuration.deprecation: hadoop.pipes.java.recordwriter is deprecated. Instead, use mapreduce.pipes.isjavarecordwriter
14/04/03 23:59:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1396578697573_0004
14/04/03 23:59:52 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
14/04/03 23:59:53 INFO impl.YarnClientImpl: Submitted application application_1396578697573_0004
14/04/03 23:59:53 INFO mapreduce.Job: The url to track the job: 
14/04/03 23:59:53 INFO mapreduce.Job: Running job: job_1396578697573_0004
14/04/04 00:00:26 INFO mapreduce.Job: Job job_1396578697573_0004 running in uber mode : false

**14/04/04 00:00:26 INFO mapreduce.Job: map 0% reduce 0%
14/04/04 00:10:53 INFO mapreduce.Job: map 100% reduce 0%
14/04/04 00:10:53 INFO mapreduce.Job: Task Id : attempt_1396578697573_0004_m_000001_0, Status : FAILED
AttemptID:attempt_1396578697573_0004_m_000001_0 Timed out after 600 secs
14/04/04 00:10:54 INFO mapreduce.Job: Task Id : attempt_1396578697573_0004_m_000000_0, Status : FAILED
AttemptID:attempt_1396578697573_0004_m_000000_0 Timed out after 600 secs
14/04/04 00:10:55 INFO mapreduce.Job: map 0% reduce 0%
14/04/04 00:21:23 INFO mapreduce.Job: map 100% reduce 0%
14/04/04 00:21:24 INFO mapreduce.Job: Task Id : attempt_1396578697573_0004_m_000000_1, Status : FAILED
AttemptID:attempt_1396578697573_0004_m_000000_1 Timed out after 600 secs
14/04/04 00:21:24 INFO mapreduce.Job: Task Id : attempt_1396578697573_0004_m_000001_1, Status : FAILED
AttemptID:attempt_1396578697573_0004_m_000001_1 Timed out after 600 secs
14/04/04 00:21:25 INFO mapreduce.Job: map 0% reduce 0%
14/04/04 00:31:53 INFO mapreduce.Job: Task Id : attempt_1396578697573_0004_m_000000_2, Status : FAILED
AttemptID:attempt_1396578697573_0004_m_000000_2 Timed out after 600 secs
14/04/04 00:31:53 INFO mapreduce.Job: Task Id : attempt_1396578697573_0004_m_000001_2, Status : FAILED
AttemptID:attempt_1396578697573_0004_m_000001_2 Timed out after 600 secs
14/04/04 00:42:24 INFO mapreduce.Job: map 100% reduce 0%
14/04/04 00:42:25 INFO mapreduce.Job: map 100% reduce 100%
14/04/04 00:42:26 INFO mapreduce.Job: Job job_1396578697573_0004 failed with state FAILED due to: Task failed task_1396578697573_0004_m_000000

Job failed as tasks failed. failedMaps:1 failedReduces:0
14/04/04 00:42:27 INFO mapreduce.Job: Counters: 9
 Job Counters
  Failed map tasks=8
  Launched map tasks=8
  Other local map tasks=6
  Data-local map tasks=2
  Total time spent by all maps in occupied slots (ms)=5017539
  Total time spent by all reduces in occupied slots (ms)=0
  Total time spent by all map tasks (ms)=5017539
  Total vcore-seconds taken by all map tasks=5017539
  Total megabyte-seconds taken by all map tasks=5137959936

Exception in thread "main" java.io.IOException: Job failed!
 at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836)
 at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:264)
 at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:503)
 at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:518)**

It runs long time but failed. Is there some wrong with my settiings ? But I run no-pipes applications well, no any error. so I think no problem with my enviromnent.

Also, no any error in namenode and datanode(one) logs. So I wonder why. Thanks in advance.


After analyzing datanode's container logs, I have found the reason why these errors. My master machine is Ubuntu 13.04 and my slave machine is Fedora 17, the two system have different core design and different library version. For example, as for libcrypto.so.10, Ubuntu has this library but Fedora hasn't, so the error occurs(slave couldn't find the accurate library). The basic problem is that "master and all slaves machines must be the SAME system", you can clone system to achieve this.

Need Your Help

Implement SQL Union Query using Query Builder in Doctrine Symfony2

php sql symfony doctrine query-builder

I'm currently using Symfony2 framework. I don't know how to join 2 of my query by Query Builder in Symfony2, I just can join them using SQL UNION query. Below is the query that returns correct resu...