hadoop - How long can we check the status upon completion of mapreduce job?

I have 3 mapreduce jobs to be executed in parallel, thus, i've done something like this

Job[] job = new Job[3];
.
.
.
job[0].submit();
job[1].submit();
job[2].submit();

So to check if all jobs are successful, i put some polling on the 3 jobs.

boolean isAllFinished = false;
while(!isAllFinished) {
    for(int i = 0; i < 3; i++) {
        log.debug("job["+i+"].isComplete() >> " + job[i].isComplete());
        isAllFinished = isAllFinished & job[i].isComplete();
    }
    Thread.sleep(1000);
}

Though, this yield positively, it failed on random occasions where I would have these kind of error/log:

14/04/08 18:43:59 DEBUG FMS: job[0].isComplete() >> false
14/04/08 18:44:00 DEBUG FMS: job[1].isComplete() >> false
14/04/08 18:44:01 DEBUG FMS: job[2].isComplete() >> false
14/04/08 18:44:12 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
14/04/08 18:44:13 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/08 18:44:14 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/04/08 18:44:15 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)

I'm thinking if there is a timeout on up to when we can check the job status via Job class. Appreciate any thoughts on this!

Answers


Hadoop job can be configured to wait till the completion using job.waitForCompletion(true);

You can try code like below :

job[0].waitForCompletion(true);
job[1].waitForCompletion(true);
job[2].waitForCompletion(true);

If you want to provide timeout, then you can use wait(timeout) method.

job[0].wait(1000);

Need Your Help

sql/php - add 4 days to unixtimestamps, with different starting days

php mysql sql unix-timestamp

I have a database with 30k+ users. I wish to add 4 days to a time I have stored for each user in unixtimestamp.