Increasing io.sort.mb

It would be highly appreciated if someone could help me to find out what went wrong in my configuration.

I wanted to increase the value of io.sort.mb and thus I added the property below in core-site.xml. io.sort.mb 350m

The runtime information I am attaching below clearly shows that the value of io.sort.mb, did not change rather the default value io.sort.mb = 100 stayed.

13/08/15 16:43:34 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1e5e96c1
13/08/15 16:43:34 INFO mapred.MapTask: numReduceTasks: 1
13/08/15 16:43:34 INFO mapred.MapTask: **io.sort.mb = 100**
13/08/15 16:43:34 INFO mapred.MapTask: data buffer = 79691776/99614720
13/08/15 16:43:34 INFO mapred.MapTask: record buffer = 262144/327680
13/08/15 16:43:34 INFO mapred.MapTask: Starting flush of map output
13/08/15 16:43:34 INFO mapred.MapTask: Finished spill 0
13/08/15 16:43:34 INFO mapred.Task: Task:attempt_local_0001_m_004609_0 is done. And is in the process of commiting

Since it was not working, I added the property in mapred-site.xml schema, however I got the same outcome as above.

Can anyone suggest me what should I do?

Thanking you in advance.

Haq

Answers


according to the article here io.sort.mb should be 10 * io.sort.factor incase you have ram. "core-site.xml"

<property>
<name>io.sort.factor</name>
<value>100</value>
<description>More streams merged at once while sorting files.</description>
</property>  

<property>
<name>io.sort.mb</name>
<value>200</value>
<description>Higher memory-limit while sorting data.</description>
</property>

trying changing sort factor also on all nodes.


this conf should be in mapred-site.xml instead of core-site.xml

refer: http://hadoop.apache.org/docs/r1.0.4/mapred-default.html


Need Your Help

libGDX - remove color from imagebutton

android button colors libgdx imagebutton

I made an Imagebutton called skillButton and changed its color by using

mgwt side navigation / menu

navigation mgwt

Is there a full example or is it even possbile to display a side navigation in mgwt like :