8.10. Recommended Memory Configurations for the MapReduce Service

  • Make sure that there is enough memory for all the processes. Remember that system processes take around 10% of the available memory.

  • For co-deploying an HBase RegionServer and MapReduce service on the same node, reduce the RegionServer's heap size (use the HBase Region Servers maximum Java heap size property to modify the RegionServer heap size).

  • For co-deploying an HBase RegionServer and the MapReduce service on the same node, or for memory intensive MapReduce applications, modify the map and reduce slots as suggested in the following example:

EXAMPLE: For co-deploying an HBase RegionServer and the MapReduce service on a machine with 16GB of available memory, the following would be a recommended configuration:

2 GB: system processes

8 GB: MapReduce slots. 6 Map + 2 Reduce slots per 1 GB task

4 GB: HBase RegionServer

1 GB: TaskTracker

1 GB: DataNode

To change the number of Map and Reduce slots based on the memory requirements of your application, use the following properties:

Number of Map slots per node

Number of Reduce slots per node


loading table of contents...