org.apache.crunch.util
Class PartitionUtils
java.lang.Object
org.apache.crunch.util.PartitionUtils
public class PartitionUtils
- extends Object
Helper functions and settings for determining the number of reducers to use in a pipeline
job created by the Crunch planner.
BYTES_PER_REDUCE_TASK
public static final String BYTES_PER_REDUCE_TASK
- See Also:
- Constant Field Values
DEFAULT_BYTES_PER_REDUCE_TASK
public static final long DEFAULT_BYTES_PER_REDUCE_TASK
- See Also:
- Constant Field Values
MAX_REDUCERS
public static final String MAX_REDUCERS
- Set an upper limit on the number of reducers the Crunch planner will set for an MR
job when it tries to determine how many reducers to use based on the input size.
- See Also:
- Constant Field Values
DEFAULT_MAX_REDUCERS
public static final int DEFAULT_MAX_REDUCERS
- See Also:
- Constant Field Values
PartitionUtils
public PartitionUtils()
getRecommendedPartitions
public static <T> int getRecommendedPartitions(PCollection<T> pcollection)
getRecommendedPartitions
public static <T> int getRecommendedPartitions(PCollection<T> pcollection,
org.apache.hadoop.conf.Configuration conf)
Copyright © 2014 The Apache Software Foundation. All Rights Reserved.