org.apache.hive.hcatalog.mapreduce

Class MultiOutputFormat

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable>

org.apache.hive.hcatalog.mapreduce.MultiOutputFormat

public class MultiOutputFormat
extends org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable>

The MultiOutputFormat class simplifies writing output data to multiple outputs.

Multiple output formats can be defined each with its own OutputFormat class, own key class and own value class. Any configuration on these output format classes can be done without interfering with other output format's configuration.

Usage pattern for job submission:

 Job job = new Job();

 FileInputFormat.setInputPath(job, inDir);

 job.setMapperClass(WordCountMap.class);
 job.setReducerClass(WordCountReduce.class);
 job.setInputFormatClass(TextInputFormat.class);
 job.setOutputFormatClass(MultiOutputFormat.class);
 // Need not define OutputKeyClass and OutputValueClass. They default to
 // Writable.class
 job.setMapOutputKeyClass(Text.class);
 job.setMapOutputValueClass(IntWritable.class);


 // Create a JobConfigurer that will configure the job with the multiple
 // output format information.
 JobConfigurer configurer = MultiOutputFormat.createConfigurer(job);

 // Defines additional single text based output 'text' for the job.
 // Any configuration for the defined OutputFormat should be done with
 // the Job obtained with configurer.getJob() method.
 configurer.addOutputFormat("text", TextOutputFormat.class,
                 IntWritable.class, Text.class);
 FileOutputFormat.setOutputPath(configurer.getJob("text"), textOutDir);

 // Defines additional sequence-file based output 'sequence' for the job
 configurer.addOutputFormat("sequence", SequenceFileOutputFormat.class,
                 Text.class, IntWritable.class);
 FileOutputFormat.setOutputPath(configurer.getJob("sequence"), seqOutDir);
 ...
 // configure method to be called on the JobConfigurer once all the
 // output formats have been defined and configured.
 configurer.configure();

 job.waitForCompletion(true);
 ...
 

Usage in Reducer:

 public class WordCountReduce extends
         Reducer<Text, IntWritable, Writable, Writable> {

     private IntWritable count = new IntWritable();

     public void reduce(Text word, Iterator<IntWritable> values,
             Context context)
             throws IOException {
         int sum = 0;
         for (IntWritable val : values) {
             sum += val.get();
         }
         count.set(sum);
         MultiOutputFormat.write("text", count, word, context);
         MultiOutputFormat.write("sequence", word, count, context);
     }

 }

 

Map only jobs:

MultiOutputFormat.write("output", key, value, context); can be called similar to a reducer in map only jobs.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MultiOutputFormat.JobConfigurer Class that supports configuration of the job for multiple output formats.
class	MultiOutputFormat.MultiOutputCommitter

Constructor Summary

Constructors

Constructor and Description
MultiOutputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
static MultiOutputFormat.JobConfigurer	createConfigurer(org.apache.hadoop.mapreduce.Job job) Get a JobConfigurer instance that will support configuration of the job for multiple output formats.
static org.apache.hadoop.mapreduce.JobContext	getJobContext(String alias, org.apache.hadoop.mapreduce.JobContext context) Get the JobContext with the related OutputFormat configuration populated given the alias and the actual JobContext
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
static org.apache.hadoop.mapreduce.TaskAttemptContext	getTaskAttemptContext(String alias, org.apache.hadoop.mapreduce.TaskAttemptContext context) Get the TaskAttemptContext with the related OutputFormat configuration populated given the alias and the actual TaskAttemptContext
static <K,V> void	write(String alias, K key, V value, org.apache.hadoop.mapreduce.TaskInputOutputContext context) Write the output key and value using the OutputFormat defined by the alias.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiOutputFormat

public MultiOutputFormat()

Method Detail

createConfigurer

public static MultiOutputFormat.JobConfigurer createConfigurer(org.apache.hadoop.mapreduce.Job job)

Get a JobConfigurer instance that will support configuration of the job for multiple output formats.

Parameters:

job - the mapreduce job to be submitted

Returns:

JobConfigurer

getJobContext

public static org.apache.hadoop.mapreduce.JobContext getJobContext(String alias,
                                                   org.apache.hadoop.mapreduce.JobContext context)

Get the JobContext with the related OutputFormat configuration populated given the alias and the actual JobContext

Parameters:

alias - the name given to the OutputFormat configuration

context - the JobContext

Returns:

a copy of the JobContext with the alias configuration populated

getTaskAttemptContext

public static org.apache.hadoop.mapreduce.TaskAttemptContext getTaskAttemptContext(String alias,
                                                                   org.apache.hadoop.mapreduce.TaskAttemptContext context)

Get the TaskAttemptContext with the related OutputFormat configuration populated given the alias and the actual TaskAttemptContext

Parameters:

alias - the name given to the OutputFormat configuration

context - the Mapper or Reducer Context

Returns:

a copy of the TaskAttemptContext with the alias configuration populated

write

public static <K,V> void write(String alias,
               K key,
               V value,
               org.apache.hadoop.mapreduce.TaskInputOutputContext context)
                  throws IOException,
                         InterruptedException

Write the output key and value using the OutputFormat defined by the alias.

Parameters:

alias - the name given to the OutputFormat configuration

key - the output key to be written

value - the output value to be written

context - the Mapper or Reducer Context

Throws:

IOException

InterruptedException

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
                      throws IOException,
                             InterruptedException

Specified by:

checkOutputSpecs in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable>

Throws:

IOException

InterruptedException

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                                      throws IOException,
                                                                                                                             InterruptedException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable>

Throws:

IOException

InterruptedException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                               throws IOException,
                                                                      InterruptedException

Specified by:

getOutputCommitter in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable>

Throws:

IOException

InterruptedException