org.apache.crunch.io

Class To

java.lang.Object

org.apache.crunch.io.To

public class To
extends Object

Static factory methods for creating common Target types.

The To class is intended to be used as part of a literate API for writing the output of Crunch pipelines to common file types. We can use the Target objects created by the factory methods in the To class with either the write method on the Pipeline class or the convenience write method on PCollection and PTable instances.

 

   Pipeline pipeline = new MRPipeline(this.getClass());
   ...
   // Write a PCollection<String> to a text file:
   PCollection<String> words = ...;
   pipeline.write(words, To.textFile("/put/my/words/here"));
   
   // Write a PTable<Text, Text> to a sequence file:
   PTable<Text, Text> textToText = ...;
   textToText.write(To.sequenceFile("/words/to/words"));
   
   // Write a PCollection<MyAvroObject> to an Avro data file:
   PCollection<MyAvroObject> objects = ...;
   objects.write(To.avroFile("/my/avro/files"));
   
   // Write a PTable to a custom FileOutputFormat:
   PTable<KeyWritable, ValueWritable> custom = ...;
   pipeline.write(custom, To.formattedFile("/custom", MyFileFormat.class));
 
 

Constructor Summary

Constructors

Constructor and Description
To()

Method Summary

Modifier and Type	Method and Description
static Target	avroFile(org.apache.hadoop.fs.Path path) Creates a Target at the given Path that writes data to Avro files.
static Target	avroFile(String pathName) Creates a Target at the given path name that writes data to Avro files.
static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> Target	formattedFile(org.apache.hadoop.fs.Path path, Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass) Creates a Target at the given Path that writes data to a custom FileOutputFormat.
static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> Target	formattedFile(String pathName, Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass) Creates a Target at the given path name that writes data to a custom FileOutputFormat.
static Target	sequenceFile(org.apache.hadoop.fs.Path path) Creates a Target at the given Path that writes data to SequenceFiles.
static Target	sequenceFile(String pathName) Creates a Target at the given path name that writes data to SequenceFiles.
static Target	textFile(org.apache.hadoop.fs.Path path) Creates a Target at the given Path that writes data to text files.
static Target	textFile(String pathName) Creates a Target at the given path name that writes data to text files.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

To

public To()

Method Detail

formattedFile

public static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> Target formattedFile(String pathName,
                                                                                                                     Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass)

Creates a Target at the given path name that writes data to a custom FileOutputFormat.

Parameters:

pathName - The name of the path to write the data to on the filesystem

formatClass - The FileOutputFormat<K, V> to write the data to

Returns:

A new Target instance

formattedFile

public static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> Target formattedFile(org.apache.hadoop.fs.Path path,
                                                                                                                     Class<? extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<K,V>> formatClass)

Creates a Target at the given Path that writes data to a custom FileOutputFormat.

Parameters:

path - The Path to write the data to

formatClass - The FileOutputFormat to write the data to

Returns:

A new Target instance

avroFile

public static Target avroFile(String pathName)

Creates a Target at the given path name that writes data to Avro files. The PType for the written data must be for Avro records.

Parameters:

pathName - The name of the path to write the data to on the filesystem

Returns:

A new Target instance

avroFile

public static Target avroFile(org.apache.hadoop.fs.Path path)

Creates a Target at the given Path that writes data to Avro files. The PType for the written data must be for Avro records.

Parameters:

path - The Path to write the data to

Returns:

A new Target instance

sequenceFile

public static Target sequenceFile(String pathName)

Creates a Target at the given path name that writes data to SequenceFiles.

Parameters:

pathName - The name of the path to write the data to on the filesystem

Returns:

A new Target instance

sequenceFile

public static Target sequenceFile(org.apache.hadoop.fs.Path path)

Creates a Target at the given Path that writes data to SequenceFiles.

Parameters:

path - The Path to write the data to

Returns:

A new Target instance

textFile

public static Target textFile(String pathName)

Creates a Target at the given path name that writes data to text files.

Parameters:

pathName - The name of the path to write the data to on the filesystem

Returns:

A new Target instance

textFile

public static Target textFile(org.apache.hadoop.fs.Path path)

Creates a Target at the given Path that writes data to text files.

Parameters:

path - The Path to write the data to

Returns:

A new Target instance