org.apache.crunch.lib.join

Class OneToManyJoin

java.lang.Object

org.apache.crunch.lib.join.OneToManyJoin

public class OneToManyJoin
extends Object

Optimized join for situations where exactly one value is being joined with any other number of values based on a common key.

Constructor Summary

Constructors

Constructor and Description
OneToManyJoin()

Method Summary

Modifier and Type	Method and Description
static <K,U,V,T> PCollection<T>	oneToManyJoin(PTable<K,U> left, PTable<K,V> right, DoFn<Pair<U,Iterable<V>>,T> postProcessFn, PType<T> ptype) Performs a join on two tables, where the left table only contains a single value per key.
static <K,U,V,T> PCollection<T>	oneToManyJoin(PTable<K,U> left, PTable<K,V> right, DoFn<Pair<U,Iterable<V>>,T> postProcessFn, PType<T> ptype, int numReducers) Supports a user-specified number of reducers for the one-to-many join.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OneToManyJoin

public OneToManyJoin()

Method Detail

oneToManyJoin

public static <K,U,V,T> PCollection<T> oneToManyJoin(PTable<K,U> left,
                                                     PTable<K,V> right,
                                                     DoFn<Pair<U,Iterable<V>>,T> postProcessFn,
                                                     PType<T> ptype)

Performs a join on two tables, where the left table only contains a single value per key.

This method accepts a DoFn, which is responsible for converting the single left-side value and the iterable of right-side values into output values.

This method of joining is useful when there is a single context value that contains a large number of related values, and all related values must be brought together, with the quantity of the right-side values being too big to fit in memory.

If there are multiple values for the same key in the left-side table, only a single one will be used.

Parameters:

left - left-side table to join

right - right-side table to join

postProcessFn - DoFn to process the results of the join

ptype - type of the output of the postProcessFn

Returns:

the post-processed output of the join

oneToManyJoin

public static <K,U,V,T> PCollection<T> oneToManyJoin(PTable<K,U> left,
                                                     PTable<K,V> right,
                                                     DoFn<Pair<U,Iterable<V>>,T> postProcessFn,
                                                     PType<T> ptype,
                                                     int numReducers)

Supports a user-specified number of reducers for the one-to-many join.

Parameters:

left - left-side table to join

right - right-side table to join

postProcessFn - DoFn to process the results of the join

ptype - type of the output of the postProcessFn

numReducers - The number of reducers to use

Returns:

the post-processed output of the join