org.apache.solr.hadoop.dedup

Class SortingUpdateConflictResolver

java.lang.Object

org.apache.solr.hadoop.dedup.SortingUpdateConflictResolver

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, UpdateConflictResolver

public class SortingUpdateConflictResolver
extends Object
implements UpdateConflictResolver, org.apache.hadoop.conf.Configurable

UpdateConflictResolver implementation that orders colliding updates ascending from least recent to most recent (partial) update, based on a configurable numeric Solr field, which defaults to the file_last_modified timestamp.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ORDER_BY_FIELD_NAME_DEFAULT
static String	ORDER_BY_FIELD_NAME_KEY

Constructor Summary

Constructors

Constructor and Description
SortingUpdateConflictResolver()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConf()
protected String	getOrderByFieldName()
Iterator<SolrInputDocument>	orderUpdates(org.apache.hadoop.io.Text key, Iterator<SolrInputDocument> updates, org.apache.hadoop.mapreduce.Reducer.Context ctx) Given a list of all colliding document updates for the same unique document key, this method returns zero or more documents in an application specific order.
void	setConf(org.apache.hadoop.conf.Configuration conf)
protected Iterator<SolrInputDocument>	sort(Iterator<SolrInputDocument> updates, String fieldName, Comparator child)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ORDER_BY_FIELD_NAME_KEY

public static final String ORDER_BY_FIELD_NAME_KEY

ORDER_BY_FIELD_NAME_DEFAULT

public static final String ORDER_BY_FIELD_NAME_DEFAULT

See Also:

Constant Field Values

Constructor Detail

SortingUpdateConflictResolver

public SortingUpdateConflictResolver()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getOrderByFieldName

protected String getOrderByFieldName()

orderUpdates

public Iterator<SolrInputDocument> orderUpdates(org.apache.hadoop.io.Text key,
                                       Iterator<SolrInputDocument> updates,
                                       org.apache.hadoop.mapreduce.Reducer.Context ctx)

Description copied from interface: UpdateConflictResolver

Given a list of all colliding document updates for the same unique document key, this method returns zero or more documents in an application specific order. The caller will then apply the updates for this key to Solr in the order returned by the orderUpdate() method.

Specified by:

orderUpdates in interface UpdateConflictResolver

Parameters:

key - the document key common to all collidingUpdates mentioned below

updates - all updates in the MapReduce job that have a key equal to uniqueKey mentioned above. The input order is unspecified.

ctx - The Context passed from the Reducer implementations.

Returns:

the order in which the updates shall be applied to Solr

sort

protected Iterator<SolrInputDocument> sort(Iterator<SolrInputDocument> updates,
                               String fieldName,
                               Comparator child)