org.apache.hive.hcatalog.api.repl

Class ReplicationTask

java.lang.Object

org.apache.hive.hcatalog.api.repl.ReplicationTask

Direct Known Subclasses:

AddPartitionReplicationTask, AlterPartitionReplicationTask, AlterTableReplicationTask, CreateTableReplicationTask, DropDatabaseReplicationTask, DropPartitionReplicationTask, DropTableReplicationTask, InsertReplicationTask, NoopReplicationTask

@InterfaceStability.Evolving
public abstract class ReplicationTask
extends Object

ReplicationTask captures the concept of what it'd take to replicate changes from one warehouse to another given a notification event that captures what changed.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	ReplicationTask.Factory

Field Summary

Fields

Modifier and Type	Field and Description
protected com.google.common.base.Function<String,String>	dbNameMapping
protected StagingDirectoryProvider	dstStagingDirProvider
protected HCatNotificationEvent	event
protected static MessageFactory	messageFactory
protected StagingDirectoryProvider	srcStagingDirProvider
protected com.google.common.base.Function<String,String>	tableNameMapping

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ReplicationTask(HCatNotificationEvent event)

Method Summary

Methods

Modifier and Type	Method and Description
static ReplicationTask	create(HCatClient client, HCatNotificationEvent event) Factory method to return appropriate subtype of ReplicationTask for given event
abstract Iterable<? extends Command>	getDstWhCommands() Returns a Iterable to send to a hive driver on the source warehouse If you *need* a List instead, you can use guava's ImmutableList.copyOf(iterable) or Lists.newArrayList(iterable) to get the underlying list, but this defeats the purpose of making this interface an Iterable rather than a List, since it is very likely that the number of Commands returned here will cause your process to run OOM.
HCatNotificationEvent	getEvent() Returns the event that this ReplicationTask is attempting to replicate
abstract Iterable<? extends Command>	getSrcWhCommands() Returns a Iterable to send to a hive driver on the source warehouse If you *need* a List instead, you can use guava's ImmutableList.copyOf(iterable) or Lists.newArrayList(iterable) to get the underlying list, but this defeats the purpose of making this interface an Iterable rather than a List, since it is very likely that the number of Commands returned here will cause your process to run OOM.
boolean	isActionable() Returns true if this ReplicationTask is prepared with all info it needs, and is ready to be used
abstract boolean	needsStagingDirs() Returns true if the replication task in question needs to create staging directories to complete its operation.
static void	resetFactory(Class<? extends ReplicationTask.Factory> factoryClass) Package scoped method used for testing - allows resetting the ReplicationTaskFactory used
protected void	validateEventType(HCatNotificationEvent event, String allowedEventType)
protected void	verifyActionable()
ReplicationTask	withDbNameMapping(com.google.common.base.Function<String,String> dbNameMapping) Allows a user to specify a db name mapping, where the the function provided maps the name of the db in the source warehouse to the name of the db in the dest warehouse.
ReplicationTask	withDstStagingDirProvider(StagingDirectoryProvider dstStagingDirProvider) See StagingDirectoryProvider
ReplicationTask	withSrcStagingDirProvider(StagingDirectoryProvider srcStagingDirProvider) See StagingDirectoryProvider
ReplicationTask	withTableNameMapping(com.google.common.base.Function<String,String> tableNameMapping) Allows a user to specify a table name mapping, where the the function provided maps the name of the table in the source warehouse to the name of the table in the dest warehouse.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

event

protected HCatNotificationEvent event

srcStagingDirProvider

protected StagingDirectoryProvider srcStagingDirProvider

dstStagingDirProvider

protected StagingDirectoryProvider dstStagingDirProvider

tableNameMapping

protected com.google.common.base.Function<String,String> tableNameMapping

dbNameMapping

protected com.google.common.base.Function<String,String> dbNameMapping

messageFactory

protected static MessageFactory messageFactory

Constructor Detail

ReplicationTask

protected ReplicationTask(HCatNotificationEvent event)

Method Detail

resetFactory

public static void resetFactory(Class<? extends ReplicationTask.Factory> factoryClass)

Package scoped method used for testing - allows resetting the ReplicationTaskFactory used

Parameters:

factoryClass - The new ReplicationTaskFactory to use.

create

public static ReplicationTask create(HCatClient client,
                     HCatNotificationEvent event)

Factory method to return appropriate subtype of ReplicationTask for given event

Parameters:

event - HCatEventMessage returned by the notification subsystem

Returns:

corresponding ReplicationTask

getEvent

public HCatNotificationEvent getEvent()

Returns the event that this ReplicationTask is attempting to replicate

Returns:

underlying event

needsStagingDirs

public abstract boolean needsStagingDirs()

Returns true if the replication task in question needs to create staging directories to complete its operation. This will mean that you will need to copy these directories over to the destination warehouse for each source-destination warehouse pair. If this is true, you will need to call .withSrcStagingDirProvider(...) and .withDstStagingDirProvider(...) before this ReplicationTask is usable

isActionable

public boolean isActionable()

Returns true if this ReplicationTask is prepared with all info it needs, and is ready to be used

withSrcStagingDirProvider

public ReplicationTask withSrcStagingDirProvider(StagingDirectoryProvider srcStagingDirProvider)

See StagingDirectoryProvider

Parameters:

srcStagingDirProvider - Staging Directory Provider for the source warehouse

Returns:

this

withDstStagingDirProvider

public ReplicationTask withDstStagingDirProvider(StagingDirectoryProvider dstStagingDirProvider)

See StagingDirectoryProvider

Parameters:

dstStagingDirProvider - Staging Directory Provider for the destination warehouse

Returns:

this replication task

withTableNameMapping

public ReplicationTask withTableNameMapping(com.google.common.base.Function<String,String> tableNameMapping)

Allows a user to specify a table name mapping, where the the function provided maps the name of the table in the source warehouse to the name of the table in the dest warehouse. It is expected that if the mapping does not exist, it should return the same name sent in. Or, if the function throws an IllegalArgumentException as well, a ReplicationTask will use the same key sent in. That way, the default will then be that the destination db name is the same as the src db name If you want to use a Map mapping instead of a Function, simply call this function as .withTableNameMapping(ReplicationUtils.mapBasedFunction(tableMap))

Parameters:

tableNameMapping -

Returns:

this replication task

withDbNameMapping

public ReplicationTask withDbNameMapping(com.google.common.base.Function<String,String> dbNameMapping)

Allows a user to specify a db name mapping, where the the function provided maps the name of the db in the source warehouse to the name of the db in the dest warehouse. It is expected that if the mapping does not exist, it should return the same name sent in. Or, if the function throws an IllegalArgumentException as well, a ReplicationTask will use the same key sent in. That way, the default will then be that the destination db name is the same as the src db name If you want to use a Map mapping instead of a Function, simply call this function as .withDbNameMapping(ReplicationUtils.mapBasedFunction(dbMap))

Parameters:

dbNameMapping -

Returns:

this replication task

verifyActionable

protected void verifyActionable()

getSrcWhCommands

public abstract Iterable<? extends Command> getSrcWhCommands()

Returns a Iterable to send to a hive driver on the source warehouse If you *need* a List instead, you can use guava's ImmutableList.copyOf(iterable) or Lists.newArrayList(iterable) to get the underlying list, but this defeats the purpose of making this interface an Iterable rather than a List, since it is very likely that the number of Commands returned here will cause your process to run OOM.

getDstWhCommands

public abstract Iterable<? extends Command> getDstWhCommands()

Returns a Iterable to send to a hive driver on the source warehouse If you *need* a List instead, you can use guava's ImmutableList.copyOf(iterable) or Lists.newArrayList(iterable) to get the underlying list, but this defeats the purpose of making this interface an Iterable rather than a List, since it is very likely that the number of Commands returned here will cause your process to run OOM.

validateEventType

protected void validateEventType(HCatNotificationEvent event,
                     String allowedEventType)