org.apache.hadoop.yarn.client.api.impl

Class AMRMClientImpl<T extends AMRMClient.ContainerRequest>

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.client.api.AMRMClient<T>

org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl<T>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class AMRMClientImpl<T extends AMRMClient.ContainerRequest>
extends AMRMClient<T>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.client.api.AMRMClient

AMRMClient.ContainerRequest

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected String	appHostName
protected int	appHostPort
protected String	appTrackingUrl
protected Set<org.apache.hadoop.yarn.api.records.ResourceRequest>	ask
protected Set<String>	blacklistAdditions
protected Set<String>	blacklistedNodes
protected Set<String>	blacklistRemovals
protected Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.Resource>>	change
protected org.apache.hadoop.yarn.api.records.Resource	clusterAvailableResources
protected int	clusterNodeCount
protected Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.Resource>>	pendingChange
protected Set<org.apache.hadoop.yarn.api.records.ContainerId>	pendingRelease
protected Set<org.apache.hadoop.yarn.api.records.ContainerId>	release
protected Map<org.apache.hadoop.yarn.api.records.Priority,Map<String,TreeMap<org.apache.hadoop.yarn.api.records.Resource,org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.ResourceRequestInfo>>>	remoteRequestsTable
protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol	rmClient

Constructor Summary

Constructors

Constructor and Description
AMRMClientImpl()

Method Summary

Methods

Modifier and Type	Method and Description
void	addContainerRequest(T req) Request containers for resources before calling allocate
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse	allocate(float progressIndicator) Request additional containers and receive new container allocations.
org.apache.hadoop.yarn.api.records.Resource	getAvailableResources() Get the currently available resources in the cluster.
int	getClusterNodeCount() Get the current number of nodes in the cluster.
List<? extends Collection<T>>	getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority, String resourceName, org.apache.hadoop.yarn.api.records.Resource capability) Get outstanding ContainerRequests matching the given parameters.
protected void	populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens)
org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse	registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) Register the application master.
void	releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Release containers assigned by the Resource Manager.
void	removeContainerRequest(T req) Remove previous container request.
protected void	removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers)
protected void	removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses)
void	requestContainerResourceChange(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.Resource capability) Request container resource change before calling allocate.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
void	unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl) Unregister the application master.
void	updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals) Update application's blacklist with addition or removal resources.

Methods inherited from class org.apache.hadoop.yarn.client.api.AMRMClient

createAMRMClient, getNMTokenCache, setNMTokenCache, waitFor, waitFor, waitFor

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

appHostName

protected String appHostName

appHostPort

protected int appHostPort

appTrackingUrl

protected String appTrackingUrl

rmClient

protected org.apache.hadoop.yarn.api.ApplicationMasterProtocol rmClient

clusterAvailableResources

protected org.apache.hadoop.yarn.api.records.Resource clusterAvailableResources

clusterNodeCount

protected int clusterNodeCount

blacklistedNodes

protected final Set<String> blacklistedNodes

blacklistAdditions

protected final Set<String> blacklistAdditions

blacklistRemovals

protected final Set<String> blacklistRemovals

remoteRequestsTable

protected final Map<org.apache.hadoop.yarn.api.records.Priority,Map<String,TreeMap<org.apache.hadoop.yarn.api.records.Resource,org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.ResourceRequestInfo>>> remoteRequestsTable

ask

protected final Set<org.apache.hadoop.yarn.api.records.ResourceRequest> ask

release

protected final Set<org.apache.hadoop.yarn.api.records.ContainerId> release

pendingRelease

protected Set<org.apache.hadoop.yarn.api.records.ContainerId> pendingRelease

change

protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.Resource>> change

pendingChange

protected final Map<org.apache.hadoop.yarn.api.records.ContainerId,AbstractMap.SimpleEntry<org.apache.hadoop.yarn.api.records.Container,org.apache.hadoop.yarn.api.records.Resource>> pendingChange

Constructor Detail

AMRMClientImpl

public AMRMClientImpl()

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

registerApplicationMaster

public org.apache.hadoop.yarn.api.protocolrecords.RegisterApplicationMasterResponse registerApplicationMaster(String appHostName,
                                                                                                     int appHostPort,
                                                                                                     String appTrackingUrl)
                                                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                                              IOException

Description copied from class: AMRMClient

Register the application master. This must be called before any other interaction

Specified by:

registerApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

appHostName - Name of the host on which master is running

appHostPort - Port master is listening on

appTrackingUrl - URL at which the master info can be seen

Returns:

RegisterApplicationMasterResponse

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

allocate

public org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse allocate(float progressIndicator)
                                                                     throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                            IOException

Description copied from class: AMRMClient

Request additional containers and receive new container allocations. Requests made via addContainerRequest are sent to the ResourceManager. New containers assigned to the master are retrieved. Status of completed containers and node health updates are also retrieved. This also doubles up as a heartbeat to the ResourceManager and must be made periodically. The call may not always return any new allocations of containers. App should not make concurrent allocate requests. May cause request loss.

Note : If the user has not removed container requests that have already been satisfied, then the re-register may end up sending the entire container requests to the RM (including matched requests). Which would mean the RM could end up giving it a lot of new allocated containers.

Specified by:

allocate in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

progressIndicator - Indicates progress made by the master

Returns:

the response of the allocate request

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

removePendingReleaseRequests

protected void removePendingReleaseRequests(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainersStatuses)

removePendingChangeRequests

protected void removePendingChangeRequests(List<org.apache.hadoop.yarn.api.records.UpdatedContainer> changedContainers)

populateNMTokens

@InterfaceAudience.Private
protected void populateNMTokens(List<org.apache.hadoop.yarn.api.records.NMToken> nmTokens)

unregisterApplicationMaster

public void unregisterApplicationMaster(org.apache.hadoop.yarn.api.records.FinalApplicationStatus appStatus,
                               String appMessage,
                               String appTrackingUrl)
                                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                                        IOException

Description copied from class: AMRMClient

Unregister the application master. This must be called in the end.

Specified by:

unregisterApplicationMaster in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

appStatus - Success/Failure status of the master

appMessage - Diagnostics message on failure

appTrackingUrl - New URL to get master info

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

addContainerRequest

public void addContainerRequest(T req)

Description copied from class: AMRMClient

Request containers for resources before calling allocate

Specified by:

addContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

req - Resource request

removeContainerRequest

public void removeContainerRequest(T req)

Description copied from class: AMRMClient

Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request

Specified by:

removeContainerRequest in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

req - Resource request

requestContainerResourceChange

public void requestContainerResourceChange(org.apache.hadoop.yarn.api.records.Container container,
                                  org.apache.hadoop.yarn.api.records.Resource capability)

Description copied from class: AMRMClient

Request container resource change before calling allocate. Any previous pending resource change request of the same container will be removed. Application that calls this method is expected to maintain the Containers that are returned from previous successful allocations or resource changes. By passing in the existing container and a target resource capability to this method, the application requests the ResourceManager to change the existing resource allocation to the target resource allocation.

Specified by:

requestContainerResourceChange in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

container - The container returned from the last successful resource allocation or resource change

capability - The target resource capability of the container

releaseAssignedContainer

public void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from class: AMRMClient

Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources

Specified by:

releaseAssignedContainer in class AMRMClient<T extends AMRMClient.ContainerRequest>

getAvailableResources

public org.apache.hadoop.yarn.api.records.Resource getAvailableResources()

Description copied from class: AMRMClient

Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made

Specified by:

getAvailableResources in class AMRMClient<T extends AMRMClient.ContainerRequest>

Returns:

Currently available resources

getClusterNodeCount

public int getClusterNodeCount()

Description copied from class: AMRMClient

Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made

Specified by:

getClusterNodeCount in class AMRMClient<T extends AMRMClient.ContainerRequest>

Returns:

Current number of nodes in the cluster

getMatchingRequests

public List<? extends Collection<T>> getMatchingRequests(org.apache.hadoop.yarn.api.records.Priority priority,
                                                String resourceName,
                                                org.apache.hadoop.yarn.api.records.Resource capability)

Description copied from class: AMRMClient

Get outstanding ContainerRequests matching the given parameters. These ContainerRequests should have been added via addContainerRequest earlier in the lifecycle. For performance, the AMRMClient may return its internal collection directly without creating a copy. Users should not perform mutable operations on the return value. Each collection in the list contains requests with identical Resource size that fit in the given capability. In a collection, requests will be returned in the same order as they were added.

Specified by:

getMatchingRequests in class AMRMClient<T extends AMRMClient.ContainerRequest>

Returns:

Collection of request matching the parameters

updateBlacklist

public void updateBlacklist(List<String> blacklistAdditions,
                   List<String> blacklistRemovals)

Description copied from class: AMRMClient

Update application's blacklist with addition or removal resources.

Specified by:

updateBlacklist in class AMRMClient<T extends AMRMClient.ContainerRequest>

Parameters:

blacklistAdditions - list of resources which should be added to the application blacklist

blacklistRemovals - list of resources which should be removed from the application blacklist