This document describes how to deploy services on YARN using the YARN Service framework.
Below is a simple service definition that launches sleep containers on YARN by writing a simple spec file and without writing any code.
{
"name": "sleeper-service",
"components" :
[
{
"name": "sleeper",
"number_of_containers": 1,
"launch_command": "sleep 900000",
"resource": {
"cpus": 1,
"memory": "256"
}
}
]
}
User can simply run a pre-built example service on YARN using below command:
yarn app -launch <service-name> <example-name>
e.g. Below command launches a sleeper service named as my-sleeper on YARN.
yarn app -launch my-sleeper sleeper
For launching docker based services using YARN Service framework, please refer to API doc.
Below steps walk you through deploying a services on YARN using CLI. Refer to Yarn Commands for the full list of commands and options.
yarn app -launch ${SERVICE_NAME} ${PATH_TO_SERVICE_DEF_FILE}
Params: - SERVICE_NAME: The name of the service. Note that this needs to be unique across running services for the current user. - PATH_TO_SERVICE_DEF: The path to the service definition file in JSON format.
For example:
yarn app -launch sleeper-service /path/to/local/sleeper.json
Increase or decrease the number of containers for a component.
yarn app -flex ${SERVICE_NAME} -component ${COMPONENT_NAME} ${NUMBER_OF_CONTAINERS}
For example, for a service named sleeper-service:
Set the sleeper component to 2 containers (absolute number).
yarn app -flex sleeper-service -component sleeper 2
Relative changes are also supported for the ${NUMBER_OF_CONTAINERS} in the flex command, such as +2 or -2.
Stopping a service will stop all containers of the service and the ApplicationMaster, but does not delete the state of a service, such as the service root folder on hdfs.
yarn app -stop ${SERVICE_NAME}
YARN API Server REST API can be activated as part of the ResourceManager.
For running inside ResourceManager, add this property to yarn-site.xml and restart ResourceManager.
<property>
<description>
Enable services rest api on ResourceManager.
</description>
<name>yarn.webapp.api-service.enable</name>
<value>true</value>
</property>
Services can be deployed on YARN through the ResourceManager web endpoint.
Refer to API doc for the detailed API specificatiosn.
POST the aforementioned example service definition to the ResourceManager api-server endpoint:
POST http://localhost:8088/app/v1/services
PUT http://localhost:8088/app/v1/services/${SERVICE_NAME}/components/${COMPONENT_NAME}
PUT Request body:
{
"name": "${COMPONENT_NAME}",
"number_of_containers": ${COUNT}
}
For example:
{
"name": "sleeper",
"number_of_containers": 2
}
Stopping a service will stop all containers of the service and the ApplicationMaster, but does not delete the state of a service, such as the service root folder on hdfs.
PUT http://localhost:8088/app/v1/services/${SERVICE_NAME}
PUT Request body:
{
"name": "${SERVICE_NAME}",
"state": "STOPPED"
}
A new service tab is added in the YARN UI2 specially to show YARN Services in a first class manner. The services framework posts the data into TimelineService and the service UI reads data from TimelineService to render its content.
Please refer to TimeLineService v2 doc for how to enable Timeline Service v2.
Set below config in yarn-site.xml and start ResourceManager. If you are building from source code, make sure you use -Pyarn-ui in the mvn command - this will generate the war file for the new YARN UI.
<property>
<description>To enable RM web ui2 application.</description>
<name>yarn.webapp.ui2.enable</name>
<value>true</value>
</property>
YARN service framework supports running in a secure(kerberized) environment. User needs to specify the kerberos principal name and keytab when they launch the service. E.g. A typical configuration looks like below:
{
"name": "sample-service",
...
...
"kerberos_principal" : {
"principal_name" : "hdfs-demo@EXAMPLE.COM",
"keytab" : "hdfs:///etc/security/keytabs/hdfs.headless.keytab"
}
}
The above example is only for a non-docker container based service. YARN Service Framework also provides first-class support for managing docker based services. Most of the steps for managing docker based services are the same except that in docker the Artifact type for a component is DOCKER and the Artifact id is the name of the docker image. For details in how to setup docker on YARN, please check Docker on YARN.
With docker support, it also opens up a set of new possibilities to implement features such as discovering service containers on YARN with DNS. Check ServiceDiscovery for more details.