Apache Slider: Dynamic YARN Applications¶

Apache Slider is a application to deploy existing distributed applications on an Apache Hadoop YARN cluster, monitor them and make them larger or smaller as desired -even while the application is running.

Applications can be stopped then started; the distribution of the deployed application across the YARN cluster is persisted —enabling a best-effort placement close to the previous locations. Applications which remember the previous placement of data (such as HBase) can exhibit fast start-up times from this feature.

YARN itself monitors the health of "YARN containers" hosting parts of the deployed application -it notifies the Slider manager application of container failure. Slider then asks YARN for a new container, into which Slider deploys a replacement for the failed component. As a result, Slider can keep the size of managed applications consistent with the specified configuration, even in the face of failures of servers in the cluster -as well as parts of the application itself

Some of the features are:

• Allows users to create on-demand applications in a YARN cluster

• Allow different users/applications to run different versions of the application.

• Allow users to configure different application instances differently

• Stop / Restart application instances as needed

• Expand / shrink application instances as needed

The Slider tool is a Java command line application.

The tool persists the information as JSON documents in HDFS.

Once the cluster has been started, the cluster can be made to grow or shrink using the Slider commands. The cluster can also be stopped and later restarted.

Get Slider¶

Download Slider now

Using¶

• Getting Started
• Man Page
• Examples
• Application Configuration
• Client Configuration
• Client Exit Codes
• Security
• Placement
• Agent to AM SSL
• How to define a new slider-packaged application
• High Availability
• Troubleshooting
• REST API

Developing¶

Slider is an open source project, we encourage and welcome development.

Start with the Developing section in the documents

Design¶

• Overview
• Application Needs
• Expected Behavior
• Service Registry
• SSL Implementation
• Role History

Disclaimer¶

Apache Slider (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache TLP sponsor. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.