Maven dependencies

For your application module

<dependency>
    <groupId>org.apache.twill</groupId>
    <artifactId>twill-api</artifactId>
    <version>0.11.0</version>
</dependency>

For the your Apache Twill YARN client module

<dependency>
    <groupId>org.apache.twill</groupId>
    <artifactId>twill-yarn</artifactId>
    <version>0.11.0</version>
</dependency>

Clone and build the Twill library

You can also clone the source and build the latest snapshot.

$ git clone https://git-wip-us.apache.org/repos/asf/twill.git twill
$ cd twill
$ mvn clean install

Quick example

Let’s begin by building a basic EchoServer in Twill. Traditionally, when you build a server as simple as this, you add logic within a Runnable implementation to run it in a Thread using an appropriate ExecutorService:

public class EchoServer implements Runnable {
  private static Logger LOG = LoggerFactory.getLogger(EchoServer.class);
  private final ServerSocket serverSocket;

  public EchoServer() {
    ...
  }

  @Override
  public void run() {
    while ( isRunning() ) {
      Socket socket = serverSocket.accept();
      ...
    }
  }
}

Our example defines an implementation of Runnable that implements the run() method. The EchoServer is now a Runnable that can be executed by an ExecutorService:

ExecutorService service = Executors.newFixedThreadPool(2);
service.submit(new EchoServer());

The above model is familiar, but now assume you want to run your EchoServer on a YARN cluster. To do this, all you need to do is implement the TwillRunnable interface similarly to how you normally implement Runnable:

public class EchoServer implements TwillRunnable {

  private static Logger LOG = LoggerFactory.getLogger(EchoServer.class);
  private final ServerSocket serverSocket;
  private final int port;

  public EchoServer() {
    ...
  }

  @Override
  public void run() {
    while ( isRunning() ) {
      Socket socket = serverSocket.accept();
      ...
    }
  }
}

To run EchoServer on the YARN cluster, you need a TwillRunnerService, which is similar to ExecutorService. You can specify the YARN cluster configuration and connection string to a running instance of a Zookeeper service to create an instance of TwillRunnerService:

TwillRunnerService runnerService = new YarnTwillRunnerService(
  new YarnConfiguration(), zkConnectStr);
runnerService.start();

Now you are read to run the EchoServer on YARN, by simply preparing and starting through the TwillRunnerService. You can also attach log handler to receives logs generated by EchoService running on some nodes in the cluster:

TwillController controller = runnerService.prepare(new EchoServer())
  .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out)))
  .start();

Now that the EchoServer is started and you can inspect or control the application through the TwillController. For example, you can attach listeners for state changes:

controller.addListener(new ListenerAdapter() {
  @Override
  public void running() {
    LOG.info('Echo Server Started');
  }
}

You can also stop the application through the TwillController:

controller.terminate().get();

This will shut down the application master and all the containers.

Hello World

To see Twill in action, you can run the “hello world” example applications located in the twill-examples module.

Prerequisites

  • Single Node or Cluster installation of Hadoop with YARN (Hadoop >= 2.2.0) set-up and running.
  • Single Node or Cluster installation of ZooKeeper set-up and running.
  • Build of Twill Library Code (minimum, build of twill-examples module)

Running the Examples

There are two example applications you can run: HelloWorld and BundledJarExample.

HelloWorld Application

The HelloWorld application creates a simple YARN application that prints a line to the log.

You can run the HelloWorld application from any node of the Hadoop cluster using the below command (be sure to add your ZooKeeper Host and Port):

$ export CP=twill-examples-yarn-0.10.0.jar:`hadoop classpath`
$ java -cp $CP org.apache.twill.example.yarn.HelloWorld {zookeeper_host:port}

If successful, you should see logs output to the terminal with details of the running application. Once the application is finished running, check the YARN logs and you should see output like the following:

14:49:45.944 [TwillContainerService] INFO  o.a.twill.example.yarn.HelloWorld - Hello World. My first distributed application.
BundledJarExample Application

The BundledJarExample application demonstrates the Twill functionality that allows you to run any Java application in Twill without worrying about library version conflicts between your application and Hadoop. The example calls the main class in a sample application Echo, which simply logs the command line argument(s) passed to it. The Echo application uses a different version of Guava from Twill and Hadoop distributions. BundledJarExample looks for the dependency in a lib folder packaged at the root of the Echo jar.

You can run the BundleJarExample application from any node of the Hadoop cluster using the below command (be sure to add your ZooKeeper Host and Port):

$ export CP=twill-examples-yarn-0.10.0.jar:`hadoop classpath`
$ java -cp $CP org.apache.twill.example.yarn.BundledJarExample {zookeeper_host:port} \
    twill-examples-echo-0.10.0.jar echo.EchoMain arg1

Like with the HelloWorld example, you should see logs output to the terminal. Once the application is complete, check the YARN logs as before and you should see output like the following:

[TwillContainerService] INFO echo.EchoMain - Hello from EchoMain: 6
err HELLO from scatch
[TwillContainerService] INFO echo.EchoMain - Got args: [arg1]

...

out HELLO from scatch
Got args: [arg1]

Congratulations! You have run your first Twill applications.

Back to top

Reflow Maven skin by Andrius Velykis.

Apache Twill, Apache, the Apache feather logo, and the Apache Twill project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.