Slider REST APIs v1¶

December 2014¶

This document covers the external REST APIs of Slider

 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and
 "OPTIONAL" in this document are to be interpreted as described in
 RFC 2119.

Introduction and Background¶

Slider 0.60 Uses Hadoop IPC for communications between the Slider client and the per-instance application master. This uses protobuf-formatted payloads, with the Hadoop IPC layer handling security: authorization, authentication and encryption.

It provides read-only views of generated content and state through a set of web service URLs. All these services are implemented by the Jersey JAX-RS engine, operating in an embedded Jetty Web engine, with the YARN AmWebFilter class redirecting all requests not coming from the RM Proxy IP address to that RM Proxy via a 302 (redirect-as-GET)

Excluding the internal-use-only the AM<->Agent channel operating on a private HTTP port, there are a number of current services

Read only access to the YARN registry: /ws/v1/slider/registry¶

This was done primarily to get a browseable view of what the YARN-913 registry was publishing in the absence of any web view built in to that registry. The unit tests against this service actually discovered some JSON marshalling problems with the registry design and Jersey integration.

The root path /v1/slider/registry lists the root path of the registry; all entries below it follow the path in the registry.

/ws/v1/slider/registry: {"nodes":["services","users"]}

Every entry in the path serves an instance of org.apache.slider.server.appmaster.web.rest.registry.PathEntryResource; a list of child directory entries, and, if present, the Yarn registry service record at that path (org.apache.hadoop.registry.client.types.ServiceRecord).

/ws/v1/slider/registry/users/stevel/services/org-apache-slider/apps-through-agent :

{
"nodes": ["components"],
"service": {
  "type": "JSONServiceRecord",
  "description": "Slider Application Master",
  "external": [
    {
      "api": "http://",
      "addressType": "uri",
      "protocolType": "webui",
      "addresses": [{"uri": "http://devix.cotham.uk:37283"}]
    },
    {
      "api": "classpath:org.apache.slider.management",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "http://devix.cotham.uk:37283/ws/v1/slider/mgmt"}]
    },
    {
      "api": "classpath:org.apache.slider.publisher",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "http://devix.cotham.uk:37283/ws/v1/slider/publisher"}]
    },
    {
      "api": "classpath:org.apache.slider.registry",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "http://devix.cotham.uk:37283/ws/v1/slider/registry"}]
    },
    {
      "api": "classpath:org.apache.slider.publisher.configurations",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "http://devix.cotham.uk:37283/ws/v1/slider/publisher/slider"}]
    },
    {
      "api": "classpath:org.apache.slider.publisher.exports",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "http://devix.cotham.uk:37283/ws/v1/slider/publisher/exports"}]
    }
  ],
  "internal": [
    {
      "api": "classpath:org.apache.slider.agents.secure",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "https://devix.cotham.uk:44586/ws/v1/slider/agents"}]
    },
    {
      "api": "classpath:org.apache.slider.agents.oneway",
      "addressType": "uri",
      "protocolType": "REST",
      "addresses": [{"uri": "https://devix.cotham.uk:36590/ws/v1/slider/agents"}]
    }
  ],
  "yarn:persistence": "application",
  "yarn:id": "application_1418130391694_0002"
}
}

The slider code makes no direct use of this registry service except for a pair of tests. The Agents MUST use the core (ZK-based) registry service to locate a restarted AM. The slider client operations to enumerate and retrieve configuration files could use it, but they currently also use the YARN registry directly.

As and when the YARN registry itself implements a REST view (possibly an R/W one), then this view will become obsolete —and possibly disabled.

Generated configurations: /ws/v1/slider/publisher¶

This service contains a one or more "published configuration sets"

http://devix.cotham.uk:8088/proxy/application_1418130391694_0002/ws/v1/slider/publisher

{
  "uris":  {
    "slider":"http://devix.cotham.uk:37283/ws/v1/slider/publisher/slider",
    "exports":"http://devix.cotham.uk:37283/ws/v1/slider/publisher/exports"
    }
}

each one contains a list of available configuration documents , such as under /ws/v1/slider/publisher/slider:

{
    "configurations":{
        "core-site":{
          "description":"Core site settings",
          "updated":0,
          "entries":{},"
          empty":true
          }
    }
}

The final path is built from the published configuration and the desired file format,the latter as a suffix. Thus the following links all server the same content, merely in different formats

/ws/v1/slider/publisher/slider/core-site.xml
/ws/v1/slider/publisher/slider/core-site.properties
/ws/v1/slider/publisher/slider/core-site.json

The path for these becomes:

/ws/v1/slider/publisher/${set-name}/{configuration}.{format}

The value of individual properties can be retrieved as JSON entries under the path

/ws/v1/slider/publisher/${set-name}/{configuration}/{key}

Thus the response to a GET of /ws/v1/slider/publisher/slider/core-site/fs.defaultFS

is

{"fs.defaultFS":"hdfs://devix.cotham.uk:8020"}

Slider uses this service to publish its configuration (core-site.xml), etc, for diagnostics —and the configurations of the deployed applications on the command line

slider registry --getconf hbase-site.xml --name hbase1 --destdir out

This configuration publishing service is listed in the YARN Registry, and could be implemented by other applications to publish their configurations —including the standard YARN services.

Application state in slider v1 format¶

The original "v1" formatted slider specification/state is provided, wrapped in HTML, at:

http://devix.cotham.uk:8088/proxy/application_1418130391694_0002/slideram/spec

This combines the declaration of what is desired with information about what has actually been provided

(this shows the merits of a single file model which can describe both the desired state and the actual state; the same architecture of the HPL Cells cloud infrastructure)

Slider 0.70: codahale metrics servlets¶

Slider 0.70 is adding metrics via the Codahale metrics library; as well as (optionally) publishing to Ganglia, the metrics service export metrics,health and liveness information, as well as a thread dump

/system/metrics :

{
  "version": "3.0.0",
  "gauges": {},
  "counters": {},
  "histograms": {},
  "meters": {},
  "timers": {}
}

/system/health :

  {"AM Health": {"healthy": true}}

As metrics are added, this content will become more useful