Apache Slider Security

This document discusses the design, implementation and use of Slider to deploy secure applications on a secure Apache Hadoop cluster.

Important:

This document does not cover Kerberos, how to secure a Hadoop cluster, Kerberos command line tools or how Hadoop uses delegation tokens to delegate permissions round a cluster. These are prerequisites &em;though some links to useful pages are listed at the bottom.

Concepts

Slider runs in secure clusters, but with restrictions

  1. The keytabs to allow the AM and components to authenticate with the master are either distributed in advance or can be distributed by Slider.
  2. Until the location of Slider node instances can be strictly limited to a set of nodes (a future YARN feature), the keytabs are required by all the nodes in the cluster, and made available to the user creating the cluster
  3. due to the way that HBase and accumulo authenticate worker nodes to the masters, any HBase node running on a server must authenticate as the same principal, and so have equal access rights to the HBase cluster.
  4. As the data directories for a slider cluster are created under the home directories of that user, the principals representing all role instances in the clusters MUST have read/write access to these files. This can be done with a shortname that matches that of the user, or by requesting that Slider create a directory with group write permissions -and using LDAP to identify the application principals as members of the same group as the user.

Security Requirements

Needs

  • Slider and deployed applications to work against secure HDFS
  • Slider to work with secure YARN.
  • Slider to start secure applications
  • Kerberos and ActiveDirectory to perform the authentication.
  • Slider to only allow cluster operations by authenticated users -command line and direct RPC.
  • Any Slider Web UI and REST API for Ambari to only allow access to authenticated users.
  • The Slider database in ~/.slider/clusters/$name/data to be writable by HBase

Short-lived Clusters

  • Cluster to remain secure for the duration of the Kerberos tokens issued to Slider.

Long-lived Clusters

  • Slider application instance and HBase instance to remain functional and secure over an indefinite period of time.

Initial Non-requirements

  • Secure audit trail of cluster operations.
  • Multiple authorized users being granted rights to a Slider Cluster (YARN admins can always kill the Slider application instance.
  • Any way to revoke certificates/rights of running containers.

Assumptions

  • Kerberos is running and that HDFS and YARN are running Kerberized.
  • LDAP cannot be assumed.
  • Credentials needed for the application can be pushed out into the local filesystems of the of the worker nodes via some external mechanism (e.g. scp), and protected by the access permissions of the native filesystem. Any user with access to these credentials is considered to have been granted such rights.
  • These credentials can outlive the duration of the application instances
  • The user running the application has the same identity as that of the application.
  • All application instances run by a single user can share the same machine-specific kerberos identities.

Design

  1. The user is expected to have their own Kerberos principal, and have used kinit or equivalent to authenticate with Kerberos and gain a (time-bounded) TGT
  2. The user is expected to have principals for every host in the cluster of the form username/hostname@REALM for component authentication. The AM authentication requirements can be satisfied with a non-host based principal (username@REALM).
  3. Separate keytabs should be generated for the AM, which contains the AM login principal, and the service components, which contain all the service principals. The keytabs can be manually distributed to all the nodes in the cluster with read access permissions to the user, or the user may elect to leverage the Slider keytab distribution mechanism.
  4. When the user creates a secure cluster, they provide the standard HBase kerberos options to identify the principals to use and the keytab location.

The Slider Client will talk to HDFS and YARN authenticating itself with the TGT, talking to the YARN and HDFS principals which it has been configured to expect.

This can be done as described in Client Configuration on the command line as

 -D yarn.resourcemanager.principal=yarn/master@LOCAL 
 -D dfs.namenode.kerberos.principal=hdfs/master@LOCAL

The Slider Client will create the cluster data directory in HDFS with rwx permissions for
user r-x for the group and --- for others. (these can be configurable as part of the cluster options),

It will then deploy the AM, which will (somehow? for how long?) retain the access rights of the user that created the cluster.

The Application Master will read in the JSON cluster specification file, and instantiate the relevant number of components.

The Keytab distribution/access Options

Rather than relying on delegation token based authentication mechanisms, the AM leverages keytab files for obtaining the principals to authenticate to the configured cluster KDC. In order to perform this login the AM requires access to a keytab file that contains the principal representing the user identity to be associated with the launched application instance (e.g. in an HBase installation you may elect to use the hbase principal for this purpose). There are two mechanisms supported for keytab access and/or distribution:

Local Keytab file access:

An application deployer may choose to pre-distribute the keytab files required to the Node Manager (NM) hosts in a Yarn cluster. In that instance the appConfig.json requires the following properties:

. . .
"components": {
    "slider-appmaster": {
        "jvm.heapsize": "256M",
        "slider.am.keytab.local.path": "/etc/security/keytabs/hbase.headless.keytab",
        "slider.keytab.principal.name" : "hbase"
    }
}

The slider.am.keytab.local.path property provides the full path to the keytab file location and is mandatory for the local lookup mechanism. The principal to leverage from the file is identified by the slider.keytab.principal.name property.

In this scenario the distribution of keytab files for the AM AND the application itself is the purview of the application deployer. So, for example, for an hbase deployment, the hbase site service keytab will have to be distributed as well and indicated in the hbase-site properties:

    . . .
    "site.hbase-site.hbase.master.kerberos.principal": "hbase/_HOST@EXAMPLE.COM",
    "site.hbase-site.hbase.master.keytab.file": "/etc/security/keytabs/hbase.service.keytab",
    . . .

Slider keytab distribution:

The deployer can select to upload the keytab files (manually or using the Slider client install-keytab option - see below) for the AM and the application to an HDFS directory (with appropriate permissions set) and slider will localize the keytab files to locations accessible by the AM or the application containers:

. . .
"components": {
    "slider-appmaster": {
        "jvm.heapsize": "256M",
        "slider.hdfs.keytab.dir": ".slider/keytabs/hbase",
        "slider.am.login.keytab.name": "hbase.headless.keytab",
        "slider.keytab.principal.name" : "hbase"
    }
}

The slider.hdfs.keytab.dir points to an HDFS path, relative to the users home directory (e.g./users/hbase`), in which slider can find all keytab files required for both AM login as well as application services. For example, for Apache HBase the uses would be the headless keytab for the AM and the service keytab for the HBase application components).

If no value is specified, a default location of .slider/keytabs/<cluster name> is assumed.

The slider.am.login.keytab.name is the name of the keytab file (mandatory property), found within the specified directory, that the AM will use to lookup up the login principal and authenticate.

When using the slider-based distribution mechanism, the keytab files for components will be accessible from a keytabs sub-directory of the container work folder and can therefore be specified relative to the $AGENT_WORK_ROOT/keytabs directory, e.g.:

    . . .
    "site.hbase-site.hbase.master.kerberos.principal": "hbase/_HOST@EXAMPLE.COM",
    "site.hbase-site.hbase.master.keytab.file": "${AGENT_WORK_ROOT}/keytabs/hbase.service.keytab",
    . . .

For both mechanisms above, the principal name used for authentication is either:

  • The principal name established on the client side before invocation of the Slider CLI (the principal used to kinit) or
  • The value specified for a slider.keytab.principal.name property.

Slider Client Keytab installation:

The Slider client can be used to install keytab files individually into a designated keytab HDFS folder. The format of the command is:

slider install-keytab —keytab <path to keytab on local file system> —folder <name of HDFS folder to store keytab> [—overwrite]

The command will store the keytab file specified by the —keytab option in to an HDFS folder that is created or exists under /user/username/.slider/keytabs named by the —folder option (e.g. if the folder name specified is HBASE the keytab will be stored in /user/username/.slider/keytabs/HBASE). The command can be used to upload keytab files individually up to HDFS. For example, if uploading both AM and HBase service keytabs to the HBASE folder, the command will be invoked twice:

slider install-keytab —keytab /my/local/keytabs/folder/hbase.headless.keytab —folder HBASE
slider install-keytab —keytab /my/local/keytabs/folder/hbase.service.keytab —folder HBASE

Subsequently, the associated hbase-site configuration properties would be:

"global": {
    . . .
     "site.hbase-site.hbase.master.kerberos.principal": "hbase/_HOST@EXAMPLE.COM",
     "site.hbase-site.hbase.master.keytab.file": "${AGENT_WORK_ROOT}/keytabs/hbase.service.keytab",
     . . .
  }
"components": {
     "slider-appmaster": {
         "jvm.heapsize": "256M",
         "slider.hdfs.keytab.dir": ".slider/keytabs/HBASE",
         "slider.am.login.keytab.name": "hbase.headless.keytab"
         `slider.keytab.principal.name` : `hbase"
     }
}

Securing communications between the Slider Client and the Slider AM.

When the AM is deployed in a secure cluster, it automatically uses Kerberos-authorized RPC channels. The client must acquire a token to talk the AM.

This is provided by the YARN Resource Manager when the client application wishes to talk with the SliderAM -a token which is only provided after the caller authenticates itself as the user that has access rights to the cluster

To allow the client to stop a Slider application instance while they are unable to acquire a token to authenticate with the AM, use the --force option.

How to enable a secure Slider client

Slider can be placed into secure mode by setting the Hadoop security options:

This can be done in slider-client.xml:

<property>
  <name>hadoop.security.authorization</name>
  <value>true</value>
</property>

<property>
  <name>hadoop.security.authentication</name>
  <value>kerberos</value>
</property>

Or it can be done on the command line

-D hadoop.security.authorization=true -D hadoop.security.authentication=kerberos

Adding Kerberos binding properties to the Slider Client JVM

The Java Kerberos library needs to know the Kerberos controller and realm to use. This should happen automatically if this is set up as the default Kerberos binding (on a Unix system this is done in /etc/krb5.conf.

If is not set up, a stack trace with kerberos classes at the top and the message java.lang.IllegalArgumentException: Can't get Kerberos realm will be printed -and the client will then fail.

The realm and controller can be defined in the Java system properties java.security.krb5.realm and java.security.krb5.kdc. These can be fixed in the JVM options, as described in the [Client Configuration] (/docs/client-configuration.html) documentation.

They can also be set on the Slider command line itself, using the -S parameter.

-S java.security.krb5.realm=MINICLUSTER  -S java.security.krb5.kdc=hadoop-kdc

Generation and deployment of application keystores/truststores

Application components may make use of keystores and truststores to establish secure communications. Given the nature of application deployments in a YARN cluster and the lack of certainty concerning the host on which a component container may be allocated, Slider provides the facility for creating and deploying the keystores and truststores that may be required.

The process of enabling application keystore/truststore generation and deployment is:

  • Set the configuration option "slider.component.security.stores.required" to "true". This optional can be set as a global property (indicating all components require stores) or can be set/overridden at the component level to selectively enable store generation for a given component.
  • Specify the password property for the component keystore or truststore or,
  • Specify the property providing the alias that references a credential managed by the Hadoop Credential Provider. This credential provides the password for securing the keystore/truststore.

Specifying a keystore/truststore password

Applications that make use of a keystore and/or truststore may already have configuration properties that reference the value for the password used to secure the given certificate store. In those instances the application configuration can reference the value of the password property in the component specific configuration section:

    "APP_COMPONENT": {
        "slider.component.security.stores.required": "true", 
        "slider.component.keystore.password.property": "site.myapp-site.app_component.keystore.password"
    }

In this example:

  • The store required property is set to "true" for the APP_COMPONENT component
  • The application has a property in its site configuration file named "app_component.keystore.password". This property is specified in the appConfig file's global section (with the "site.myapp-site" prefix), and is referenced here to indicate to Slider which application property provides the store password.

Specifying a keystore/truststore Credential Provider alias

Applications that utilize the Credential Provider API to retrieve application passwords can specify the following configuration:

  • Indicate the credential storage path in the credentials section of the app configuration file:
      "credentials": {
    "jceks://hdfs/user/${USER}/myapp.jceks": ["app_component.keystore.password.alias"]
  }

If you specify a list of aliases and are making use of the Slider CLI for application deployment, you will be prompted to enter a value for the passwords specified if no password matching a configured alias is found in the credential store. However, any mechanism available for pre-populating the credential store may be utilized.

  • Reference the alias to use for securing the keystore/truststore in the component's configuration section:
    "APP_COMPONENT": {
    "slider.component.security.stores.required": "true", 
    "slider.component.keystore.credential.alias.property": "app_component.keystore.password.alias"
}

At runtime, Slider will read the credential mapped to the alias (in this case, "app_component.keystore.password.alias"), and leverage the password stored to secure the generated keystore.

Zookeeper binding

Sometimes problems authenticating with Zookeeper have surfaced, with the error being about an unknown JAAS Config

2015-12-15 13:01:29,247 [main] ERROR zk.RegistrySecurity (validateContext(703)) - Entry "Client" not found; JAAS config = ; java.security.auth.login.config=(undefined) 2015-12-15 13:01:31,066 [main-SendThread(devix:2181)] WARN zookeeper.ClientCnxn (startConnect(957)) - SASL configuration failed: javax.security.auth.login.LoginException: Client cannot SASL-authenticate because the specified JAAS configuration section 'Client' could not be found. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2015-12-15 13:01:31,069 [main-EventThread] ERROR curator.ConnectionState (checkState(244)) - Authentication failed

Important: Java Cryptography Package

When trying to talk to a secure cluster you may see the message:

No valid credentials provided (Mechanism level: Illegal key size)]

or

No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

This means that the JRE does not have the extended cryptography package needed to work with the keys that Kerberos needs. This must be downloaded from Oracle (or other supplier of the JVM) and installed according to the accompanying instructions.

  1. Hadoop and Kerberos: The Madness Beyond the Gate
  2. Adding Security to Apache Hadoop
  3. The Role of Delegation Tokens in Apache Hadoop Security
  4. Chapter 8. Secure Apache HBase
  5. Hadoop Operations p135+
  6. Java Kerberos Requirements
  7. Troubleshooting Kerberos on Java
  8. For OS/X users, the GUI ticket viewer is /System/Library/CoreServices/Ticket\ Viewer.app