Simplified Application Packaging

An application may be a simple command as all of its dependencies are pre-installed. In these cases creating a full application package zip file may be too much. Similarly, applications artifacts can be organized in a folder that the developer is continously editing while she is creating/debugging a package. The need to zip up the package and uploading to FS explicitly can be eliminated. The following simplifications are designed:

  • eliminate the need for a zip package
  • only mandate a metadata file (allow json structure as well)
  • allow reading an application definition from a local folder

Examples

A minimal application specified through metadata

It is assummed that all binary dependencies are installed on the host machines.

Application specification

metainfo.json

{
"schemaVersion": "2.1",
"application": {
    "name": "MEMCACHED",
    "components": [
        {
            "name": "MEMCACHED",
            "commands": [
                {
                    "exec": "/usr/jdk64/jdk1.7.0_67/bin/java -classpath /usr/myapps/memcached/*:/usr/hadoop/lib/* com.thimbleware.jmemcached.Main"
                }
            ]
         }
    ]
}
}

At this time a resources.json is required but there is a plan to eliminate it by introducing default resource requirements through slider-client.xml. A common resources.json file is specified here

create

slider create memcachedmin --metainfo metainfo.json --resources resources.json

A minimal application with exports

It is assummed that all binary dependencies are installed on the host machines. Application asks Slider to allocate a dynamic port and then export it to the registry. You can use the same syntax as specified in specifying exports.

Application specification

metainfo.json

{
"schemaVersion": "2.1",
"application": {
    "name": "MEMCACHED",
    "exportGroups": [
        {
            "name": "Servers",
            "exports": [
                {
                    "name": "host_port",
                    "value": "${MEMCACHED_HOST}:${site.global.port}"
                }
            ]
        }
    ],
    "components": [
        {
            "name": "MEMCACHED",
            "compExports": "Servers-host_port",
            "commands": [
                {
                    "exec": "/usr/jdk64/jdk1.7.0_67/bin/java -classpath /usr/myapps/memcached/*:/usr/hdp/current/hadoop-client/lib/* com.thimbleware.jmemcached.Main --port={$conf:@//site/global/port}"
                }
            ]
         }
    ]
}
}

appConfig.json The intent to get a dynamically allocated port is specified via appConfig.json file.

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "site.global.port": "${MEMCACHED.ALLOCATED_PORT}{PER_CONTAINER}"
  }
}

create

slider create memcachedmin --metainfo metainfo.json --resources resources.json --template appConfig.json

A application with libraries

In this case, the application requires some jars to be uploaded and be made available locally. The application definition, in this case can be specified as a folder and uploaded jars can be referred to from the app_root. At this point, the layout is close to the .zip package except you do not need to explicitly package it up. Slider will package the content and upload it.

Application specification

metainfo.json

{
"schemaVersion": "2.1",
"application": {
    "name": "MEMCACHED",
    "exportGroups": [
        {
            "name": "Servers",
            "exports": [
                {
                    "name": "host_port",
                    "value": "${MEMCACHED_HOST}:${site.global.port}"
                }
            ]
        }
    ],
    "components": [
        {
            "name": "MEMCACHED",
            "compExports": "Servers-host_port",
            "commands": [
                {
                    "exec": "/usr/jdk64/jdk1.7.0_67/bin/java -classpath {$conf:@//site/global/app_root}/*:/usr/hdp/current/hadoop-client/lib/* com.thimbleware.jmemcached.Main --port={$conf:@//site/global/port}"
                }
            ]
         }
    ],
    "packages": [
        {
            "type": "folder",
            "name": "files/jmemcached-1.0.0"
        }
    ]

}
}

appConfig.json The intent to get a dynamically allocated port is specified via appConfig.json file.

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "site.global.port": "${MEMCACHED.ALLOCATED_PORT}{PER_CONTAINER}"
  }
}

folder contents:

./package
./package/files
./package/files/jmemcached-1.0.0
./package/files/jmemcached-1.0.0/jmemcached-cli-1.0.0.jar
./package/files/jmemcached-1.0.0/jmemcached-core-1.0.0.jar
./metainfo.json

create

slider create memcachedjar –appdef /usr/work/package --resources resources.json --template appConfig.json

Common resources.json

Here is the common resources json file for all samples.

resources.json

{
"schema" : "http://example.org/specification/v2.0.0",
"metadata" : {
},
"global" : {
},
"components": {
  "slider-appmaster": {
  },
  "MEMCACHED": {
    "yarn.role.priority": "1",
    "yarn.component.instances": "1",
    "yarn.memory": "256"
  }
}