Apache Slider Release Process

This is the current release process. It uses the Apache Ant build file bin/release.xml, and attempts to automate nearly all the process, including

  1. Building the source and JAR packaging.
  2. Validating the source packaging.
  3. Uploading the JAR artifacts to a Nexus staging repository
  4. Copying the RC source artifacts to a local subversion repository -then committing
  5. Generating template voting, result and announcement messages.
  6. Downloading the RC artifacts from the apache distribution repository.

It also adds some extra checks to verify past problems have not re-occurred. In particular, the release process will fail if the source tree has any uncommitted files, or if the commit checksum of the HEAD commit is not found on github.

The core release process, then, is one of preparing the source tree and a configuration properties file to manage the release, issuing an ant release command, then, after its completion, following a sequence of operations, some manual.

Release Workflow

  1. Prepare the build.
  2. Build and publish a Release Candidate (RC).
  3. Get RC voted in as a release in the slider-dev list. If the vote fails: fix the issues then repeat.
  4. Report the successful vote to the incubator list and ask for a vote there.
  5. If successful: publish the RC as the release artifacts.

This workflow is designed to be as automated as possible. In order to do so, all aspects of the automation which require configuration must be configured by a set of ant properties. These should all be defined in the file ../release.properties —that is, a file one directory above the SCM managed source tree(s). Placing the file there guarantees that it will not be unintentionally included in the release, and allows it to be shared across local copies of the source tree, specifically any live local repository in which you have been developing, and a read-only release repository.

When this documentation says "set the release property X to Y", it means "add an entry X=Y to the file ../release.properties

Preparing the build

Preparing the build means getting the tools ready, checking out the relevant repositories, making sure the tests are working, and other actions which can be done before even starting the core release process.

Tooling

As well as everything needed to build slider, there are some extra requirements for releasing:

  1. Shell: bash
  2. svn and git on the command line, with write access to the ASF repositories.
  3. Ant 1.8+. The release process is too complex for Maven.
  4. OS/X and windows: Atlassian SourceTree.

Create/Start a JIRA issue for the release

Create a JIRA for the release, estimate 12h

This seems pessimistic, but there may be surprises. Sometimes it has taken days.

Choose your versions numbers

Choose the version number for the release and its successor for the development branch. The development branch must have a -SNAPSHOT suffix and be greater than the release; the release must not have a -SNAPSHOT suffix.

Example:

  • Release: 0.61.0-incubating
  • Development: 0.62.0-incubating-SNAPSHOT

Make sure the tests are working

Make sure that you are building with the JDK of the minimum supported version for Slider. Code compiled with later major versions of Java don't work on earlier versions.

Check the Jenkins builds to verify that the build is stable.

Check out the latest version of the branch to be released, run the tests. You should release a checked out version of the code that is not the one you are developing on (ideally, a clean VM), to ensure that you aren't releasing a slightly modified version of your own, and that you haven't accidentally included passwords or other test run details into the build resource tree.

The slider-funtest functional test package is used to run functional tests against a running Apache Hadoop YARN cluster. It needs to be configured according to the instructions in testing to create Apache HBase and Apache Accumulo clusters in the YARN cluster.

Make sure that the integration tests are passing (and not being skipped) before starting to make a release

mvn clean test integration-test -Dslider.conf.dir=${your-config-dir}

A good test matrix is:

  • Insecure, Secure
  • Linux, Windows
  • Java 7, Java 8

A pair of VMs are sufficient to cover this.

It is wise to restart all YARN services in the VMs before running the tests, for better independence of test runs.

Prepare Signing keys

Verify your GPG key is provided to Apache.

Apache verifies that distributions are correctly signed. Login to https://id.apache.org and verify the fingerprint of GPG key used to sign above is provided. (gpg --fingerprint)

If you have no GPG key, add your fingerprint to the relevant field.

If you are creating a new key, remember what password/passphrase you have given it. Then generate a revocation certificate and store it somewhere, so that you can revoke the published key.

A new key will not be trusted by anyone else. You should introduce to the Apache web of trust by getting other developers to sign it. This can be done in person, or perhaps over a video conference in which key details can be confirmed.

Setup Maven Publishing settings

You need to provide credentials to the ASF maven repositories for publishing/releasing artifacts, using your ASF username and password.

The ASF docs say use the maven encryption feature. Our policy is: encrypt your HDD in its entirety —you should be doing that anyway.

In ~/.m2/settings.xml:

<settings>
  <servers>
    <!-- To publish a snapshot-->
    <server>
      <id>apache.snapshots.https</id>
      <username>YOUR_ASF_USERNAME</username>
      <password>YOUR_SECRET_PASSWORD</password>
    <server>
      <id>apache.snapshots.https</id>
      <username>YOUR_ASF_USERNAME</username>
      <password>YOUR_SECRET_PASSWORD</password>
    </server>
    <!-- To stage a release -->
    <server>
      <id>apache.staging.https</id>
      <username>YOUR_ASF_USERNAME</username>
      <password>YOUR_SECRET_PASSWORD</password>
    </server>
    <!-- To finalise the release-->
    <server>
      <id>apache.releases.https</id>
      <username>YOUR_ASF_USERNAME</username>
      <password>YOUR_SECRET_PASSWORD</password>
    </server>
  </servers>
</settings>

Create the release.properties file

Create a file ../release.properties -that is, one directory above the source tree you are using. This file will be shared across two cloned copies of the apache git repository.

Check out the Apache SVN-based distribution project

RCs are copied to the SVN distribution repository and published for distribution. Once an RC is voted in, it can be renamed.

To begin, you need to check out the SVN repository somewhere

https://dist.apache.org/repos/dist/dev/incubator/slider

Set the release property svn.publish.dir to this value

svn.publish.dir=/Users/stevel/Java/Apache/slider-dist

Make sure your PGP signature is in the distribution directory

Look for your public key in the KEYS file in the slider-dist directory. If it is not there, append it it to the file

Make sure there are no open staging repositories

This causes Nexus-related confusion., one in which it's not obvious where newly-staged artifacts have been published.

Log in to: https://repository.apache.org/index.html#stagingRepositories and close+drop all staging slider repositories.

Wait until there are none shown in the UI before proceeding.

Configure the build in release.properties

Create a file ../release.properties —that is, one directory above the source tree you are using. (That's to keep the source tree uncontaminated)

slider.release.branch=branches/branch-0.9
slider.release.version=0.90.2-incubating
slider.develop.version=0.91.0-incubating-SNAPSHOT
release.jira=SLIDER-1014
release.rc.suffix=-RC0
svn.publish.dir=~/svn/slider-dist

Create the release in JIRA

Tell JIRA there's a new release. This moves it from "unreleased" to "released", creating a URL listing all issues fixed.

This URL is be used in the vote emails to highlight changes.

Important: To perform this task, you must be registered as an admin for the Slider project on JIRA. If you are not: ask someone who is to do it for you.

  1. Go to the Slider Versions page
  2. Locate the pending release. Make sure it is in the correct order of releases: all previous releases MUST be below it; future releases above it. The immediate next release MUST be directly above the pending release.
  3. On the pending release i. Set the release date to the current day. i. Select the drop down "gear-cog" menu to the right of the page, then the "Release" option. i. Carrying all forward issues that haven't been completed *except the JIRA covering the release itself.
  4. Go to the Slider versions page
  5. Locate the release just issued and click through it.
  6. Get from the page link, work out the "version" of the release artifact

Set the release property release.jira.version to the version field

release.jira.version=12334370

It may seem premature to mark the release as complete, but our process closes it early so that the changes page is generated. It also forces all outstanding issues into the next version —which is where they will have to be fixed.

Build the release notes

Create a markdown release note which will be used for summarizing changes.

Include

  1. A quick summary
  2. The Hadoop version against which this release was built
  3. The URL to the JIRA release

The multi-line release notes should go into the slider SVN repo under site/trunk/content/release_notes.

(this is yet to be automated —it should be possible)

Building a Release Candidate

Review your release settings

The init target loads in the properties, builds up some new values and displays them.

ant -f bin/release.xml init

Verify the license of dependent artifacts

Maven can check that all dependent artifacts have valid licenses

For modules which publish JARs (slider-core and slider-funtest), run the third party check for JARs. If you run this at the top level project it will do it for the relevant sub projects.

ant -f bin/release.xml license-check

This will build everything then generate license files under each module in target/generated-sources/license/THIRD-PARTY.txt. Inspect it and make sure there are no GPL or LGPL dependencies which do not also have a license option which is permitted by the ASF.

Dependencies without a license are warned about. To fix this, make sure that the (generated or updated) file src/license/THIRD-PARTY.properties has an entry for every dependency without a license, declaring what their license is.

Update the maven artifact version to that of the new release

ant -f bin/release.xml set-to-release-version
ant -f bin/release.xml git-commit git-push -Dgit.commit.text="updating to release version"

Create the release branch

Create the release branch and push it to the apache repository:

ant -f bin/release.xml git-create-release-branch

This branch is where all the release work will take place. That can include patching and cherry-picking from the development branch.

Return to the develop branch and increment its version

ant -f bin/release.xml git-switch-develop-branch set-to-new-develop-version
ant -f bin/release.xml git-commit -Dgit.commit.text="updating to new develop version"
ant -f bin/release.xml git-push

At this point the slider version in develop/ is that for ongoing development; that in the new release branch set to the release version. The git history is such that if you merge all changes in the release branch back into develop/, the slider artifact version in the POM files will remain unchanged —the switch to the release version is a commit shared across both branches.

Check out a clean source tree for the build

Adjacent to your normal development directory, check out a new copy of slider

git clone https://git-wip-us.apache.org/repos/asf/incubator-slider.git slider-release

This ensures that there are no .git-ignored files in the source tree, files which may end up in the source distribution.

This source tree should be viewed as read-only: all changes to the branch must be done in your development directory, committed and pushed to the ASF repository, then retrieved by updating the branch in the release directory.

Assuming the two source trees are adjacent, the existing ../release.properties file is now loaded by the new version.

In slider-release, Check out the specific branch for the release.

Example

cd slider-release
git checkout -t origin/branches/branch-0.9

For the test of this document, all instructions are to be executed in the this slider-release directory, in the release branch.

Build and Release to Nexus and SVN

ant -f bin/release.xml clean release

Follow the instructions printed for the manual steps

To see them again:

ant -f bin/release.xml print-nexus-instructions

The Nexus UI gives the name of the repository, such as orgapacheslider-1010

In release.properties set nexus.staging.repository to this value:

nexus.staging.repository=orgapacheslider-1010

Begin the slider-dev vote

ant -f bin/release.xml generate-vote-text

This saves the vote file to target/vote.txt

The git checksum in this message is automatically calculated (the git-version) target. You must not make a release from a repository which has modified files, as the source archive and the checksummed version will differ.

Load this and use it as the text for an email to send to the vote. The address and subject line are included in the text.

Don't forget to you vote yourself!

Tip: Validating the artifacts

There's a small project slider-dependency-check which verifies that slider artifacts can be downloaded from the public or staging repositories. Clone this project and read its instructions to verify that everything went up to the staging repo.

Await the slider-dev vote

Await the declared time for the slider vote to complete

Add all the results as release properties

vote.result.positive.binding=0
vote.result.zero.binding=0
vote.result.negative.binding=0
vote.result.negative=0
vote.result.positive=0
vote.result.zero=0
vote.result.comment=(any other commentary, half votes, etc)

If there aren't any non-binding votes, you can skip listing the totals; the binding values will propagate over. Once non-binding results come in, you need to increment both of the fields with every binding vote.

The vote.result.comment property is for any optional text to be included in the results.

Generate the result message

Generate the email to send to the developer list

ant -f bin/release.xml generate-result-message

Load the text for this message from the generated file (target/vote-results.txt) and send it out.

If the vote was a failure, the text will indicate this and the build will will fail.

Follow the instructions in How to clean up after a failed/cancelled vote to clean up.

You now have to fix the build. Do this in your R/W source tree, not slider-release, and propagate the changes to the release source tree via git commit, pushes and pulls.

Then repeat the entire release process from Build and Release to Nexus and SVN.

Call for an incubator vote

If the slider-dev vote passes, call for (the binding) vote on the incubator general list. This vote tends to be a review of artifacts and licensing, rather than at the build and test level.

The incubator vote email must include references to the VOTE and RESULT threads from the developer mailing list.

  1. Go to the Slider mail archives.
  2. Locate the email thread of the vote, and get the permanent link to the email at the start of the vote. Save this in the property mail.vote.thread
  3. Repeat for the email of the results, setting the release property mail.vote.result.thread

Example:

mail.vote.thread=http://mail-archives.apache.org/mod_mbox/incubator-slider-dev/201512.mbox/%3C7EBD312A-E115-4A99-AAE7-1BF553E3B28D%40hortonworks.com%3E
mail.vote.result.thread=http://mail-archives.apache.org/mod_mbox/incubator-slider-dev/201601.mbox/%3CC28233B7-9016-44F4-9A6B-1EBC8BFCDF08%40hortonworks.com%3E

Build the incubator vote into the text file, target/vote-incubator.txt

ant -f bin/release.xml generate-incubator-vote

Use this file as the basis of an email to the incubator general list.

Process the results of the incubator vote

Await 72h for the results of the incubator vote —you can of course finish early if a binding Incubator PMC (IPMC) member votes against it.

Record the vote into vote.incubator.result. release properties:

vote.incubator.result.positive.binding=0
vote.incubator.result.zero.binding=0
vote.incubator.result.negative.binding=0
vote.incubator.result.positive=0
vote.incubator.result.zero=0
vote.incubator.result.negative=0
vote.incubator.result.comment=(any other commentary, half votes, etc)

If there aren't any non-binding votes, you can skip listing the totals; the binding values will propagate over. Once non-binding results come in, you need to increment both of the fields with every binding vote.

The vote.incubator.result.comment property is for any optional text to be included in the results.

Then: create the release message

ant -f bin/release.xml generate-incubator-result-message

This build target will fail if there were any binding -1 votes or not enough positive votes.

If the vote failed,

  1. Follow the instructions in How to clean up after a failed/cancelled vote to clean up.
  2. address the issues.
  3. repeat the entire release process from Build and Release to Nexus and SVN.

Commit the final svn release

Assuming the vote succeeded, you can now commit the artifacts

ant -f bin/release.xml svn-release-final-artifacts

Release the staged Nexus artifacts

Now release the nexus-staged artifacts to the central repository.

  1. Log in to the ASF Nexus server
  2. Locate and select the staged Slider repository
  3. In the toolbar above, press the Release button

Releasing "can take time", as the web site will inform you.

Await the artifacts to propagate

The released artifacts may take up to 24 hours to replicate to the mirrors.

The probe for these artifacts is:

ant -f bin/release.xml check-release-urls

Only once it has succeeded can the announcement be made

Announce the release

The released artifacts may take up to 24 hours to replicate to the mirrors.

Generate the release message:

ant -f bin/release.xml generate-announcement-message

Then follow the instructions on where to email it.

Update the Slider Web Site

Update the Slider website as needed. Most of the documents should have been updated by now to reflect the released version. The remaining changes should really be modifying the references to the latest release to the new version.

The download details for the latest release needs to be updated after mirrors are updated in .

In particular, review content/downloads/index.md

There's also a reference to the latest version in templates/skeleton.html

After any changes have been committed to SVN, wait 60-90s, then trigger a site rebuild.

Finish the JIRA

Log the time, close the issue.

Example release.properties file

Here is the final release.properties file from the 0.90.2 release

slider.release.version=0.90.2-incubating
slider.develop.version=0.91.0-incubating-SNAPSHOT
release.jira=SLIDER-1014
release.jira.version=12334370
release.rc.suffix=-RC1
svn.publish.dir=/Users/stevel/Projects/Apache/slider-dist
git.release.branch=branches/branch-0.90
git.version=9bb379f83c78b61aeb4110b93796bc02c08c4226
nexus.staging.repository=orgapacheslider-1013

vote.result.positive.binding=3
vote.result.zero.binding=0
vote.result.negative.binding=0
vote.result.comment=

mail.vote.thread=http://mail-archives.apache.org/mod_mbox/incubator-slider-dev/201512.mbox/%3C7EBD312A-E115-4A99-AAE7-1BF553E3B28D%40hortonworks.com%3E
mail.vote.result.thread=http://mail-archives.apache.org/mod_mbox/incubator-slider-dev/201601.mbox/%3CC28233B7-9016-44F4-9A6B-1EBC8BFCDF08%40hortonworks.com%3E

vote.incubator.result.positive.binding=4
vote.incubator.result.zero.binding=0
vote.incubator.result.negative.binding=0
vote.incubator.result.comment=(and a +0.5 vote from Daniel Gruno)

How to clean up after a failed/cancelled vote

If a vote fails you must:

  1. Clean up the release artifacts in the svn distribution directory

        ant -f bin/release.xml svn-delete-rc-artifacts

  2. Log in to the ASF Nexus server.

  3. Locate and select the staged Slider repository.
  4. In the toolbar above, press the Drop button.

Notes

  1. The maven package and deploy operations both rebuild their artifacts; if you re-run them between stages they may produce different artifacts.

  2. You can set the property git.version if you wish to build a vote message from a different git version from the current build, but as you must be calling for a vote on the version released, only do this if you are modifying the release.xml release file)

  3. You can create the release.properties file elsewhere and point to it via the ant release.properties property option. This would need to be set in build.properties or on the command line.

    ant -f bin/release.xml -Drelease.properties=~/release.properties

It's easier to use a symbolic link here.

  1. The slider POMs offer different profiles of Hadoop versions to build against. The instructions below use the default. If a profile is needed, append it to all the maven commands.

    mvn.extra.args=-Prelease-2.8

  2. If you are not building against a stable Hadoop release

    1. Check out the Hadoop branch you intend to build and test against —and include in the redistributable artifacts.
    2. Build it via its own mvn clean install -DskipTests
    3. Note the git revision number of this build —for documentation.
    4. Maybe: tag that revision so you can easily revert to it.
    5. set the -Dhadoop.version property to the targeted Hadoop version in the mvn.extra.args release property
        mvn.extra.args=-Dhadoop.version=2.8.12

Unstable Hadoop versions MUST NOT be used for anything declared as a stable Slider release.