Apache Slider Release Process v1¶

This was release process for Slider versions up to and including Slider 0.60, which was released in November 2014.

The process was superceded in January 2015 —later releases follow the process described in Releasing.

The process has two phases.

Phase I: Merge develop branch with master so that you can release from master
Phase II: Perform Apache release from master

Tooling¶

As well as everything needed to build slider, there are some extra requirements for releasing:

Shell: (Currently: Bash)
git flow
OS/X and windows: Atlassian SourceTree. This can perform the git flow operations, as well as show the state of your git graph.

Before you begin¶

1. Check out the latest version of the develop branch, run the tests. This should be done on a checked out version of the code that is not the one you are developing on (ideally, a clean VM), to ensure that you aren't releasing a slightly modified version of your own, and that you haven't accidentally included passwords or other test run details into the build resource tree.

The slider-funtest functional test package is used to run functional tests against a running Apache Hadoop YARN cluster. It needs to be configured according to the instructions in testing to create HBase and Accumulo clusters in the YARN cluster.

Make sure that the integration tests are passing (and not being skipped) before starting to make a release

2. Check out the latest version of origin/master.

3. Make sure there are no uncommitted files in your local repo.

4. If you are not building against a stable Hadoop release

Check out the hadoop branch you intend to build and test against —and include in the redistributable artifacts.
Build it via mvn clean install -DskipTests
Note the git revision number of this build -for documentation.
Maybe: tag that revision so you can easily revert to it

Profiles¶

The slider POMs offer different profiles of hadoop versions to build against. The instructions below do not choose one. If a profile is needed, append it to all the maven commands.

e.g. mvn clean install -DskipTests -Prelease-2.6

Phase - I¶

Step #0: Sync up with the ASF repo. You do not want to release the wrong code.

git checkout develop
git pull origin
git push origin

Step #1: Create a JIRA for the release, estimate 3h (so you don't try to skip the tests)

export SLIDER_RELEASE_JIRA=SLIDER-544

Step #2: Check everything in. Git flow won't let you progress without this.

Step #4: Increment those version numbers using the maven versions plugin

export SLIDER_RELEASE=0.60.0-incubating

mvn versions:set -DnewVersion=$SLIDER_RELEASE

Step #5: Commit the changed POM files

git status

git add --all

git commit -m "$SLIDER_RELEASE_JIRA updating release POMs for $SLIDER_RELEASE"

Step #3: Create and check out a release branch

git checkout -b releases/slider-0.60

Step #6: Do a final test run to make sure nothing is broken

In the slider directory, run:

mvn clean install -DskipTests

(Make sure the build isn't trying to pull down its JARs from the Apache Staging repo) Once everything is built- including .tar files, run the tests

mvn test

Then the integration tests

mvn integration-test -Dslider.conf.dir=${your-config-dir}

This will run the functional tests as well as the slider-core tests.

It is wise to reset any VMs here, and on live clusters kill all running jobs. This stops functional tests failing because the job doesn't get started before the tests time out.

As the test run takes 30-60+ minutes, now is a good time to consider finalizing the release notes.

Step #7: Build the release package

Run a clean install, followed by a site build and package goal:

mvn clean install -DskipTests
mvn site:site site:stage package -DskipTests

Step #8: Validate the tarball

Look in slider-assembly/target to find the .tar.gz file, and the expanded version of it. Inspect that expanded version to make sure that everything looks good -and that the versions of all the dependent artifacts look good too: there must be no -SNAPSHOT dependencies. [Exception, when the release is deliberately made against an unreleased version of Hadoop or other dependencies. in that situation the unreleased dependency is expected to consist of -SNAPSHOT artifacts]

tip: the lib dir can be viewed with

ls -l slider-assembly/target/slider-$SLIDER_RELEASE-all/slider-$SLIDER_RELEASE/lib

Important Verify that there is a slider-agent.tar.gz file. The two-phase install/site build gets it in there, but doing a single build doesn't always seem to.

Step #9: Build the release notes

Create a a one-line plain text release note for commits and tags and a multi-line markdown release note which will be used for artifacts.

Release against hadoop 2.6.0

The multi-line release notes should go into the slider SVN repo under site/trunk/content/release_notes.

Commit the documentation changes:

svn commit -m "$SLIDER_RELEASE_JIRA updating release notes for slider-$SLIDER_RELEASE"

Step #10: Get back to the develop branch

git checkout develop

You will now be back on the develop branch.

Step #11: update maven versions

Switch back to develop and update its version number past the release number

export SLIDER_DEV=0.61.0-incubating-SNAPSHOT
mvn versions:set -DnewVersion=$SLIDER_DEV
git commit -a -m "$SLIDER_RELEASE_JIRA updating development POMs to $SLIDER_DEV"

Step #12: Push the release and develop branches to github

git push origin develop releases/slider-0.60

(assuming that origin maps to https://git-wip-us.apache.org/repos/asf/incubator-slider.git; you can check this with git remote -v

Step #13: Finish the JIRA

Log the time, close the issue. This should normally be the end of a sprint -so wrap that up too.

Step #14: Get back to developing!

Check out the develop branch and purge all release artifacts

git checkout develop
git pull origin
mvn clean

At this point you are ready for Apache release from master.

Phase - II: publish to the ASF¶

This phase of the process creates the artifacts for the ASF. This is done in a separate directory to ensure that nothing accidentally gets in.

Preamble: set up the RC Version

export RELEASE_RC=rc0

Step 1

Get a clean git clone in its own directory.

mkdir release
cd release
git clone https://git-wip-us.apache.org/repos/asf/incubator-slider.git slider-$SLIDER_RELEASE
cd slider-$SLIDER_RELEASE

Step 2

Ensure you are in the master branch and it's clean. Beware that the git clean command will remove any unknown files to Git in your current repository.

git checkout releases/slider-0.60
git pull origin
git clean -fxd

Step 3

Ensure the rat check is clean.

mvn clean apache-rat:check -Prat

Step 4

Tag for the release. Start with RC0 and move to RC1, RC2 if you need to restart the release process for this version.

git tag -a release-$SLIDER_RELEASE-$RELEASE_RC -m 'Slider $SLIDER_RELEASE-$RELEASE_RC'
git push --tags origin

Step 5

Create the release source tarball.

mvn clean install -DskipTests -Papache-release -Prat

You should find a *.tar.gz at target folder (e.g. ./target/apache-slider-$SLIDER_RELEASE-source-release.tar.gz and .zip) Rename it with incubator in the name.

cd target

rm rat.txt
rm .plxarc
rm -rf archive-tmp
rm -rf maven-shared-archive-resources

Step 6 This is now automated: skip

Sign the tarball

export ASF_USER=stevel

gpg2  --armor  --default-key $ASF_USER@apache.org -v --output apache-slider-$SLIDER_RELEASE-source-release.tar.gz.asc --detach-sig apache-slider-$SLIDER_RELEASE-source-release.tar.gz
gpg2  --armor  --default-key $ASF_USER@apache.org -v  --output apache-slider-$SLIDER_RELEASE-source-release.zip.asc --detach-sig apache-slider-$SLIDER_RELEASE-source-release.zip

Step 7

Generate MD5 checksums. md5sum and sha1sum or gpg are all capable of creating checksum files.

Either

openssl md5 apache-slider-$SLIDER_RELEASE-source-release.tar.gz > apache-slider-$SLIDER_RELEASE-source-release.tar.gz.md5
openssl sha1 apache-slider-$SLIDER_RELEASE-source-release.tar.gz > apache-slider-$SLIDER_RELEASE-source-release.tar.gz.sha
openssl md5 apache-slider-$SLIDER_RELEASE-source-release.zip > apache-slider-$SLIDER_RELEASE-source-release.zip.md5
openssl sha1 apache-slider-$SLIDER_RELEASE-source-release.zip > apache-slider-$SLIDER_RELEASE-source-release.zip.sha

 md5sum apache-slider-$SLIDER_RELEASE-source-release.tar.gz > apache-slider-$SLIDER_RELEASE-source-release.tar.gz.md5
 sha1sum apache-slider-$SLIDER_RELEASE-source-release.tar.gz > apache-slider-$SLIDER_RELEASE-source-release.tar.gz.sha
 sha1sum apache-slider-$SLIDER_RELEASE-source-release.zip > apache-slider-$SLIDER_RELEASE-source-release.zip.sha
 md5sum apache-slider-$SLIDER_RELEASE-source-release.zip > apache-slider-$SLIDER_RELEASE-source-release.zip.md5

Step 8

Upload the artifacts to your apache home:

The artifacts then need to be copied over to ~/public_html/slider-release-$SLIDER_RELEASE-rcX

cd ..

mv target slider-release-$SLIDER_RELEASE-$RELEASE_RC

scp -r slider-release-$SLIDER_RELEASE-$RELEASE_RC $ASF_USER@people.apache.org:public_html/slider/

(if you don't get the authentication for the copy, go to https://id.apache.org/ and add your ssh keys to your profile)

The entire directory should now be uploaded, check it:

echo http://people.apache.org/~$ASF_USER/slider/slider-release-$SLIDER_RELEASE-$RELEASE_RC

on OS/X, you can have it opened in your chosen browser

open http://people.apache.org/~$ASF_USER/slider/slider-release-$SLIDER_RELEASE-$RELEASE_RC

Step 9

(omitted)

Step 10

Call for a release VOTE to the dev mailing list (dev@slider.incubator.apache.org).

To build the URLs, echo them then verify in your browser that they are present

echo  [VOTE] Apache Slider Incubating Release $SLIDER_RELEASE $RELEASE_RC

echo "This is a call for a vote on Apache Slider $SLIDER_RELEASE release"

echo "Artifacts at http://people.apache.org/~$ASF_USER/slider/slider-release-$SLIDER_RELEASE-$RELEASE_RC"

echo "Git source tag: https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=shortlog;h=refs/tags/release-$SLIDER_RELEASE-$RELEASE_RC"

echo "PGP keys at http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=$ASF_USER@apache.org"

Sample:

```

Title: [VOTE] Apache Slider Incubating Release $SLIDER_RELEASE $RELEASE_RC

Hello folks,

This is a call for a vote on Apache Slider $SLIDER_RELEASE release. Thanks to everyone who have contributed to this release.

Git source tag: https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;a=shortlog;h=refs/tags/release-$SLIDER_RELEASE-$RELEASE_RC

Staging site: http://people.apache.org/~$ASF_USER/slider-release-$SLIDER_RELEASE-$RELEASE_RC"

PGP release keys (signed using $ASF_USER) http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=$ASF_USER@apache.org

The issues fixed in this release are listed at at: https://issues.apache.org/jira/browse/SLIDER/fixforversion/

Note that this is a source only release and we are voting on the source.

Build instructions at: http://slider.incubator.apache.org/developing/building.html

Vote will be open for 72 hours

[ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why)

```

CHECK ALL THE URLS FIRST *

Step 11

If the VOTE passes, call for VOTE from IPMC. Include the VOTE and RESULT thread from the dev alias. If that VOTE passes with the IPMC send a RESULT mail.

Sample:

Title: [RESULT][VOTE] Apache Slider $SLIDER_RELEASE $RELEASE_RC

Thanks for the votes.
The VOTE passes with 5, +1 votes from IPMC.

List any IPMC votes.

Other +1 votes:

List all non-IPMC votes.

Step 12

Verify your PGP key is provided to Apache.

Apache verifies that distributions are correctly signed. Login to https://id.apache.org and verify the fingerprint of PGP key used to sign above is provided. (gpg --fingerprint) Upload your PGP public key only to /home/$ASF_USER/.pgpkey

scp public-key.asc people.apache.org:.pgpkey

Step 13

Publish the release as below:

svn co https://dist.apache.org/repos/dist/release/incubator/slider slider

cd slider

Make sure your PGP keys is in the KEYS file. The instructions to add a key are at the top of the file

Then...

mkdir $SLIDER_RELEASE $SLIDER_RELEASE/src
cd $SLIDER_RELEASE/src
scp ~/public_html/$SLIDER_RELEASE-$RELEASE_RC/* .
cd ..
svn add $SLIDER_RELEASE
svn commit -m "SLIDER_RELEASE_JIRA Committing Release $SLIDER_RELEASE"

Step 14

Create the release tag:

On branch master
git tag -a $SLIDER_RELEASE -m 'Slider $SLIDER_RELEASE'
git push --tags origin

Step 15

Update the Slider website as needed. Most of the documents would have been updated by now to reflect the released version. The remaining changes should really be modifying the references to the latest release to the new version.

The download details for the latest release needs to be updated after mirrors are updated in Step 12.

Verify the download links at Slider Page.

Step 16

Send an announcement:

The bits published in Step 13 may take up to 24 hours to replicate to the mirrors.

Send an announcement to the Slider dev, Slider user, and the general Apache announcement (announcement@apache.org) list after the bits have replicated. To send the message to announcement@apache.org, you will need to send the message from your apache.org email address.

Sample:

Title: [ANNOUNCE] Apache Slider x.y.z-incubating.

The Apache Slider team is proud to announce Apache Slider incubation release version x.y.z.

Apache Slider (incubating) is a YARN application which deploys existing distributed applications on YARN,
monitors them, and makes them larger or smaller as desired - even while the application is running.

The release bits are at:  http://www.apache.org/dyn/closer.cgi/incubator/slider/x.y.z-incubating/

To use the released bits please use the following documentation: http://slider.incubator.apache.org/docs/getting_started.html

We would like to thank all the contributors that made the release possible.

Regards,
The Slider Team

Step 17

Tell JIRA there's a new release. You must be registered as an admin for the Slider project on JIRA.

Go to the Slider Versions page

Release the version you've done, carrying forward issues that haven't been done yet.

Hotfixing a release¶

If you need to change a release —such as to issue a new RC, you must create a hotfix release which updates the master/ and develop/ branches. you can then create a new release off the last commit off the hotfix branch (the one which is then merged into both branches).

Create a JIRA for the hotfix, estimate as much time as the main release and any time needed to actually fix the code.
set the release env var to it.

export SLIDER_RELEASE_JIRA=SLIDER-349
Create the hotfix branch with the JIRA version
Fix things.
Rerun all the tests.
Check in the fixes
Note the commit number of the final commit
With git flow, finish the hotfix
start a git flow release using the specific commit of the hotfix branch, rather than HEAD of the develop branch.
Note the current version of the develop artifacts; set $SLIDER_DEV to it. The POMs will be returned to this after the release process.
Create a new release version (e.g. slider-0.50.2). This can before that of the current develop branch:
```
export SLIDER_RELEASE=0.50.2-incubating
```
Repeat the release process from step 4 to step 10.

When you get to the end of release phase, the end of release process will fail due to conflict between POM versions.

Resolve the conflict by accepting the new updates: the maven version will be set to the $SLIDER_RELEASE value, even though it is now older than the actual development version.
verify that $SLIDER_DEV is set to the version you want to return the develop/ branch to:
```
echo $SLIDER_DEV
```

reset the versions

mvn versions:set -DnewVersion=$SLIDER_DEV
git commit -a -m "$SLIDER_RELEASE_JIRA reverting development POMs to $SLIDER_DEV"

Continue the release process from step 12.

Publishing artifacts¶

Once the vote has (successfully) completed on both slider-dev and incubator-general, the artifacts may be published in two ways

tarballs for download via the apache distribution system
JAR files for download via the maven repository

Publishing files for apache release¶

See the ASF release guidelines