HOWTO
Here’s some miscellaneous documentation about using Avatica.
- Building from a source distribution
- Building from git
- Running tests
- Contributing
- Getting started
- Advanced topics for developers
- Advanced topics for committers
- Set up PGP signing keys (for Calcite committers)
- Run a GPG agent
- Set up Maven repository credentials (for Calcite committers)
- Making a snapshot (for Calcite committers)
- Making a release (for Calcite committers)
- Cleaning up after a failed release attempt (for Calcite committers)
- Validate a release
- Get approval for a release via Apache voting process (for Calcite committers)
- Publishing a release (for Calcite committers)
- Publishing the web site (for Calcite committers)
Building from a source distribution
Prerequisites are maven (3.2.1 or later) and Java (JDK 8 or later) on your path.
Unpack the source distribution .tar.gz
file,
cd
to the root directory of the unpacked source,
then build using maven:
Running tests describes how to run more or fewer tests.
Building from git
Prerequisites are git, maven (3.2.1 or later) and Java (JDK 8 or later) on your path.
Create a local copy of the github repository,
cd
to its root directory,
then build using maven:
Running tests describes how to run more or fewer tests.
Running tests
The test suite will run by default when you build, unless you specify
-DskipTests
.
Running tests in your environment
By default, invoking the verify
Maven lifecycle phase will also cause checkstyle
rules to be run. It is expected that contributions pass the checkstyle rules; however,
it is common to ignore these while working on a feature/bug and fix them at the end.
To run tests in docker:
- You will need to have docker and Docker Compose installed.
Contributing
See the developers guide.
Getting started
See the developers guide.
Advanced topics for developers
The following sections might be of interest if you are adding features to particular parts of the code base. You don’t need to understand these topics if you are just building from source and running tests.
Advanced topics for committers
The following sections are of interest to Calcite committers and in particular release managers.
Set up PGP signing keys (for Calcite committers)
Follow instructions here to
create a key pair. (On Mac OS X, I did brew install gpg
and
gpg --gen-key
.)
Add your public key to the
KEYS
file by following instructions in the KEYS
file.
(The KEYS
file is not present in the git repo or in a release tar
ball because that would be
redundant.)
Run a GPG agent
By default, Maven plugins which require you to unlock a GPG secret key
will prompt you in the terminal. To prevent you from having to enter
this password numerous times, it is highly recommended to install and
run gpg-agent
.
This can be started automatically via an ~/.xsession
on Linux or some
scripting in your shell’s configuration script of choice (e.g. ~/.bashrc
or ~/.zshrc
)
Also, ensure that default-cache-ttl 6000
is set in ~/.gnupg/gpg-agent.conf
to guarantee that your credentials will be cached for the duration of the build.
Set up Maven repository credentials (for Calcite committers)
Follow the instructions here to add your credentials to your maven configuration.
Making a snapshot (for Calcite committers)
Before you start:
- Set up signing keys as described above.
- Make sure you are using JDK 8 (not 9 or 10).
When the dry-run has succeeded, change install
to deploy
.
Making a release (for Calcite committers)
Before you start:
- Set up signing keys as described above.
- Make sure you are using JDK 8 (not 9 or 10).
- Check that
README
,site/_docs/howto.md
,site/_docs/docker_images.md
have the correct version number. - Check that
NOTICE
has the current copyright year. - Set
version.major
andversion.minor
inpom.xml
. - Add release notes to
site/_docs/history.md
. Include the commit history, and say which versions of Java, Guava and operating systems the release is tested against. - Generate a report of vulnerabilities that occur among dependencies,
using
./mvnw verify -Ppedantic
. - Make sure that every “resolved” JIRA case (including duplicates) has a fix version assigned (most likely the version we are just about to release)
Create a release branch named after the release, e.g.
branch-avatica-1.9
, and push it to Apache.
We will use the branch for the entire the release process. Meanwhile,
we do not allow commits to the master branch. After the release is
final, we can use git merge --ff-only
to append the changes on the
release branch onto the master branch. (Apache does not allow reverts
to the master branch, which makes it difficult to clean up the kind of
messy commits that inevitably happen while you are trying to finalize
a release.)
Now, set up your environment and do a dry run. The dry run will not commit any changes back to git and gives you the opportunity to verify that the release process will complete as expected.
If any of the steps fail, clean up (see below), fix the problem, and start again from the top.
To perform the dry-run, you can either use your environment or the release script and docker.
To perform the dry-run directly in your environment:
To perform the dry-run in docker:
-
You will need to have docker and Docker Compose installed.
-
The script expects you to mount your
~/.gnupg
directory into the/.gnupg
directory in the container. Once mounted into the container, the script will make a copy of the contents and move it to a different location, so that it will not modify the contents of your original~/.gnupg
directory during the build.
Check the artifacts:
- In the
target
directory should be these 6 files, among others:- apache-calcite-avatica-X.Y.Z-src.tar.gz
- apache-calcite-avatica-X.Y.Z-src.tar.gz.asc
- apache-calcite-avatica-X.Y.Z-src.tar.gz.sha256
- Note that the file names start
apache-calcite-avatica-
. - In the source distro
.tar.gz
(currently there is no binary distro), check that all files belong to a directory calledapache-calcite-avatica-X.Y.Z-src
. - That directory must contain files
NOTICE
,LICENSE
,README
,README.md
- Check that the version in
README
is correct
- Check that the version in
- Make sure that there is no
KEYS
file in the source distros - For each .jar (for example
core/target/avatica-core-X.Y.Z.jar
andserver/target/avatica-server-X.Y.Z-sources.jar
), verify that theMETA-INF
directory contains the correct contents forDEPENDENCIES
,LICENSE
andNOTICE
per the source/classes contained. Refer to the ASF licensing documentation on what is required. - Check PGP, per this
If something is not correct, you can invoke the release:clean
mojo to remove the
generated files from your workspace:
If you are building directly in your environment:
If you are building using docker:
If successful, remove the -DdryRun
flag and run the release for real.
To build directly in your environment:
To build using docker:
Verify the staged artifacts in the Nexus repository:
- Go to https://repository.apache.org/ and login
- Under
Build Promotion
, clickStaging Repositories
- In the
Staging Repositories
tab there should be a line with profileorg.apache.calcite
- Navigate through the artifact tree and make sure the .jar, .pom, .asc files are present
- Check the box on in the first column of the row, and press the ‘Close’ button to publish the repository at https://repository.apache.org/content/repositories/orgapachecalcite-1000 (or a similar URL)
Upload the artifacts via subversion to a staging area, https://dist.apache.org/repos/dist/dev/calcite/apache-calcite-avatica-X.Y.Z-rcN:
Cleaning up after a failed release attempt (for Calcite committers)
Validate a release
Get approval for a release via Apache voting process (for Calcite committers)
Release vote on dev list
After vote finishes, send out the result:
Use the Apache URL shortener to generate shortened URLs for the vote proposal and result emails. Examples: s.apache.org/calcite-1.2-vote and s.apache.org/calcite-1.2-result.
Publishing a release (for Calcite committers)
After a successful release vote, we need to push the release out to mirrors, and other tasks.
Choose a release date. This is based on the time when you expect to announce the release. This is usually a day after the vote closes. Remember that UTC date changes at 4pm Pacific time.
In JIRA, search for all issues resolved in this release, and do a bulk update changing their status to “Closed”, with a change comment “Resolved in release X.Y.Z (YYYY-MM-DD)” (fill in release number and date appropriately). Uncheck “Send mail for this update”.
Promote the staged nexus artifacts.
- Go to https://repository.apache.org/ and login
- Under “Build Promotion” click “Staging Repositories”
- In the line with “orgapachecalcite-xxxx”, check the box
- Press “Release” button
Tip: Push the git tag only after the staged nexus artifacts are promoted in the repository. This is because pushing the tag triggers Docker Hub to start building the docker images immediately and the build will pull in the promoted artifacts. If the artifacts are not yet available, the build on Docker Hub will fail.
Copy the Git tag:
Check the artifacts into svn.
Svnpubsub will publish to the release repo and propagate to the mirrors within 24 hours.
If there are now more than 2 releases, clear out the oldest ones:
The old releases will remain available in the release archive.
Add a release note by copying
site/_posts/2016-11-01-release-1.9.0.md,
generate the javadoc and copy to site/target/avatica/apidocs
and site/target/avatica/testapidocs
,
publish the site,
and check that it appears in the contents in news.
After 24 hours, announce the release by sending an email to announce@apache.org. You can use the 1.8.0 announcement as a template. Be sure to include a brief description of the project.
Publishing the web site (for Calcite committers)
See instructions in site/README.md.