Developer Documentation
Contributing
Ways to Contribute
While the concept of an open source contribution can refer to doing
development work on the code base, there are many other ways outside of coding to contribute to Apache TinkerPop.
Participating on the various mailing lists, offering ideas, reporting bugs, writing documentation are all welcome
contributions to the project that help improve the TinkerPop. This section of the document is designed to help
provide some structure for potential contributors and to give them ideas for how they could get started becoming more
involved in the TinkerPop community.
Note
|
As a quick acknowledgment, this section was influenced by Apache Spark’s well organized "Contributing to Spark" document. |
Mailing List Participation
TinkerPop has two mailing lists: gremlin-users and dev. Subscribing to and participating on one or both of these mailing lists is a great way to contribute to TinkerPop. Helping users with their problems by answering their questions on gremlin-users is a great help to the community and an easy way for the community to become familiar with the contributor.
The dev@tinkerpop.apache.org mailing list is where all design discussion, early feature announcements, release discussions and other similar communication takes place. Having insight into the project at this level, will yield a lot of information about the day-to-day course of TinkerPop and many of the other ways contributors can be involved require a subscription to this list.
Testing Releases
Proposed TinkerPop releases are not official until after they are evaluated by the community. The process of evaluation occurs on the dev@tinkerpop.apache.org mailing list where the community is invited to review release artifacts and vote if they should be released or not. Anyone is free to comment and provide a vote (+1/+0/-1) on a release. Note that votes of non-PMC members are considered non-binding, but are certainly considered in the review process.
Reviewing Changes
Virtually all changes to TinkerPop’s source code are performed via GitHub pull requests. When a pull request is issue it undergoes a review process prior to be merged to a release branch. Anyone is free to comment and provide a vote (+1/+0/-1) on a pull request. Note that votes of non-committers are considered non-binding, but are certainly considered in the review process.
Writing Documentation
Proposing a documentation change is quite similar to providing a source code change, which is described below in more detail. Note that there is a difference between project documentation and the TinkerPop web site. The project documentation refers to the documentation that is published per release like, tutorials, reference documentation, and other similar pages. This content is not updated after release and remains static bound to that version. The TinkerPop web site refers to the static home page and its related content, such as Understanding Gremlin, Provider Listing and Graphic Usage Policies and other similar pages. This content is not bound to any particular version and can be published at any time where new content simply replaces old content.
For both types of documentation, changes can be submitted via pull request. For project documentation, TinkerPop has a robust documentation system that is based on asciidoc. The content can be found in the docs/src. Recall that this documentation is version specific, so consider the appropriate branch on which to submit the pull request so that the documentation is reflective of the version it is tied to. To view generated documentation locally, read more about environment configurations in the Documentation Environment section.
For web site changes, the process is largely the same except that the documentation system is HTML based instead of
Asciidoc. The content can be found in the source control tree at docs/site.
The web site is always published from the master
branch as it is not bound to a version, so there is no need to
submit a pull request to any other branches besides that one. If the change to this documentation involves changing
provider listings in some way, first propose the those changes on the dev@tinkerpop.apache.org mailing list and achieve
consensus (wait time is 72 hours as usual). At that point, pull requests or direct changes can be submitted.
Again, see the Documentation Environment section for more information on how to generate
the site locally.
Reporting Bugs
Well formed bug reports (especially when accompanied by a pull request that fixes the problem) are welcome contributions that ultimately help improve the quality of TinkerPop. Of course, a bug report is only as good as its reproduction steps and the surrounding details of the issue. When a bug is encountered, it should be documented in JIRA, where it will likely trigger some discussion and ultimately a resolution.
TinkerPop Libraries
TinkerPop is more than just the core source code repository. It is a rich ecosystem of libraries developed by providers, who look to use or extend upon the TinkerPop interfaces. TinkerPop is highly focused on its interfaces and a small set of reference implementations of those interfaces as part of its core code base. As such, large and independent new functionality is often rejected from inclusion in TinkerPop itself, however, it is very much encouraged that such a project stand on its own as separate repository. Projects like this that meet the TinkerPop listing policy can be added to the TinkerPop web site and promoted there. Please see the current listing of such libraries on the TinkerPop home page to get an idea as to what kinds of libraries are available from the community.
Considering Code Changes
Code changes can take a fair bit of effort from both the contributor doing the work, as well as the people who will be reviewing and testing that work. It is beneficial to all involved that some basic considerations be made by the contributor to ensure the best use is made of everyone’s time.
Note
|
For those who are trying to find a place to start to contribute, consider looking at unresolved issues that have the "trivial" priority as these issues are specifically set aside as low-hanging fruit for newcomers. |
Before proceeding, contributors should evaluate if the proposed change is likely to be relevant, new and actionable:
-
Is it clear that code must change? Proposing a JIRA issue and pull request is appropriate only when a clear problem or change has been identified. When in doubt, email dev@tinkerpop.apache.org first about the possible change.
-
Search the mailing list archives for related discussions. Often, the problem has been discussed before, with a resolution that doesn’t require a code change, or recording what kinds of changes will not be accepted as a resolution.
-
Search JIRA for existing issues.
-
Is the scope of the change matched to the contributor’s level of experience? Anyone is qualified to suggest a typo fix, but refactoring a core feature in serialization or OLAP requires much more understanding of TinkerPop. Some changes require building up experience first.
Before considering how to contribute code, it is useful to understand how code is reviewed, and why changes may be rejected. Simply put, changes that have many or large positives, and few negative effects or risks, are much more likely to be merged, and merged quickly. Risky and less valuable changes are very unlikely to be merged, and may be rejected outright rather than receive iterations of review.
Positives
-
Change has already been discussed and is known to committers
-
Fixes the root cause of a bug in existing functionality
-
Adds functionality or fixes a problem needed by a large number of users
-
Simple, targeted
-
Easily tested; has tests
-
Reduces complexity and lines of code
Negatives, Risks
-
Band-aids a symptom of a bug only
-
Introduces complex new functionality, especially an API that needs to be supported
-
Adds complexity that only helps a niche use case
-
Adds user-space functionality that does not need to be maintained in TinkerPop, but could be hosted externally and promoted in provider listings
-
Changes a public API or semantics thus introducing a breaking change
-
Adds large dependencies
-
Changes versions of existing dependencies
-
Adds a large amount of code
-
Makes lots of modifications in one "big bang" change
Contributing Code Changes
Generally, TinkerPop uses JIRA to track logical issues, including bugs and improvements, and uses GitHub pull requests to manage the review and merge of specific code changes. That is, JIRA issues are used to describe what should be fixed or changed, and high-level approaches, and pull requests describe how to implement that change in the project’s source code.
As a first step to making a contribution, consider JIRA:
-
Find the existing JIRA ticket that the change pertains to.
-
Do not create a new ticket if creating a change to address an existing issue in JIRA; add to the existing discussion and work instead.
-
Look for existing pull requests that are linked from the ticket, to understand if someone is already working on the JIRA.
-
-
If the change is new, then it usually needs a new ticket. However, trivial changes, where the change is virtually the same as the how it should change do not require a JIRA (e.g. "Fix typos in Foo javadoc").
-
If required, create a new ticket:
-
Provide a descriptive Title and a detailed Description. For bug reports, this should ideally include a short reproduction of the problem.
-
Set required fields - these are detailed later in this document in the Issue Tracker Conventions section.
-
-
If the change is a large change, consider inviting discussion on the issue at dev@tinkerpop.apache.org first before proceeding to implement the change.
Next, make changes and prepare a pull request:
-
Fork and then clone the Apache TinkerPop GitHub repository if not already done.
-
Make changes in the fork
-
It is typically best to create a branch for the changes. Consider naming that branch after the JIRA issue number to easily track what that branch is for.
-
Consider which release branch (e.g.
master
,tp31
etc) to create the development branch from in the first place. In other words, is the change to be targeted at a specific TinkerPop version (e.g. a patch to an older version)? When in doubt, please ask on dev@tinkerpop.apache.org.
-
-
Build the project and run tests.
-
A simple build can be accomplished with maven:
mvn clean install
. -
Often, a "simple build" isn’t sufficient and integration tests are required:
mvn clean install -DskipIntegrationTests=false -DincludeNeo4j
. Note that Hadoop must be running for the integration tests to execute. -
Docker can help simplify building and testing:
docker/build.sh -t -i -n
-
Please see the Building and Testing section for more building and testing options.
-
-
Consider whether documentation or tests need to be added or updated as part of the change, and add them as needed.
-
Nearly all changes should include a modification to the
CHANGELOG.asciidoc
file - one more entries to help summarize the change. -
Some changes will require updates to the "upgrade documentation" - usually reserved for major new features and breaking changes.
-
Docker can help simplify documentation generation:
docker/build.sh -d
-
Please see the Building and Testing section for more documentation generation options.
-
-
Open the pull request against the appropriate branch on the Apache TinkerPop repository.
-
Target the pull request at the appropriate branch in TinkerPop’s repository
-
Prefix the name of the pull request with the JIRA issue number (include a brief description after that).
-
Include a link to the ticket in JIRA in the pull request description.
-
Include a rough synopsis of how the changes were tested. This might be as simple as "Ran mvn clean install to success and performed manual testing in the Gremlin Console".
-
Include other descriptive elements about the change if they are not already included in the JIRA ticket.
-
Automated builds will occur with Travis. Please be sure that the pull request passes those builds and correct them if there are problems.
-
Once the pull request has been placed it will go into review:
-
Other reviewers, including committers, may comment on the changes and suggest modifications. Changes can be added by simply pushing more commits to the same branch.
-
Lively, polite, rapid technical debate is encouraged from everyone in the community. The outcome may be a rejection of the entire change.
-
Reviewers can indicate that a change looks suitable for merging with by providing a "+1". Please see the Review then Commit process for more details.
-
Sometimes, other changes will be merged which conflict with your pull request’s changes. The PR can’t be merged until the conflict is resolved. This can be resolved with "git fetch origin" followed by "git merge origin/master" and resolving the conflicts by hand, then pushing the result to your branch. Or more nicely, consider rebasing changes and force pushing the branch.
-
Try to be responsive to the discussion rather than let days pass between replies
On successful review, the pull request will be merged to the main repository and the JIRA issue will be closed.
Development Environment
System Configuration
At a minimum, development of TinkerPop requires Java 1.8.0_40+
and Maven 3.2.5+. Maven is used as the common build system, which even
controls the builds of non-JVM GLVs
such as gremlin-python
. Java and Maven are described as a "minimum" for a development environment, because they
will only build JVM portions of TinkerPop and many integration tests will not fire with this simple setup. It is
possible to get a clean and successful build with this minimum, but it will not be possible to build non-JVM aspects
of the project and those will go untested.
To gain the ability to execute all aspects of the TinkerPop build system, other environmental configurations must be established. Those prerequisites are defined in the following subsections.
Important
|
For those who intend to offer a contribution, building with a minimal configuration may not be sufficient when submitting a pull request. Consider setting up the full environment. |
Note
|
For those using Windows, efforts have been made to keep the build OS independent, but, in practice, it is likely that TinkerPop’s build system will only allow for a minimum build at best. |
Documentation Environment
The documentation generation process is not Maven-based and uses shell scripts to process the project’s asciidoc. The scripts should work on Mac and Linux.
To generate documentation, it is required that Hadoop 2.7.x is running in
pseudo-distributed
mode. Be sure to set the HADOOP_GREMLIN_LIBS
environment variable as described in the
reference documentation. It is also important
to set the CLASSPATH
to point at the directory containing the Hadoop configuration files, like mapred-site.xml
.
Also note that awk version 4.0.1
is required for documentation generation.
Documentation can be generated locally with:
bin/process-docs.sh
Documentation is generated to the target/docs
directory. It is also possible to generate documentation locally with
Docker. docker/build.sh -d
.
To generate the web site locally, there is no need for any of the above infrastructure. Site generation is a simple shell script:
bin/generate-home.sh
The site will be generated to the target/site/home
directory.
Python Environment
As of TinkerPop 3.2.2, the build optionally requires Python 2.x to work with the
gremlin-python
module. If Python is not installed, TinkerPop will still build with Maven, but native Python tests and
Java tests that require Python code will be skipped. Developers should also install pip
and virtualenv (version 15.0.2 - older versions may cause build failures).
Once the Python environment is established, the full building and testing of gremlin-python
may commence. It can be
done manually from the command line with:
mvn clean install -Pglv-python
which enables the "glv-python" Maven profile or in a more automated fashion simply add a .glv
file to the root of the
gremlin-python
module which will signify to Maven that the environment is Python-ready. The .glv
file need not have
any contents and is ignored by Git. A standard mvn clean install
will then build gremlin-python
in full.
As of TinkerPop 3.2.5, the build also requires Python to execute gremlin-console
integration tests. The integration
test is configured by a "console-integration-tests" Maven profile. This profile can be activated manually or can more
simply piggy-back on the .glv
file in gremlin-python
. Note that unlike gremlin-python
the tests are actually
integration tests and therefore must be actively switched on with -DskipIntegrationTests=false
:
mvn clean install -pl gremlin-console -DskipIntegrationTests=false
DotNet Environment
The build optionally requires .NET Core SDK (>=2.1.101) to work with the
gremlin-dotnet
module. If .NET Core SDK is not installed, TinkerPop will still build with Maven, but .NET projects
will be skipped.
gremlin-dotnet
can be build and tested from the command line with:
mvn clean install -Pgremlin-dotnet
which enables the "gremlin-dotnet" Maven profile or in a more automated fashion simply add a .glv
 file to the src
and test
directories of the gremlin-dotnet
module  which will signify to Maven that the environment is .NET-ready.
The .glv
 file need not have any contents and is ignored by Git. A standard mvn clean install
 will then build
gremlin-dotnet
in full.
For those who will release TinkerPop, it is also necessary to install Mono. The
release process is known to work with 5.0.1, so it is best to probably install that version. Release managers should
probably also do an install of nuget 3.4.4 as it
will help with environmental setup. To get an environment ready to deploy to NuGet, it is necessary to have a
NuGet API key (PMC members who have NuGet accounts can help with that). The API key should be added to NuGet.Config
with the following:
mono nuget.exe setApiKey [your-api-key]
This should update ~/.config/NuGet/NuGet.Config
a file with an entry containing the encrypted API key. On
mvn deploy
, this file will be referenced on the automated nuget push
.
See release documentation for more information on configuration for release.
JavaScript Environment
When building gremlin-javascript
, mvn command will include a local copy of Node.js runtime and npm inside your project
using com.github.eirslett:frontend-maven-plugin
plugin. This copy of the Node.js runtime will not affect any
other existing Node.js runtime instances in your machine.
The release manager should have the authentication token set, for more information see the Release Environment section below.
Release Environment
This section is only useful to TinkerPop release managers and describes prerequisites related to deploying an official release of TinkerPop. All Apache releases must be signed. Please see this guide in the Apache documentation for instructions on to set up gpg. Keys should be added to KEYS files in both the development and release distribution directories and committed using Apache Subversion (SVN).
Uploading to pypi uses twine which is automatically installed by the build
process in maven. Twine refers to HOME/.pypirc
file for configuration on the pypi deploy environments and username
and password combinations. The file typically looks like this:
[distutils]
index-servers=
pypi
pypitest
[pypitest]
repository = https://testpypi.python.org/pypi
username = <username>
password =
[pypi]
repository = https://pypi.python.org/pypi
username = <username>
password =
The release manager shall use the project’s pypi credentials, which are available in the PMC repository.
The password
should be left blank so the deployment process in Maven will prompt for it at deployment time.
To deploy gremlin-javascript
on the npm registry, the release manager must set the
authentication information on the ~/.npmrc file. The easiest way to do that is to use the npm adduser
command. This
must be done only once, as the auth token doesn’t have an expiration date and it’s stored on your file system.
Building and Testing
The following commands are a mix of Maven flags and shell scripts that handle different build operations
-
Build project:
mvn clean install
-
Build a specific module (e.g.
gremlin-server
) within the project:mvn clean install -pl gremlin-server
-
Specify specific tests in a TinkerPop Suite to run with the
GREMLIN_TESTS
environment variable, along with the Maven project list argument, e.g.:export GREMLIN_TESTS='org.apache.tinkerpop.gremlin.process.traversal.step.map.PathTest$Traversals,org.apache.tinkerpop.gremlin.process.traversal.PathTest' mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test
-
Clean the
.groovy/grapes/org.apache.tinkerpop
directory on build:mvn clean install -DcleanGrapes
-
Turn off "heavy" logging in the "process" tests:
mvn clean install -DargLine="-DmuteTestLogs=true"
-
The test suite for
neo4j-gremlin
is disabled by default - to turn it on:mvn clean install -DincludeNeo4j
-
-
Regenerate test data (only necessary given changes to IO classes):
mvn clean install -Dio
fromtinkergraph-gremlin
directory-
If there are changes to the Gryo format, it may be necessary to generate the Grateful Dead dataset from GraphSON (see
IoDataGenerationTest.shouldWriteGratefulDead
)
-
-
Check license headers are present:
mvn apache-rat:check
-
Build AsciiDocs (see Documentation Environment):
bin/process-docs.sh
-
Build AsciiDocs (but don’t evaluate code blocks):
bin/process-docs.sh --dryRun
-
Build AsciiDocs (but don’t evaluate code blocks in specific files):
bin/process-docs.sh --dryRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,…
-
Build AsciiDocs (but evaluate code blocks only in specific files):
bin/process-docs.sh --fullRun docs/src/reference/the-graph.asciidoc,docs/src/tutorial/getting-started,…
-
Process a single AsciiDoc file: docs/preprocessor/preprocess-file.sh `pwd`/gremlin-console/target/apache-tinkerpop-gremlin-console-*-standalone "" "*" `pwd`/docs/src/xyz.asciidoc
-
-
Build JavaDocs:
mvn process-resources -Djavadoc
-
Check for Apache License headers:
mvn apache-rat:check
-
Check for newer dependencies:
mvn versions:display-dependency-updates
ormvn versions:display-plugin-updates
-
Deploy JavaDocs/AsciiDocs:
bin/publish-docs.sh svn-username
-
Integration Tests:
mvn verify -DskipIntegrationTests=false
-
Execute with the
-DincludeNeo4j
option to include transactional tests. -
Execute with the
-DuseEpoll
option to try to use Netty native transport (works on Linux, but will fallback to Java NIO on other OS).
-
-
Benchmarks:
mvn verify -DskipBenchmarks=false
-
Reports are generated to the console and to
gremlin-tools/gremlin-benchmark/target/reports/benchmark
.
-
-
Test coverage report:
mvn clean install -Dcoverage
- note that theinstall
is necessary because report aggregation is bound to that part of the lifecycle.-
Reports are generated to
gremlin-tools/gremlin-coverage/target/site
.
-
-
cd site
-
Generate web site locally:
bin/generate-home.sh
-
Publish web site:
bin/publish-home.sh <username>
-
Docker Integration
TinkerPop provides a shell script, that can start several build tasks within a Docker container. The required Docker images will be built automatically if they don’t exist yet. Thus the first invocation of the Docker script is expected to take some time.
The script can be found under PROJECT_HOME/docker/build.sh
. The following tasks are currently
supported:
-
run standard test suite
-
run integration tests
-
build Java docs
-
build user docs
A list of command line options is provided by docker/build.sh --help
. The container will install,
configure and start all required dependencies, such as Hadoop.
Options can be passed to Docker by setting the TINKERPOP_DOCKER_OPTS
environment variable. A speed boost can
be gained at the expense of memory by using tmpfs and the special directory /usr/src/tinkermem
.
TINKERPOP_DOCKER_OPTS="--tmpfs /usr/src/tinkermem:exec,mode=0755,rw,noatime,size=2000m"
TINKERPOP_DOCKER_OPTS="--sysctl net.ipv6.conf.all.disable_ipv6=1 --sysctl net.ipv6.conf.default.disable_ipv6=1"
A custom maven settings.xml can be supplied, for example, to point to a local proxy. Copy the settings.xml
to the
PROJECT_HOME/
directory. The Docker script will detect and copy it to the running container.
If the container is used to generate the user docs, it will start a web server and show the URL that is used to host the HTML docs.
After finishing all tasks, the script will immediately destroy the container.
IDE Setup with Intellij
This section refers specifically to setup within Intellij. TinkerPop has a module called gremlin-shaded
which
contains shaded dependencies for some libraries that are widely used and tend to introduce conflicts. To ensure
that Intellij properly interprets this module after importing the Maven pom.xml
perform the following steps:
-
Build
gremlin-shaded
from the command line withmvn clean install
. -
Right-click on the
gremlin-shaded
module in the project viewer of Intellij and select "Remove module". If this menu option is not available (as is the case in newer versions of Intellij - first noticed in 13.1.5), then open the "Maven Projects" side panel, right click thegremlin-shaded
module and select "Ignore Project". -
In the "Maven Projects" Tool window and click the tool button for "Reimport All Maven projects" (go to
View | Tool Windows | Maven Projects
on the main menu if this panel is not activated). -
At this point it should be possible to compile and run the tests within Intellij, but in the worst case, use
File | Invalidate Caches/Restart
to ensure that indices properly rebuild.
Note that it may be necessary to re-execute these steps if the gremlin-shaded
pom.xml
is ever updated.
Developers working on the neo4j-gremlin
module should enabled the include-neo4j
Maven profile in Intellij.
This will ensure that tests will properly execute within the IDE.
If Intellij complains about "duplicate sources" for the Groovy files when attempting to compile/run tests, then install the GMavenPlus Intellij plugin.
The gremlin-core
module uses a Java annotation processor to help support DSLs. To support this capability be sure
that:
-
File | Settings | Compiler | Annotation Processors
has the checkbox with the "Enable annotation processing" checked. Intellij should be able to detect the processor automatically on build. -
The
gremlin-core/target
directory should not be hidden andtarget/classes
,target/generated-sources
and `target/generated-test-sources should be marked as "Generated Sources Root". If they are not setup that way by Intellij by default then simply right-click on them use the "Mark Directory with" option to make the appropriate selections.
For Committers
![business gremlin](../../images/business-gremlin.png)
The guidelines that follow apply to those with commit access to the main repository:
Communication
TinkerPop has a user mailing list and a dev@tinkerpop.apache.org">developer mailing list. As a committer, it is a good idea to join both.
It would also be helpful to join the public TinkerPop HipChat room for developer discussion. This helps contributors to communicate in a more real-time way. Anyone can join as a guest, but for regular contributors it may be best to request that an Apache HipChat account be created.
Occasionally, online meetings via video conference are held. These meetings are schedule via the dev mailing list about a week before they are to occur to find a day and time that is available for those interested in attending. On the day of the meeting, the meeting organizer will create a Google Hangout (or similar video conferencing link) to post to the TinkerPop room in HipChat. At that point, all who are interested can attend. Meeting minutes should be taken and added to the Meetings section of this document using the pattern already established.
Release Notes
There is a two-pronged approach to maintaining the change log and preparing the release notes.
-
For work that is documented in JIRA, run the release notes report to include all of the tickets targeted for a specific release. This report can be included in the release announcement.
-
The manual change log (
CHANGELOG.asciidoc
) can be used to highlight large changes, describe themes (e.g. "We focused on performance improvements") or to give voice to undocumented changes.
Given the dependence on the JIRA report for generating additions to the CHANGELOG.asciidoc
,
which uses the title of the issue as the line presented in the release note report, titles should
be edited prior to release to be useful in that context. In other words, an issue title should
be understandable as a change in the fewest words possible while still conveying the gist of the
change.
Changes that break the public APIs should be marked with a "breaking" label and should be distinguished from other changes in the release notes.
Branches
TinkerPop has several release branches:
-
tp30
- 3.0.x (no longer maintained) -
tp31
- 3.1.x (no longer maintained) -
tp32
- 3.2.x (bug fixes and documentation updates only) -
tp33
- 3.3.x (current development) -
master
- 3.4.x (future development)
Changes to tp32
should merge to tp33
and then tp33
to master
. Please read more about this process in the
Pull Requests section.
Other branches may be created for collaborating on features or for RFC’s that other developers may want to inspect. It is suggested that the JIRA issue ID be used as the prefix, since that triggers certain automation, and it provides a way to account for the branch lifecycle, i.e. "Who’s branch is this, and can I delete it?"
For branches that are NOT associated with JIRA issues, developers should utilize their Apache ID as a branch name prefix. This provides a unique namespace, and also a way to account for the branch lifecycle.
Developers should remove their own branches when they are no longer needed.
Tags
Tags are used for milestones, release candidates, and approved releases. Please refrain from creating arbitrary tags, as they produce permanent clutter.
Issue Tracker Conventions
TinkerPop uses Apache JIRA as its issue tracker. JIRA is a very robust piece of software with many options and configurations. To simplify usage and ensure consistency across issues, the following conventions should be adhered to:
-
An issue’s "status" should generally be in one of two states:
open
orclosed
(reopened
is equivalent toopen
for our purposes).-
An
open
issue is newly created, under consideration or otherwise in progress. -
A
closed
issue is completed for purposes of release (i.e. code, testing, and documentation complete). -
Issues in a
resolved
state should immediately be evaluated for movement toclosed
- issue becomeresolved
by those who don’t have the permissions toclose
.
-
-
An issue’s "type" should be one of two options:
bug
orimprovement
.-
A
bug
has a very specific meaning, referring to an error that prevents usage of TinkerPop AND does not have a reasonable workaround. Given that definition, abug
should generally have very high priority for a fix. -
Everything else is an
improvement
in the sense that any other work is an enhancement to the current codebase.
-
-
The "component" should be representative of the primary area of code that it applies to and all issues should have this property set.
-
Issues are not assigned "labels" with two exceptions:
-
The "breaking" label which marks an issue as one that is representative of a change in the API that might affect users or providers. This label is important when organizing release notes.
-
The "deprecation" label which is assigned to an issue that includes changes to deprecate a portion of the API.
-
-
The "affects/fix version(s)" fields should be appropriately set, where the "fix version" implies the version on which that particular issue will completed. This is a field usually only set by committers.
-
The "priority" field can be arbitrarily applied with one exception. The "trivial" option should be reserved for tasks that are "easy" for a potential new contributor to jump into and do not have significant impact to urgently required improvements.
Code Style
Contributors should examine the current code base to determine what the code style patterns are and should match their style to what is already present. Of specific note however, TinkerPop does not use "import wildcards" - IDEs should be adjusted accordingly to not auto-wildcard the imports.
Build Server
Deprecation
When possible, committers should avoid direct "breaking" change (e.g. removing a method from a class) and favor deprecation. Deprecation should come with sufficient documentation and notice especially when the change involves public APIs that might be utilized by users or implemented by providers:
-
Mark the code with the
@Deprecated
annotation. -
Use javadoc to further document the change with the following content:
-
@deprecated As of release 3.3.3-SNAPSHOT, replaced by {@link SomeOtherClass#someNewMethod()}
- if the method is not replaced then the comment can simply read "not replaced". Additional comments that provide more context are encouraged. -
@see <a href="https://issues.apache.org/jira/browse/TINKERPOP-XXX">TINKERPOP-XXX</a>
- supply a link to the JIRA issue for reference - the issue should include the "deprecation" label.
-
-
Be sure that deprecated methods are still under test - consider using javadoc/comments in the tests themselves to call out this fact.
-
Create a new JIRA issue to track removal of the deprecation for future evaluation.
-
Update the "upgrade documentation" to reflect the API change and how the reader should resolve it.
The JIRA issues that track removal of deprecated methods should be periodically evaluated to determine if it is prudent to schedule them into a release.
Developing Tests
TinkerPop has a wide variety of test types that help validate its internal code as well as external provider code.
There are "unit tests" and "integration tests". Unit tests execute on standard runs of mvn clean install
. These
tests tend to run quickly and provide a reasonable level of coverage and confidence in the code base. Integration
tests are disabled by default and must be explicitly turned on with a special build property by adding
-DskipIntegrationTests=false
to the mvn
execution. Integration tests run slower and may require external
components to be running when they are executed. They are "marked" as separate from unit tests by inclusion of the
suffix "IntegrateTest".
Here are some other points to consider when developing tests:
-
Avoid use of
println
in tests and prefer use of a SLF4jLogger
instance so that outputs can be controlled in a standard way. -
If it is necessary to create files on the filesystem, do not hardcode directories - instead, use the
TestHelper
to create directory structures.TestHelper
will properly create file system structure in the appropriate build directory thus allowing proper clean-up between test runs. -
If writing tests in one of the test suites, like
gremlin-test
, it is important to remember that if a newGraph
instance is constructed within the test manually, that it be closed on exit of that test. Failing to do this cleanup can cause problems for some graph providers. -
Tests that are designed to use a
GraphProvider
implementation in conjunction withAbstractGremlinTest
and are in the/test
directory should not be named withTest
as the suffix, as this will cause them to execute in some environments without aGraphProvider
being initialized by a suite. These types of tests should be suffixed withCheck
instead. Please see NativeNeo4jStructureCheck for an example.
Gremlin Language Test Cases
Test cases for the Gremlin Language currently requires that the newly developed test be added in three places:
-
As a test written in Java in the
gremlin-test
module within the subpackages oforg.apache.tinkerpop.gremlin.process.traversal.step
-
As a test written in Gherkin in the
gremlin-test
module in the/features
subdirectory
When writing a Java test case for a Gremlin step, be sure to use the following conventions.
-
The name of the traversal generator should start with
get
, useX
for brackets,_
for space, and the Gremlin-Groovy sugar syntax.-
get_g_V_hasLabelXpersonX_groupXaX_byXageX_byXsumX_name()
-
-
When creating a test for a step that has both a barrier and sideEffect form (e.g.
group()
,groupCount()
, etc.), test both representations.-
get_g_V_groupCount_byXnameX()
-
get_g_V_groupCountXaX_byXnameX_capXaX()
-
-
The name of the actual test case should be the name of the traversal generator minus the
get_
prefix. -
The Gremlin-Groovy version of the test should use the sugar syntax in order to test sugar (as Gremlin-Java8 tests test standard syntax).
-
g.V.age.sum
-
-
Avoid using lambdas in the test case unless that is explicitly what is being tested as OLAP systems will typically not be able to execute those tests.
-
AbstractGremlinProcessTest
has various static methods to make writing a test case easy.-
checkResults(Arrays.asList("marko","josh"), traversal)
-
checkMap(new HashMap<String,Long>() {{ put("marko",1l); }}, traversal.next())
-
Gherkin tests follow some important conventions and have a sub-language that must be adhered to for the tests to
function properly. Note that Gherkin tests are designed to support the testing of GLVs and at some point will likely
replace the Java tests. If a new Java test is added and an associated Gherkin tests is not, the overall build will
fail the FeatureCoverageTest
of gremlin-test
which validates that all tests written in Java are also implemented
in Gherkin.
The basic syntax of a Gherkin test is as follows:
Scenario: g_VX1X_unionXrepeatXoutX_timesX2X__outX_name
Given the modern graph
And using the parameter v1Id defined as "v[marko].id"
And the traversal of
"""
g.V(v1Id).union(__.repeat(__.out()).times(2), __.out()).values("name")
"""
When iterated to list
Then the result should be unordered
| result |
| ripple |
| lop |
| lop |
| vadas |
| josh |
Scenario Name
The name of the scenario needs to match the name of the Java test. If it does not then the FeatureCoverageTest
will
fail.
Given
"Given" sets the context of the test. Specifically, it establishes the graph that will be used for the test. It conforms to the pattern of "Given the xxx graph" where the "xxx" may be one of the following:
-
empty
-
modern
-
classic
-
crew
-
sink
-
grateful
Never modify the data of any of the graphs except for the "empty" graph. The "empty" graph is the only graph that is guaranteed to be refreshed between tests. The "empty" graph maybe be modified by the traversal under test or by an additional "Given" option:
Given the empty graph
And the graph initializer of
"""
g.addV("person").property(T.id, 1).property("name", "marko").property("age", 29).as("marko").
addV("person").property(T.id, 2).property("name", "vadas").property("age", 27).as("vadas").
addV("software").property(T.id, 3).property("name", "lop").property("lang", "java").as("lop").
addV("person").property(T.id, 4).property("name","josh").property("age", 32).as("josh").
addV("software").property(T.id, 5).property("name", "ripple").property("lang", "java").as("ripple").
addV("person").property(T.id, 6).property("name", "peter").property("age", 35).as('peter').
addE("knows").from("marko").to("vadas").property(T.id, 7).property("weight", 0.5).
addE("knows").from("marko").to("josh").property(T.id, 8).property("weight", 1.0).
addE("created").from("marko").to("lop").property(T.id, 9).property("weight", 0.4).
addE("created").from("josh").to("ripple").property(T.id, 10).property("weight", 1.0).
addE("created").from("josh").to("lop").property(T.id, 11).property("weight", 0.4).
addE("created").from("peter").to("lop").property(T.id, 12).property("weight", 0.2)
"""
The above configuration will use the "empty" graph and initialize it with the specified traversal. In this case, that traversal loads the "empty" graph with the "modern" graph.
Once the graph for the test is defined, the context can be expanded to include parameters that will be applied to the traversal under test. Any variable value being used in the traversal under test, especially ones that require a specific type, should be defined as parameters. The structure for parameter definition looks like this:
Given the modern graph
And using the parameter v1Id defined as "v[marko].id"
In the above example, "v1Id" is the name of the parameter that will be used in the traversal. The end of that line in quotes is the value of that parameter and should use the type system notation that has been developed for the TinkerPop Gherkin tests. The type system notation ensures that different language variants have the ability to construct the appropriate types expected by the tests.
The syntax of the type notation involves a prefix character to help denote the type, a value between two square brackets, optionally suffixed with some additional notation depending on the primary type.
-
Edge - e[xxx] - The "xxx" should be replaced with a representation of an edge in the form of the
vertex_name-edgelabel→vertex_name
. This syntax may also include the.id
suffix which would indicate getting the edge identifier or the.sid
suffix which gets a string representation of the edge identifier. -
Lambda - c[xxx] - The "xxx" should contain a lambda written in Groovy.
-
List - l[xxx,yyy,zzz,…] - A comma separated collection of values that make up the list should be added to between the square brackets. These values respect the type system thus allowing for creation of lists of vertices, edges, maps, and any other available type.
-
Map - m[xxx] - The "xxx" should be replaced with a JSON string. Note that keys and values will be parsed using the type notation system so that it is possible to have maps containing arbitrary keys and values.
-
Numeric - d[xxx].y - The "xxx" should be replaced with a number. The suffix denoted by "y" should always be included to further qualify the type of numeric. The following options are available:
-
d - 32-bit Double
-
f - 32-bit Float
-
i - 32-bit Integer
-
l - 64-bit Long
-
m - Arbitrary-precision signed decimal numbers (i.e. BigDecimal in Java)
-
-
Path - p[xxx,yyy,zzz,…] - A comma separated collection of values that make up the
Path
should be added to between the square brackets. These values respect the type system thus allowing for creation ofPath
of vertices, edges, maps, and any other available type. -
Set - s[xxx,yyy,zzz,…] - A comma separated collection of values that make up the set should be added to between the square brackets. These values respect the type system thus allowing for creation of sets of vertices, edges, maps, and any other available type.
-
String - Any value not using the system notation will be interpreted as a string.
-
T - t[xxx] - The "xxx" should be replaced with a value of the
T
enum, such asid
orlabel
. -
Vertex - v[xxx] - The "xxx" should be replaced with the "name" property of a vertex in the graph. This syntax may include the
.id
suffix which would indicate getting the vertex identifier or the.sid
suffix which gets a string representation of the edge identifier.
Finally, specify the traversal under test with the "Given" option "and the traversal":
And the traversal of
"""
g.V(v1Id).union(__.repeat(__.out()).times(2), __.out()).values("name")
"""
It will be the results of this traversal that end up being asserted by Gherkin. When writing these test traversals,
be sure to always use the method and enum prefixes. For example, use __.out()
for an anonymous traversal rather
than just out()
and prefer Scope.local
rather than just local
.
If a particular test cannot be written in Gherkin for some reason or cannot be otherwise supported by a GLV, first, consider whether or not this test can be re-written in Java so that it will work for GLVs and then, second, if it cannot, then use the following syntax for unsupported tests:
Scenario: g_V_outXcreatedX_groupCountXxX_capXxX
Given an unsupported test
Then nothing should happen because
"""
The result returned is not supported under GraphSON 2.x and therefore cannot be properly asserted. More
specifically it has vertex keys which basically get toString()'d under GraphSON 2.x. This test can be supported
with GraphSON 3.x.
"""
When
The "When" options get the result from the traversal in preparation for assertion. There are two options to iterate:
-
"When iterated to list" - iterates the entire traversal into a list result that is asserted
-
"When iterated next" - gets the first value from the traversal as the result to be asserted
There should be only one "When" defined in a scenario.
Then
The "Then" options handle the assertion of the result. There are several options to consider:
-
"the result should have a count of xxx" - assumes a list value in the result and counts the number of values in it
-
"the result should be empty" - no results
-
"the result should be ordered" - the exact results and should appear in the order presented
-
"the result should be unordered" - the exact results but can appear any order
-
"the result should be of" - results can be any of the specified values and in any order (use when guarantees regarding the exact results cannot be pre-determined easily - see the
range()
step tests for examples)
These final three types of assertions mentioned above should be followed by a Gherkin table that has one column, where each row value in that column represents a value to assert in the result. These values are type notation respected as shown in the following example:
Then the result should be unordered
| result |
| ripple |
| lop |
| lop |
| vadas |
| josh |
Another method of assertion is to test mutations in the original graph. Again, mutations should only occur on the "empty" graph, but they can be validated as follows:
Scenario: g_V_outE_drop
Given the empty graph
And the graph initializer of
"""
g.addV().as("a").addV().as("b").addE("knows").to("a")
"""
And the traversal of
"""
g.V().outE().drop()
"""
When iterated to list
Then the result should be empty
And the graph should return 2 for count of "g.V()"
And the graph should return 0 for count of "g.E()"
Developing Benchmarks
Benchmarks are a useful tool to track performance between TinkerPop versions and also as tools to aid development decision making. TinkerPop uses OpenJDK JMH for benchmark development. The JMH framework provides tools for writing robust benchmarking code that avoid many of the pitfalls inherent in benchmarking JIT compiled code on the JVM. Example JMH benchmarks can be found here.
TinkerPop benchmarks live in the gremlin-benchmark
module and can either be run from within your IDE or as a standalone
uber-jar. The uber-jar is the JMH recommended approach and also makes it easy to distribute artifacts to various environments
to gather benchmarking numbers. Having said that, in most cases it should be sufficient to run it from within the IDE.
Benchmarks will not run by default because they are time consuming. To enable benchmarks during the test phase do
-DskipBenchmarks=false
. To change the number of warmup iterations, measurement iterations, and forks you can do
mvn clean test -DskipBenchmarks=false -DdefaultForks=5 -DmeasureIterations=20 -DwarmupIterations=20
. Benchmark results
will be output by default to the benchmarks
directory in JSON format.
Benchmarks may also be run from the command line using the JMH runner. Build the uber-jar and simply run
java -jar gremlin-benchmark-TP-VERSION.jar
. To see a list of JMH runner options, add the -h
flag.
The JUnit/JMH integration was inspired by the Netty projects microbenchmarking suite. Please refer to the Netty
docs for more details. Presently there are 3 abstract benchmark classes
that may be used as building blocks for your benchmarks; AbstractBenchmarkBase
, AbstractGraphBenchmark
, and
AbstractGraphMutateBenchmark
.
-
AbstractBenchmarkBase
- extend when your benchmark does not require a graph instance -
AbstractGraphBenchmark
- extend when you are benchmarking read operations against a graph -
AbstractGraphMutateBenchmark
- extend when you are benchmarking graph mutation operations eg.g.addV()
,graph.addVertex()
Review then Commit
Code modifications must go through a review-then-committ (RTC) process before being merged into a release branch. All committers should follow the pattern below, where "you" refers to the committer wanting to put code into a release branch.
-
Make a JIRA ticket for the software problem you want to solve (i.e. a fix).
-
Fork the release branch that the fix will be put into.
-
The branch name should be the JIRA issue identifier (e.g.
TINKERPOP-XXX
).
-
-
Develop your fix in your branch.
-
When your fix is complete and ready to merge, issue a pull request.
-
Be certain that the test suite is passing.
-
If you updated documentation, be sure that the
process-docs.sh
is building the documentation correctly.
-
-
Before you can merge your branch into the release branch, you must have at least 3 +1 consensus votes from other committers.
-
Please see the Apache Software Foundations regulations regarding Voting on Code Modifications.
-
-
Votes are issued by TinkerPop committers as comments to the pull request.
-
Once 3 +1 votes are received, you are responsible for merging to the release branch and handling any merge conflicts.
-
If there is a higher version release branch that requires your fix (e.g.
3.y-1.z
fix going to a3.y.z
release), be sure to merge to that release branch as well.
-
-
Be conscious of deleting your branch if it is no longer going to be used so stale branches don’t pollute the repository.
Note
|
These steps also generally apply to external pull requests from those who are not official Apache committers. In this case, the person responsible for the merge after voting is typically the first person available who is knowledgeable in the area that the pull request affects. Any additional coordination on merging can be handled via the pull request comment system. |
The following exceptions to the RTC (review-then-commit) model presented above are itemized below. It is up to the committer to self-regulate as the itemization below is not complete and only hints at the types of commits that do not require a review.
-
You are responsible for a release and need to manipulate files accordingly for the release.
-
Gremlin.version()
, CHANGELOG dates,pom.xml
version bumps, etc.
-
-
You are doing an minor change and it is obvious that an RTC is not required (would be a pointless burden to the community).
-
The fix is under the commit-then-review (CTR) policy and lazy consensus is sufficient, where a single -1 vote requires you to revert your changes.
-
Adding a test case, fixing spelling/grammar mistakes in the documentation, fixing LICENSE/NOTICE/etc. files, fixing a minor issue in an already merged branch.
-
When the committer chooses CTR, it is considered good form to include something in the commit message that explains that CTR was invoked and the reason for doing so. For example, "Invoking CTR as this change encompasses minor adjustments to text formatting."
Pull Requests
When submitting a pull request to one of the release branches, be sure it uses the following style:
-
The title of the pull request is the JIRA ticket number + "colon" + the title of the JIRA ticket.
-
The first line of the pull request message should contain a link to the JIRA ticket.
-
Discuss what you did to solve the problem articulated in the JIRA ticket.
-
Discuss any "extra" work done that go beyond the assumed requirements of the JIRA ticket.
-
Be sure to explain what you did to prove that the issue is resolved.
-
Test cases written.
-
Integration tests run (if required for the work accomplished).
-
Documentation building (if required for the work accomplished).
-
Any manual testing (though this should be embodied in a test case).
-
-
Notes about what you will do when you merge to the respective release branch (e.g. update CHANGELOG).
-
These types of "on merge tweaks" are typically done to extremely dynamic files to combat and merge conflicts.
-
-
If you are a TinkerPop committer, you can VOTE on your own pull request, so please do so.
A pull request will typically be made to a target branch. Assuming that branch is upstream of other release branches (e.g. a pull request made to for the branch containing 3.2.x must merge to the branch that releases 3.3.x), it is important to be sure that those changes are merged to the downstream release branches. Typically, this process is best handled by multiple pull requests: one to each release branch.
As an example, consider a situation where there is a feature branch named "TINKERPOP-1234" that contains a fix for
the tp32
branch:
`git checkout -b TINKERPOP-1234 tp32`
// do a bunch of stuff to implement TINKERPOP-1234 and commit/push
git checkout -b <TINKERPOP-1234-master> master
git merge TINKERPOP-1234
At this point, there are two branches, with the same set of commits going to tp32
and master
. Voting will occur
on both pull requests. After a successful vote, it is time to merge. If there are no conflicts, then simply git merge
both pull requests to their respective branches. If there are conflicts, then there is some added work to do - time to
rebase:
git checkout TINKERPOP-1234
git rebase origin/tp32
Depending on the conflict, it might be a good idea to re-test before going any further, otherwise:
git push origin TINKERPOP-1234 --force
Now, git rebase
has re-written the commit history, which makes a mess of the other pull request to master. This
problem is rectified by essentially re-issuing the PR:
git checkout TINKERPOP-1234-master
git reset --hard origin/master
git merge TINKERPOP-1234
Again, depending on the changes, it may make sense to re-test at this point, otherwise:
git push origin TINKERPOP-1234-master --force
It should not be safe to merge both pull requests to their release branches.
Important
|
Always take a moment to review the commits in a particular pull request. Be sure that they are all related to the work that was done and that no extraneous commits are present that cannot be explained. Ensuring a pull request only contains the expected commits is the responsibility of the committer as well as the reviewer. |
Dependencies
There are many dependencies on other open source libraries in TinkerPop modules. When adding dependencies or altering the version of a dependency, developers must consider the implications that may apply to the TinkerPop LICENSE and NOTICE files. There are two implications to consider:
-
Does the dependency fit an Apache approved license?
-
Given the addition or modification to a dependency, does it mean any change for TinkerPop LICENSE and NOTICE files?
Understanding these implications is important for insuring that TinkerPop stays compliant with the Apache 2 license that it releases under.
Regarding the first item, refer to the Apache Legal for a list of approved licenses that are compatible with the Apache 2 license.
The second item requires a bit more effort to follow. The Apache website offers a how-to guide on the approach to maintaining appropriate LICENSE and NOTICE files, but this guide is designed to offer some more specific guidance as it pertains to TinkerPop and its distribution.
To get started, TinkerPop has both "source" and "binary" LICENSE/NOTICE files:
Source LICENSE and NOTICE
As dependencies are not typically added to the source distribution (i.e. the source zip distribution), there is
typically no need to edit source LICENSE/NOTICE when editing a TinkerPop pom.xml
. These files only need to be edited
if the distribution has a file added to it. Such a situation may arise from several scenarios, but it would most
likely come from the addition of a source file from another library.
-
If the file being bundled is Apache licensed, then add an entry to NOTICE.
-
If the file being bundled is under a different approved license, then add an entry to LICENSE and include a copy of that LICENSE in the root
/licenses
directory of the code repository.
Binary LICENSE and NOTICE
The binary LICENSE/NOTICE is perhaps most impacted by changes to the various pom.xml
files. After altering the
pom.xml
file of any module, build both Gremlin Console and Gremlin Server and examine the contents of both binary
distributions, either:
-
target/apache-tinkerpop-gremlin-console-3.3.3-SNAPSHOT-distribution.zip
-
target/apache-tinkerpop-gremlin-server-3.3.3-SNAPSHOT-distribution.zip
Apache licensed software does not need to be included in LICENSE, but if the new dependency is an Apache-approved
license (e.g. BSD, MIT) then it should be added in the pattern already defined. A copy of the LICENSE should be
added to the <project>/src/main/static/licenses
directory of the code repository.
To determine if changes are required to the NOTICE, first check if the bundled jar has a NOTICE file in it (typically
found in /META-INF
directory of the jar).
-
If the bundled file does not have a NOTICE, then no changes to TinkerPop’s NOTICE are required.
-
If the NOTICE of the file being bundled is NOT Apache licensed, then there is no change to TinkerPop’s NOTICE.
-
If the NOTICE of the file being bundled is Apache licensed, then include the copyright notification in TinkerPop’s NOTICE.
-
If the NOTICE of the file being bundled is Apache licensed AND is an Apache Software Foundation project, then ONLY include the portion of that NOTICE in TinkerPop’s NOTICE that is unrelated to the Apache boilerplate NOTICE. If there is no such portion that is different than the boilerplate then this NOTICE can be excluded (i.e. don’t alter TinkerPop’s NOTICE at all).
Please refer to the Modifications to Notice section of the Apache "Licensing How-to" for more information.
Documentation
The documentation for TinkerPop is stored in the git repository in docs/src/
and are then split into several
subdirectories, each representing a "book" (or its own publishable body of work). If a new AsciiDoc file is added to
a book, then it should also be included in the index.asciidoc
file for that book, otherwise the preprocessor will
ignore it. Likewise, if a whole new book (subdirectory) is added, it must include an index.asciidoc
file to be
recognized by the AsciiDoc preprocessor.
Adding a book also requires a change to the root pom.xml
file. Find the "asciidoc" Maven profile and add a new
<execution>
to the asciidoctor-maven-plugin
configuration. For each book in docs/src/
, there should be a
related <execution>
that generates the HTML from the AsciiDoc. Follows the patterns already established by
the existing <execution>
entries, paying special attention to the pathing of the '<sourceDirectory>',
<outputDirectory>
and <imagesdir>
. Note that the <outputDirectory>
represents where the book will exist when
uploaded to the server and should preserve the directory structure in git as referenced in <sourceDirectory>
.
Please see the Building and Testing section for more information on how to generate the documentation.
Site
The content for the TinkerPop home page and related pages that make up the web site at tinkerpop.apache.org
is stored in the git repository under /docs/site
. In this way, it becomes easier for the community to provide content
presented there, because the content can be accepted via the standard workflow of a pull request. To generate the site
for local viewing, run bin/generate-home.sh
, which will build the site in target/site/
. PMC members can officially
publish the site with bin/publish-home.sh <username>
.
"Publishing" does not publish documentation (e.g. reference docs, javadocs, etc) and only publishes what is generated
from the content in /docs/site
. Publishing the site can be performed out of band with the release cycle and is no
way tied to a version. The master
branch should always be considered the "current" web site and publishing should
only happen from that branch.
Logging
TinkerPop uses SLF4j for logging and typically leans back on Log4j as the implementation. Configuring log outputs
for debugging purposes within tests can be altered by editing the log4j-test.properties
file in each module’s test
resources. That file gets copied to the target/test-classes
on build and surefire and failsafe plugins in maven
are then configured to point at that area of the file system for those configuration files. The properties files
can be edited to fine tune control of the log output, but generally speaking the current configuration is likely
best for everyone’s general purposes, so if changes are made please revert them prior to commit.
IO Documentation and Testing
The IO Documentation provides more details into GraphML, GraphSON
and Gryo with a special focus on the needs of developers who are working directly with these formats. GraphSON gets
the greatest focus here as it is used as the primary IO format for GLVs.
This documentation is largely generated from the gremlin-io-test
module found under gremlin-tools
. The
gremlin-io-test
module also includes a testing framework which validates that formats don’t break between TinkerPop
versions. Unfortunately, this module requires some maintenance to ensure that the documentation and tests both stay
updated.
The gremlin-io-test
module contains a set of files in the test resources that are statically bound to the version in
which they were generated. Older versions should never be modified. The only time changes to these resources should be
accepted should be for the current SNAPSHOT
version. The test resources are generated from the Model
class which
contains the objects that will undergo serialization for purpose of testing. Note that these same objects in the
Model
are also used to generate documentation.
To generate these test resources and documentation snippets based on the Model
, use this Maven command:
mvn clean install -pl :gremlin-io-test -Dio
This command will generate two directories in the /target
output directory of gremlin-io-test
: test-case-data
and dev-docs
. The contents of test-case-data
represents the serialized Model
objects that can be copied to the
test resources and the contents of the dev-docs
contains asciidoc snippets that can be copied to the IO documentation.
Generating data files in the fashion mentioned above with Maven is only good for versions of TinkerPop on the 3.3.x
line because the gremlin-io-test
module did not exist in 3.2.x. Of course, compatibility is still tested back to
those older versions. To generate test data from 3.2.x, there are Groovy scripts in the comments of the
gryo.asciidoc
and graphson.asciidoc
files that can copy/pasted to the Gremlin Console. They will generate the
batch of test files needed for gremlin-io-test
.
When does this command need to be executed?
-
If a new object is added to the
Model
- in this case, the newly created data files should be copied to the appropriate test resource directory for the currentSNAPSHOT
version and the appropriate asciidoc snippet added to the IO asciidocs. -
After the release of a new TinkerPop version - in this case, a new test resource directory should be created for the
SNAPSHOT
version and the generatedtest-case-data
copied in appropriately.
The second case, does require some additional discussion. When a new version is added the following classes will need to be updated in the following ways:
GryoCompatibility - Include new Gryo 1.0 and 3.0 enums for the current SNAPSHOT
.
V1D0_3_3_x("3.3.x", "1.0", "v1d0"),
V3D0_3_3_x("3.3.x", "3.0", "v3d0")
GraphSONCompatibility - Include new GraphSON enums for each of the various GraphSON configurations and versions.
V1D0_3_3_x("3.3.x", "1.0", "v1d0"),
V2D0_PARTIAL_3_3_x("3.3.x", "2.0", "v2d0-partial"),
V2D0_NO_TYPE_3_3_x("3.3.x", "2.0", "v2d0-no-types"),
V3D0_PARTIAL_3_3_x("3.3.x", "3.0", "v3d0");
GryoCompatibilityTest - Add the newly included GryoCompatibility
enums to the test parameters being careful to
match the appropriate "mapper" to the right version.
GraphSONUntypedCompatibilityTest - Add the newly included GraphSON 1.0 and 2.0 "untyped" enums to the test parameters being careful to match the appropriate "mapper" to the right version.
GraphSONTypedCompatibilityTest - Add the newly included GraphSON 3.0 and 2.0 "typed" enums to the test parameters being careful to match the appropriate "mapper" to the right version.
At this point, all of the IO tests are rigged up properly and assuming the test resources are available a standard
mvn clean install
should execute the compatibility tests and validate that everything is working as expected and
that there are no breaks in serialization processes.
For Providers
Release Process
This document describes the steps required to release a version of TinkerPop. The release is handled by a "release manager" (a committer fulfills this role), who ensures that the steps in this document are executed. The process is multi-phased and can therefore take several weeks to complete given the time needed for Apache voting and community feedback. Once a release point has been identified, the following phases represent the flow of "release":
-
Release manager key setup.
-
Pre-flight check.
-
Optionally, produce a release candidate for community feedback.
-
Submit the official release for PMC vote.
-
Release and promote.
Note
|
It might be helpful to use this document as generated from the currently release as opposed to one generate
from a previous version or from recent SNAPSHOT . When using one generated for release, all the "versions" in the
commands end up being set to the version that is being released, making cut and paste of those commands less labor
intensive and error prone.
|
Important
|
The following instructions assume that the release manager’s environment is setup
properly for release and includes a .glv file in gremlin-python as described in the Python Environment
section, so that the gremlin-python module builds in full.
|
Development Versions
A "development version" or snapshot (in Java parlance) is not an "official" release. Artifacts produced for a snapshot are solely for the convenience of providers and other developers who want to use the latest releases of TinkerPop. These releases do not require a VOTE and do not require a "release manager". Any PMC member can deploy them. It is important to note that these releases cannot be promoted outside of the developer mailing list and should not be recommended for use beyond the purpose of evaluation and testing.
Important
|
A development release must contain the suffix "-SNAPSHOT" in the pom.xml .
|
For JVM-based artifacts, simply use the following command:
mvn clean deploy
and artifacts will be pushed to the Apache Snapshot Repository. Python development artifacts must be generated and deployed separately with additional commands:
mvn clean install -Pglv-python
mvn deploy -pl gremlin-python -Dpypi
Python does not use the snapshot model the JVM does, however, the build is smart in that it will dynamically
generate a development version number for the Python artifacts when "-SNAPSHOT" is in the pom.xml
. The previous
command will push the development version to pypi for distribution.
Use the testpypi
test environment by updating the gremlin-python/pom.xml
to verify the mvn deploy
command works.
Important
|
The clean in the above commands is more important to the pypi deployment because the process will deploy
anything found in the target/python-packaged/dist directory. Since the names of the artifacts are based on
timestamps, they will not overwrite one another and multiple artifacts will get uploaded.
|
For .NET and NuGet, development artifacts can be created as follows:
mvn clean install -pl :gremlin-dotnet,:gremlin-dotnet-source -Dnuget
mvn deploy -pl :gremlin-dotnet-source -Dnuget
As with PyPi, NuGet does not support a snapshot model as Java does. The commands above will dynamically generate a
version number when "-SNAPSHOT" is in the pom.xml
. Use the staging.nuget.org
environment by updating the
gremlin-dot-source/pom.xml
to ensure the mvn deploy
command works.
Important
|
These commands will dynamically edit the gremlin-dotnet-source/Gremlin.Net.csproj . Take care to commit
or not commit changes related to that as necessary.
|
Release Manager Requirements
If this is your first time as release manager, you will need to setup keys for signing purposes per the Apache release process. Generally speaking, this will mean that you will need to generate a key-pair and then upload your public key to a public keyserver.
For a general overview of key basics, refer to this. For detailed step-by-step instructions, please follow the instructions here.
After completing the key-pair setup instructions, be sure to add yourself to the PGP signature
section of bin/validate-distribution.sh
.
echo -n " * PGP signature ... "
[ `gpg ${ZIP_FILENAME}.asc 2>&1 | grep -c '^gpg: Good signature from "Stephen Mallette <spmallette@apache.org>"$'` -eq 1 ] || \
[ `gpg ${ZIP_FILENAME}.asc 2>&1 | grep -c '^gpg: Good signature from "Marko Rodriguez <okram@apache.org>"$'` -eq 1 ] || \
[ `gpg ${ZIP_FILENAME}.asc 2>&1 | grep -c '^gpg: Good signature from "Theodore Ratte Wilmes (CODE SIGNING KEY) <twilmes@apache.org>"'` -eq 1 ] || \
{ echo "failed"; exit 1; }
echo "OK"
Pre-flight Check
The "pre-flight check" is a list of things performed by the release manager during the weeks leading up to a scheduled day to release. These checks will help to ensure that that release day goes smoothly by identifying problems up early and communicating with other members of the community.
-
Fourteen days before release, issue an email to the dev mailing list to remind the community of the pending release.
-
Note any important issues open in JIRA in that post.
-
Request review and update of the "upgrade documentation" and CHANGELOG.
-
-
Seven days before release, announce the code freeze on the dev mailing list to remind the community that the branch under release is protected. Tweaks to documentation and other odds and ends related to release are still allowed during this period.
-
At some point during the week:
-
Run the full integration test suite:
mvn clean install -DskipIntegrationTests=false -DincludeNeo4j
-
Deploy a final SNAPSHOT to the snapshot repository as well as GLV pre-releases.
-
Review LICENSE and NOTICE files to make sure that no changes are needed.
-
Review javadoc filters on the "Core API" docs to be sure nothing needs to change.
-
Review JIRA tickets in the release and ensure that:
-
All tickets categorized by having a "Component" assigned.
-
All tickets are either of type "Bug" or "Improvement".
-
All tickets where work was completed are "Closed"
-
Search for "closed the pull request" in comments for hints on possible tickets that were left open by mistake.
-
Look for tickets marked as "Resolved" as some users might not have rights to mark as "Closed" - convert these to "Closed".
-
-
All tickets not marked "Fixed", "Done", or "Implemented" for their Resolution should not have a Fix Version assigned (use common sense when reviewing these tickets before removing the Fix Version as it is possible the incorrect Resolution may have been assigned).
-
-
Update
gremlin-io-test
(from 3.3.0 on - this module was not available prior to the 3.3.x line):-
Build new test files with
mvn clean install -pl gremlin-tools/gremlin-io-test -Dio
-
If necessary copy relevant portions of IO dev documentation from
target/dev-docs
to the actual asciidoc file atdocs/src/dev/io
(this is basically just for GraphSON). -
Generate test files for other lines of TinkerPop releases that don’t have
gremlin-io-test
to automate that process (refers to the 3.2.x line):-
Scripts to "back generate" test data files can be found in the comments of
docs/src/dev/io/graphson.asciidoc`and `docs/src/dev/io/gryo.asciidoc
and can be pasted into a Gremlin Console instance that matches the target version expected of the test files. Once generated these files can be copied to the branch that requires them. -
It is worth double checking the output file counts against past versions as the number of files should match. Obviously, if new test types were added to the model then those would have to be accounted for. Update the scripts as necessary.
-
The
CompatibilitiesTest
will need to be updated - it is a troublesome test which needs to be adjusted on each release (doesn’t appear to be a good way to test all of its features without this hassle).
-
-
Commit and push all files to the repository after doing a build with
mvn clean install
to verify that there are no problems.
-
-
-
When all documentation changes are in place, use
bin/publish-docs.sh
to deploy a finalSNAPSHOT
representation of the docs and thus validate that there are no issues with the documentation generation process. Request review of the published documentation on the dev mailing list.
Release Candidate
A release candidate is an unofficial release that is represented by a tagged version in the Git repository. It is offered in cases where there is significant change in a particular version and the potential for upgrades and problems might be high. Release candidates do not require a vote thread. Lazy consensus is acceptable for confirming their deployment.
-
mvn clean install -DincludeNeo4j
-
mvn verify -DskipIntegrationTests=false -DincludeNeo4j
-
-
bin/publish-docs.sh <username>
- note that under a release candidate the documentation is published as SNAPSHOT -
mvn versions:set -DnewVersion=xx.yy.zz -DgenerateBackupPoms=false
to update the project files to reference a non-SNAPSHOT version -
pushd gremlin-console/bin; ln -fs ../target/apache-tinkerpop-gremlin-console-xx.yy.zz-standalone/bin/gremlin.sh gremlin.sh; popd
-
git diff
and review the updated files -
git commit -a -m "TinkerPop xx.yy.zz release"
andgit push
-
git tag -a -m "TinkerPop xx.yy.zz release" xx.yy.zz
andgit push --tags
-
mvn clean install
-
mvn versions:set -DnewVersion=xx.yy.zz-SNAPSHOT -DgenerateBackupPoms=false
to go back to SNAPSHOT -
pushd gremlin-console/bin; ln -fs ../target/apache-tinkerpop-gremlin-console-xx.yy.zz-SNAPSHOT-standalone/bin/gremlin.sh gremlin.sh; popd
-
git commit -a -m "Returned to xx.yy.zz-SNAPSHOT"
andgit push
-
Announce the release candidate to
dev
mailing list and await feedback -
Repeat as required or proceed to the next phase
Note that release candidates need not require the release of all artifacts. For example, if the risk is with one particular GLV, then a release candidate can be prepared of just that particular artifact. In those cases, a tag on the commit that represents the release candidate is sufficient and does not necessarily require that the versions be bumped to reflect the actual "-rc" version. In other words, if preparing a release candidate for .NET, then there is no need to go through the processing of bumping versions in Java artifacts and all GLVs. Nor is it necessary to alter the version of .NET to include the release candidate versioning. It can simply be altered locally by the release manager and deployed.
PMC Vote
This section describes the process that process that prepares a release for VOTE by the community. If there are multiple releases (as there usually are) being prepared, it may be best for downstream releases to wait for upstream releases to complete their process to assure that no last minute commits are required to get the upstream release completed. It is up to the discretion of the release managers to decide how they wish to proceed with respect to preparing releases in parallel or in a more serial fashion.
-
By this point, the testing performed during the code freeze should have validated the release. If however there are additional tests to perform that the release manager feels are relevant, they should be performed now. In other words, there is no need to rebuild the
SNAPSHOT
yet another time unless there are circumstances that would call its validity into question. -
Update
CHANGELOG.asciidoc
:-
Update the release date - the release date is the date of the vote. Double check the header to be sure that it is formatted properly - the easiest way to is to view the
CHANGELOG.asciidoc
to be sure that it produces a link in the GitHub viewer. -
Generate the JIRA release notes report for the current version and append them to the
CHANGELOG.asciidoc
.-
Use an "advanced" search to filter out JIRA issues already released on other versions. For example:
project = TINKERPOP and status = Closed AND fixVersion = 3.2.0 AND fixVersion not in (3.1.3, 3.1.2, 3.1.1, 3.1.0) ORDER BY type, Id ASC
. -
Consider use of an "Excel" export to organize and prepare the JIRA tickets to be pasted to
CHANGELOG.asciidoc
. This formula can help construct each line item for the CHANGELOG if columnA
is the issue number,B
is the issue title andD
is the label field:="* "&A2&" "&B2&(IF(D2="breaking"," *(breaking)*",""))
-
Be sure to include a link to other versions in the
CHANGELOG.asciidoc
that were previously released while the current release was under development as this new release will have those changes included within it. Please see 3.2.1 for an example.
-
-
Format "breaking" changes to be clearly marked (use JIRA and the "breaking" label to identify those)
-
-
Update "upgrade documentation":
-
Update the release date.
-
Update the link to
CHANGELOG.asciidoc
- this link may already be correct but will not exist until the repository is tagged.
-
-
Update homepage with references in
/site
to latest distribution and to other internal links elsewhere on the page.-
This step should only be performed by the release manager for the newest line of code (i.e. if release 3.3.x, 3.2.x and 3.1.x, then only do this step for 3.3.x, but update the site for 3.2.x and 3.1.x).
-
Update the
template/header-footer.html
. -
Update
index.html
. -
Update Downloads page, when moving "Current Releases" to "Archived Releases" recall that the hyperlink must change to point to version in the Apache Archives.
-
Preview changes locally with
bin/generate-home.sh
then commit changes to git.
-
-
mvn versions:set -DnewVersion=xx.yy.zz -DgenerateBackupPoms=false
to update project files to reference the non-SNAPSHOT version -
pushd gremlin-console/bin; ln -fs ../target/apache-tinkerpop-gremlin-console-xx.yy.zz-standalone/bin/gremlin.sh gremlin.sh; popd
-
git diff
and review the updated files -
mvn clean install
- need to build first so that the right version of the console is used withbin/publish-docs.sh
-
This step should update the Gremlin.Net project file and Gremlin Javascript package file with the newly bumped version.
-
-
git commit -a -m "TinkerPop xx.yy.zz release"
and push -
bin/process-docs.sh
and validate the generated documentation locally. Don’t rely on "BUILD SUCCESS" - scroll up through logs to ensure there were no errors and view the HTML directly. Code blocks that did not execute properly have a gray background and do not show the results of the commands. -
bin/publish-docs.sh <username>
- Note that this step requires no additional processing as the previous step handled document generation and this step now merely needs to upload what was generated. Note that this step will be responsible for generating javadoc and without that the binary distributions won’t contain that documentation. -
mvn deploy -Papache-release -DcreateChecksum=true -DskipTests
- deploy signed artifacts with checksums to Apache Nexus. -
Review generated artifacts to be sure they have both javadocs and asciidocs present (request another committer to review as well) then "close" the repo - if the repo is left open it will be automatically dropped after five days and closing the repo will allow it to stay available for a full ninety days which is more than enough time to complete a vote. Do NOT "release" the repository at this time.
-
Upload artifacts to
https://dist.apache.org/repos/dist/dev/tinkerpop
for[VOTE]
review.-
Use
svn rm
to delete past versions that were up for review in the same line of code. In other words, if uploading 3.2.3 then remove instances of 3.2.2 or any other past 3.2.x releases. -
svn co --depth empty https://dist.apache.org/repos/dist/dev/tinkerpop/ dev
andmkdir dev/xx.yy.zz
-
cp ~/.m2/repository/org/apache/tinkerpop/gremlin-console/xx.yy.zz/gremlin-console-xx.yy.zz-distribution.zip* dev/xx.yy.zz
-
cp ~/.m2/repository/org/apache/tinkerpop/gremlin-server/xx.yy.zz/gremlin-server-xx.yy.zz-distribution.zip* dev/xx.yy.zz
-
cp ~/.m2/repository/org/apache/tinkerpop/tinkerpop/xx.yy.zz/tinkerpop-xx.yy.zz-source-release.zip* dev/xx.yy.zz
-
`rm -f dev/*.md5
-
cd dev/xx.yy.zz
-
ls * | xargs -n1 -I {} echo "mv apache-tinkerpop-{} {}" | sed -e 's/distribution/bin/' -e 's/source-release/src/' -e 's/tinkerpop-tinkerpop/tinkerpop/' -e s'/^\(.*\) \(.*\) \(.*\)$/\1 \3 \2/' | /bin/bash
-
cd ..; svn add xx.yy.zz/; svn ci -m "TinkerPop xx.yy.zz release"
-
-
Execute
bin/validate-distribution.sh
and any other relevant testing. -
git tag -a -m "TinkerPop xx.yy.zz release" xx.yy.zz
andgit push --tags
-
Submit for
[VOTE]
atdev@tinkerpop.apache.org
(see email template below) -
Wait for vote acceptance (72 hours)
Release & Promote
-
Login to Apache Nexus and release the previously closed repository.
-
Deploy the GLVs
-
This build will likely occur from the tag for the release, so be sure to checkout the tag first before executing this step.
-
mvn clean install -DskipTests -Dnuget
-
mvn deploy -pl gremlin-python -DskipTests -Dpypi
-
mvn deploy -pl :gremlin-dotnet-source -DskipTests -Dnuget
-
mvn deploy -pl gremlin-javascript -DskipTests -Dnpm
-
-
svn co --depth empty https://dist.apache.org/repos/dist/dev/tinkerpop dev; svn up dev/xx.yy.zz
-
svn co --depth empty https://dist.apache.org/repos/dist/release/tinkerpop release; mkdir release/xx.yy.zz
-
Copy release files from
dev/xx.yy.zz
torelease/xx.yy.zz
. -
cd release; svn add xx.yy.zz/; svn ci -m "TinkerPop xx.yy.zz release"
-
Wait for Apache Sonatype to sync the artifacts to Maven Central at (http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/).
-
Report the release through reporter.apache.org (an email reminder should arrive shortly following the svn command above to do the release)
-
Wait for zip distributions to to sync to the Apache mirrors (i.e ensure the download links work from a mirror).
-
bin/publish-home.sh <username>
to publish the updated web site with new releases. -
Execute
bin/update-current-docs.sh
to migrate to the latest documentation set for/current
. -
This step should only occur after the website is updated and all links are working. If there are releases present in SVN that represents lines of code that are no longer under development, then remove those releases. In other words, if
3.2.0
is present and3.2.1
is released then remove3.2.0
. However, if3.1.3
is present and that line of code is still under potential development, it may stay. -
Announce release on
dev@
/gremlin-users@
mailing lists and tweet from@apachetinkerpop
Post-release Tasks
A number of administration tasks should be taken care of after release is public. Some of these items can be performed during the VOTE period at the release manager’s discretion, though it may be wise to wait until a successful VOTE is eminent before reopening development:
-
Perform JIRA administration tasks:
-
"Release" the current version and set the "release date"
-
If there is to be a follow on release in the current line of code, create that new version specifying the "start date"
-
-
Prepare Git administration tasks. Apply the following steps as needed per release branch:
-
Make the appropriate branching changes as required by the release and bump the version to
SNAPSHOT
withmvn versions:set -DnewVersion=xx.yy.zz-SNAPSHOT -DgenerateBackupPoms=false
. -
pushd gremlin-console/bin; ln -fs ../target/apache-tinkerpop-gremlin-console-xx.yy.zz-SNAPSHOT-standalone/bin/gremlin.sh gremlin.sh; popd
-
Update CHANGELOG and upgrade docs to have the appropriate headers for the next version.
-
mvn clean install -DskipTests
- need to build first so that the right version of the console is used withbin/publish-docs.sh
-
mvn deploy -DskipTests
- deploy the newSNAPSHOT
-
bin/process-docs.sh
and validate the generatedSNAPSHOT
documentation locally and thenbin/publish-docs.sh <username>
-
Commit and push the
SNAPSHOT
changes to git
-
-
Send email to advise that code freeze is lifted.
-
Generate a list of dead branches that will be automatically deleted and post them as a DISCUSS thread for review, then once consensus is reached removed those branches.
-
Set up the IO tests for the current
SNAPSHOT
as discussed in the IO Documentation and Testing Section
Email Templates
Release VOTE
Subject: [VOTE] TinkerPop xx.yy.zz Release
Hello,
We are happy to announce that TinkerPop xx.yy.zz is ready for release.
The release artifacts can be found at this location:
https://dist.apache.org/repos/dist/dev/tinkerpop/xx.yy.zz/
The source distribution is provided by:
apache-tinkerpop-xx.yy.zz-src.zip
Two binary distributions are provided for user convenience:
apache-tinkerpop-gremlin-console-xx.yy.zz-bin.zip
apache-tinkerpop-gremlin-server-xx.yy.zz-bin.zip
The GPG key used to sign the release artifacts is available at:
https://dist.apache.org/repos/dist/dev/tinkerpop/KEYS
The online docs can be found here:
http://tinkerpop.apache.org/docs/xx.yy.zz/ (user docs)
http://tinkerpop.apache.org/docs/xx.yy.zz/upgrade/ (upgrade docs)
http://tinkerpop.apache.org/javadocs/xx.yy.zz/core/ (core javadoc)
http://tinkerpop.apache.org/javadocs/xx.yy.zz/full/ (full javadoc)
The tag in Apache Git can be found here:
https://git-wip-us.apache.org/repos/asf?p=tinkerpop.git;XXXXXXXXXXXXXXXXXX
The release notes are available here:
https://github.com/apache/tinkerpop/blob/master/CHANGELOG.asciidoc#XXXXXXXXXXXXXXXXXX
The [VOTE] will be open for the next 72 hours --- closing <DayOfTheWeek> (<Month> <Day> <Year>) at <Time> <TimeZone>.
My vote is +1.
Thank you very much,
<TinkerPop Committer Name>
Dev Release RESULT VOTE
Subject: [RESULT][VOTE] TinkerPop xx.yy.zz Release
This vote is now closed with a total of X +1s, no +0s and no -1s. The results are:
BINDING VOTES:
+1 (X -- list of voters)
0 (0)
-1 (0)
NON-BINDING VOTES:
+1 (X -- list of voters)
0 (0)
-1 (0)
Thank you very much,
<TinkerPop Committer Name>
General Release Announcement
The template below refers to the "name of release line" and the "release line logo". Every release line has a name and logo. For example, 3.1.x had the name, "A 187 On The Undercover Gremlinz" and the logo shown here in the upgrade documentation.
Subject: TinkerPop xx.yy.zz Released: [name of release line]
Hello,
TinkerPop xx.yy.zz has just been released. [some text to introduce the release - e.g. whether or not
there is breaking change, an important game-changing feature or two, etc.]
The release artifacts can be found at this location:
https://www.apache.org/dyn/closer.lua/tinkerpop/xx.yy.zz/apache-tinkerpop-gremlin-console-xx.yy.zz-bin.zip
https://www.apache.org/dyn/closer.lua/tinkerpop/xx.yy.zz/apache-tinkerpop-gremlin-server-xx.yy.zz-bin.zip
https://www.apache.org/dyn/closer.lua/tinkerpop/xx.yy.zz/apache-tinkerpop-xx.yy.zz-src.zip
The online docs can be found here:
http://tinkerpop.apache.org/docs/xx.yy.zz/reference/ (user docs)
http://tinkerpop.apache.org/docs/xx.yy.zz/upgrade/#XXXXXXXXXXXXXXXXXX (upgrade docs)
http://tinkerpop.apache.org/javadocs/xx.yy.zz/core/ (core javadoc)
http://tinkerpop.apache.org/javadocs/xx.yy.zz/full/ (full javadoc)
http://tinkerpop.apache.org/docs/xx.yy.zz/some-new-content/ (some new content) [NEW!]
The release notes are available here:
https://github.com/apache/tinkerpop/blob/xx.yy.zz/CHANGELOG.asciidoc#XXXXXXXXXXXXXXXXXX
The Central Maven repo has sync'd as well:
https://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/xx.yy.zz/
Python artifacts are available in pypi:
https://pypi.python.org/pypi/gremlinpython/xx.yy.zz
.NET artifacts are available in NuGet:
https://www.nuget.org/packages/Gremlin.Net/xx.yy.zz
Javascript artifacts are available in npm:
https://www.npmjs.com/package/gremlin
[include the release line logo image]
Administration
New Committers
When a candidate is identified by a PMC member as someone who might be a good official committer to TinkerPop, the PMC member should open a DISCUSS thread on the private TinkerPop mailing list. The thread should provide some background and reasoning for why that member believes the candidate would be a good committer. Given enough time for feedback on the candidate and presuming there is still positive interest in doing so, a VOTE thread on the private TinkerPop mailing list is started to get the official stance. As per usual, the VOTE will be made open for no less than 72 hours.
If the VOTE closes with a successful positive vote to make the candidate a committer, then send the following email to the candidate and copy the private TinkerPop mailing list:
SUBJECT: Invitation to become TinkerPop committer: [candidate name]
Hello,
The TinkerPop Project Management Committee (PMC) hereby offers you committer
privileges to the project. These privileges are offered on the understanding
that you'll use them reasonably and with common sense. We like to work on
trust rather than unnecessary constraints.
Being a committer enables you to more easily make changes without needing to
go through the patch submission process.
Being a committer does not require you to participate any more than you already
do. It does tend to make one even more committed. You will probably find that
you spend more time here.
Of course, you can decline and instead remain as a contributor, participating
as you do now.
A. This personal invitation is a chance for you to accept or decline in
private. Either way, please let us know in reply to the
private@tinkerpop.apache.org address only.
B. If you are accepting, the next step is to register an iCLA with the Apache
Software Foundation:
1. Details of the iCLA and the forms are found through this link:
http://www.apache.org/licenses/#clas.
2. The form (text or PDF version) provides instructions for its completion
and return to the Secretary of the ASF at secretary@apache.org.
3. When you complete iCLA ensure that you include "Apache TinkerPop" in the
"notify project" project field and choose a preferred unique Apache id.
Look to see if your preferred id is already taken at
http://people.apache.org/committer-index.html. This will allow the
Secretary to notify the PMC when your iCLA has been recorded.
When recording of your iCLA is noticed, you will receive a follow-up message
with the next steps for establishing you as a committer.
Assuming the iCLA included items from step 3, the secretary will handle account creation, otherwise it will be up to the PMC chair to handle such things. Once the account is established the PMC chair will then need to:
-
Validate that the iCLA is on file either through svn or through people.apache.org.
-
Request that the account be created. The PMC Chair may do this through the Account Request Form.
-
Once verified, provide the new committer access to the repository, which is most easily done through Whimsy.
-
Send an announcement email to the developer and user mailing lists with the template below.
SUBJECT: New Committer: [committer name]
The Project Management Committee (PMC) for Apache TinkerPop has asked
[committer name] to become a committer and we are pleased to announce their
acceptance.
[describe the nature of the committer's work in the community]
Being a committer enables easier contribution to the project since there is no
need to work via the patch submission process. This should enable better
productivity.
Finally, new committers should be sent an email that covers some of the administrative elements of their new role:
-
TinkerPop Developer Documentation
-
Bring specific attention to the "committer" section which describes our general policies.
-
New PMC Members
The process for bringing on new PMC members is not so different from the one for new committers. The process begins with a DISCUSS thread to the private mailing list for building consensus followed by a VOTE thread to confirm. Presuming the new PMC member is a committer already (which is mostly likely for TinkerPop), there should be no need for any additional paperwork. On successful vote however, a NOTICE should be sent to the board@apache.org (copying private@tinkerpop.apache.org):
Subject [NOTICE] [name of PMC new member] for Apache TinkerPop PMC
Apache TinkerPop proposes to invite [name of PMC new member] ([ApacheId]) to join the PMC.
The vote result is available here:
[include the link to the vote result thread from private]
If the candidate does not (yet) have an Apache account, then please note that fact in the notification email. Provide the board 72 hours to respond with objection before extending the invite to the new PMC member. Once that time has elapsed, the invite may be sent to the new PMC member. If they accept, then the PMC chair should refer to the How to Add a New PMC Member section in the Apache docs for how to complete the process. Use Whimsy to update PMC membership.
Board Reports
The PMC Chair is responsible for submitting a report to the board on a quarterly basis. TinkerPop reports on the following months: January, April, July, October. The report has the following format:
## Description:
Apache TinkerPop is a graph computing framework for both graph databases
(OLTP) and graph analytic systems (OLAP).
## Activity:
<discuss general project health, project direction, community growth/activity, etc.>
## Issues:
There are no issues requiring board attention at this time.
## Releases:
- 3.3.3-SNAPSHOT (<month> <day>, <year>)
- 3.3.3-SNAPSHOT (<month> <day>, <year>)
## PMC/Committer:
- Last PMC addition was <name> - <month> <year>
- Last committer addition was <name> - <month> <year>
## Links
[1] <hyperlink to external reference, if a reference was made in the report>
A draft of the report should be sent to the TinkerPop developer mailing list for review at least three days prior to
submitting to the board. The final report should be sent in plain-text format to board@apache.org
and should be
committed to the appropriate meeting agenda in SVN at:
https://svn.apache.org/repos/private/foundation/board
Meetings
November 23, 2015
The meeting was scheduled for 12:00pm EST, started at 12:05pm when sufficient attendance to constitute a quorum was reached and was held via Google Hangout hosted by Marko Rodriguez. Meeting adjourned at 2:35pm EST.
Committers/PPMC Members
-
Daniel Kuppitz
-
Stephen Mallette
-
Dylan Millikin
-
Jason Plurad
-
Marko Rodriguez
-
Ted Wilmes
Guests
None
Minutes
-
Performed a full review of the list of unresolved tickets in JIRA. The summary of the changes can be found below. Note that releases and assignments made today are not meant to be set in stone. The assignments could change given community discussion or unforeseen events.
-
Decided to add version
3.2.0-incubating
to JIRA. This version was added to help better categorize the available issues. It was explicitly decided that no development was expected to be performed on these issue at this time and that we would wait until3.1.1-incubating
was released before thinking too hard on that body of work. This decision means that themaster
branch will continue to be the current branch of development for the time being and be bound to3.1.x
line of code. -
Discussed use of JIRA dependency links to show "blockers" and dependencies among different JIRA tickets.
-
Discussed a target date for upcoming releases:
-
3.1.1-incubating
- February 1, 2016 -
3.2.0-incubating
- May, 1, 2016
-
The following tickets were assigned to 3.1.1-incubating
:
ID | Description | Type |
---|---|---|
TINKERPOP-379 |
MessageScope.Local.setStaticMessage(M msg) |
Improvement |
TINKERPOP-659 |
GraphConfiguration Class |
Improvement |
TINKERPOP-736 |
Automatic Traversal rewriting |
Improvement |
TINKERPOP-763 |
IsStep broken when profiling is enabled. |
Bug |
TINKERPOP-818 |
Consider a P.instanceOf() |
Improvement |
TINKERPOP-824 |
Do we need runtime BigDecimal in more places? |
Improvement |
TINKERPOP-842 |
MapReduceHelper should sort respective of the number of reduce tasks |
Improvement |
TINKERPOP-859 |
Provide a more general way to set log levels in plugins |
Improvement |
TINKERPOP-860 |
Bindings applied to the PluginAcceptor should appear to Gremlin Server |
Improvement |
TINKERPOP-882 |
Develop a less error prone way for rewriting strategies |
Improvement |
TINKERPOP-886 |
Allow any GraphReader/Writer to be persistence engine for TinkerGraph |
Improvement |
TINKERPOP-891 |
Re-examine Sandboxing Abstractions |
Improvement |
TINKERPOP-893 |
Validate dependency grabs that have TinkerPop dependencies |
Improvement |
TINKERPOP-912 |
Improve the ability to embed Gremlin Server with Channelizer injection |
Improvement |
TINKERPOP-920 |
Test case needed for ensuring same cardinality for key. |
Improvement |
TINKERPOP-927 |
bin/publish-docs.sh should only upload diffs. |
Improvement |
TINKERPOP-930 |
Tie Alias to Transaction Manager in Gremlin Server |
Improvement |
TINKERPOP-932 |
Add ability to cancel script execution associated with a Gremlin Server Session |
Improvement |
TINKERPOP-937 |
Move the implementations sections of the primary documentation to "provider tutorials." |
Improvement |
TINKERPOP-938 |
Add a "clear SNAPSHOT jars" section to the process-docs.sh. |
Improvement |
TINKERPOP-939 |
Neo4jGraph should support HighAvailability (Neo4jHA). |
Improvement |
TINKERPOP-941 |
Improve error message for wrong order().by() arguments |
Improvement |
TINKERPOP-943 |
Warn if Gremlin Server is running prior to generating docs |
Improvement |
TINKERPOP-951 |
Barrier steps provide unexpected results in Gremlin OLAP |
Bug |
TINKERPOP-954 |
Consistent test directory usage |
Improvement |
TINKERPOP-956 |
Connection errors tend to force a complete close of the channel |
Improvement |
TINKERPOP-958 |
Improve usability of .profile() step. |
Improvement |
TINKERPOP-964 |
Test XXXGraphComputer on a Hadoop2 cluster (non-pseudocluster). |
Improvement |
TINKERPOP-967 |
Support nested-repeat() structures |
Improvement |
TINKERPOP-973 |
BLVP shouldn’t clear configuration properties |
Bug |
TINKERPOP-976 |
Fail earlier if invalid version is supplied in validate-distribution.sh |
Bug |
TINKERPOP-978 |
Native TinkerGraph Serializers for GraphSON |
Improvement |
TINKERPOP-979 |
ComputerVerificationStrategy not picking up Order local traversal |
Bug |
TINKERPOP-980 |
Add a service script or daemon mode in the distribution |
Improvement |
TINKERPOP-981 |
Deprecate support for credentialsDbLocation in Gremlin Server Config |
Improvement |
TINKERPOP-983 |
Provide a way to track open Graph instances in tests |
Improvement |
TINKERPOP-984 |
Use GraphProvider for id conversion in Groovy Environment test suite |
Improvement |
TINKERPOP-986 |
Manage binary LICENSE/NOTICE files with bin/checkLicenseNotice.sh |
Improvement |
TINKERPOP-988 |
SparkGraphComputer.submit shouldn’t use ForkJoinPool.commonPool |
Improvement |
The following items were identified for 3.2.0-incubating
:
ID | Description | Type |
---|---|---|
TINKERPOP-331 |
TraverserConverterStep (proposal) |
Improvement |
TINKERPOP-620 |
Commutative Step Marker interface |
Improvement |
TINKERPOP-669 |
Consider Required TraversalStrategies |
Improvement |
TINKERPOP-691 |
TP3 is too prescriptive in exception |
Improvement |
TINKERPOP-740 |
Serializer Handshake |
Improvement |
TINKERPOP-743 |
Support "barrier syntax" in step labels. |
Improvement |
TINKERPOP-761 |
Some basic mathematical functions / steps |
Improvement |
TINKERPOP-786 |
Patterns for DSL Development |
Improvement |
TINKERPOP-789 |
Choose then Enforce Semantics for Graph.close() |
Improvement |
TINKERPOP-790 |
Implement AutoCloseable on TraversalSource |
Improvement |
TINKERPOP-800 |
[Proposal] Domain/Range checking during traversal construction. |
Improvement |
TINKERPOP-813 |
[Proposal] Make the Gremlin Graph Traversal Machine and Instruction Set Explicit |
Improvement |
TINKERPOP-819 |
Mapping Cardinality Interface |
Improvement |
TINKERPOP-831 |
How should OLAP treat Collection<Element> objects? No contract is specified. |
Bug |
TINKERPOP-844 |
PropertyMapStep should reuse PropertiesStep |
Improvement |
TINKERPOP-864 |
Operator.mean would be nice. |
Improvement |
TINKERPOP-871 |
RuntimeStrategy as the general model for all such execution time rewrites/re-orders |
Improvement |
TINKERPOP-872 |
Remove GroupCountStep in favor of new Reduce-based GroupStep |
Improvement |
TINKERPOP-873 |
FoldStep should default to using BulkSet for efficiency. |
Improvement |
TINKERPOP-878 |
Refactor Gremlin Server integration tests to be Client parameterized |
Improvement |
TINKERPOP-889 |
Support for partitioned vertices in GraphComputer |
Improvement |
TINKERPOP-890 |
Remove the concept of branch/ package. |
Improvement |
TINKERPOP-919 |
Features needs to specify whether 2 vertex properties with same key/value is allowed. |
Improvement |
TINKERPOP-940 |
Convert LocalTraversals to MatchSteps in OLAP and Solve the StarGraph Problem |
Improvement |
TINKERPOP-942 |
Use EventStrategy to solve OLAP bulk mutation of OLTP. |
Improvement |
TINKERPOP-944 |
Consider deprecating or better enforcing Graph.Exceptions.elementNotFound |
Improvement |
TINKERPOP-946 |
Traversal respecting Thread.interrupt() |
Improvement |
TINKERPOP-947 |
Enforce semantics of threaded transactions as manual |
Improvement |
TINKERPOP-960 |
Add a Bulk class which is used by Traverser |
Improvement |
TINKERPOP-962 |
Provide "vertex query" selectivity when importing data in OLAP. |
Improvement |
TINKERPOP-963 |
SubgraphTraversalAnalyzer to determine what is really required from a traversal. |
Improvement |
TINKERPOP-966 |
Support reversible traversals in MatchStep (and respective MatchAlgorithms) |
Improvement |
TINKERPOP-969 |
respawn |
Improvement |
TINKERPOP-971 |
TraversalSource should be fluent like GraphComputer |
Improvement |
TINKERPOP-974 |
Saving headless traversals for reuse (clone Iterator Fun) |
Improvement |
TINKERPOP-982 |
valuesDecr, valuesIncr, keysDecr, and valuesDecr is lame. |
Improvement |
The following issues were simply closed during review - the reasons for closing can be found in the comments of the issues themselves:
ID | Description | Type |
---|---|---|
TINKERPOP-375 |
Better Methods for Managing ClassPath for Plugins |
Improvement |
TINKERPOP-487 |
User Supplied Ids and IO |
Improvement |
TINKERPOP-509 |
Subgraph support for VertexProgram |
|
TINKERPOP-604 |
DetachedEdge.attach(Vertex) is too slow. |
Improvement |
TINKERPOP-795 |
Provide Traverser.setPath() |
Improvement |
TINKERPOP-862 |
Add a TraversalSourceStrategy that provides "locked" values. |
Improvement |
TINKERPOP-894 |
Shorthand for install of TinkerPop dependencies |
Improvement |
TINKERPOP-936 |
Check feature requirements before opening graph during tests |
Improvement |
October 29, 2015
The meeting was scheduled for 1:00pm EST, started on time and was held via Google Hangout hosted by Stephen Mallette. Meeting adjourned at 1:45pm EST.
Committers/PPMC Members
-
Daniel Kuppitz
-
Stephen Mallette
-
Marko Rodriguez
Guests
None
Minutes
-
Reviewed the scope of 3.1.0-incubating in JIRA in the context of the upcoming release date on November 16, 2015.
-
It was noted that with the new one week code freeze policy that the real cut-off date for work is November 9, 2015.
-
There was general consensus on the call that work should be accomplished in such a way that the code review process not drag into the code freeze time period. In other words, pull requests to the release branch should be completed well ahead of the 9th.
Upon review of the tickets in JIRA assigned to 3.1.0-incubating, the following were removed from the 3.1.0-incubating roadmap:
ID | Description | Removal Reasoning |
---|---|---|
TINKERPOP-891 |
Re-examine Sandboxing Abstractions |
Low priority and time limitations. |
TINKERPOP-890 |
Remove the concept of branch/ package. |
Awaiting step library definition in 3.2.0-incubating. |
TINKERPOP-873 |
FoldStep should default to using BulkSet for efficiency. |
Awaiting step library definition in 3.2.0-incubating. |
TINKERPOP-864 |
Operator.mean would be nice. |
Awaiting step library definition in 3.2.0-incubating. |
TINKERPOP-862 |
Add a TraversalSourceStrategy that provides "locked" values. |
Low priority and time limitations. |
TINKERPOP-854 |
Remove PropertyMapStep in favor of multi-instruction construction. |
Non-trivial given time limitations. |
TINKERPOP-789 |
Choose then Enforce Semantics for Graph.close() |
Non-trivial given time limitations. |
TINKERPOP-768 |
MatchStep in OLAP should be smart about current vertex. |
Non-trivial given time limitations. |
TINKERPOP-705 |
Make use of a MemoryRDD instead of Accumulators in SparkGraphComputer? |
Non-trivial given time limitations. |
TINKERPOP-659 |
GraphConfiguration Class |
Non-trivial given time limitations. |
TINKERPOP-623 |
More output for OLAP jobs |
Not clearly defined given time limitations. |
TINKERPOP-620 |
Commutative Step Marker interface |
Awaiting step library definition in 3.2.0-incubating |
TINKERPOP-550 |
Gremlin IO needs to support both OLTP and OLAP naturally. |
Not clearly defined given time limitations. |
TINKERPOP-479 |
Consider Providing "getOrCreate" Functionality |
Not clearly defined given time limitations. |