------
 Developers centre - Testing Plugins Strategies
 ------
 Vincent Siveton
 ------
 2008-01-01
 ------

~~ Licensed to the Apache Software Foundation (ASF) under one
~~ or more contributor license agreements.  See the NOTICE file
~~ distributed with this work for additional information
~~ regarding copyright ownership.  The ASF licenses this file
~~ to you under the Apache License, Version 2.0 (the
~~ "License"); you may not use this file except in compliance
~~ with the License.  You may obtain a copy of the License at
~~
~~   http://www.apache.org/licenses/LICENSE-2.0
~~
~~ Unless required by applicable law or agreed to in writing,
~~ software distributed under the License is distributed on an
~~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~~ KIND, either express or implied.  See the License for the
~~ specific language governing permissions and limitations
~~ under the License.

~~ NOTE: For help with the syntax of this file, see:
~~ http://maven.apache.org/doxia/references/apt-format.html

Introduction

 Currently, Maven only supports unit testing out of the box. This document is intended to help Maven Developers to
 test Plugins with Unit Tests, Integration Tests or Functional tests.

 <<Note: There are a lot of different ways to test a Maven plugin.>>  For a review of different strategies and tools, please refer to {{{http://docs.codehaus.org/display/MAVENUSER/Review+of+Plugin+Testing+Strategies}Review of Plugin Testing Strategies}}

Testing Styles: Unit Testing vs. Functional/Integration Testing

 A unit test attempts to verify a mojo as an isolated unit, by mocking out the rest of the Maven environment.  A mojo unit test does not attempt to run your plugin in the context of a real Maven build.  Unit tests are designed to be fast.

 A functional/integration test attempts to use a mojo in a real Maven build, by launching a real instance of Maven in a real project.  Normally this requires you to construct special dummy Maven projects with real POM files.  Often this requires you to have already installed your plugin into your local repository so it can be used in a real Maven build.  Functional tests run much more slowly than unit tests, but they can catch bugs that you may not catch with unit tests.

 The general wisdom is that your code should be mostly tested with unit tests, but should also have some functional tests.

Unit Tests

* Using JUnit alone

 In principle, you can write a unit test of a plugin Mojo the same way you'd write any other JUnit test case, by writing a class that <<<extends TestCase>>>.

 However, most mojos need more information to work properly.  For example, you'll probably need to inject a reference to a MavenProject, so your mojo can query project variables.

* Using PlexusTestCase

 Mojo variables are injected using Plexus, and many Mojos are written to take specific advantage of the Plexus container (by executing a lifecycle or having various injected dependencies).

 If you all you need is Plexus container services, you can write your class with <<<extends PlexusTestCase>>> instead of TestCase.

 With that said, if you need to inject Maven objects into your mojo, you'll probably prefer to use the maven-plugin-testing-harness.

* maven-plugin-testing-harness

 The {{{/plugin-testing/maven-plugin-testing-harness/}maven-plugin-testing-harness}}
 is explicitly intended to test the <<<org.apache.maven.reporting.AbstractMavenReport#execute()>>> implementation.

 In general, you need to include <<<maven-plugin-testing-harness>>> as dependency, and create a *MojoTest (by convention)
 class which <<<extends AbstractMojoTestCase>>>.

+-----+
...
  <dependencies>
    ...
    <dependency>
      <groupId>org.apache.maven.shared</groupId>
      <artifactId>maven-plugin-testing-harness</artifactId>
      <version>1.0-beta-1</version>
      <scope>test</scope>
    </dependency>
    ...
  </dependencies>
...
+-----+
+-----+
public class YourMojoTest
    extends AbstractMojoTestCase
{
    /**
     * @see junit.framework.TestCase#setUp()
     */
    protected void setUp() throws Exception
    {
        // required for mojo lookups to work
        super.setUp();
    }

    /**
     * @throws Exception
     */
    public void testMojoGoal() throws Exception
    {
        File testPom = new File( getBasedir(),
          "src/test/resources/unit/basic-test/basic-test-plugin-config.xml" );

        YourMojo mojo = (YourMojo) lookupMojo( "yourGoal", testPom );

        assertNotNull( mojo );
    }
}
+-----+

 For more information, please refer to {{{http://cwiki.apache.org/confluence/display/MAVENOLD/Maven+Plugin+Harness}Maven Plugin Harness Wiki}}

Integration/Functional testing

* maven-verifier

 maven-verifier tests are run using JUnit or TestNG, and provide a simple class allowing you to launch Maven and assert on its log file and built artifacts.  It also provides a ResourceExtractor, which extracts a Maven project from your src/test/resources directory into a temporary working directory where you can do tricky stuff with it.

 Maven itself uses maven-verifier to run its core integration tests.  For more information, please refer to {{{http://cwiki.apache.org/confluence/display/MAVENOLD/Creating+a+Maven+Integration+Test}Creating a Maven Integration Test}}.

+-----+
public class TrivialMavenVerifierTest extends TestCase
{
    public void testMyPlugin()
        throws Exception
    {
        // Check in your dummy Maven project in /src/test/resources/...
        // The testdir is computed from the location of this
        // file.
        File testDir = ResourceExtractor.simpleExtractResources( getClass(), "/my-dummy-maven-project" );

        Verifier verifier;

        /*
         * We must first make sure that any artifact created
         * by this test has been removed from the local
         * repository. Failing to do this could cause
         * unstable test results. Fortunately, the verifier
         * makes it easy to do this.
         */
        verifier = new Verifier( testDir.getAbsolutePath() );
        verifier.deleteArtifact( "org.apache.maven.its.itsample", "parent", "1.0", "pom" );
        verifier.deleteArtifact( "org.apache.maven.its.itsample", "checkstyle-test", "1.0", "jar" );
        verifier.deleteArtifact( "org.apache.maven.its.itsample", "checkstyle-assembly", "1.0", "jar" );

        /*
         * The Command Line Options (CLI) are passed to the
         * verifier as a list. This is handy for things like
         * redefining the local repository if needed. In
         * this case, we use the -N flag so that Maven won't
         * recurse. We are only installing the parent pom to
         * the local repo here.
         */
        List cliOptions = new ArrayList();
        cliOptions.add( "-N" );
        verifier.executeGoal( "install" );

        /*
         * This is the simplest way to check a build
         * succeeded. It is also the simplest way to create
         * an IT test: make the build pass when the test
         * should pass, and make the build fail when the
         * test should fail. There are other methods
         * supported by the verifier. They can be seen here:
         * http://maven.apache.org/shared/maven-verifier/apidocs/index.html
         */
        verifier.verifyErrorFreeLog();

        /*
         * Reset the streams before executing the verifier
         * again.
         */
        verifier.resetStreams();

        /*
         * The verifier also supports beanshell scripts for
         * verification of more complex scenarios. There are
         * plenty of examples in the core-it tests here:
         * http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk
         */
    }
+-----+

 <<Note>>: maven-verifier and maven-verifier-plugin sound similar, but are totally different unrelated pieces of code.  maven-verifier-plugin simply verifies the existence/absence of files on the filesystem.  You could use it for functional testing, but you may need more features than maven-verifier-plugin provides.

* maven-invoker-plugin

 You can use {{{https://svn.apache.org/repos/asf/maven/plugins/trunk/maven-invoker-plugin/}maven-invoker-plugin}}
 to invoke Maven and to provide some BeanShell tests.  Tests written in this way don't run under JUnit/TestNG; instead, they're run by Maven itself.

+-----+
<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-invoker-plugin</artifactId>
        <version>1.0-SNAPSHOT</version>
        <configuration>
          <debug>true</debug>
          <projectsDirectory>src/it</projectsDirectory>
          <pomIncludes>
            <pomInclude>**/pom.xml</pomInclude>
          </pomIncludes>
          <postBuildHookScript>verify.bsh</postBuildHookScript>
        </configuration>
        <executions>
          <execution>
            <phase>integration-test</phase>
            <goals>
              <goal>run</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      ...
    </plugins>
  </build>
  ...
</project>
+-----+

* shitty-maven-plugin

 The {{{http://mojo.codehaus.org/shitty-maven-plugin/}shitty-maven-plugin}} (Super Helpful Integration Testing ThingY) provides a simple way to run integration tests for a project (single module or multi-module).

 shitty-maven-plugin does many of the same things as the maven-invoker-plugin (it supports Groovy tests instead of BeanShell tests), though it has some features that aren't available in maven-invoker-plugin.  Notably, it provides some advanced setup steps to automatically install your plugin using a special version name ("TESTING"), so your dummy projects can depend on that version explicitly.

* maven-it-plugin

 <<Note>>: maven-it-plugin is not at 1.0 yet.  Use it at your own risk.

 The {{{https://svn.apache.org/repos/asf/maven/sandbox/trunk/plugins/maven-it-plugin/}maven-it-plugin}} is used directly
 in the <<<integration-test>>> phase.

 <<Note>>: this it plugin can not be used to test a plugin that is being built for the first time (i.e. with no
 release). In this case, you could, for instance, defined an it-snapshot of the plugin with <<<maven-install-plugin>>>.
 during the <<<pre-integration-test>>> phase.

+-----+
<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...
  <artifactId>maven-XXX-plugin</artifactId>
  <packaging>maven-plugin</packaging>
  <version>1.0-SNAPSHOT</version>
  <description>Test Report plugin in the site phase</description>
  ...
  <build>
    <plugins>
      ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-it-plugin</artifactId>
        <version>1.0-alpha-1-SNAPSHOT</version>
        <configuration>
          <integrationTestsDirectory>${project.basedir}/src/it</integrationTestsDirectory>
          <includes>
            <include>**/pom.xml</include>
          </includes>
          <goals>site</goals>
        </configuration>
        <executions>
          <execution>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <!-- Need to install IT snapshot of maven-XXX-plugin -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-install-plugin</artifactId>
        <version>2.2-SNAPSHOT</version>
        <executions>
          <execution>
            <id>it-test</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>install-file</goal>
            </goals>
            <configuration>
              <file>${project.build.directory}/maven-XXX-plugin-1.0-SNAPSHOT.jar</file>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-XXX-plugin</artifactId>
              <version>1.0-it-SNAPSHOT</version> <!-- IT SNAPSHOT -->
              <packaging>maven-plugin</packaging>
              <pomFile>${project.basedir}/pom.xml</pomFile>
            </configuration>
          </execution>
        </executions>
      </plugin>

      <!-- Testing the result of the it pom.xml -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.4.2</version>
        <executions>
          <execution>
            <id>it-test</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <includes>
                <include>**/*TestIt.class</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>
+-----+

 The it pom should use the it snapshot:

+-----+
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
  <reporting>
    <outputDirectory>
      ${project.basedir}/../../../../../target/it/it1/target/site
    </outputDirectory>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-XXX-plugin</artifactId>
        <version>1.0-it-SNAPSHOT</version>
      </plugin>
    </plugins>
  </reporting>
...
</project>
+-----+

* maven-plugin-management-plugin

 The {{{https://svn.apache.org/repos/asf/maven/sandbox/trunk/plugins/maven-plugin-management-plugin/}maven-plugin-management-plugin}}
 is to stage/unstage a plugin into the local repository for pre/post-integration-test.

 You need to configure the <<<maven-plugin-test-plugin>>> and the <<<maven-surefire-plugin>>>:

+-----+
<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...
  <artifactId>maven-XXX-plugin</artifactId>
  <packaging>maven-plugin</packaging>
  <version>1.0-SNAPSHOT</version>
  <description>Test Report plugin in the site phase</description>
  <prerequisites>
    <maven>2.0.4</maven>
  </prerequisites>
  ...
  <dependencies>
    ...
    <!-- Due to the Maven 2.0.4 prerequisites.
    By default, maven-plugin-test-plugin uses 2.0.1 -->
    <dependency>
      <groupId>org.apache.maven</groupId>
      <artifactId>maven-embedder</artifactId>
      <version>2.0.4</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.maven</groupId>
      <artifactId>maven-core</artifactId>
      <version>2.0.4</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-plugin-management-plugin</artifactId>
        <version>1.0-SNAPSHOT</version>
        <executions>
          <execution>
            <id>pre-it-test</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>stage</goal>
            </goals>
          </execution>
          <execution>
            <id>post-it-test</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>unstage</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <!-- Testing the result of the it pom.xml -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.4.2</version>
        <executions>
          <execution>
            <id>it-test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <includes>
                <include>**/*TestIt.class</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>
+-----+