h1. Testing

Untested code is broken code, so we take testing seriously.  Off the bat you
get to use either JUnit or TestNG for writing unit tests and integration tests.
And you can also add your own framework, or even script tests using Ruby.  But
first, let's start with the basics.


h2. Writing Tests

Each project has a @TestTask@ that you can access using the @test@ method.
@TestTask@ reflects on the fact that each project has one task responsible for
getting the tests to run and acting on the results.  But in fact there are
several tasks that do all the work, and a @test@ task coordinates all of that.

The first two tasks to execute are @test.compile@ and @test.resources@.  They
work similar to @compile@ and @resources@, but uses a different set of
directories.  For example, Java tests compile from the @src/test/java@
directory into the @target/test/classes@ directory, while resources are copied
from @src/test/resources@ into @target/test/resources@.

The @test.compile@ task will run the @compile@ task first, then use the same
dependencies to compile the test classes.  That much you already assumed.  It
also adds the test framework (e.g. JUnit, TestNG) and JMock to the dependency
list.  Less work for you.

If you need more dependencies, the best way to add them is by calling
@test.with@.  This method adds dependencies to both @compile.dependencies@ (for
compiling) and @test.dependencies@ (for running).  You can manage these two
dependency lists separately, but using @test.with@ is good enough in more
cases.

Once compiled, the @test@ task runs all the tests.


h2. Using JUnit

The default test framework for Java projects is JUnit 4.

When you use JUnit, the dependencies includes JUnit, and Buildr picks up all
test classes from the project by looking for classes that either subclass 
@junit.framework.TestCase@, include methods annotated with @org.junit.Test@,
or test suites annotated with @org.org.junit.runner.RunWith@.

The JUnit test framework supports the following options:

|_. Option        |_. Value |
| @:fork@         | VM forking, defaults to true. |
| @:clonevm@      | If true clone the VM each time it is forked. |
| @:properties@   | Hash of system properties available to the test case. |
| @:environment@  | Hash of environment variables available to the test case. |
| @:java_args@    | Arguments passed as is to the JVM. |

For example, to pass properties to the test case:

{{{!ruby
test.using :properties=>{ :currency=>'USD' }
}}}

There are benefits to running test cases in separate VMs.  The default forking
mode is @:once@, and you can change it by setting the @:fork@ option.

|_. :fork=> |_. Behavior |
| @:once@   | Create one VM to run all test classes in the project, separate
VMs for each project. |
| @:each@   | Create one VM for each test case class.  Slow but provides the
best isolation between test classes. |
| @false@   | Without forking, Buildr runs all test cases in a single VM.  This
option runs fastest, but at the risk of running out of memory and causing test
cases to interfere with each other. |

You can see your tests running in the console, and if any tests fail, Buildr
will show a list of the failed test classes.  In addition, JUnit produces text
and XML report files in the project's @reports/junit@ directory.  You can use
that to get around too-much-stuff-in-my-console, or when using an automated
test system.

In addition, you can get a consolidated XML or HTML report by running the
@junit:report@ task.  For example:

{{{!sh
$ buildr test junit:report test=all
$ firefox report/junit/html/index.html
}}}

The @junit:report@ task generates a report from all tests run so far.  If you
run tests in a couple of projects, it will generate a report only for these two
projects.  The example above runs tests in all the projects before generating
the reports.


h2. Using TestNG

You can also use TestNG in your project by telling your project to use TestNG:

{{{!ruby
test.using :testng
}}}

Like all other options you can set with @test.using@, it affects the projects
and all its sub-projects, so you only need to do this once at the top-most
project to use TestNG throughout.  You can also mix TestNG and JUnit by setting
different projects to use different frameworks, but you can't mix both
frameworks in the same project.  (And yes, @test.using :junit@ will switch a
project back to using JUnit)

TestNG works much like JUnit, it gets included in the dependency list, Buildr
picks test classes that contain methods annotated with 
@org.testng.annotations.Test@, and generates test reports in
the @reports/testng@ directory.  At the moment we don't have consolidated HTML
reports for TestNG.

The TestNG test framework supports the following options:

|_. Option        |_. Value |
| @:properties@   | Hash of system properties available to the test case. |
| @:java_args@    | Arguments passed as is to the JVM. |


h2. Excluding Tests and Ignoring Failures

If you have a lot of tests that are failing or just hanging there collecting
dusts, you can tell Buildr to ignore them.  You can either tell Buildr to only
run specific tests, for example:

{{{!ruby
test.include 'com.acme.tests.passing.*'
}}}

Or tell it to exclude specific tests, for example:

{{{!ruby
test.exclude '*FailingTest', '*FailingWorseTest'
}}}

Note that we're always using the package qualified class name, and you can use
star (@*@) to substitute for any set of characters.

When tests fail, Buildr fails the @test@ task.  This is usually a good thing,
but you can also tell Buildr to ignore failures by resetting the
@:fail_on_failure@ option:

{{{!ruby
test.using :fail_on_failure=>false
}}}

Besides giving you a free pass to ignore failures, you can use it for other
causes, for example, to be somewhat forgiving:

{{{!ruby
test do
  fail 'More than 3 tests failed!' if test.failed_tests.size > 3
end
}}}

The @failed_tests@ collection holds the names of all classes with failed tests.
And there's @classes@, which holds the names of all test classes.  Ruby
arithmetic allows you to get the name of all passed test classes with a simple
@test.classes – test.failed_tests@.  We'll let you imagine creative use for
these two.


h2. Running Tests

It's a good idea to run tests every time you change the source code, so we
wired the @build@ task to run the @test@ task at the end of the build.  And
conveniently enough, the @build@ task is the default task, so another way to
build changes in your code and run your tests:

{{{!sh
$ buildr
}}}

That only works with the local @build@ task and any local task that depends on
it, like @package@, @install@ and @upload@.  Each project also has its own
@build@ task that does not invoke the @test@ task, so @buildr build@ will run
the tests cases, but @buildr foo:build@ will not.

While it's a good idea to always run your tests, it's not always possible.
There are two ways you can get @build@ to not run the @test@ task.  You can set
the environment variable @test@ to @no@ (but @skip@ and @off@ will also work).
You can do that when running Buildr:

{{{!sh
$ buildr test=no
}}}

Or set it once in your environment:

{{{!sh
$ export TEST=no
$ buildr
}}}

If you're feeling really adventurous, you can also disable tests from your
Buildfile or @buildr.rb@ file, by setting @options.test = false@. We didn't say
it's a good idea, we're just giving you the option.

The @test@ task is just smart enough to run all the tests it finds, but will
accept include/exclude patterns.  Often enough you're only working on one
broken test and you only want to run that one test.  Better than changing your
Buildfile, you can run the @test@ task with a pattern.  For example:

{{{!sh
$ buildr test:KillerAppTest
}}}

Buildr will then run only tests that match the pattern @KillerAppTest@.  It
uses pattern matching, so @test:Foo@ will run @com.acme.FooTest@ and
@com.acme.FooBarTest@.  With Java, you can use this to pick a class name, or a
package name to run all tests in that package, or any such combination.  In
fact, you can specify several patterns separated with commas.  For example:

{{{!sh
$ buildr test:FooTest,BarTest
}}}

As you probably noticed, Buildr will stop your build at the first test that
fails.  We think it's a good idea, except when it's not.  If you're using a
continuous build system, you'll want a report of all the failed tests without
stopping at the first failure.  To make that happen, set the environment
variable @test@ to "all", or the Buildr @options.test@ option to @:all@.  For
example:

{{{!sh
$ buildr package test=all
}}}

We're using @package@ and not @build@ above.  When using a continuous build
system, you want to make sure that packages are created, contain the right
files, and also run the integration tests.


h2. Integration Tests

So far we talked about unit tests.  Unit tests are run in isolation on the
specific project they test, in an isolated environment, generally with minimal
setup and teardown.  You get a sense of that when we told you tests run after
the @build@ task, and include JMock in the dependency list.

In contrast, integration tests are run with a number of components, in an
environment that resembles production, often with more complicates setup and
teardown procedures.  In this section we'll talk about the differences between
running unit and integration tests.

You write integration tests much the same way as you write unit tests, using
@test.compile@ and @test.resources@.  However, you need to tell Buildr that
your tests will execute during integration test.  To do so, add the following
line in your project definition:

{{{!ruby
test.using :integration
}}}

Typically you'll use unit tests in projects that create internal modules, such
as JARs, and integration tests in projects that create components, such as WARs
and EARs.  You only need to use the @:integration@ option with the later.

To run integration tests on the current project:

{{{!sh
$ buildr integration
}}}

You can also run specific tests cases, for example:

{{{!sh
$ buildr integration:ClientTest
}}}

If you run the @package@ task (or any task that depends on it, like @install@
and @upload@), Buildr will first run the @build@ task and all its unit tests,
and then create the packages and run the integration tests.  That gives you
full coverage for your tests and ready to release packages.  As with unit
tests, you can set the environment variable @test@ to "no" to skip integration
tests, or "all" to ignore failures.


h2. Using Setup and Teardown

Some tests need you to setup an environment before they run, and tear it down
afterwards.  The test frameworks (JUnit, TestNG) allow you to do that for each
test.  Buildr provides two additional mechanisms for dealing with more
complicated setup and teardown procedures.

Integration tests run a setup task before the tests, and a teardown task
afterwards.  You can use this task to setup a Web server for testing your Web
components, or a database server for testing persistence.  You can access
either task by calling @integration.setup@ and @integration.teardown@.  For
example:

{{{!ruby
integration.setup { server.start ; server.deploy }
integration.teardown { server.stop }
}}}

Depending on your build, you may want to enhance the setup/teardown tasks from
within a project, for example, to populate the database with data used by that
project's test, or from outside the project definition, for example, to start
and stop the Web server.

Likewise, each project has its own setup and teardown tasks that are run before
and after tests for that specific project.  You can access these tasks using
@test.setup@ and @test.teardown@.


h2. Testing Your Build

So you got the build running and all the tests pass, binaries are shipping when
you find out some glaring omissions.  The license file is empty, the localized
messages for Japanese are missing, the CSS files are not where you expect them
to be.  The fact is, some errors slip by unit and integration tests.  So how do
we make sure the same mistake doesn't happen again?

Each project has a @check@ task that runs just after packaging.  You can use
this task to verify that your build created the files you wanted it to create.
And to make it extremely convenient, we introduced the notion of expectations.

You use the @check@ method to express and expectation.  Buildr will then run
all these expectations against your project, and fail at the first expectation
that doesn't match.  An expectation says three things.  Let's look at a few
examples:

{{{!ruby
check package(:war), 'should exist' do
  it.should exist
end
check package(:war), 'should contain a manifest' do
  it.should contain('META-INF/MANIFEST.MF')
end
check package(:war).path('WEB-INF'), 'should contain files' do
  it.should_not be_empty
end
check package(:war).path('WEB-INF/classes'), 'should contain classes' do 
  it.should contain('**/*.class')
end
check package(:war).entry('META-INF/MANIFEST'), 'should have license' do
  it.should contain(/Copyright (C) 2007/)
end
check file('target/classes'), 'should contain class files' do
  it.should contain('**/*.class')
end
check file('target/classes/killerapp/Code.class'), 'should exist' do
  it.should exist
end
}}}

The first argument is the subject, or the project if you skip the first
argument.  The second argument is the description, optional, but we recommend
using it.  The method @it@ returns the subject.

You can also write the first expectation like this:

{{{!ruby
check do
  package(:jar).should exist
end
}}}

We recommend using the subject and description, they make your build easier to
read and maintain, and produce better error messages.

There are two methods you can call on just about any object, called @should@
and @should_not@.  Each method takes an argument, a matcher, and executes that
matcher.  If the matcher returns false, @should@ fails.  You can figure out
what @should_not@ does in the same case.

Buildr provides the following matchers:

|_. Method |_. Checks that ... |
| @exist@   | Given a file task checks that the file (or directory) exists. |
| @empty@   | Given a file task checks that the file (or directory) is empty. |
| @contain@ | Given a file task referencing a file, checks its contents, using
string or regular expression.  For a file task referencing a directory, checks
that it contains the specified files; global patterns using @*@ and @**@ are
allowed. |

All these matchers operate against a file task.  If you run them against a
ZipTask (including JAR, WAR, etc) they can also check the contents of the ZIP
file.  And as you can see in the examples above, you can also run them against
a path in a ZIP file, checking its contents as if it was a directory, or
against an entry in a ZIP file, checking the content of that file.

p(note). The @package@ method returns a package task based on packaging type,
identifier, group, version and classifier.  The last four are inferred, but if
you create a package with different specifications (for example, you specify a
classifier) your checks must call @package@ with the same qualifying arguments
to return the very same package task.

Buildr expectations are based on RSpec.  "RSpec":http://rspec.info/ is the
behavior-driven development framework we use to test Buildr itself.  Check the
RSpec documentation if want to see all the supported matchers, or want to write
your own.


h2. Behaviour-Driven Development

Buildr supports several Behaviour-Driven Development(BDD) frameworks for
testing your projects.  Buildr follows each framework naming conventions,
searching for files under the @src/spec/{lang}@ directory.


h4. JBehave 

"JBehave":http://jbehave.org/ is a pure Java BDD framework, stories and
behaviour specifications are written in the Java language. 

To use JBehave in your project you can select it with @test.using :jbehave@.

This framework will search for the following patterns under your project:

{{{
src/spec/java/**/*Behaviour.java
}}}

Supports the following options:

|_. Option        |_. Value |
| @:properties@   | Hash of system properties available to the test case. |
| @:java_args@    | Arguments passed as is to the JVM. |


h4. RSpec

"RSpec":http://rspec.info/ is the de-facto BDD framework for ruby. It's the
framework used to test Buildr itself. 

Specifications are written in "Ruby":http://www.ruby-lang.org/en/ language, but
are run by using "JRuby":http://jruby.codehaus.org/.  That means you have
access to all your Java classes and any Java or Ruby tool out there.

To use this framework in your project you can select it with @test.using
:rspec@.

This framework will search for the following patterns under your project:

{{{
src/spec/ruby/**/*_spec.rb
}}}

Supports the following options:

|_. Option        |_. Value |
| @:properties@   | Hash of system properties available to the test case. |
| @:java_args@    | Arguments passed as is to the JVM. |


h4. EasyB

"EasyB":http://www.easyb.org/ is a BDD framework using
"Groovy":http://groovy.codehaus.org/. 

Specifications are written in the Groovy language, of course you get seamless
Java integration as with all things groovy. 

To use this framework in your project you can select it with @test.using
:easyb@.

This framework will search for the following patterns under your project:

{{{
src/spec/groovy/**/*Behavior.groovy
src/spec/groovy/**/*Story.groovy
}}}

Supports the following options:

|_. Option        |_. Value |
| @:properties@   | Hash of system properties available to the test case. |
| @:java_args@    | Arguments passed as is to the JVM. |
| @:format@       | Report format, either @:txt@ or @:xml@ |


Next, let's talk about "customizing your environment and using
profiles":settings_profiles.html
