Output flaky re-run information on the screen

When rerunFailingTestsCount is set to a value larger than 0 and the test fails, then it will be re-run and each run information will also be output. Each run with its number and trimmed stack trace will be output.

If the test passes in its first run, then the output on the screen will be identical to the case where rerunFailingTestsCount is not used.

It the test fails in the first run, then there are two possible cases:

1) The test passes in one of its re-runs: the last run will be marked as PASS

For example, a test passed in its second run will output on the screen:

  Run 1: ...
  Run 2: PASS

Then this test will be counted as a flaky test. The build will be successful, but in the end of the summary of all tests run, the number of flaky tests will be output on the screen, for example:

  Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1

2) The test fails in all of the re-runs:

For example, a test failed in all the three runs, then all the runs will be output on the screen:

  Run 1: ...
  Run 2: ...
  Run 3: ...

Then this build will be marked as failure. The type of the failure (failed test or test in error) of this test depends on its first failure.

Output flaky re-run information in test report xml

When rerunFailingTestsCount is set to a value larger than 0, the output xml of test report will also be extended to include information of each re-run.

If the test passes in its first run, then the output xml report will be identical to the case where rerunFailingTestsCount is not used.

It the test fails in the first run, then there are also two possible cases.

1) The test passes in one of its re-runs:

flakyFailure and flakyError elements will be used in the generated xml report to include information of each failing re-runs. system-out and system-err will also be used inside each flakyFailure or flakyError to include information of System.out and System.err output. The original system-out and system-err elements will be retained on the top level under testcase for the last successful run.

For example:

<testcase name=".." classname=".." time="0.1">
  <flakyFailure message="exception message" type="assertion exception">
    <stackTrace>flaky failure stack trace</stackTrace>
    <system-out> flaky failure System.out log message </system-out>
  </flakyFailure>
  <system-out> success </system-out>
</testcase>

In the xml report, the running time of a flaky test will be the running time of the last successful run.

2) The test fails in all of the re-runs:

failure and error elements will still be used in the generated xml report to include information for the first failing run, the same as without using rerunFailingTestsCount. rerunFailure and rerunError elements will be used in the generated xml report to include information of each subsequent failing re-runs. system-out and system-err will also be used inside each flakyFailure or flakyError to include information of System.out and System.err output. The original system-out and system-err elements will be retained on the top level under testcase for the first original failing run.

For example:

<testcase name=".." classname=".." time="0.1">
  <failure message="" type=""> first failure stack trace </failure>
  <system-out> first failure </system-out>
  <rerunFailure message="exception message" type="assertion exception">
    <stackTrace>rerun failure stack trace</stackTrace>
    <system-out> rerun failure </system-out>
  </rerunFailure>
</testcase>

In the xml report, the running time of a failing test with re-runs will be the running time of the first failing run.

Re-run execution in JUnit Providers

The provider surefire-junit4 executes individual test class and consequently re-runs failed tests. The provider surefire-junit47 executes all test classes and re-runs failed tests afterwards.

Re-run execution in Cucumber JVM

Since of 2.21.0 the provider surefire-junit47 can rerun scenarios created by cucumber-jvm 2.0.0 and higher.

Re-run and skip execution

Since of 2.19.1 you can use parameters skipAfterFailureCount and rerunFailingTestsCount together. This is enabled by providers surefire-junit4 and surefire-junit47. You can run again failed tests and skip the rest of the test-set if errors or failures reached skipAfterFailureCount. Notice that failed tests within re-run phase are not included in skipAfterFailureCount.