Table Of Contents

• Overview
• Why Use JUnitPerf?
• Downloading JUnitPerf
• Installing JUnitPerf
• Building And Testing JUnitPerf
• Using JUnitPerf
• Writing Effective JUnitPerf Tests
• Limitations
• Support
• Donate
• Consulting Services
• License
• Acknowledgments
• Resources
• Release History

Overview

JUnitPerf is a collection of JUnit test decorators used to measure the performance and scalability of functionality contained within existing JUnit tests.

JUnitPerf contains the following JUnit test decorators:

• TimedTest

  A TimedTest is a test decorator that runs a test and measures the elapsed time of the test.

  A TimedTest is constructed with a specified maximum elapsed time. By default, a TimedTest will wait for the completion of its decorated test and then fail if the maximum elapsed time was exceeded. Alternatively, a TimedTest can be constructed to immediately signal a failure when the maximum elapsed time of its decorated test is exceeded.

• LoadTest

  A LoadTest is a test decorator that runs a test with a simulated number of concurrent users and iterations.

Why Use JUnitPerf?

JUnitPerf tests transparently decorate existing JUnit tests. This decoration-based design allows performance testing to be dynamically added to an existing JUnit test without affecting the use of the JUnit test independent of its performance. By decorating existing JUnit tests, it's quick and easy to compose a set of performance tests into a performance test suite. The performance test suite can then be run automatically and independent of your other JUnit tests. In fact, you generally want to avoid grouping your JUnitPerf tests with your other JUnit tests so that you can run the test suites independently and at different frequencies. Long-running performance tests will slow you down and undoubtedly tempt you to abandon unit testing altogether, so try to schedule them to run at times when they won't interfere with your refactoring pace.

JUnitPerf tests are intended to be used specifically in situations where you have quantitative performance and/or scalability requirements that you'd like to keep in check while refactoring code. For example, you might write a JUnitPerf test to ensure that refactoring an algorithm didn't incur undesirable performance overhead in a performance-critical code section. You might also write a JUnitPerf test to ensure that refactoring a resource pool didn't adversely affect the scalability of the pool under load.

It's important to maintain a pragmatic approach when writing JUnitPerf tests to maximize the return on your testing investment. Traditional performance profiling tools and techniques should be employed first to identify which areas of code exhibit the highest potential for performance and scalability problems. JUnitPerf tests can then be written to automatically test and check that requirements are being met now and in the future.

Here's an example usage scenario:

You've built a well-factored chunk of software, complete with the necessary suite of JUnit tests to validate the software. At this point in the process you've gained as much knowledge about the design as possible.

You then use a performance profiling tool to isolate where the software is spending most of its time. Based on your knowledge of the design you're better equipped to make realistic estimates of the desired performance and scalability. And, since your refactorings have formed clear and succinct methods, your profiler is able to point you towards smaller sections of code to tune.

You then write a JUnitPerf test with the desired performance and scalability tolerances for the code to be tuned. Without making any changes to the code, the JUnitPerf test should fail, proving that the test is written properly. You then make the tuning changes in small steps.

After each step you compile and rerun the JUnitPerf test. If you've improved performance to the expected degree, the test passes. If you haven't improved performance to the expected degree, the test fails and you continue the tuning process until the test passes. In the future, when the code is again refactored, you re-run the test. If the test fails, the previously defined performance limits have been exceeded, so you back out the change and continue refactoring until the test passes.

Downloading JUnitPerf

JUnitPerf 1.9 is the latest major version release. It includes all the minor version changes.

This version requires Java 2 and JUnit 3.5 (or higher).

The distribution contains a JAR file, source code, sample tests, API documentation, and this document.

Installing JUnitPerf

To install JUnitPerf, follow these steps:

1. Unzip the junitperf.zip distribution file to a directory referred to as %JUNITPERF_HOME%.
2. Add JUnitPerf to the classpath:

   set CLASSPATH=%CLASSPATH%;%JUNITPERF_HOME%\lib\junitperf.jar

To install JUnitPerf, follow these steps:

1. Unzip the junitperf.zip distribution file to a directory referred to as $JUNITPERF_HOME.
2. Change file permissions:

   chmod -R a+x $JUNITPERF_HOME

3. Add JUnitPerf to the classpath:

   export CLASSPATH=$CLASSPATH:$JUNITPERF_HOME/lib/junitperf.jar

Building And Testing JUnitPerf

The JUnitPerf distribution includes the pre-built classes in the $JUNITPERF_HOME/lib/junitperf.jar file.

An Ant build file is included in $JUNITPERF_HOME/build.xml to build the $JUNITPERF_HOME/lib/junitperf.jar file from the included source code.

Building

To build JUnitPerf, use:

cd $JUNITPERF_HOME
ant jar

Testing

The JUnitPerf distribution includes JUnit test cases to validate the integrity of JUnitPerf.

To test JUnitPerf, use:

cd $JUNITPERF_HOME
ant test

Using JUnitPerf

The easiest way to describe how to use JUnitPerf is to show examples of each type of test decorator.

The $JUNITPERF_HOME/samples directory contains the set of example JUnitPerf tests described in this section.

TimedTest

A TimedTest test decorator is constructed with an existing JUnit test and a maximum elapsed time in milliseconds.

For example, to create a timed test that waits for the completion of the ExampleTestCase.testOneSecondResponse() method and then fails if the elapsed time exceeded 1 second, use:

long maxElapsedTime = 1000;
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test timedTest = new TimedTest(testCase, maxElapsedTime);

Alternatively, to create a timed test that fails immediately when the elapsed time of the ExampleTestCase.testOneSecondResponse() test method exceeds 1 second, use:

long maxElapsedTime = 1000;
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test timedTest = new TimedTest(testCase, maxElapsedTime, false);

The following is an example test that creates a TimedTest to test the performance of the functionality being unit tested in the ExampleTestCase.testOneSecondResponse() method. The timed test waits for the method under test to complete, and then fails if the elapsed time exceeded 1 second.

import com.clarkware.junitperf.*;

import junit.framework.Test;

public class ExampleTimedTest {

    public static Test suite() {
        
        long maxElapsedTime = 1000;
        
        Test testCase = new ExampleTestCase("testOneSecondResponse");
        Test timedTest = new TimedTest(testCase, maxElapsedTime);
        
        return timedTest;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}

The granularity of the test decoration design offered by JUnit, and used by JUnitPerf, imposes some limitations. The elapsed time measured by a TimedTest decorating a single testXXX() method of a TestCase includes the total time of the setUp(), testXXX(), and tearDown() methods. The elapsed time measured by a TimedTest decorating a TestSuite includes the total time of all setUp(), testXXX(), and tearDown() methods for all the Test instances in the TestSuite. Therefore, the expected elapsed time measurements should be adjusted accordingly to account for the set-up and tear-down costs of the decorated test.

A LoadTest is a test decorator that runs a test with a simulated number of concurrent users and iterations.

In its simplest form, a LoadTest is constructed with a test to decorate and the number of concurrent users. By default, each user runs one iteration of the test.

For example, to create a load test of 10 concurrent users with each user running the ExampleTestCase.testOneSecondResponse() method once and all users starting simultaneously, use:

int users = 10;
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test loadTest = new LoadTest(testCase, users);

The load can be ramped by specifying a pluggable Timer instance that prescribes the delay between the addition of each concurrent user. A ConstantTimer has a constant delay, with a zero value indicating that all users will be started simultaneously. A RandomTimer has a random delay with a uniformly distributed variation.

For example, to create a load test of 10 concurrent users with each user running the ExampleTestCase.testOneSecondResponse() method once and with a 1 second delay between the addition of users, use:

int users = 10;
Timer timer = new ConstantTimer(1000);
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test loadTest = new LoadTest(testCase, users, timer);

In order to simulate each concurrent user running a test for a specified number of iterations, a LoadTest can be constructed to decorate a RepeatedTest. Alternatively, a LoadTest convenience constructor specifying the number of iterations is provided which creates a RepeatedTest.

For example, to create a load test of 10 concurrent users with each user running the ExampleTestCase.testOneSecondResponse() method for 20 iterations, and with a 1 second delay between the addition of users, use:

int users = 10;
int iterations = 20;
Timer timer = new ConstantTimer(1000);
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test repeatedTest = new RepeatedTest(testCase, iterations);
Test loadTest = new LoadTest(repeatedTest, users, timer);

int users = 10;
int iterations = 20;
Timer timer = new ConstantTimer(1000);
Test testCase = new ExampleTestCase("testOneSecondResponse");
Test loadTest = new LoadTest(testCase, users, iterations, timer);

If a test case intended to be decorated as a LoadTest contains test-specific state in the setUp() method, then the TestFactory should be used to ensure that each concurrent user thread uses a thread-local instance of the test. For example, to create a load test of 10 concurrent users with each user running a thread-local instance of ExampleStatefulTest, use:

int users = 10;
Test factory = new TestFactory(ExampleStatefulTest.class);
Test loadTest = new LoadTest(factory, users);

int users = 10;
Test factory = new TestMethodFactory(ExampleStatefulTest.class, "testSomething");
Test loadTest = new LoadTest(factory, users);

The following is an example test that creates a LoadTest to test the scalability of the functionality being unit tested in the ExampleTestCase.testOneSecondResponse() test method. The LoadTest adds 10 concurrent users without delay, with each user running the test method once. The LoadTest itself is decorated with a TimedTest to test the throughput of the ExampleTestCase.testOneSecondResponse() test method under load. The test will fail if the total elapsed time of the entire load test exceeds 1.5 seconds.

import com.clarkware.junitperf.*;

import junit.framework.Test;

public class ExampleThroughputUnderLoadTest {

    public static Test suite() {
     
        int maxUsers = 10;
        long maxElapsedTime = 1500;
        
        Test testCase = new ExampleTestCase("testOneSecondResponse");
        Test loadTest = new LoadTest(testCase, maxUsers);
        Test timedTest = new TimedTest(loadTest, maxElapsedTime);

        return timedTest;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}

In the following example, the order of test decoration is reversed. The TimedTest measures the elapsed time of the ExampleTestCase.testOneSecondResponse() method. The LoadTest then decorates the TimedTest to simulate a 10-user load on the ExampleTestCase.testOneSecondResponse() method. The test will fail if any user's response time exceeds 1 second.

import com.clarkware.junitperf.*;

import junit.framework.Test;

public class ExampleResponseTimeUnderLoadTest {

    public static Test suite() {
     
        int maxUsers = 10;
        long maxElapsedTime = 1000;
        
        Test testCase = new ExampleTestCase("testOneSecondResponse");
        Test timedTest = new TimedTest(testCase, maxElapsedTime);
        Test loadTest = new LoadTest(timedTest, maxUsers);

        return loadTest;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}

The following is an example TestCase that combines the ExampleTimedTest and ExampleLoadTest into a single test suite that can be run automatically to run all performance-related tests:

import junit.framework.Test;
import junit.framework.TestSuite;

public class ExamplePerfTestSuite {

    public static Test suite() {

        TestSuite suite = new TestSuite();
        suite.addTest(ExampleTimedTest.suite());
        suite.addTest(ExampleLoadTest.suite());
        
        return suite;
    }
    
    public static void main(String args[]) {
        junit.textui.TestRunner.run(suite());
    }
}

Writing Effective JUnitPerf Tests

Waiting Timed Tests

By default, a TimedTest will wait for the completion of its decorated test and then fail if the maximum elapsed time was exceeded. This type of waiting timed test always allows its decorated test to accumulate all test results until test completion and check the accumulated test results.

If the test decorated by a waiting timed test spawns threads, either directly or indirectly, then the decorated test must wait for those threads to run to completion and return control to the timed test. Otherwise, the timed test will wait indefinitely. As a general rule, unit tests should always wait for spawned threads to run to completion, using Thread.join() for example, in order to accurately assert test results.

Non-Waiting Timed Tests

Alternatively, a TimedTest can be constructed to immediately signal a failure when the maximum elapsed time of its decorated test is exceeded. This type of non-waiting timed test will not wait for its decorated test to run to completion if the maximum elapsed time is exceeded. Non-waiting timed tests are more efficient than waiting timed tests in that non-waiting timed tests don't waste time waiting for the decorated test to complete only then to signal a failure, if necessary. However, unlike waiting timed tests, test results from a decorated test will not be accumulated after the expiration of the maximum elapsed time in a non-waiting timed test.

Non-Atomic Load Tests

By default, a LoadTest does not enforce test atomicity (as defined in transaction processing) if its decorated test spawns threads, either directly or indirectly. This type of non-atomic load test assumes that its decorated test is transactionally complete when control is returned. For example, if the decorated test spawns threads and then returns control without waiting for its spawned threads to complete, then the decorated test is assumed to be transactionally complete.

As a general rule, unit tests should always wait for spawned threads to run to completion, using Thread.join() for example, in order to accurately assert test results. However, in certain environments this isn't always possible. For example, as a result of a distributed lookup of an Enterprise JavaBean (EJB), an application server may spawn a new thread to handle the request. If the new thread belongs to the same ThreadGroup as the thread running the decorated test (the default), then a non-atomic load test will simply wait for the completion of all threads spawned directly by the load test and the new (rogue) thread is ignored.

To summarize, non-atomic load tests only wait for the completion of threads spawned directly by the load test to simulate more than one concurrent user.

Atomic Load Tests

If threads are integral to the successful completion of a decorated test, meaning that the decorated test should not be treated as complete until all of its threads run to completion, then setEnforceTestAtomicity(true) should be invoked to enforce test atomicity (as defined in transaction processing). This effectively causes the atomic load test to wait for the completion of all threads belonging to the same ThreadGroup as the thread running the decorated test. Atomic load tests also treat any premature thread exit as a test failure. If a thread dies abruptly, then all other threads belonging to the same ThreadGroup as the thread running the decorated test will be interrupted immediately.

If a decorated test spawns threads belonging to the same ThreadGroup as the thread running the decorated test (the default), then the atomic load test will wait indefinitely for the spawned thread to complete.

To summarize, atomic load tests wait for the completion of all threads belonging to the same ThreadGroup as the threads spawned directly by the load test to simulate more than one concurrent user.

Limitations

JUnitPerf has the following known limitations:

• The elapsed time measured by a TimedTest decorating a single testXXX() method of a TestCase includes the total time of the setUp(), testXXX(), and tearDown() methods, as this is the granularity offered by decorating any Test instance. The expected elapsed time measurements should be adjusted accordingly to account for the set-up and tear-down costs of the decorated test.
• JUnitPerf is not intended to be a full-fledged load testing or performance profiling tool, nor is it intended to replace the use of these tools. JUnitPerf should be used to write localized performance unit tests to help developers refactor responsibly.
• The performance of your tests can degrade significantly if too many concurrent users are cooperating in a load test. The actual threshold number is JVM specific.

Support

If you have any questions, comments, enhancement requests, success stories, or bug reports regarding JUnitPerf, or if you want to be notified when new versions of JUnitPerf are available, please email mike@clarkware.com. Your information will be kept private.

A mailing list is also available to discuss JUnitPerf or to be notified when new versions of JUnitPerf are available.

Donate

Please support the ongoing development of JUnitPerf by purchasing a copy of the book Pragmatic Project Automation.

Thanks in advance!

Consulting Services

I offer customized consulting, training, and mentoring to help you and your team quickly become productive using testing tools that any budget can afford. My services are always tailored to your specific needs to deliver maximum business value.

Contact me by email for more details.

License

JUnitPerf is licensed under the BSD License.

Acknowledgments

Many thanks to Ervin Varga for improving thread safety and test atomicity by suggesting the use of a ThreadGroup to catch and handle thread exceptions. Ervin also proposed the idea and provided the implementation for the TimedTest signaling a failure immediately if the maximum time is exceeded and the TestFactory. His review of JUnitPerf and his invaluable contributions are much appreciated!

Resources

• JUnit Primer
  This article demonstrates how to write and run simple test cases and test suites using the JUnit testing framework.
  Mike Clark, Clarkware Consulting, Inc.

• Test-Driven Development: A Practical Guide
  David Astels (Prentice Hall, 2003)
  Includes a section written by yours truly on how to use JUnitPerf for continuous performance testing.

• Java Extreme Programming Cookbook
  Eric Burke, Brian Coyner (O'Reilly & Associates, 2003)
  Includes a chapter of JUnitPerf recipes.

• Java Tools for Extreme Programming: Mastering Open Source Tools Including Ant, JUnit, and Cactus
  Richard Hightower, Nicholas Lesiecki (John Wiley & Sons, 2001)
  Includes a chapter on how to use JUnitPerf with HttpUnit.

• http://www.junit.org
  The JUnit Website

Release History

Version 1.9 Date 02.16.2004 Description Bug fix

Changes When using the Swing test runner, the progress bar now turns red when a TimedTest fails. In prior releases, when a test failed a failure message was printed but the progress bar stayed green. Thanks to those folks who do use the graphical runner for performance tests for pointing out this bug!

Version 1.8 Date 09.03.2002 Description Usability upgrades

Changes Documentation edits and additions. Added the ExampleThroughputUnderLoadTest and ExampleResponseTimeUnderLoadTest to the samples. Added the TimedTest.setQuiet() method to optionally disable output of the test's elapsed time.

Version 1.7 Date 02.26.2002 Description Added TestMethodFactory

Changes The TestMethodFactory can be used to load test a single test method while ensuring that each concurrent user thread uses a thread-local instance of the test.

Version 1.6 Date 11.23.2001 Description Maintenance upgrade.

Changes If a threaded test in a load test has been stopped, either by using the "Stop" button of the Swing UI or using the haltonfailure="yes" attribute of a JUnit Ant task, the stopped or failed test is cancelled and the currently active threaded tests of the load test are allowed to complete. Prior to this upgrade, if a threaded test was stopped, the load test would hang while waiting for the threaded test to report its completion.

Version 1.5 Date 09.08.2001 Description Usability upgrade.

Changes Added the TestFactory class to allow stateful tests to be decorated as LoadTest instances. Use of a TestFactory ensures that each LoadTest thread uses its own decorated test instance.

Version 1.4 Date 06.12.2001 Description Design and usability upgrades.

Changes A TimedTest can now be constructed to fail immediately if the maximum elapsed time of the decorated test is exceeded. In other words, the TimedTest will not wait for the decorated test to run to completion if the maximum elapsed time is exceeded. The TimedTest.outOfTime() method was added to unambiguously determine whether the test failed due to the maximum elapsed time being exceeded or the test itself failing. The LoadTest class now supports enforcing test atomicity using the setEnforceTestAtomicity() method. By default, test atomicity is not enforced for test cases that spawn threads, either directly or indirectly. The TimedTest.toString() method now includes an indication of whether the timed test will wait for test completion (WAITING) or wait for the maximum elapsed time to expire (NON-WAITING). The LoadTest.toString() method now includes an indication of whether the load test will enforce test atomicity by waiting for control to return (ATOMIC) or waiting for all threaded tests to complete (NON-ATOMIC).

Version 1.3 Date 05.11.2001 Description Design and usability upgrades.

Changes The LoadTest class now employs a ThreadBarrier to allow threads spawned directly by a load test to properly signal their completion. Threads spawned by decorated tests, either directly or indirectly, without a specified thread group are added to the ThreadedTestGroup by default. This was causing the active count of the thread group to never fall to 0, thereby causing the load test to hang indefinitely.

Version 1.2 Date 04.23.2001 Description Design and usability upgrades.

Changes Replaced the ThreadBarrier with a ThreadedTestGroup to catch and handle uncaught exceptions thrown by threads spawned by ThreadedTest. This improves thread safety and supports test atomicity (as defined by transaction processing) when enabled. Added several variants of LoadTest constructors for convenience and extensibility. Updated JUnitPerf.html and ExampleLoadTest.java to include more examples for constructing LoadTest instances with various constructors.

Version 1.1 Date 03.03.2001

Description Initial public release.