This page last changed on Feb 25, 2010 by ritchiem.

Qpid has a substantial server performance test suite based around the junit-toolkit library.

The test cases fall into the following categories:

Throughput - straight line 0-600,000 messages/second in how long?
Latency - how quickly can I get a few messages?
Reliability - burn in, stress and soak tests

Building and running the tests

The performance tests live in qpid/java/perftests. When they are built the scripts are generated and placed into qpid/java/build/bin/perftests. These are detailed in the relevant list of test cases above. There are 3 convinience scripts which correspond to the categories to run all the throughput, latency or reliability tests.

Test case implementation

All test cases utilise one of a couple of test classes, with a different set of command line parameters.

The following parameters may be passed to the test runner to control its operation:

Parameter Meaning
-c pattern The number of tests to run concurrently.
-r num The number of times to repeat each test.
-d duration The length of time to run the tests for.
-t name The name of the test case to execute.
-s pattern The size parameter to run tests with.
-o dir The name of the directory to output test timings to.
--csv Output test results in CSV format.
--xml Output test results in XML format.
-v Verbose mode.

Here are some examples:

-c [10:20:30:40:50] Runs the test with 10,20,...,50 threads.
-s [1:100]:samples=10 Runs the test with ten different size parameters evenly spaced between 1 and 100.
-s [1:1000000]:samples=10:exp Runs the test with ten different size parameters exponentially spaced between 1 and 1000000.
-r 10 Runs each test ten times.
-d 10H Runs the test repeatedly for 10 hours.
-d 1M, -r 10 Runs the test repeatedly for 1 minute but only takes a timing sample every 10 test runs.
-r 10, -c [1:5:10:50], -s [100:1000:10000] Runs 12 test cycles (4 concurrency samples * 3 size sample), with 10 repeats each. In total the test will be run 199 times (3 + 15 + 30 + 150)

The test runner also accepts name=value properties on the end of the command line, and these are passed to the test code as parameters to control the operation of the test. The following properties may be set:

Parameter Default Comments
messageSize 0 Message size in bytes. Not including any headers.
destinationName ping The root name to use to generate destination names to ping.
persistent false Determines whether peristent delivery is used.
transacted false Determines whether messages are sent/received in transactions.
broker tcp://localhost:5672 Determines the broker to connect to.
virtualHost test Determines the virtual host to send all ping over.
rate 0 The maximum rate (in hertz) to send messages at. 0 means no limit.
verbose false The verbose flag for debugging. Prints to console on every message.
pubsub false Whether to ping topics or queues. Uses p2p by default.
failAfterCommit false Whether to prompt user to kill broker after a commit batch.
failBeforeCommit false Whether to prompt user to kill broker before a commit batch.
failAfterSend false Whether to prompt user to kill broker after a send.
failBeforeSend false Whether to prompt user to kill broker before a send.
failOnce true Whether to prompt for failover only once.
username guest The username to access the broker with.
password guest The password to access the broker with.
selector null Not used. Defines a message selector to filter pings with.
destinationCount 1 The number of destinations to send pings to.
numConsumers 1 The number of consumers on each destination.
timeout 30000 In milliseconds. The timeout to stop waiting for replies.
commitBatchSize 1 The number of messages per transaction in transactional mode.
uniqueDests true Whether each receivers only listens to one ping destination or all.
durableDests false Whether or not durable destinations are used.
ackMode AUTO_ACK

The message acknowledgement mode. Possible values are:

      0 - SESSION_TRANSACTED
      1 - AUTO_ACKNOWLEDGE
      2 - CLIENT_ACKNOWLEDGE
      3 - DUPS_OK_ACKNOWLEDGE
      257 - NO_ACKNOWLEDGE
      258 - PRE_ACKNOWLEDGE
consTransacted false

Whether or not consumers use transactions. Defaults to the same value
as the 'transacted' option if not seperately defined.

consAckMode AUTO_ACK

The message acknowledgement mode for consumers. Defaults to the same
value as 'ackMode' if not seperately defined.

maxPending 0

The maximum size in bytes, of messages sent but not yet received.
Limits the volume of messages currently buffered on the client
or broker. Can help scale test clients by limiting amount of buffered
data to avoid out of memory errors.

Added for 0.7

Parameter Default Comments
numConsumers <int> Augmented Allow a value of 0, meaning no consumers.
consumeOnly boolean Disable all message sending. Message counts are used by consumers as validation.
preFill int Message count to preFill the destination with before the test start.
delayBeforeConsume int Delay in ms to wait after the preFill has occured before the test starts.

Test case output

Test cases output data in csv format. The extratThroughputResults.sh script can be used to interpret the data and output the average throughput rate.

This page ignores the existence of the C++ perftest, latencytest, etc.

I assume that these client-side tools are broadly similar, but written in different languages? If so, the documentation for each should point to the other, and we should think about how to organize the content so that users can find both.

Posted by jonathan.robie@redhat.com at May 18, 2009 08:17

Yeah, I forget that cwiki would flatten this down - in my head this is sitting beneath java developer docs.

I don't think either set of tooling is a strict subset of the other, but I'm not really sure yet since the qpid/java/perftests stuff isn't very well documented. :/

It might make sense to have all the docs for this sort of thing in a seperate bit of their own, there's no real reason for the split that I can see.

Posted by aidan at May 18, 2009 08:24
Document generated by Confluence on May 26, 2010 10:34