17. Help! My boss wants me to load test our application!¶

This is a fairly open-ended proposition. There are a number of questions to be asked first, and additionally a number of resources that will be needed. You will need some hardware to run the benchmarks/load-tests from. A number of tools will prove useful. There are a number of products to consider. And finally, why is Java a good choice to implement a load-testing/Benchmarking product.

17.1 Questions to ask¶

What is our anticipated average number of users (normal load) ?

What is our anticipated peak number of users ?

When is a good time to load-test our application (i.e. off-hours or week-ends), bearing in mind that this may very well crash one or more of our servers ?

Does our application have state ? If so, how does our application manage it (cookies, session-rewriting, or some other method) ?

What is the testing intended to achieve?

17.2 Resources¶

The following resources will prove very helpful. Bear in mind that if you cannot locate these resources, you will become these resources. As you already have your work cut out for you, it is worth knowing who the following people are, so that you can ask them for help if you need it.

17.2.1 Network¶

Who knows our network topology ? If you run into any firewall or proxy issues, this will become very important. As well, a private testing network (which will therefore have very low network latency) would be a very nice thing. Knowing who can set one up for you (if you feel that this is necessary) will be very useful. If the application doesn't scale as expected, who can add additional hardware ?

17.2.2 Application¶

Who knows how our application functions ? The normal sequence is

test (low-volume - can we benchmark our application?)
benchmark (the average number of users)
load-test (the maximum number of users)
test destructively (what is our hard limit?)

The test process may progress from black-box testing to white-box testing (the difference is that the first requires no knowledge of the application [it is treated as a "black box"] while the second requires some knowledge of the application). It is not uncommon to discover problems with the application during this process, so be prepared to defend your work.

17.3 What platform should I use to run the benchmarks/load-tests ?¶

This should be a widely-used piece of hardware, with a standard (i.e. vanilla) software installation. Remember, if you publish your results, the first thing your clients will do is hire a graduate student to verify them. You might as well make it as easy for this person as you possibly can.

For Windows, Windows XP Professional should be a minimum (the others do not multi-thread past 50-60 connections, and you probably anticipate more users than that).

Good free platforms include the linuxes, the BSDs, and Solaris Intel. If you have a little more money, there are commercial linuxes. This may be worth it if you need the support.

For non-Windows platforms, investigate "ulimit -n unlimited" with a view to including it in your user account startup scripts (.bashrc or .cshrc scripts for the testing account).

As you progress to larger-scale benchmarks/load-tests, this platform will become the limiting factor. So it's worth using the best hardware and software that you have available. Remember to include the hardware/software configuration in your published benchmarks.

When you need a lot of machines or want to test the network latency, Cloud can help you. JMeter can easily be installed on Cloud instances as it runs on nearly any architecture available in the Cloud. JMeter is also supported within Commercial Cloud PAAS if you don't want to manage it yourself.

Don't forget JMeter batch (NON-GUI) mode. This mode should be used during load testing for many reasons:

If you have a powerful server that supports Java but perhaps does not have a fast graphics implementation, or where you need to login remotely.
Batch (non-GUI) mode can reduce the network traffic compared with using a remote display or client-server mode.
Java AWT Thread used for GUI mode can alter injection behaviour by blocking sometimes

The batch log file can then be loaded into JMeter on a workstation for analysis, or you can use CSV output and import the data into a spreadsheet.

Remember GUI mode is for Script creation and debugging, not for load testing

17.4 Tools¶

The following tools will all prove useful. It is definitely worthwhile to become familiar with them. This should include trying them out, and reading the appropriate documentation (man-pages, info-files, application --help messages, and any supplied documentation).

17.4.1 ping¶

This can be used to establish whether or not you can reach your target site. Options can be specified so that 'ping' provides the same type of route reporting as 'traceroute'.

17.4.2 nslookup/dig¶

While the user will normally use a human-readable internet address, you may wish to avoid the overhead of DNS lookups when performing benchmarking/load-testing. These can be used to determine the unique address (dotted quad) of your target site.

17.4.3 traceroute¶

If you cannot "ping" your target site, this may be used to determine the problem (possibly a firewall or a proxy). It can also be used to estimate the overall network latency (running locally should give the lowest possible network latency - remember that your users will be running over a possibly busy internet). Generally, the fewer hops the better.

17.5 How can I enhance JMeter ?¶

There a lot of open-source and commercial plugins that can enhance JMeter, let's mention here the main open-source ones:

17.5.1 JMeter-Plugins¶

This non official project is THE companion to core JMeter.
It provides many useful extensions, among which:

Active Threads Over Time Graph Listener
Response Times vs Threads Graph Listener
Transaction Throughput vs Threads Graph Listener
GraphGenerator listener to create graphs at end of a load test
Selenium WebDriver Sampler
....

17.5.2 JMeter Plugin for Maven¶

This non official plugin allows you to run your automated JMeter tests through Maven.

17.5.3 JMeter Performance Plugin¶

This non official plugin allows you to capture reports from JMeter and JUnit.
Jenkins will generate graphic charts with the trend report of performance and robustness. It includes the feature of setting the final build status as good, unstable or failed, based on the reported error percentage.

17.5.4 JMeter plugin for AWS¶

This non official plugin automates running Apache JMeter on Amazon EC2

17.6 Why Java ?¶

Why not Perl or C ?

Well, Perl might be a very good choice except that the Benchmark package seems to give fairly fuzzy results. Also, simulating multiple users with Perl is a tricky proposition (multiple connections can be simulated by forking many processes from a shell script, but these will not be threads, they will be processes). However, the Perl community is very large. If you find that someone has already written something that seems useful, this could be a very good solution.

C, of course, is a very good choice (check out the Apache ab tool). But be prepared to write all of the custom networking, threading, and state management code that you will need to benchmark your application.

Java gives you (for free) the custom networking, threading, and state management code that you will need to benchmark your application. Java is aware of HTTP, FTP, and HTTPS - as well as RMI, IIOP, and JDBC (not to mention cookies, URL-encoding, and URL-rewriting). In addition Java gives you automatic garbage-collection, and byte-code level security.