15. Remote Testing
In the event that your JMeter client machine is unable, performance-wise, to simulate enough users to stress your server or is limited at network level, an option exists to control multiple, remote JMeter engines from a single JMeter client. By running JMeter remotely, you can replicate a test across many low-end computers and thus simulate a larger load on the server. One instance of the JMeter client can control any number of remote JMeter instances, and collect all the data from them. This offers the following features:
- Saving of test samples to the local machine
- Managment of multiple JMeterEngines from a single machine
- No need to copy the test plan to each server - the client sends it to all the servers
However, remote mode does use more resources than running the same number of non-GUI tests independently. If many server instances are used, the client JMeter can become overloaded, as can the client network connection. This has been improved by switching to Stripped modes (see below) but you should always check that your client is not overloaded.
Note that while you can execute the JMeterEngine on your application server, you need to be mindful of the fact that this will be adding processing overhead on the application server and thus your testing results will be somewhat tainted. The recommended approach is to have one or more machines on the same Ethernet segment as your application server that you configure to run the JMeter Engine. This will minimize the impact of the network on the test results without impacting the performance of the application server itself.
Step 0: Configure the nodes
Make sure that all the nodes (client and servers) :
- are running exactly the same version of JMeter.
- are using the same version of Java on all systems. Using different versions of Java may work but is discouraged.
If the test uses any data files, note that these are not sent across by the client so make sure that these are available in the appropriate directory on each server. If necessary you can define different values for properties by editing the user.properties or system.properties files on each server. These properties will be picked up when the server is started and may be used in the test plan to affect its behaviour (e.g. connecting to a different remote server). Alternatively use different content in any datafiles used by the test (e.g. if each server must use unique ids, divide these between the data files)
Step 1: Start the servers
To run JMeter in remote node, start the JMeter server component on all machines you wish to run on by running the JMETER_HOME/bin/jmeter-server (unix) or JMETER_HOME/bin/jmeter-server.bat (windows) script.
Note that there can only be one JMeter server on each node unless different RMI ports are used.
Since JMeter 2.3.1, the JMeter server application starts the RMI registry itself; there is no need to start RMI registry separately. To revert to the previous behaviour, define the JMeter property server.rmi.create=false on the server host systems.
By default, RMI uses a dynamic port for the JMeter server engine. This can cause problems for firewalls, so with versions of JMeter after 2.3.2 you can define the JMeter property server.rmi.localport to control this port number. If this is non-zero, it will be used as the local port number for the server engine.
Step 2: Add the server IP to your client's Properties File
Edit the properties file on the controlling JMeter machine. In /bin/jmeter.properties, find the property named, "remote_hosts", and add the value of your running JMeter server's IP address. Multiple such servers can be added, comma-delimited.
Note that you can use the -R command line option instead to specify the remote host(s) to use. This has the same effect as using -r and -Jremote_hosts={serverlist}. E.g. jmeter -Rhost1,127.0.0.1,host2
If you define the JMeter property server.exitaftertest=true, then the server will exit after it runs a single test. See also the -X flag (described below)
Step 3a: Start the JMeter Client from a GUI client to check configuration
Now you are ready to start the controlling JMeter client. For MS-Windows, start the client with the script "bin/jmeter.bat". For UNIX, use the script "bin/jmeter". You will notice that the Run menu contains two new sub-menus: "Remote Start" and "Remote Stop" (see figure 1). These menus contain the client that you set in the properties file. Use the remote start and stop instead of the normal JMeter start and stop menu items.
![Figure 1 - Run Menu](../images/screenshots/remote/run-menu00.gif)
Step 3b: Start the JMeter from a non-GUI Client
GUI mode should only be used for debugging, as a better alternative, you should start the test on remote server(s) from a non-GUI (command-line) client. The command to do this is:
jmeter -n -t script.jmx -r or jmeter -n -t script.jmx -R server1,server2... Other flags that may be useful: -Gproperty=value - define a property in all the servers (may appear more than once) -X - Exit remote servers at the end of the test.The first example will start the test on whatever servers are defined in the JMeter property remote_hosts;
The second example will define remote_hosts from the list of servers and then start the test on the remote servers.
The command-line client will exit when all the remote servers have stopped.
15.1 Doing it Manually¶
In some cases, the jmeter-server script may not work for you (if you are using an OS platform not anticipated by the JMeter developers). Here is how to start the JMeter servers (step 1 above) with a more manual process:
Step 1a: Start the RMI Registry
Since JMeter 2.3.1, the RMI registry is started by the JMeter server, so this section does not apply in the normal case. To revert to the previous behaviour, define the JMeter property server.rmi.create=false on the server host systems and follow the instructions below.
JMeter uses Remote Method Invocation (RMI) as the remote communication mechanism. Therefore, you need to run the RMI Registry application (which is named, "rmiregistry") that comes with the JDK and is located in the "bin" directory. Before running rmiregistry, make sure that the following jars are in your system claspath:
- JMETER_HOME/lib/ext/ApacheJMeter_core.jar
- JMETER_HOME/lib/jorphan.jar
- JMETER_HOME/lib/logkit-2.0.jar
Step 1b: Start the JMeter Server
Once the RMI Registry application is running, start the JMeter Server. Use the "-s" option with the jmeter startup script ("jmeter -s").
Steps 2 and 3 remain the same.
15.2 Tips¶
JMeter/RMI requires a connection from the client to the server. This will use the port you chose, default 1099.
JMeter/RMI also requires a reverse connection in order to return sample results from the server to the client.
This will use a high-numbered port.
This port can be controlled by jmeter property called client.rmi.localport in jmeter.properties.
If there are any firewalls or other network filters between JMeter client and server,
you will need to make sure that they are set up to allow the connections through.
If necessary, use monitoring software to show what traffic is being generated.
If you're running Suse Linux, these tips may help. The default installation may enable the firewall. In that case, remote testing will not work properly. The following tips were contributed by Sergey Ten.
If you see connections refused, turn on debugging by passing the following options.
rmiregistry -J-Dsun.rmi.log.debug=true -J-Dsun.rmi.server.exceptionTrace=true -J-Dsun.rmi.loader.logLevel=verbose -J-Dsun.rmi.dgc.logLevel=verbose -J-Dsun.rmi.transport.logLevel=verbose -J-Dsun.rmi.transport.tcp.logLevel=verboseSince JMeter 2.3.1, the RMI registry is started by the server; however the options can still be passed in from the JMeter command line. For example: "jmeter -s -Dsun.rmi.loader.logLevel=verbose" (i.e. omit the -J prefixes). Alternatively the properties can be defined in the system.properties file.
The solution to the problem is to remove the loopbacks 127.0.0.1 and 127.0.0.2 from etc/hosts. What happens is jmeter-server can't connect to rmiregistry if 127.0.0.2 loopback is not available. Use the following settings to fix the problem.
Replace
- `dirname $0`/jmeter -s "$@"
With
- HOST="-Djava.rmi.server.hostname=[computer_name][computer_domain]
- -Djava.security.policy=`dirname $0`/[policy_file]"
- `dirname $0`/jmeter $HOST -s "$@"
Also create a policy file and add [computer_name][computer_domain] line to /etc/hosts.
In order to better support SSH-tunneling of the RMI communication channels used in remote testing, since JMeter 2.6:
- a new property "client.rmi.localport" can be set to control the RMI port used by the RemoteSampleListenerImpl
- To support tunneling RMI traffic over an SSH tunnel as the remote endpoint using a port on the local machine, loopback interface is now allowed to be used if it has been specified directly using the Java System Property "java.rmi.server.hostname" parameter.
15.3 Using a different port¶
By default, JMeter uses the standard RMI port 1099. It is possible to change this. For this to work successfully, all the following need to agree:
- On the server, start rmiregistry using the new port number
- On the server, start JMeter with the property server_port defined
- On the client, update the remote_hosts property to include the new remote host:port settings
Since Jmeter 2.1.1, the jmeter-server scripts provide support for changing the port. For example, assume you want to use port 1664 (perhaps 1099 is already used).
On Windows (in a DOS box) C:\JMETER> SET SERVER_PORT=1664 C:\JMETER> JMETER-SERVER [other options] On Unix: $ SERVER_PORT=1664 jmeter-server [other options] [N.B. use upper case for the environment variable]
In both cases, the script starts rmiregistry on the specified port, and then starts JMeter in server mode, having defined the "server_port" property.
The chosen port will be logged in the server jmeter.log file (rmiregistry does not create a log file).
15.4 Using a different sample sender¶
Listeners in the test plan send their results back to the client JMeter which writes the results to the specified files By default, samples are sent back synchronously as they are generated. This can affect the maximum throughput of the server test; the sample result has to be sent back before the thread can continue. There are some JMeter properties that can be set to alter this behaviour.
- mode - sample sending mode - default is StrippedBatch since 2.9. This should be set on the client node.
- Standard - send samples synchronously as soon as they are generated
- Hold - hold samples in an array until the end of a run. This may use a lot of memory on the server and is discouraged.
- DiskStore - store samples in a disk file (under java.io.temp) until the end of a run. The serialised data file is deleted on JVM exit.
- StrippedDiskStore - remove responseData from succesful samples, and use DiskStore sender to send them.
- Batch - send saved samples when either the count (num_sample_threshold) or time (time_threshold) exceeds a threshold,
at which point the samples are sent synchronously.
The thresholds can be configured on the server using the following properties:
- num_sample_threshold - number of samples to accumulate, default 100
- time_threshold - time threshold, default 60000 ms = 60 seconds
See also the Asynch mode, described below.
- Statistical - send a summary sample when either the count or time exceeds a threshold.
The samples are summarised by thread group name and sample label.
The following fields are accumulated:
- elapsed time
- latency
- bytes
- sample count
- error count
- Stripped - remove responseData from succesful samples
- StrippedBatch - remove responseData from succesful samples, and use Batch sender to send them.
- Asynch - samples are temporarily stored in a local queue. A separate worker thread sends the samples. This allows the test thread to continue without waiting for the result to be sent back to the client. However, if samples are being created faster than they can be sent, the queue will eventually fill up, and the sampler thread will block until some samples can be drained from the queue. This mode is useful for smoothing out peaks in sample generation. The queue size can be adjusted by setting the JMeter property asynch.batch.queue.size (default 100) on the server node.
- StrippedAsynch - remove responseData from succesful samples, and use Async sender to send them.
- Custom implementation : set the mode parameter to your custom sample sender class name. This must implement the interface SampleSender and have a constructor which takes a single parameter of type RemoteSampleListener.
This is not really a problem as there is always a more efficient way to implement this feature.
The following properties apply to the Batch and Statistical modes:
- num_sample_threshold - number of samples in a batch (default 100)
- time_threshold - number of milliseconds to wait (default 60 seconds)
15.5 Dealing with nodes that failed starting¶
For large-scale tests there is a chance that some part of remote servers will be unavailable or down. For example, when you use automation script to allocate many cloud machines and use them as generators, some of requested machines might fail booting because of cloud's issues. Since JMeter 2.13 there are new properties to control this behaviour.
First what you might want is to retry initialization attempts in hope that failed nodes just slightly delayed their boot. To enable retries, you should set client.tries property to total number of connection attempts. By default it does only one attempt. To control retry delay, set the client.retries_delay property to number of milliseconds to sleep between attempts.
Finally, you might still want to run the test with those generators that succeeded initialization and skipping failed nodes. To enable that, set the client.continue_on_fail=true property.