Performance

This page covers setting up and running performance testing using JMeter.

JMeter is open-source software and works by simulating high volumes of HTTP (Hypertext Transfer Protocol) requests (usually POST requests) to a web server, and provides a way to view the outputs.

It allows you to record test steps, edit each request, and then execute the test using a given number of threads (users).

While there is plenty of documentation on the web, this page intends to show what's important for testing GOV.UK services and common pitfalls, without going into unnecessary detail.

JMeter site

JMeter user manual

Contents

• What you'll need
• Install JMeter
• Create a standard test plan
• Configure Firefox to record tests
• Set up your test environment
• Record a test
• Delete the items you don't want
• Make the tests repeatable
• Add assertions
• Run a test from the command line
• Upload the tests to GitHub
• Run the tests from Jenkins
• Review the results
• Pitfalls

What you'll need

• An off-network machine. This guide assumes you're using a Mac, but most of the guidance is the same for a PC.
• To record and debug tests, you can install JMeter on your own machine.
• To execute tests, you'll need to run them from a dedicated performance test server. Speak to another member of the QA team to discuss how this was set up.
• Install a recent version of Firefox onto your machine, to help with recording tests.
• Access to the command prompt or terminal on your machine
• A livelike preproduction environment, with stubs for any external services - for example Ordnance Survey address lookups
• Access to Jenkins in order to run tests from a performance test server

This is what a typical JMeter test plan looks like:

Install JMeter

• Ensure that the version of JMeter on the server you'll be executing from, and the version you're writing the tests on, will be compatible.
• On a Mac, we recommend using Homebrew to make it easier to install things. The rest of this section assumes you're using it.

In Terminal, type:

brew install jmeter

JMeter will be installed in the following hidden folder. (The shortcut to view hidden files in Finder is Cmd Shift . )

Macintosh HD/usr/local/Cellar/jmeter

It is also worth installing plugins to give you more options to run JMeter tests and view reports. Follow the instructions on JMeter Plugins to do this.

To start JMeter, type jmeter from any folder in Terminal.

Create a standard test plan

Your test plan will need to include several standard elements which need to be set up for any new test.

You will need to use the JMeter GUI (graphical user interface) to record and debug scripts, but not to execute full performance tests. This is because the GUI uses a lot of memory itself and usually becomes unresponsive when a large number of tests have been run and are stored in memory.

Remember to save the whole test plan regularly in case JMeter crashes. This will be a file with extension .jmx.

Action	Details
Add a thread group	Right click Test Plan > Add > Threads > Thread Group This tells JMeter how many users send how many requests over time. For example, 5 users starting 1 second apart, running twice.
Add HTTP Request Defaults	Right click Thread Group > Add > Config Element > HTTP Request Defaults This defines the default properties for each request so that they don't have to be repeated in each test step. Under the HTTP Request, enter the root URL for your test environment.
Add a cookie handler	Right click Thread Group > Add > Config Element > HTTP Cookie Handler Required so that the tests can support cookies. This should work out of the box with no changes.
Add a transaction controller	Thread Group > Add > Logic Controller > Transaction Controller, clicking generate parent sample This is the "folder" where your recorded tests will be stored.
Configure JMeter to record tests	Right click Test Plan> Add > Non test elements > HTTP(S) Script Recorder. Choose a port number which you're not already using on a local test environment. 8000 is often good. Set the Target Controller as the transaction controller created above.
Add a results tree	Right click Thread Group > Add > Listener > View Results Tree This shows your test outputs (pass/fail) and captures the HTTP request and response for each step, including an HTML view. You'll use this later if you need to share data between steps.

At this stage, your test plan should hopefully look something like this:

Configure Firefox to record tests

Set the proxy

To record tests, you'll need to configure a proxy in Firefox through which requests and responses can be read by JMeter.

On Firefox, go to preferences > general > network settings and set:

• HTTP proxy: 127.0.0.1 (this represents your local machine)
• Port: 8000 (or whichever one you chose for the HTTP Test Script Recorder)
• Tick the option to allow HTTPS.

There is a tool called FoxyProxy which allows for quick proxy switching.

Trust the JMeter certificate

For Firefox to allow you to record tests, you need to trust the certificate issued by JMeter. In the JMeter recorder, click Start and you'll see a window giving details of the certificate. This will be stored in JMeter's install directory, under /bin.

You'll need Firefox to trust the Apache certificate authority. Certificate instructions

Set up your test environment

You will want to run performance tests in an environment which is as close to live as possible. In most of our services this is the preproduction environment.

However, it is important that you do not send large numbers of calls to any external services, as this will break the licence agreement we have with them. For example, if your service includes an address lookup, it is most likely calling Ordnance Survey. This is usually fine in functional testing with small volumes, but not for performance testing.

Speak to your developers to work out how to mock responses for any external services.

Record a test

Once you're satisfied that Firefox can record tests, go to the Test Script Recorder and click Start.

Carry out the main steps you plan to run in Firefox and click Stop when finished.

Delete the items you don't want

At each step, JMeter will generate more files than you actually need to run the tests.

You should be able to delete the following:

• Google Analytics and Google Tag Manager requests. Check the server name under the requests and delete them.
• Anything generated by the browser. For example, Firefox may send telemetry data for its own analytics.
• Some GET requests. Particularly where a services uses redirects, each page may include both a GET and POST request. Unless you need the GET request to navigate from one page to another, you will mainly only need the POST requests to rerun the tests. It's usually fine to delete any GET requests which are greyed out (disabled) by default.
• Redirects. Some services include automatic redirects as part of a journey. For example, there might be a session token included in the URL, and accessing the /start link redirects to /(token)/start. You should make the POST request from the original URL in JMeter and ignore the redirect URL.
• Cascading Style Sheet files (CSS)
• Images

If you accidentally delete something you needed, you can always re-record that step and drag it to the right place.

On a Mac, the backspace key does not work to delete tests. If you want to select and delete lots of steps at once, use Cmd X to cut them.

Make the tests repeatable

The first time you try to run a test that you've recorded, it will usually fail.

You will often need to read a dynamically-generated token, which changes on every test run, from one step and pass it into subsequent steps.

While the exact process varies by service, the process usually follows a similar pattern. Here are two examples of passing a token through a journey:

Passing a token generated in the HTML

Sometimes, a token will appear in the HTML first POST request and will need to be passed through the journey in all subsequent POST requests.

1. Run the tests once and go to the results tree.

2. Find the first step where a token appears in the HTML.

3. Use trial and error to find a regular expression (regex) which matches the HTML containing the token, for example, _token" value="(.*)". The expression (.*) means any number of any characters, and represents the gap which contains the token:

4. Under the same test step that was viewed in the results tree, add a Regular Expression Extractor. Include the regex you used in the last step and set the variable name to match the name of the token, for example, example_token. Under template, set $1$. The other values are optional:

5. Under the next step that contains a POST request with parameters, call the variable you created from the last step in the format ${example_token}:

If done correctly and you've captured all the tokens, this step should pass when you next run the test.

If not then re-run the test and look at the response. If you see an error such as "422 unprocessable entity" then this could mean that there is another token, or you've captured the current one incorrectly.

Passing a token through a URL

Sometimes a token generated at the start of a journey will be included in the URL and need to be included in all subsequent URLs.

1. Looking at the results tree, find the first step which introduces a new token. For example, something that gets included in every URL when running through a user journey.

2. Add a regular expression extractor to the end of this step, and save the data to a variable. For example:

• One URL includes a token ABCDEF and has path: /fo/ABCDEF/start
• The next URL includes the same token and has path: /fo/ABCDEF/location
• In the regular expression extractor, select URL and search for /fo/(.*) to capture the URL
• This saves what's in (.*) as a variable whose name you can select, such as url_token

3. In the next POST request, set the URL as /fo/${url_token}/location:

Advice on adding dynamic URLs

Add assertions

While the intent of a performance test is not to test functionality, it is useful to add a small number of assertions into a performance test to ensure that it is on the right track. For example, this might be a check that you can see the text "Registration complete" at the end of a journey, and perhaps a check mid-way through.

Do this by adding a Response Assertion under the step you want to check.

Run a test from the command line

Once you're happy that the test will run reliably using a small number of users, the next step is to trigger the test from the Command Line Interface (CLI mode). Explanation of terms used in CLI mode

In Terminal, navigate to (JMeter version)/bin and enter the following. This assumes that the tests are stored in a folder called Tests which is two folders up from the JMeter /bin folder, but your configuration may be different.

jmeter -n -t "../../Tests/project-name/Testplan.jmx" -l ../../Tests/project-name/testresults.jtl

This will produce a raw reports file that you can open up in JMeter after you finish testing.

To generate a full HTML report as well, create an empty htmlreport folder in your test folder and use:

jmeter -n -t "../../Tests/project-name/Testplan.jmx" -l ../../Tests/water-abstraction/testresults.jtl -e -o  "../../Tests/water-abstraction/htmlreport"

Remember to use straight quotes (not smart quotes) here, and make sure that the paths actually point to the correct folder and file.

If you are re-running a test then remember to move the original HTML report first, otherwise it may get overwritten or cause an error.

Upload the tests to GitHub

JMeter tests are stored in a private repository. Contact another member of the team for access.

Run the tests from Jenkins

Once the test suite is working locally, it should be run from a dedicated performance test server, to more accurately simulate external input.

These tests are triggered from Jenkins and call the test from the GitHub repository containing the .jmx file.

The easiest way to set this up is to follow an existing example - speak to another team member.

Review the results

Once the tests have finished running, you can open your test plan in the JMeter GUI and add any graphs to help you visualise the results. Under the graph you've chosen, load data from the test results file. This will have a .jtl extension.

Right click Thread Group > Add > Listener and choose the graphs you want.

The following graphs have been found to be useful in the past:

• Active threads over time
• Response times over time
• Response times percentiles
• Transactions per second

You may need to install these on top of JMeter. Use Options > Plugin Manager to find and install these. On older versions of JMeter these graphs are prefixed with jp@gc

Take care not to open more graphs than you need at once, as they can cause JMeter to become unresponsive.

Pitfalls

• If the versions of JMeter you use to write and run the tests are significantly different, they may not work. Try to use the same version where possible, or at least the same major version (such as version 5.x.x)
• On older versions of JMeter, there is an icon of a floppy disk with a pencil on it. DO NOT CLICK THIS. It can overwrite your test suite with a single element from it, making you lose your work.
• Test reports will not be generated if there are output files from a previous run in the output folders. Make sure that all required folders exist, but are empty, before running a test.
• HTML reports don't work in IE11. Use Firefox instead.
• Variables cannot be redefined during execution. More advice on variables. If you want to change a parameter during execution, you can add a "User Parameter" preprocessor to the appropriate step instead.