Skip to content

Commit

Permalink
Remove multivariate test
Browse files Browse the repository at this point in the history
* The multivariate test framework dynamically updates page content
after the page has loaded to show variants to users which are tracked
to GA
* GOV.UK is the main user of this and have created a server-side
alternative

An example of a test that was created in an inaccessible way:
alphagov/frontend#1228 (review)
4
  • Loading branch information
fofr committed Jun 20, 2017
1 parent b7a7f68 commit 86f1122
Show file tree
Hide file tree
Showing 5 changed files with 0 additions and 559 deletions.
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,6 @@ In production:
* [JavaScript](/docs/javascript.md)
* [Modules](/docs/javascript.md#modules)
* [Media player](/docs/javascript.md#media-player)
* [Multivariate test framework](/docs/javascript.md#multivariate-test-framework)
* [Primary links](/docs/javascript.md#primary-links)
* [Stick at top when scrolling](/docs/javascript.md#stick-at-top-when-scrolling)
* [Selection buttons](/docs/javascript.md#selection-buttons)
Expand Down
149 changes: 0 additions & 149 deletions docs/javascript.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,155 +134,6 @@ Accesible Media Player][nomensa] repository.

[nomensa]: https://github.com/nomensa/Accessible-Media-Player/tree/master/example

## Multivariate test framework

`GOVUK.MultiVariateTest` runs split tests to display different content, layouts etc to users.

It randomly assigns a user a cohort on first execution by setting a cookie, and on every execution sets a session level custom variable on Google Analytics to mark which cohort a user is in. This can be used to segment users in GA.

A simple content replacement test can be done by defining a set of cohorts with content. E.g.:

```javascript
var test = new GOVUK.MultivariateTest({
el: '.car-tax-button',
name: 'car_tax_button_text',
cohorts: {
pay_your_car_tax: {html: "Pay Your Car Tax"},
give_us_money: {html: "Give Us Money Or We Will Crush Your Car"}
}
});
```

A more complex test can be done by defining callbacks for what to do
when a user is in each cohort:

```javascript
var test = new GOVUK.MultivariateTest({
name: 'car_tax_button_text',
cohorts: {
pay_your_car_tax: {callback: function() { ... }},
give_us_money: {callback: function() { ... }}
}
});
```

If you want one cohort to appear 25% of the time then you can optionally weight
that cohort:

```javascript
var test = new GOVUK.MultivariateTest({
name: 'car_tax_button_text',
cohorts: {
pay_your_car_tax: {weight: 25, callback: function() { ... }}, // 25%
give_us_money: {weight: 75, callback: function() { ... }} // 75%
}
});
```

If you want to set the cookie expiration then you can optionally set cookieDuration as follows:

```javascript
var test = new GOVUK.MultivariateTest({
name: 'car_tax_button_text',
cookieDuration: 14,
cohorts: {
pay_your_car_tax: {weight: 25, callback: function() { ... }}, // 25%
give_us_money: {weight: 75, callback: function() { ... }} // 75%
}
});
```
Here, it is set to expire in 14 days time. if this option is not set the default cookie expiration (30 days) take effect.

If you have a complex test, it may be worth extending MultivariateTest with
your own. Callbacks can be strings which will call a method of that name
on the current object.

Takes these options:
- `el`: Element to run this test on (optional)
- `name`: The name of the text (alphanumeric and underscores), which will be part of the cookie.
- `defaultWeight`: Number of times each cohorts should appear in an array the random cohort is picked from, to be used in conjunction with weights on individual cohorts.
- `cohorts`: An object that maps cohort name to an object that defines the cohort. Name must be same format as test name. Object contains keys (all optional):
- `html`: HTML to fill element with when this cohort is picked.
- `callback`: Function to call when this cohort is chosen. If it is a string, that method on the test object is called.
- `weight`: Number of times this cohort should appear in an array the random cohort is picked from, defaults to the `defaultWeight` of the test.

Full documentation on how to design multivariate tests, use the data in GA and construct hypothesis tests is on its way soon.

### Reporting to Google Content Experiments
The toolkit includes a library for multivariate testing that is capable of reporting data into [Google Content Experiments](https://developers.google.com/analytics/devguides/platform/experiments-overview).

#### To create a new experiment

1. Log in to Google Universal Analytics, select "UA - 1. GOV.UK(Entire Site - Filtered)".

2. In the left column, click on Behaviour, then Experiments and follow [these instructions](https://support.google.com/analytics/answer/1745152?hl=en-GB) to set up your experiment; you will need to have edit permissions on the Universal Analytics profile. If you cannot see a "Create experiment" button, this means you don't have these permissions; you can ask someone from the Performance Analyst team to set the experiment up for you.

3. In step number 2, in our case the address of the web pages will purely be used as descriptions in reports, so we recommend you pick the addresses accordingly, ie: "www.gov.uk" and "www.gov.uk/?=variation1".

4. In step number 3, "Setting up your experiment code", select "Manually insert the code" and make a note of the Experiment ID number located under the script window.

5. Add the below code to the page you want to test.
- the contentExperimentId is the Experiment ID you retrieved in step 3.
- the variantId is 0 for the original variant, 1 for the first modified variant, 2 for the second modified variant, etc.
- see section above for other elements.
This code requires analytics to be loaded in order to run; static is the app that would load the analytics by default, which automatically happens before experiments are run.

6. Check that it works: launch the app, open the page of your app, and check that there is a cookie with the name you had picked in your experiment. You can delete the cookie and refresh the page to check whether you can be assigned to another cohort. Then in your browser console, go to the Networks tab and search for xid and xvar. The values should correspond to your contentExperimentId and the cohort your cookie indicates you were assigned to.

7. If it works, in Google Universal Analytics, click on "Next Step" and then "Start Experiment". You can ignore the error messages relative to Experiment Code Validation as they don't concern us in our setup.

```js
var test = new GOVUK.MultivariateTest({
name: 'car_tax_button_text',
contentExperimentId: "Your_Experiment_ID",
cohorts: {
pay_your_car_tax: {weight: 25, variantId: 0},
give_us_money: {weight: 25, variantId: 1}
}
});

```

### Using Google custom dimensions with your own statistical model

It is possible to use Google custom dimensions for determining the results of
the multivariate test (as an alternative to Google Content Experiments). This
may be appropriate if you wish to build the statistical model yourself or you
aren't able to use Content Experiments for some reason.

This requires setting the optional `customDimensionIndex` variable:

```js
var test = new GOVUK.MultivariateTest({
name: 'car_tax_button_text',
customDimensionIndex: 555,
cohorts: {
pay_your_car_tax: {weight: 25},
give_us_money: {weight: 50}
}
});
```

`customDimensionIndex` is the index of the custom variable in Google Analytics. GA only gives 50 integer slots to each account, and it is important that a unique integer is assigned to each test. Current contact for assigning a custom var slot for GOV.UK is: Tim Leighton-Boyce <tim.leighton-boyce@digital.cabinet-office.gov.uk>

For certain types of tests where the custom dimensions in Google Analytics can only have one of three scopes (hit, session, or user). Measuring performance of A/B tests, and want to compare test results for session-scoped custom dimensions (eg: index 222) against hit-scoped ones (eg: index 223) may become important.

Instead of supplying an integer (like the above example), `customDimensionIndex` can also accept an array of dimension indexes ([222, 223]), see below for more.

Make sure to check GA debugger that these values are being sent before deploying.

```js
var test = new GOVUK.MultivariateTest({
name: 'car_tax_button_text',
customDimensionIndex: [222, 223],
cohorts: {
pay_your_car_tax: {weight: 25},
give_us_money: {weight: 50}
}
});
```


## Primary Links

`GOVUK.PrimaryList` hides elements in a list which don't have a supplied
Expand Down
145 changes: 0 additions & 145 deletions javascripts/govuk/multivariate-test.js

This file was deleted.

2 changes: 0 additions & 2 deletions spec/manifest.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ var manifest = {
'../../node_modules/jquery/dist/jquery.js',
'../../javascripts/govuk/modules.js',
'../../javascripts/govuk/modules/auto-track-event.js',
'../../javascripts/govuk/multivariate-test.js',
'../../javascripts/govuk/primary-links.js',
'../../javascripts/govuk/shim-links-with-button-role.js',
'../../javascripts/govuk/show-hide-content.js',
Expand All @@ -22,7 +21,6 @@ var manifest = {
test: [
'../unit/modules.spec.js',
'../unit/Modules/auto-track-event.spec.js',
'../unit/multivariate-test.spec.js',
'../unit/primary-links.spec.js',
'../unit/shim-links-with-button-role.spec.js',
'../unit/show-hide-content.spec.js',
Expand Down
Loading

0 comments on commit 86f1122

Please sign in to comment.