diff --git a/README.md b/README.md index b0781dc7..f0e97734 100644 --- a/README.md +++ b/README.md @@ -141,7 +141,6 @@ In production: * [JavaScript](/docs/javascript.md) * [Modules](/docs/javascript.md#modules) * [Media player](/docs/javascript.md#media-player) - * [Multivariate test framework](/docs/javascript.md#multivariate-test-framework) * [Primary links](/docs/javascript.md#primary-links) * [Stick at top when scrolling](/docs/javascript.md#stick-at-top-when-scrolling) * [Selection buttons](/docs/javascript.md#selection-buttons) diff --git a/docs/javascript.md b/docs/javascript.md index 84afd2ed..c24d2530 100644 --- a/docs/javascript.md +++ b/docs/javascript.md @@ -134,155 +134,6 @@ Accesible Media Player][nomensa] repository. [nomensa]: https://github.com/nomensa/Accessible-Media-Player/tree/master/example -## Multivariate test framework - -`GOVUK.MultiVariateTest` runs split tests to display different content, layouts etc to users. - -It randomly assigns a user a cohort on first execution by setting a cookie, and on every execution sets a session level custom variable on Google Analytics to mark which cohort a user is in. This can be used to segment users in GA. - -A simple content replacement test can be done by defining a set of cohorts with content. E.g.: - -```javascript -var test = new GOVUK.MultivariateTest({ - el: '.car-tax-button', - name: 'car_tax_button_text', - cohorts: { - pay_your_car_tax: {html: "Pay Your Car Tax"}, - give_us_money: {html: "Give Us Money Or We Will Crush Your Car"} - } -}); -``` - -A more complex test can be done by defining callbacks for what to do -when a user is in each cohort: - -```javascript -var test = new GOVUK.MultivariateTest({ - name: 'car_tax_button_text', - cohorts: { - pay_your_car_tax: {callback: function() { ... }}, - give_us_money: {callback: function() { ... }} - } -}); -``` - -If you want one cohort to appear 25% of the time then you can optionally weight -that cohort: - -```javascript -var test = new GOVUK.MultivariateTest({ - name: 'car_tax_button_text', - cohorts: { - pay_your_car_tax: {weight: 25, callback: function() { ... }}, // 25% - give_us_money: {weight: 75, callback: function() { ... }} // 75% - } -}); -``` - -If you want to set the cookie expiration then you can optionally set cookieDuration as follows: - -```javascript -var test = new GOVUK.MultivariateTest({ - name: 'car_tax_button_text', - cookieDuration: 14, - cohorts: { - pay_your_car_tax: {weight: 25, callback: function() { ... }}, // 25% - give_us_money: {weight: 75, callback: function() { ... }} // 75% - } -}); -``` -Here, it is set to expire in 14 days time. if this option is not set the default cookie expiration (30 days) take effect. - -If you have a complex test, it may be worth extending MultivariateTest with -your own. Callbacks can be strings which will call a method of that name -on the current object. - -Takes these options: - - `el`: Element to run this test on (optional) - - `name`: The name of the text (alphanumeric and underscores), which will be part of the cookie. - - `defaultWeight`: Number of times each cohorts should appear in an array the random cohort is picked from, to be used in conjunction with weights on individual cohorts. - - `cohorts`: An object that maps cohort name to an object that defines the cohort. Name must be same format as test name. Object contains keys (all optional): - - `html`: HTML to fill element with when this cohort is picked. - - `callback`: Function to call when this cohort is chosen. If it is a string, that method on the test object is called. - - `weight`: Number of times this cohort should appear in an array the random cohort is picked from, defaults to the `defaultWeight` of the test. - -Full documentation on how to design multivariate tests, use the data in GA and construct hypothesis tests is on its way soon. - -### Reporting to Google Content Experiments -The toolkit includes a library for multivariate testing that is capable of reporting data into [Google Content Experiments](https://developers.google.com/analytics/devguides/platform/experiments-overview). - -#### To create a new experiment - -1. Log in to Google Universal Analytics, select "UA - 1. GOV.UK(Entire Site - Filtered)". - -2. In the left column, click on Behaviour, then Experiments and follow [these instructions](https://support.google.com/analytics/answer/1745152?hl=en-GB) to set up your experiment; you will need to have edit permissions on the Universal Analytics profile. If you cannot see a "Create experiment" button, this means you don't have these permissions; you can ask someone from the Performance Analyst team to set the experiment up for you. - -3. In step number 2, in our case the address of the web pages will purely be used as descriptions in reports, so we recommend you pick the addresses accordingly, ie: "www.gov.uk" and "www.gov.uk/?=variation1". - -4. In step number 3, "Setting up your experiment code", select "Manually insert the code" and make a note of the Experiment ID number located under the script window. - -5. Add the below code to the page you want to test. - - the contentExperimentId is the Experiment ID you retrieved in step 3. - - the variantId is 0 for the original variant, 1 for the first modified variant, 2 for the second modified variant, etc. - - see section above for other elements. -This code requires analytics to be loaded in order to run; static is the app that would load the analytics by default, which automatically happens before experiments are run. - -6. Check that it works: launch the app, open the page of your app, and check that there is a cookie with the name you had picked in your experiment. You can delete the cookie and refresh the page to check whether you can be assigned to another cohort. Then in your browser console, go to the Networks tab and search for xid and xvar. The values should correspond to your contentExperimentId and the cohort your cookie indicates you were assigned to. - -7. If it works, in Google Universal Analytics, click on "Next Step" and then "Start Experiment". You can ignore the error messages relative to Experiment Code Validation as they don't concern us in our setup. - -```js -var test = new GOVUK.MultivariateTest({ - name: 'car_tax_button_text', - contentExperimentId: "Your_Experiment_ID", - cohorts: { - pay_your_car_tax: {weight: 25, variantId: 0}, - give_us_money: {weight: 25, variantId: 1} - } -}); - -``` - -### Using Google custom dimensions with your own statistical model - -It is possible to use Google custom dimensions for determining the results of -the multivariate test (as an alternative to Google Content Experiments). This -may be appropriate if you wish to build the statistical model yourself or you -aren't able to use Content Experiments for some reason. - -This requires setting the optional `customDimensionIndex` variable: - -```js -var test = new GOVUK.MultivariateTest({ - name: 'car_tax_button_text', - customDimensionIndex: 555, - cohorts: { - pay_your_car_tax: {weight: 25}, - give_us_money: {weight: 50} - } -}); -``` - -`customDimensionIndex` is the index of the custom variable in Google Analytics. GA only gives 50 integer slots to each account, and it is important that a unique integer is assigned to each test. Current contact for assigning a custom var slot for GOV.UK is: Tim Leighton-Boyce - -For certain types of tests where the custom dimensions in Google Analytics can only have one of three scopes (hit, session, or user). Measuring performance of A/B tests, and want to compare test results for session-scoped custom dimensions (eg: index 222) against hit-scoped ones (eg: index 223) may become important. - -Instead of supplying an integer (like the above example), `customDimensionIndex` can also accept an array of dimension indexes ([222, 223]), see below for more. - -Make sure to check GA debugger that these values are being sent before deploying. - -```js -var test = new GOVUK.MultivariateTest({ - name: 'car_tax_button_text', - customDimensionIndex: [222, 223], - cohorts: { - pay_your_car_tax: {weight: 25}, - give_us_money: {weight: 50} - } -}); -``` - - ## Primary Links `GOVUK.PrimaryList` hides elements in a list which don't have a supplied diff --git a/javascripts/govuk/multivariate-test.js b/javascripts/govuk/multivariate-test.js deleted file mode 100644 index e591b64c..00000000 --- a/javascripts/govuk/multivariate-test.js +++ /dev/null @@ -1,145 +0,0 @@ -;(function (global) { - 'use strict' - - var $ = global.jQuery - var GOVUK = global.GOVUK || {} - - // A multivariate test framework - // - // Based loosely on https://github.com/jamesyu/cohorts - // - // Full documentation is in README.md. - // - function MultivariateTest (options) { - this.$el = $(options.el) - this._loadOption(options, 'name') - this._loadOption(options, 'customDimensionIndex', null) - this._loadOption(options, 'cohorts') - this._loadOption(options, 'runImmediately', true) - this._loadOption(options, 'defaultWeight', 1) - this._loadOption(options, 'contentExperimentId', null) - this._loadOption(options, 'cookieDuration', 30) - - if (this.runImmediately) { - this.run() - } - } - - MultivariateTest.prototype._loadOption = function (options, key, defaultValue) { - if (options[key] !== undefined) { - this[key] = options[key] - } - if (this[key] === undefined) { - if (defaultValue === undefined) { - throw new Error(key + ' option is required for a multivariate test') - } else { - this[key] = defaultValue - } - } - } - - MultivariateTest.prototype.run = function () { - var cohort = this.getCohort() - if (cohort) { - this.setUpContentExperiment(cohort) - this.setCustomVar(cohort) - this.executeCohort(cohort) - this.createDummyEvent(cohort) - } - } - - MultivariateTest.prototype.executeCohort = function (cohort) { - var cohortObj = this.cohorts[cohort] - if (cohortObj.callback) { - if (typeof cohortObj.callback === 'string') { - this[cohortObj.callback]() - } else { - cohortObj.callback() - } - } - if (cohortObj.html) { - this.$el.html(cohortObj.html) - this.$el.show() - } - } - - // Get the current cohort or assign one if it has not been already - MultivariateTest.prototype.getCohort = function () { - var cohort = GOVUK.cookie(this.cookieName()) - if (!cohort || !this.cohorts[cohort]) { - cohort = this.chooseRandomCohort() - GOVUK.cookie(this.cookieName(), cohort, {days: this.cookieDuration}) - } - return cohort - } - - MultivariateTest.prototype.setCustomVar = function (cohort) { - if (this.customDimensionIndex && - this.customDimensionIndex.constructor === Array) { - for (var index = 0; index < this.customDimensionIndex.length; index++) { - this.setDimension(cohort, this.customDimensionIndex[index]) - } - } else if (this.customDimensionIndex) { - this.setDimension(cohort, this.customDimensionIndex) - } - } - - MultivariateTest.prototype.setDimension = function (cohort, dimension) { - GOVUK.analytics.setDimension( - dimension, - this.cookieName() + '__' + cohort - ) - } - - MultivariateTest.prototype.setUpContentExperiment = function (cohort) { - var contentExperimentId = this.contentExperimentId - var cohortVariantId = this.cohorts[cohort]['variantId'] - if (typeof contentExperimentId !== 'undefined' && - typeof cohortVariantId !== 'undefined' && - typeof window.ga === 'function') { - window.ga('set', 'expId', contentExperimentId) - window.ga('set', 'expVar', cohortVariantId) - }; - } - - MultivariateTest.prototype.createDummyEvent = function (cohort) { - // Fire off a dummy event to set the custom var and the content experiment on the page. - // Ideally we'd be able to call setCustomVar before trackPageview, - // but would need reordering the existing GA code. - GOVUK.analytics.trackEvent(this.cookieName(), 'run', {nonInteraction: true}) - } - - MultivariateTest.prototype.weightedCohortNames = function () { - var names = [] - var defaultWeight = this.defaultWeight - - $.each(this.cohorts, function (key, cohortSettings) { - var numberForCohort, i - - if (typeof cohortSettings.weight === 'undefined') { - numberForCohort = defaultWeight - } else { - numberForCohort = cohortSettings.weight - } - - for (i = 0; i < numberForCohort; i++) { - names.push(key) - } - }) - - return names - } - - MultivariateTest.prototype.chooseRandomCohort = function () { - var names = this.weightedCohortNames() - return names[Math.floor(Math.random() * names.length)] - } - - MultivariateTest.prototype.cookieName = function () { - return 'multivariatetest_cohort_' + this.name - } - - GOVUK.MultivariateTest = MultivariateTest - - global.GOVUK = GOVUK -})(window) diff --git a/spec/manifest.js b/spec/manifest.js index aea25a44..9ab0cb12 100644 --- a/spec/manifest.js +++ b/spec/manifest.js @@ -4,7 +4,6 @@ var manifest = { '../../node_modules/jquery/dist/jquery.js', '../../javascripts/govuk/modules.js', '../../javascripts/govuk/modules/auto-track-event.js', - '../../javascripts/govuk/multivariate-test.js', '../../javascripts/govuk/primary-links.js', '../../javascripts/govuk/shim-links-with-button-role.js', '../../javascripts/govuk/show-hide-content.js', @@ -22,7 +21,6 @@ var manifest = { test: [ '../unit/modules.spec.js', '../unit/Modules/auto-track-event.spec.js', - '../unit/multivariate-test.spec.js', '../unit/primary-links.spec.js', '../unit/shim-links-with-button-role.spec.js', '../unit/show-hide-content.spec.js', diff --git a/spec/unit/multivariate-test.spec.js b/spec/unit/multivariate-test.spec.js deleted file mode 100644 index df529518..00000000 --- a/spec/unit/multivariate-test.spec.js +++ /dev/null @@ -1,262 +0,0 @@ -/* global describe it expect beforeEach afterEach jasmine spyOn */ - -var $ = window.jQuery - -describe('MultivariateTest', function () { - 'use strict' - var GOVUK = window.GOVUK - - beforeEach(function () { - GOVUK.cookie = jasmine.createSpy('GOVUK.cookie') - GOVUK.analytics = {setDimension: function () {}, trackEvent: function () {}} - spyOn(GOVUK.analytics, 'setDimension') - spyOn(GOVUK.analytics, 'trackEvent') - }) - - afterEach(function () { - delete GOVUK.analytics - }) - - describe('#run', function () { - it('should pick a random cohort on first run', function () { - GOVUK.cookie.and.returnValue(null) - var fooSpy = jasmine.createSpy('fooSpy') - var barSpy = jasmine.createSpy('barSpy') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - cohorts: { - foo: {callback: fooSpy}, - bar: {callback: barSpy} - } - }) - - expect(GOVUK.cookie.calls.count()).toEqual(2) - expect(GOVUK.cookie.calls.argsFor(1)[0]).toEqual('multivariatetest_cohort_stuff') - if (GOVUK.cookie.calls.argsFor(1)[1] === 'foo') { - expect(fooSpy).toHaveBeenCalled() - } else { - expect(barSpy).toHaveBeenCalled() - } - }) - - it('should use an existing cohort choice on subsequent runs', function () { - GOVUK.cookie.and.returnValue('foo') - var fooSpy = jasmine.createSpy('fooSpy') - var barSpy = jasmine.createSpy('barSpy') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - cohorts: { - foo: {callback: fooSpy}, - bar: {callback: barSpy} - } - }) - expect(fooSpy).toHaveBeenCalled() - }) - - it('should set a custom var with the name and cohort if one is defined', function () { - GOVUK.cookie.and.returnValue('foo') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - cohorts: { - foo: {}, - bar: {} - }, - customDimensionIndex: 2 - }) - expect(GOVUK.analytics.setDimension).toHaveBeenCalledWith( - 2, - 'multivariatetest_cohort_stuff__foo' - ) - }) - - it('should be able to set multiple custom vars with the name and cohort if one is defined as an array', function () { - GOVUK.cookie.and.returnValue('foo') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - cohorts: { - foo: {}, - bar: {} - }, - customDimensionIndex: [2, 3] - }) - expect(GOVUK.analytics.setDimension).toHaveBeenCalledWith( - 2, - 'multivariatetest_cohort_stuff__foo' - ) - expect(GOVUK.analytics.setDimension).toHaveBeenCalledWith( - 3, - 'multivariatetest_cohort_stuff__foo' - ) - }) - - it('should trigger an event to track that the test has been run', function () { - GOVUK.cookie.and.returnValue('foo') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - cohorts: { - foo: {}, - bar: {} - } - }) - expect(GOVUK.analytics.trackEvent).toHaveBeenCalledWith( - 'multivariatetest_cohort_stuff', - 'run', - {nonInteraction: true} - ) - }) - - it('should set html for a cohort', function () { - GOVUK.cookie.and.returnValue('foo') - var $el = $('
') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - el: $el, - cohorts: { - foo: {html: 'foo'}, - bar: {html: 'bar'} - } - }) - expect($el.html()).toEqual('foo') - }) - - it('should call the callback for a cohort', function () { - var fooSpy = jasmine.createSpy('fooSpy') - var barSpy = jasmine.createSpy('barSpy') - GOVUK.cookie.and.returnValue('bar') - var $el = $('
') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - el: $el, - cohorts: { - foo: {callback: fooSpy}, - bar: {callback: barSpy} - } - }) - expect(barSpy).toHaveBeenCalled() - }) - - it('should call the callback for a cohort if it is a string', function () { - GOVUK.cookie.and.returnValue('foo') - var test = new GOVUK.MultivariateTest({ - name: 'stuff', - cohorts: { - foo: {callback: 'fooCallback'}, - bar: {} - }, - runImmediately: false - }) - test.fooCallback = jasmine.createSpy('fooCallback') - test.run() - expect(test.fooCallback).toHaveBeenCalled() - }) - - it("should assign 30 if cookieDuration isn't defined", function () { - GOVUK.cookie.and.returnValue('foo') - var test = new GOVUK.MultivariateTest({ - name: 'cookie_duration_test', - cohorts: { - foo: {callback: function () {}} - } - }) - expect(test.cookieDuration).toEqual(30) - }) - - it("should assign the user's cookie duration, when cookieDuration is defined", function () { - GOVUK.cookie.and.returnValue('foo') - var test = new GOVUK.MultivariateTest({ - name: 'cookie_duration_test', - cookieDuration: 14, - cohorts: { - foo: {callback: function () {}} - } - }) - expect(test.cookieDuration).toEqual(14) - }) - - it('should assign a new random cohort if the assigned cohort does not exist', function () { - var fooSpy = jasmine.createSpy('fooSpy') - var barSpy = jasmine.createSpy('barSpy') - GOVUK.cookie.and.returnValue('baz') - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - cohorts: { - foo: {callback: fooSpy}, - bar: {callback: barSpy} - } - }) - if (GOVUK.cookie.calls.argsFor(1)[1] === 'foo') { - expect(fooSpy).toHaveBeenCalled() - } else { - expect(barSpy).toHaveBeenCalled() - } - }) - }) - - describe('#weightedCohortNames', function () { - it('should return the weighted names of the cohorts when no weights are defined', function () { - var test = new GOVUK.MultivariateTest({ - name: 'stuff', - cohorts: {foo: {}, bar: {}, baz: {}} - }) - expect(test.weightedCohortNames()).toEqual(['foo', 'bar', 'baz']) - }) - - it('should return the weighted names of the cohorts when weights are defined', function () { - var test = new GOVUK.MultivariateTest({ - name: 'stuff', - cohorts: {foo: { weight: 2 }, bar: { weight: 1 }, baz: { weight: 3 }} - }) - expect(test.weightedCohortNames()).toEqual(['foo', 'foo', 'bar', 'baz', 'baz', 'baz']) - }) - - it('should return the weighted names of the cohorts using default weighting', function () { - var test = new GOVUK.MultivariateTest({ - name: 'stuff', - defaultWeight: 2, - cohorts: {foo: {}, bar: {}, baz: {}} - }) - expect(test.weightedCohortNames()).toEqual(['foo', 'foo', 'bar', 'bar', 'baz', 'baz']) - }) - - it('should return the weighted names of the cohorts using default weighting or defined weighting', function () { - var test = new GOVUK.MultivariateTest({ - name: 'stuff', - defaultWeight: 2, - cohorts: {foo: {}, bar: { weight: 1 }, baz: {}} - }) - expect(test.weightedCohortNames()).toEqual(['foo', 'foo', 'bar', 'baz', 'baz']) - }) - }) - - describe('#chooseRandomCohort', function () { - it('should choose a random cohort', function () { - var test = new GOVUK.MultivariateTest({ - name: 'stuff', - cohorts: {foo: {}, bar: {}} - }) - expect(['foo', 'bar']).toContain(test.chooseRandomCohort()) - }) - }) - - describe('Google Content Experiment Integration', function () { - beforeEach(function () { - window.ga = function () {} - spyOn(window, 'ga') - }) - - it('should report the experiment data to Google', function () { - new GOVUK.MultivariateTest({ // eslint-disable-line no-new - name: 'stuff', - contentExperimentId: 'asdfsadasdfa', - cohorts: {foo: {variantId: 0, weight: 1}, bar: {variantId: 1, weight: 0}} - }) - expect(window.ga.calls.first().args).toEqual(['set', 'expId', 'asdfsadasdfa']) - expect(window.ga.calls.mostRecent().args).toEqual(['set', 'expVar', 0]) - expect(GOVUK.analytics.trackEvent).toHaveBeenCalledWith( - 'multivariatetest_cohort_stuff', - 'run', - {nonInteraction: true} - ) - }) - }) -})