Getting started with A/B testing: Part 2 – Start the test

With our goals recording, we went in search of an ideal candidate/ a pain point we knew needed improving.  At NICE we revolve around providing guidance on a range of health issues. Within each piece of guidance is key recommendations that Doctors and care professionals need to know when using the guidance. Our pain point and thus hypothesis was progression to recommendations information (located on a sub page) wasn’t being located.

If you need help finding your pain point, Peep Laja has written an excellent post this time on 10 Google Analytics reports that tell you were your site is leaking money.

Test 1

We have a candidate and have now written a solid hypothesis; that being:

if the recommendations link is highlighted in the menu navigation, we can increase the progression rate to recommendations as users will readily identify recommendations

We made two variants, one with the link bold (same order) and one with an amended order (no bold)

Tests were started with the additional integration to GA custom dimensions activated. Why? Well A/B tests are set up with one main goal (normally increase click through of a certain button or link) however websites have several organisational objectives. By activating a custom dimension we can see did the variants, more so the winning variation, affect the bigger objectives. Peep once again explains more in his post “How to Anaylze your A/B test result with Google Analytics”

Problem 1: It soon became apparent  the chosen candidate footfall wasn’t as big as expected. (No ones fault as in previous weeks footfall was great) This would mean the time to complete the test was going to take longer.

optimizely traffic allocation screen

Problem 2: To compound the above by adding a second variant we reduced the allocation of traffic by a 3rd to the original and variant 1 thus extending the test even further.

Leasons learnt

Pretty obvious but really do your homework on footfall/ sample size/ statistical significance. For us, it was liasing with those who are in the know about the content. Listening to them about swings in traffic especially political/ press influence. Also use the sample size tools your testing tool provide.