Getting started with A/B testing: Part 3 – Results

So our test has been up and running. Previously I mentioned an issue with numbers; partly because of an additional variation thinning out traffic, but also picking a page that wasn’t getting enough traffic.

After waiting an additional week we have some results. So of…

Optimizely our preferred tool (assume others have similar feature) based on the data entered let us know if we have hit significance (NB always run tests to your business cycle as well as significance). Variation 2 hit that; however variation 1 didn’t. That said results speak for themselves.

optimizely results test 1

optimizely results test 1

Tests are about learning not just optimising and that was the case for Test 1. Both variations had a negative improvement. Was this a case that the hypothesis wasn’t valid? Or was the result swayed by issues mentioned above? Time to iterate.

Test 2

We have a now picked a guidance page with a much higher footfall. We have also only created one variation. Our hypothesis remains:

if the recommendations link is highlighted in the menu navigation, we can increase the progression rate to recommendations as users will readily identify recommendations

This time the one variant contained an amended order.

Significance was met very quickly (days) however we ran the test for just over two weeks as per business cycle. Results spoke for themselves, but this time a very positive improvement.

optimizely results test 2

optimizely results test 2

Conversion rate was increased from 45.92% to 57.27% a 24.7% improvement.

Lessons learnt

For starters as Adam Avramescu (optimizely) said during #CROday “test, learn, iterate”. Don’t stop at one test, keep going. Secondly carefully pick your page to test with.

test, learn, iterate

We, NICE, are now testing regularly (last test +30.8% increase in conversions) and in a position to keep iterating. Of course we are always learning!

Tagged with: , , , , , , , , ,
Posted in Insights, Optimisation

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

About me

Engineering experiences with @Valtech. Advocate for customer-centric insight driven design. #UX #LeanUX #Agile #Insights #Analytics #CRO Speaker and #SitecoreMVP

Follow me:

Enter your email address to follow this blog and receive notifications of new posts by email.

%d bloggers like this: