When the Tests Lie

We all know split testing is the key to success on the internet. But what happens when tests lie?

Testing can not be overdone. It is the surest way to success, and we hammer that home time and time again. A process is only as good as the way in which it is carried out, so be careful. Poorly designed tests can give you information that is skewed or unreliable, so make sure you approach testing with the following in mind:

Use the right type of test (A/B vs. multivariate test)

Multivariates (MVTs) let you test multiple elements on a page simultaneously. A/B split tests allow you to pit two completely different versions (or more) against each other in a contest to find the most successful.

MTVs require a large amount of visitors to obtain accurate data, but is ideal for finalizing page design in preparation for a more comprehensive A/B split test. a.

Stay away from testing software. Testing ten thousand variants at once just muddles the picture, keep it under a dozen variants for a MVT test unless you have such massive amounts of traffic that hundred will still yield accurate data.

Don’t do tests that take too long This is especially true if you are trying to work around seasonal or holiday fluctuations. A few months is plenty, and you can work your tests in between holidays for non-seasonal products. (Lengthy tests can also end up skewed by a sudden slump in the economy.)

Monitor your test

Particularly if you are using Google’s Website Optimizer, you should track carefully how your test is going. Don’t jump the gun – Google isn’t always right – but watch for trends over the weeks after you make a change. tool.

Follow up.

If you test and use the data to double your conversion rates, great. Then you quit testing. Not so great! What is to say you couldn’t triple your rates if you keep going?! Finish one test and move on the next – always be testing is the motto!

These tips will help you get the most accurate data, and you won’t be in the uncomfortable position of wondering if you tests results are lying to you!