How to Become a Split Testing Expert

Every now and then, two landing page designs are pitted against each other until one is left alive and standing — sorry, I got ahead of myself back there — what I meant was these two designs battle for conversions. This is familiar to us in the design parlance as A/B testing, or split testing. It is where web designers work with marketers and growth enthusiasts to test and see which version of the design and copy does a better job of communicating with the visitors and persuading them enough to convert into the metrics set as goals. These conversions can come in the form of signing up, subscribing to a newsletter, or may be something else that the marketers define. The changes into these versions can be as rudimentary as changing the color of call to action buttons, or altering a few words in the copy to revamping the whole design.

A/B testing helps you say “this works” without sounding like a psychic.

Why do Split Testing?

Split Testing - red vs green

Making split testing part of your design process can help you identify how your design communicates with the visitors, and be able to scientifically come up with pertinent design decisions using data you get from your users. To put it simply, it’s like having a scientist come over and tell you which shirt reveals your eyes while you are getting dressed.

The reason most people put up a website is to earn money. I mean, almost everyone I know who works in our industry puts up a website in order to earn the bucks, and I know that there are articles saying it is easy, but in reality, it’s more complicated that you think. In principle, there are two main ways to attain your financial goals for the website:

  1. Get more people to visit your site
  2. Increase the rate of conversion

The first can be easier than the second, but it can be quite expensive. You can easily put up a campaign on Google Adwords, or any other advertising platform out there, but it will only get you to a step, and if you’re pressed to take a leap instead of just a step, you’d have to up your game.

Or as Mark Watney said in The Martian, you have to “science the sh*t out of this.”

That is where our friend split testing comes in. I know, I know, design is probably the least science-y thing you can think of but using different landing page versions will help you make more informed choices.

To put it simply, would you rather tell your investors (or maybe even yourself) that you used a neon green call to action button because your friend, who happens to be an “expert” in web design, told you? Or would you say that it’s backed up by actual data? What would make you sleep better at night? I’ll pick the one that had ran in actual tests, and what my users really wanted.

Split Testing helps you make these decisions, and save you a lot of money and time. Using your resources and your limited time to change bits of code in the design, and present them to your visitors, will help you get the pulse of what they want and how they want it. And this sense of user-gratification is what makes web design exciting.

When to Start Split Testing?


Now that we’ve answered the IF question of A/B Testing, we ask, WHEN?

Generally speaking, you need to start testing as soon as you have enough visitors, and conversions. Meaning, you can only do split testing with your users, when you have users to begin with. The sample size of the visitors will, of course, vary depending on the users you actually have. But it is ideal to plan on at least a thousand visitors for each variant with 150 targeted conversions per variant. You can increase the sample size depending on your user base, and for that you can use a sample-size calculator.

Typically, it would take a few hours to a month to get informed results. For that, you would have to test different metrics like click through rates of your CTAs, the conversion rate of specific forms, and more.

Doing Your First Test


Now that you are itching to test the heck out of your design, you want have to remember a few things first:

Remember that this is your first test, so don’t change the whole design, for now. It is advisable to start small in order to get your toes wet. Put your focus on the low-hanging fruits like the copy in your h1 and h3 headings, the copy and color of your call to action buttons, the position, style, and size of different elements, and the length of your website.

These small variations can help you big time, as they are usually the things that influence users the most. Remember that A/B testing is not about the big guns, it’s about paying attention to details because that is what impacts if users convert or not.  Start with using two variations of the design. Don’t get into a royal rumble just yet because you need to feel the pulse of your users first. Plus, you wouldn’t want the sample size to get smaller and smaller by dividing it, as doing so will diminish the validity of the results.

Once your time frame is over, you can study the data and declare a statistical winner. You would then have to apply the best variants of the design permanently.

Then…rinse and repeat.

After the Low-Hanging Fruits

Now that you got the hang of A/B testing, you can start hitting the bigger fruits in that tree. For that, you can start focusing on

  1. Features or Benefits? This question is one of the best ones in A/B testing. Do your users want you to highlight the product features like a check list? Or the benefits of the product for your users? Which one performs better? Test to find out.
  2. Boost Page Load. Make the landing page simple, and fast to load. Try to target a load time of under one second.
  3. Videos or Photos? Use different variants where there are videos and large photos in your header. The videos make it personal for your user to connect with, but it can turn off other users because your page basically becomes heavier. Plus, some viewers may not like watching videos.
  4. Ask new customers. When you get new users, you can ask them what made them sign up, and what they think is the most valuable part of the design. From that, you can test a variant of your design to test if these factors really make the visitors into actual users.
  5. Determine what confuses your new customers. Aside from asking what made them register, you can also be bold and ask what confuses them in your design. From there, you can start removing or altering these elements and start testing.

Remember, you can have more options than included above. You would just have to be resourceful and innovative on how to get what your users want.


Guidelines to make your Split Testing Effective

split testing image 5

Remember that you are not playing with a piñata with your designs. You can’t just blindly swing at the air hoping for the slim chance of hitting. You have to maximize your efforts and be efficient in utilizing your resources and time. Following a method that can both extrapolate a good amount of data is what you want to do, so follow these guidelines:

  1. Know thy website – Understand your design and be bold enough to identify which areas of it might suck. You could ask a few people about what they think will be detrimental to your efforts, or, from the analytics, you can see which parts of your design performs poorly. From this, you can know which things to test.
  2. “Guess” scientifically – using data, think of changes you can implement in the design to improve conversion. Use the data at your disposal and don’t just try to feel the vibe of your users. What pages, menus, widgets, or CSS elements to add? Which ones work? One good tip for this part is to see what performs well in your design, and try to incorporate them to the areas that suck.
  3. Use different alternatives for your hypothesis in case something does not work out as planned.
  4. Test your hypothesis – make small changes in the design. Do the results validate your hypothesis or contradict it?
  5. Analyze the results – after testing the designs with your users, you will see which ones perform best. Then ask yourself, what can I draw out from these data? What changes can I implement in order to find the sweet spot for the design?

Tools for Split Testing

Of course, this guide wouldn’t be complete unless we mention the tools you can use to start administering A/B tests on your designs. Here are some of the best that we have personally used before:

Visual Website Optimizer

VWO is an A/B testing tool that is easy to use. Anyone from agencies, businesses to individuals can use it. And it’s pretty straightforward: just create variations of your website, identify what your goals are, (for example, sign up for a newsletter, download something, or maybe make a purchase) and then VWO will split your traffic into these versions. If you want to try it, you can do a free trial where you can run a single test on 1000 visitors. The premium versions start at $49 annually where you can do a test on 10,000 visitors and up to three simultaneous tests. You can also use other features for free including Significance Calculator, Landing Page Analyzer, A/B Ideafox -Case Study Search Engine, and the A/B Split Multivariate Test Duration Calculator.

Split Test Calculator

This tool is for those who don’t want to deal with calculations. Basically, what this does is tell you which of your tests did well when the sample sizes are different. To use it, just key in the total number of visitors and how many of them have performed the actions you want them to do and it will automagically tell you which one worked better than the other.


Optimizely is another tool that allows you to perform A/B testing on both web and mobile versions of your website. It works by adding a JavaScript snippet on your site and change the versions of the visitors who are included in the variation sets. It then collects the data on your visitors and the conversions and gives you the winners and losers using their stats engine. It basically automates everything from running tests, to crunching up the numbers for you to just make the decision. In terms of mobile apps, this tool runs through an SDK for both iOS and Android.

Did we forget something? Let us know in the comments section.


There is no design to rule them all, at least not yet. Different things work for different people, and using split testing on your designs will help you become a smarter designer as it gives you an idea what the user wants and how he or she wants it. By understanding these ideas, you can optimize your web design and become more effective in meeting the goals of both you and your clients. So start doing A/B tests today, who knows, you might find it addicting. After all, as Sherlock Holmes once exclaimed, “Data! Data! Data! I cannot make bricks without clay” and you, my dear friend, should know where to find that clay in order to build those bricks needed for the house. Good luck!