Investing in CRO or A/B Testing Software? You (Decision Maker) Must Know This

You must have come across A/B testing before. While it might not always have seemed like a top priority due to its focus on gradual results, it’s now gaining momentum, with 77% of companies now running A/B tests on their websites. 

CRO, often mentioned alongside A/B testing, is also crucial, as 55% of businesses are increasing their CRO budgets. 

Are they the same? No. 

Think of it this way: if conversion optimization were a puzzle set, A/B testing would be a vital missing piece. 

In this blog, we tell you how they’re different but also connected. This way, you’ll have all the info you need before deciding on CRO or A/B testing software. 

Download Free: A/B Testing Guide

1) A/B testing and CRO aren’t the same

A/B testing is a technique, and CRO is the process that involves the technique.

For sure, A/B testing has revolutionized the way online businesses look at growth. It has enabled them to make decisions reliably to improve their conversion rates. The reliability seemed magical.
However, the industry hit disillusionment and then realized that A/B testing is only an enabler and not a panacea.

A/B testing is exactly what it says—a test for ideas. It’s not a magic wand nor a design roulette that works on luck.

From the initial experience, a series of crucial realizations should follow:

• A/B tests are only as good as the hypotheses they test.
• Hypotheses are only as good as the research behind them.
• Sustainable business growth is only as good as the management of this entire process called CRO.

A/B testing is an indispensable component of CRO, but it’s only one component. Below we outline the CRO process and how an A/B test fits into its overall framework:

1. Generate ideas through research

The initial step in conversion rate optimization testing is to gather ideas through research. Many marketers jump straight into CRO testing without this crucial step. Without research-backed ideas and usability testing, it’s challenging to understand what might be wrong with your website or app and how to fix it. 

Where to start? Leverage quantitative analytics tools like heatmaps and user recordings. These tools help you delve into user behavior and form hypotheses that lead to the creation of impactful tests.  

2. Prioritize your ideas

Once you’ve generated a bunch of test ideas, it’s time to prioritize. Focus on the ideas that have the most significant impact on your core business metrics, such as purchases or form submissions. Define your objective for the CRO test, the hypothesis detailing what you expect to happen, the metric you’re measuring, and the expected outcome.

3. Plan and run the test

With your ideas prioritized, it’s time to plan and run the test. Determine the test type, and the pages where it will run, design the control and variant, estimate the duration, and determine statistical significance. Use a testing tool like VWO, which offers a comprehensive testing platform – from research and test execution to results analysis and results documentation. Take a free trial and explore the features. 

Further, ensure the test is ready for launch by performing the final checks. Notify stakeholders about the CRO test, as it may impact other teams and their metrics. This also provides an additional set of eyes to identify any issues.

4. Analyze the results

Once your test reaches statistical significance, analyze the results. Examine how your metrics were affected, confirm whether your hypothesis held true, and identify key insights. If the variation you tested won, proceed with implementing it. If not, consider it an opportunity to learn valuable insights about your target audience.

5. Optimize based on insights

Use the insights gained from your results to optimize your website or app. For example, if your data shows that scrolling is more effective than clicks for improving add-to-cart and transactions, implement the scrolling experience for all visitors. This iterative process ensures ongoing improvement and user satisfaction.

Does the distinction make sense now? Conversion rate optimization test is a process that needs to be repeated, but A/B testing is a technique — a statistical tool. Both these terms are often confused for good reason. For seasoned, successful testers, good CRO tests are inconceivable without a good process.

Watch this video to understand how CRO can lead to increased revenue growth.

2) You need more than an A/B testing tool

We have already established that whether you talk about “A/B testing” or “Conversion Optimization” in your internal discussions, what you need in the end is a process. A systematic approach to hunt down barriers to improve conversion rates on your website and eliminate these barriers. Such an approach requires:

• Research to construct testable hypotheses
• Prioritization of hypotheses to decide what to test first
• Testing and Analysis of results to derive learning for future tests

There are multiple tools that will help accomplish specific aspects of this process. Also, there are project management tools that help you set up a process for integrating these steps.

Here is a list of the stages involved in making a sound business decision.

The primary objective of such a detailed process is to ensure “sound” decisions. These decisions minimize the probability of loss and maximize the probability of gain from your CRO test.

You need to perform all the steps in one place with unrestricted access to data collection, manipulation, and validation.

A system that enables you to make such decisions requires a platform with connected capabilities. Think of any marketing automation software such as Hubspot, Marketo, or Eloqua. These are all platforms with a network of tools that communicate with each other such as email, social media, content, and paid channels.

3) Decide what you want to know through A/B testing

Until some time ago, conversion rate optimization split testing or A/B testing tools ran tests the same way. The underlying statistical framework used to interpret numbers and choose a “winning variation” was the same. Currently, A/B testing tools and CRO platforms such as VWO offer you a choice. With choice comes the responsibility to evaluate the pros and cons and then choose what best suits your needs.

The choices are:

• Frequentist method of computing
• Bayesian method of computing

Both these methods present different approaches to testing and ask different questions from the data they collect. Therefore, these methods fundamentally affect the level of optimization you can achieve with existing data.

Are you looking to find the infallible truth, or are you looking to make the smartest business decision based on your conversion rate optimization tests?

What the image says: The former conclusion of 16.6% (5 times/30 days) that “Glasses” found out first is the Frequentist way to look at things. (The example is a bit wrong because 5 times is probably too low to be statistically significant, but it works to demonstrate the thinking.) The latter conclusion of 75% is Bayesian reasoning. This reasoning considers supplementary information (that Glasses complimented her eyes) that was available. Which one seems a smarter, yet very reasonable, conclusion?

Let’s consider that your team sets up a test between a control (A) and a variation (B).

Here’s what these two methods find out for you:

Frequentist method

“What are the chances of observing a result at least as large as this, if the test was done between A and an identical copy of A and on the same sample size?”

This question doesn’t sound very straightforward, because it is not.

This method helps statisticians and truth-seekers, incrementally reduce the improbability of a wrong conclusion to help arrive at the “truth.” The Frequentist method was designed to understand the probability of drawing the wrong conclusion. It is a cautious, and somewhat slower, approach to arrive at “the truth.”

Let’s say the p-value for a test is computed as 0.03. When the test is stopped, the conversion rates of A and B are Ca and Cb.

Frequentist testing tools interpret this figure and display the result as “Chance to beat A is 97%.” However, what the p-value indicates is that if you had conducted the same test a 1000 times between A and a copy of A, at 30 instances, you would observe a result at least as extreme as the difference between Cb and Ca.

This is not the same as saying “B will beat A by a margin of Cb minus Ca 97% of the times.” All that the p-value represents is the improbability of A and B to be identical.

The p-value does not discuss the probability of the variations being different, or by how much. This value also does not tell you what you should do, although these are probably what you want to know.

The frequentist aims for universally acceptable conclusions, ones that will stand up to adversarial scrutiny. The FDA for example doesn’t care about Pfizer’s prior opinion of how well its new drug will work, it wants objective proof. Pfizer, on the other hand, may care very much about its own opinions in planning future drug development.

Bradley Effron explains the Frequentist vs Bayesian thinking in his presidential address at the American Sociological Association.

This analysis does not mean that the Frequentist results have no meaning in a business setup. These comparative results let you be fairly sure that one choice is better than another. However, that’s an indirect way of saying it, and any attempt to understand “by how much” leads to hidden fallacies.

Bayesian method

Bayesian lets you ask the question:

“What choice should I make right now to make the most money? Continue the test, stop the test with A, or stop the test with B?”

This is made possible because Bayesian approaches the outcome—“conversion rate”—as a fluctuating variable. With additional data that it gathers, Bayesian updates the probability of the outcome—the conversion rate—to lie between a range.

For example, before observing any data, one might think all conversion rates (for two variations A and B) are equally likely. At this time, both A and B would have a 50% chance of being the winner.

At this time, the financial risk is also high—it’s totally plausible that B has a 20% conversion rate and A has a 5% conversion rate, so choosing A would be a bad decision! The opposite situation is also very plausible, in which case B is a bad decision. At this point, we should just wait and continue the test.

Then when new information is updated (such as an A/B test), that opinion changes. After obtaining data, we become confident that B has a 90% chance of beating A. We also come to believe that A has a conversion rate between, for example, 3% and 6%, while B has a conversion rate between 5.9% and 9%.

At this point, we’ve eliminated the risk involved in choosing B. It’s still theoretically possible that A is better—A might have a 5.97% conversion rate while B has only 5.93%. But the financial cost of that possibility is very low, so choosing B is a very safe business decision. Not the truth maybe, but a wise business decision.

Chris Stucchio, Former Director of Data Sciences at VWO

Bradley needs to be quoted again where he summarizes the differences in the approach:

The Bayesian-Frequentist debate reflects two different attitudes to the process of doing science, both quite legitimate. Bayesian statistics is well suited to individual researchers, or a research group, trying to use all the information at its disposal to make the quickest possible progress. In pursuing progress, Bayesians tend to be aggressive and optimistic with their modeling assumptions. Frequentist statisticians are more cautious and defensive.

Most tools in the market make use of Frequentist, while some like VWO have graduated to Bayesian; still others use both in conjunction.

One is mathematical—it’s the difference between “proving” a scientific hypothesis and making a business decision. There are many cases where the statistics strongly support “choose B” in order to make money, but only weakly support “B is the best is a true statement”. B has a 50% chance of beating A by a lot (say B is 15% better) and a 50% chance of being approximately the same (say 0.25% worse to 0.25% better). In that case, it’s a great business decision to choose B—maybe you win something, maybe you lose nothing.

Chris Stucchio, explains on ConversionXL why Bayesian trumps the Frequentist approach when it comes to making business decisions, where the onus is not on finding the ‘truth’ but on making the most rational business decision.

On the business front, here’s how VWO SmartStats, VWO’s Bayesian engine fares against our older Frequentist engine.

If you are interested in learning more about the Bayesian approach, you can find a detailed and intuitive approach to understanding the Bayesian method here.

And here’s Chris explaining the inherent limitation of Frequentist and how Bayesian saves the day.

Download Free: A/B Testing Guide

4) You will need more than a platform; You need people

Someday, we might have platforms that don’t just enable you to make decisions but also make decisions for your CRO testing. Until then, you need more than just platforms—a set of people who possess a unique set of skills to use the platform and move the needle.

The skills that a CRO team must possess can be broadly grouped into the following:

Strategy development

You will need a plan for your CRO tests from determining the goals or key metrics of the program prioritizing activities such as hypothesis building to A/B testing to leveraging the learning to fuel further tests. Such ownership requires context about your business and at the very least, your organization’s conversion funnels.

Data analysis

CRO is as much about being proactive as it is about reacting to obvious conversion issues. Data analysis involves monitoring of website data and user behavior to identify areas of improvement—leaks and areas of friction in your conversion funnel. A data analyst needs to make sense of CRO jargon and ensure that everyone in the team understands the test results in simple terms.

Conversion-centered designing

Design is a crucial aspect of CRO, as it is the culmination of the proposed changes and acts as the interface between your understanding of your users and their interaction with your business. Such responsibility requires you to have knowledge of persuasive design principles and psychological triggers.

Persuasive copywriting

Copy is the other element that your website visitors interact with apart from design, and it requires just as much thought and expertise. Some key aspects of conversion optimization that persuasive copy can help with are reducing visitors’ anxieties, reflecting their motivation, and delighting them.

Web development

You also need a web developer. Translating your design and copy into an error-free experience lies in the hands of this developer. From creating variations and setting up and tracking events and goals to transferring data from one system to another, the developer’s role in a CRO test is indispensable.

To provide an in-depth and comprehensive account of what it takes to run a successful optimization program, check out our interview with Michal Parizek, senior eCommerce and Optimization Specialist at Avast.

CRO course & certification

Consider investing in upskilling your existing team by exploring conversion rate optimization courses. These courses can provide exposure to the best CRO practices and insights into how CRO experts tackle challenges. To choose the right course for your team, here are some pointers:

• Evaluate your team’s current expertise and identify weak points. Seek courses that specifically address these areas.
• Check the recency of the courses and ensure their relevance to current marketing conditions.
• Look for courses that offer hands-on training with CRO tools or provide assignments that allow learners to apply their knowledge in practical situations.
• Assess the domain expertise and authority of the course creators. Consider their professional experience and the brands they’ve collaborated with in the past.

If you’ve got in-house CRO experts, leverage their knowledge to create a conversion rate optimization training kit for your team. This tailored resource serves as a customized roadmap for upskilling, aligning with your specific business needs. To ensure ongoing proficiency, consider organizing periodic conversion rate optimization training tests. These tests not only help gauge your team’s grasp of domain knowledge but also provide opportunities for continuous improvement.

Recognizing that conversion optimization training is a crucial element in honing your team’s skillset, investing in it becomes imperative. The dividends you’ll reap from more effective CRO campaigns make this investment not just worthwhile but integral to the success of your business.

Brands that benefited from full-fledged CRO campaigns

Bear Mattress

Bear Mattress, a New Jersey-based company, had a clear goal – to increase revenue by optimizing its product page’s cross-sell section. Through behavioral analysis, they observed poor visitor engagement with the ‘Frequently Bought with Mattress’ cross-sell listings.  They also found that the “Frequently Bought with Mattress” section lacked pictures, making it less interesting for customers. 

The hypothesis was that a variation addressing these issues would enhance engagement and revenue. In a 19-day A/B test, the variation outperformed the control, leading to a remarkable 24.18% increase in purchases and a 16.21% revenue rise. 

The team analyzed the results and found that a user-friendly interface in a niche like sports mattresses not only enhances upselling opportunities but also improves the overall user experience.

Bear Mattress’s success story showcases the effectiveness of a structured optimization program in increasing revenue. It’s about making informed decisions, testing hypotheses, and delivering measurable results while keeping the user experience at the forefront. 

RuneScape

RuneScape is a popular online role-playing game with 250 million players globally. Its CRO team devised an optimization campaign that aimed to increase revenue by boosting purchases. 

Upon reviewing the Treasure Hunter page, the team found that many website visitors were clicking on the “Get Keys” section, even though it wasn’t clickable.

They assumed users wanted direct access to keys or sought more information. Session recordings revealed that website visitors returned to the Treasure Hunter page from the Payment page. This suggested inadequate information about treasure chest packages. The hypothesis emerged that offering package details on the Treasure Hunter page would enhance Payment page conversions.

To test the hypothesis, a variation was created, featuring four treasure chest packages. In an A/B test, the variation outperformed the original, increasing purchases by nearly 10%. 

Encouraged by this success, the team introduced a “Recommended” package variation, which also outperformed the original, leading to a further 6% increase in purchases.

Truckstop

Truckstop, a well-known manufacturing company in the US, embraced continuous optimization, with VWO serving as a cornerstone for cultivating an experimentation culture. 

Their dedicated CRO team leveraged session recording to understand user behavior. They analyzed over 1000 recordings to gather valuable insights and came up with a campaign idea. 

It was hypothesized that a pop-up demo form on the existing page would boost the generation of MQLs. The goal was to increase the number of demo requests. To achieve this, the team introduced a pop-up demo request form to the broker load board page. The variation outperformed the control and resulted in a 26% increase in conversions.

A structured CRO program encompasses visitor research, hypothesis formulation, prioritization, testing, deployment, and learning from CRO tests. Truckstop.com shows us a good example of how to do this right.

Can you run other types of tests in your CRO program?

As mentioned earlier, the test is just one component of your CRO campaign, and it comes in various types. If you assumed it was only about CRO split testing or A/B testing, you’d be mistaken. Here are some other tests you can plan and launch as part of your CRO process.

A/B/n test

In an A/B/n test, multiple alternative versions (n) are compared to the original (A) to determine the best-performing variation. Each variation represents a different complete version of the web page, landing page or email. And the variations can include different headlines, images, layouts, or other substantial changes.

However, it comes with challenges, such as a reduced sample size for each variation, longer testing durations, and the need to carefully analyze results to attribute changes to specific elements accurately.

Multivariate test

Multivariate testing is a type of CRO test for figuring out which combinations of different page elements work best together. This approach is quite efficient because it allows you to test many combinations at the same time, which saves time and resources. 

But here’s the catch about conversion rate optimization multivariate testing. Testing numerous elements simultaneously can demand substantial website traffic, potentially extending the testing duration to achieve trustworthy results.

Multi-armed bandit test

A multi-armed bandit test uses machine learning to dynamically steer traffic towards variations that are doing well and reduce traffic to those underperforming. This approach delivers quicker results as you don’t have to wait for a single winning variation to emerge.

Further, in website optimization, contextual bandits take things up a notch by considering real-time user context data. For example, if you know a customer has made a previous purchase, you can use contextual bandits to offer them a discount for more purchases.

There are some downsides to these CRO tests as well. It isn’t easy to reach statistical significance with MAB tests. They’re not the best for optimizing multiple metrics. Plus, post-test analysis can be tricky when there isn’t enough data for the low-performing variations.

Are you wondering how to pick the perfect CRO tool for your business in this sea of options? With so many choices flooding the market, it’s easy to feel overwhelmed and hesitant about where to invest. We’ve delved into the nitty-gritty of selecting the ideal CRO tool in our blog post. Get your answer and make the right choice!

Wrapping up

A/B testing is a test; CRO is a potent growth discipline.

Here’s what we covered in this post:

• The difference between A/B testing and Conversion Rate Optimization
• For sustained growth, you’ll need a connected platform rather than an A/B testing tool
• Different CRO  testing platforms answer different questions, so choose what you need to be answered
• CRO doesn’t end with a platform, you’ll need people who can run CRO

Currently, most organizations that believe they are doing CRO are only doing patchy A/B testing. With time, it will evolve into the discipline it needs to be. When it becomes that, where do you want to be—in the gallery or in the race?
Carpe diem.

---

Frequently asked questions