How To Go From Zero to One in Your Server-Side Experimentation Journey

Think of your journey as a Netflix user. If you’re anything like me, you might watch a wildlife documentary on your phone while sipping your morning coffee. Dinner might be accompanied by an old-time favourite like Forrest Gump on the laptop. Weekend nights would be spent toggling between your profile and that of your kids’, while trying out new Netflix shows, preferably on a larger screen.  

Now let’s say Netflix is running a country-specific discount campaign. If you are part of this experiment campaign run by Netflix, how do they ensure that you are part of the same campaign every time you log in irrespective of the device and profile you’re using, and see the same promotion everywhere? How do they ensure that your experience with the variation you’re being served is seamless each time and that how you engage with the variation is tracked consistently?

The answer lies in omnichannel experimentation, which is a typical use case of server-side testing. 

Should you prefer server-side testing over client-side?

The Netflix example stated above would be extremely complicated to carry out on the client-side and could hamper the user experience. On the server side, it is relatively easy to run and ensures a consistent experience for users. It also ensures minimal impact on page performance. Besides this, it eradicates any privacy-related issues because there is no activity on the browser as such. 

There are other use-cases where server-side testing is recommended for its robustness and flexibility. We will talk about these in this article. But first, what exactly is server-side testing, and more importantly, who is it for?

In server-side testing, the test variations are processed on the web server. When a visitor lands on a page being tested, a variation is fetched directly from the server and delivered to the visitor’s browser. No subsequent modifications then take place on the front end or the browser. Contrary to this, in client-side testing, the original page loads first in the visitor’s browser and your experimentation platform creates the variation on the front-end itself using JavaScript. Let’s understand the scope of these two forms of testing with an example. 

Imagine Mike and Bob are two friends who are trying to experiment with the workings of a new car. Mike is at the wheel and has access to brakes, accelerator, dashboard, and the like. Bob has a view of the internal components like the engine, radiator, battery, etc. Both can influence the car in different ways. What Bob does with his access to the car components can reflect on the outside to Mike. The changes that Mike tests are based on his visibility of the car. From the car buyer’s perspective, the outcome of experiments run by both Bob and Mike can serve equally important, but different purposes. 

Therefore, you don’t have to choose one form of testing over the other. The use-cases are different and the teams using the tools are different. Server-side testing is an experimentation avenue for developers and product managers, just like client-side testing is used more often by marketers. 

Which problems can be addressed with server-side testing?

Server-side tests run by product teams solve problems concerning a myriad of industries from eCommerce and SaaS to banking and media. Some important use-cases where server-side testing is recommended over client-side testing in different industries are described below: 

Product recommendation

Which set of recommended products entices your visitors to purchase more? Server-side testing allows you to test multiple product recommendation algorithms to determine the selection that leads to an increase in sales and revenue. For instance, you can test whether a layout that promotes similar products works better than a layout that promotes the most popular ones. You can also decide whether to upsell or cross-sell based on your server-side experiment findings. 

Shipping fee

What is the ideal cart value that should qualify orders for free shipping? You can test out various thresholds to determine the one that positively influences customers’ purchase decisions. 

Search algorithms

Experimenting with your search algorithm requires modification to your existing code and the flexibility to test deep. You want your visitors to be able to quickly find what they are looking for and you can test your search algorithm on the server-side to achieve this. 

Form length

Free trial and demo request forms are crucial to SaaS businesses. But what is the ideal form length that ensures lesser drop-offs while also capturing all the required information? You can test out the non-mandatory fields via client-side testing. If your field is obligatory, just hiding the field using JavaScript will not work as form validation using server-side logic will fail. Hence server-side testing is recommended to experiment with mandatory fields to optimize your form length and complexity. 

Deals and discounts

While the style, look and feel, and placement of deals on your home page can be tested easily on the client-side, there are other important factors like the value of the discount, its duration, or eligibility criteria to consider. You can test on the server-side to determine the optimal value and ensure that they are consistent across channels for a particular visitor. 

Sales incentives

Testing dynamic incentives like limited-period offers or stock clearances require the flexibility of server-side testing because of the granularity involved.  

Subscription flows

How many steps should ideally be involved in a subscription process? Should social logins be provided? Experimenting with the subscription flow can help answer these questions.

Paywalls

Server-side testing allows you to test various paywall configurations in a fool-proof manner. As a publisher, you can run server-side tests to experiment with your gated content and monetize it. Running the same test on the client side is not recommended, because visitors can make their way around a paywall by deleting or opting out of cookies. 

Mobile banking

Multiple elements can be optimized within the sign-up process for a loan or a credit card. But when it comes to mobile banking, data security becomes paramount. With client-side testing, sensitive data collected by banks or financial institutions could be at a vulnerability risk. To avoid this risk, server-side experiments are usually recommended for banking applications. 

Let us now understand how you can run Feature tests on the server side and the advantages of doing so with VWO. 

How VWO makes server-side tests easier

For the server-side use-cases described above, VWO gives you the flexibility to structure your campaign either as A/B tests or Feature tests. Feature tests are used to validate values of feature parameters and give you the control to quickly configure a feature, without writing code. In some use-cases like testing which search algorithm is better, it is possible to structure the campaign both as an A/B test or Feature test. 

For instance, say you want to evaluate three vendors for the search algorithm they have built for your website.

Feature testing enables a product manager like you to quickly test and conclude with minimum dependency on engineering and maximum control of the configuration. With VWO’s Feature testing capabilities, you get a set framework where you need to write less code because the platform does most of the heavy lifting for you. In Feature testing, the algorithm can be defined as a feature variable and be configured in control and variation of an experiment from the platform setup flow itself to test which search algorithm is more efficient.

This experiment can be done via server-side A/B testing also. VWO facilitates the traffic distribution and experimentation statistics model capability via its server-side SDKs. Engineering teams can use the same to insert the code of the search algorithms and test which is more impactful.

Here are some other scenarios where Feature testing comes in handy. Say that a third-party vendor handling mobile recharges wants to charge a nominal amount to users per recharge. They want to test the appropriate amount for the same. Or a company like Airbnb, where the property charges are handled by the owner, wants to add in a cleaning fee and see if that impacts the number of bookings. This is a typical experimentation use-case for various companies to find the sweet spot where a service charge can be inserted without impacting the north star metric. It could be in the form of a convenience fee, facility fee, covid fee, packaging fee, or something similar. 

Complex use-cases like the one described above are super easy to test in VWO. Here’s an explainer video that shows you how you can quickly create a convenience fee feature and assign a value to it (in this case the amount of the fee). You can link your hypothesis of identifying the fee that adds to revenue without impacting the number of bookings, choose the environment you’re running the test on, and enable your variations. Once you do that, you’re provided with a campaign code that goes on your server. All that remains is for you to define the goals you want to track and segment your audiences if you wish to – that’s it, your campaign is ready.

If you are a product manager and you see on the dashboard that variation 3 is not working for the users; it is negatively impacting revenue, you can kill it right there by simply disabling the variation in VWO. As shown in the screenshot below, this does not impact the code and doesn’t require your engineering team to make any changes. You need to turn it off, click ‘save’, and the variation stops receiving traffic.

Essentially, the code needs to be implemented only once per campaign.

Should you build or buy a platform to run server-side tests?

Let’s put an end to the build vs. buy debate. VWO is not just a random number generator that shows different variations to different audiences and captures conversion events. VWO is a complete experimentation platform with a robust statistical model. To weigh in on whether to build a server-side testing mechanism in-house or invest in a platform like VWO, you need to consider three primary factors:

1. Cost of ownership 

Even when companies manage to build the required infrastructure in-house, they still need to manage and scale it. Paying your development teams to build and maintain an experimentation engine like VWO instead of focusing on their core jobs is likely to end up being more time-consuming and costly for you than investing in VWO.

2. Ease of use

You could build a solution that shows a certain variation to a certain audience – but would you have an easy-to-use interface that can be controlled not only by the engineering teams but also by product managers? If not, that’s another blocker for you to run server-side tests. 

3. Intuitive reporting

Typically an in-house solution would give you basic things like the number of visitors and the conversions that are coming from a particular variation. But what you need is a statistically significant result. You need your reports to be powered by a Bayesian statistics engine like VWO SmartStats. That’s where the gap lies – you can build a basic solution that’s hard to maintain and you can spend time and resources on deciphering the p-values. Or you can choose a solution like VWO where there’s a team dedicated to maintaining and scaling it and has spent years on the Bayesian algorithm to give you easily interpretable results. The in-app dashboard in VWO enables even your non-technical team members to understand results; they don’t need to rely on the Analytics team to keep track of experiments or create result dashboards – thus saving time and reducing the cost of experimentation.

4. Error-free mechanism

Building a server-side testing solution in-house can be prone to errors, and on that scale, the errors might not be easily discoverable. Compare that to the quality of a platform that is used by global brands and you’re ensured that the chances of errors creeping in are minuscule. Any errors, if at all, are flagged and rectified at the earliest by a capable support team available to you. 

Besides, when you invest in a managed platform like VWO, important best practices are built into the product. You don’t have to worry about removing the outliers from your results, visualizing your data, or about issues arising due to version updates.

Must-have capabilities to run complex server-side tests with integrity

Running server-side experiments can be very fruitful when executed correctly. To do so, you need to have the right set of feature capabilities. Some of these are laid out below:

1. Randomization of visitors in each test – In testing, when you’re bucketing your audience into campaigns, the randomization of visitors needs to be truly random and not pseudo-random.
2. Consistent omnichannel experience – While the bucketing of users needs to be random, you also need to ensure that one user experiences the same variation each time they log in, irrespective of the device they are using. The experiment should carry forward without any glitches. 
3. Mutually exclusive campaigns – Let’s say you have three factors to consider while determining whether a user should be part of your test. These could be usage regularity, low likelihood of churn, and timezone. Apart from taking into consideration these variables, you also need to determine exclusivity – so how many tests can the user fulfilling these conditions be a part of? This needs to be determined in a way that doesn’t lead to skewed data and enables you to attribute the improvement in conversion rate to the right campaign without bias. 
4. Standardized naming convention – Whether you’re setting up a new feature to test or a feature flag, you need to follow a standard naming convention to avoid mix-ups and cases of initializing the wrong features or tests. 
5. Unique and hassle-free campaign identifiers – You should use an alphanumeric key to uniquely identify the test in your code and avoid any hassles at a later stage. 
6. Choosing the right environment – You should specify the environment in which you’re running your test – for instance, you can deploy the test in the staging or QA environment for your QA team to validate the experiment. A sanity check of your test is crucial to its success and you should have the option to choose the right environment for it. 
7. Logical traffic allocation – When you’re running multiple campaigns or when you have an important event announcement like a Black Week Sale, for instance, you need not include the entire set of visitors landing on your page in your test. You should choose the percentage of traffic that you want to include in your test campaign and also how you want to distribute this traffic among the variations. 
8. Calculation of the time to reach statistical significance – Your estimated time for your test to reach statistical significance should be determined by the present conversion rate of your primary goal and the minimum improvement that you wish to achieve through your variations. It should also consider a 95% probability to beat the baseline conversion rate. 

These are some of the best practices and must-have features of server-side testing – the actual list is much longer. As mentioned earlier, you could either build these capabilities in-house, or you could use VWO where we do the job for you. 

In conclusion

Whether you’re a developer or a product manager, you don’t need to limit your test ideas. You can run complex tests without fearing performance or privacy issues with server-side testing and solve real issues faced by your customers. You can optimize every digital touchpoint so that your customers experience only the best.  

If you’re using a platform like VWO, the complexity of the test will not overwhelm you – because every input of yours in the campaign is intuitive and a good practice that powers your test. To understand more about how you can run server-side tests with ease using VWO, request a demo with our product experts.