A/B Testing with ABJS module

ABJS is a contrib Drupal module, and, without any requirements or ties to paid services, is as low cost as you can get. As we’ll see, it’s pretty basic but it really lets you get down to building your own understanding of how A/B testing works. The beauty of ABJS is in its simplicity. The settings pages are fairly self-explanatory, which is really helpful. Let’s set up a basic A/B test to show how things work.

Setting up our first experience

In our test, we’re going to split the site 50:50 in order to test an alternate homepage design. Go to /admin/config/user-interface/abjs and get a feel for things. See the tabs across the top? The best way to set up a new test is to work backwards. That’s because your Tests will need to reference your Conditions and Experiences - and you’ll need to create them before you can use them.

First up, create an Experience. Experiences make the actual A’s and B’s of your A/B tests. Go to the Experiences tab. Give your experience a very clear and helpful name. Our first one will be our normal homepage experience. Making a ‘normal’ experience allows us to explicitly log our page views for our analytics.

The Javascript we set up for our site looks like this:

if (typeof(ga) !== "undefined") {
  ga('set', 'dimension1', 'normal');
}
window.Drupal = window.Drupal || { 'settings': {}, 'behaviors': {}, 'locale': {} };
window.Drupal.behaviors.abtesting = {
    attach: function(context, settings) {
        jQuery('#request-a-quote-form').on('submit', function() {
            ga('send', 'event','Productfinder', 'ID search', 'Homepage');
        });
    }
}

1. We came to the Drupal Behavior format after realising that the ABJS scripts in the header would run before the Google Analytics scripts, and we would need the GA script to run before we could log our analytics. You could probably do this another way if you wanted, but this is easy enough to copy and paste for now.

2. Set our custom GA dimension.

3. This ends up happening before Drupal is actually ready to accept and execute behaviors, hence the very careful creation/copy of the window.Drupal variable.

4. Add a submit handler to our form, which will send an event to GA. This is the thing we’re trying to measure. Hopefully our alternate version of the page will result in more clicks on this button, and we’ll be able to track those in GA.

If you copy & paste the above, you'll want to make your tweaks:

1. Change the first call to ga() in line 2 to be a dimension you’ve set up in Google Analytics (Go do that now! Their support articles are really good, so I won’t explain that here).

2. Set the value for that call to be the value you want (‘normal’ may well be fine).

3. Change the event values in the final call to ga() to send the event values you want or need. They don’t need to be fancy, just unique - you just need to be able to track them in GA. Now, go create an “Alternate homepage experience” experience.

Set up your Alternate Experience

This is the Experience for the change / difference you're wanting to test.

Copy the JS from your ‘Normal’ experience, and tweak it to:

1. Have a different value for your GA dimension

2. Make an actual change to your page. Put this after the jQuery/ga call.

Now go create your condition(s).

All your experience Javascript will be added to every page, so this is where you make sure that you’re only running things where and when you really want to. Again, you write up some Javascript; it will return a Boolean value indicating to ABJS whether you want your test to be run. In our example, we’re just testing the homepage. So we’ll just do this:

return (window.location.pathname == "/");

Don't forget to set a helpful and clear name for your Condition, so it's easy to select later.

Test time!

At last, you can now go set up your Test.

1. Give your test a name. Make it helpful, so that you know which test does what when you have multiple tests later :) Perhaps name it after the change you’re making in your alternate test.

2. Select your two experiences in the two select boxes. Don’t let the fraction field confuse you - this is just the proportion of people you want to be diverted to each of your two experiences. This can be super helpful if you want, for example, to test something on a small proportion of users. For us doing our small, low-cost A/B test on our small client’s site, we want to maximise our data. So we’re doing 50:50 - this means fractions of 0.5 and 0.5 You can have multiple experiences here, which is pretty neat. So if you want to test multiple variations of your homepage, you can! Go for it!

3. Double check your Javascript :) Go execute it in the browser console or something, to make sure it works. No point taking your site down accidentally!

4. Set your test to 'Active'! This will add your tests to the site, so you can start collecting data! Now is the time to go to Google Analytics and watch the data pour in (for some definition of pour, depending on how busy your site is right now!).

Analysing your data

A key thing to remember when watching your analytics is that things change all the time, and sometimes randomness can be responsible for what you’re seeing. Or maybe there was a seasonal spike in sales which increased revenue that week, or maybe you’re not filtering your users to the right segment… A few recommendations for you:

• Create segments for your two dimension values, so you can easily filter your data.
• Data can be misleading. Always check other angles before declaring you’ve fixed the problem.
• Run your numbers to check the Statistical Significance. If you don’t have tens of thousands of samples, your results may just be random chatter rather than necessarily related to the changes your Experience made. Either remember your A-Level statistics, or go use an online calculator. I recommend https://measuringu.com/statistically-significant/ for a good explainer.
• If your site is not high traffic, you may need to run your tests for weeks or even months to get enough data to clearly show whether there's a difference. Or, it may be worth deciding that there's not clearly a difference, so it's worth testing something else.
   

So, we’ve created some tests with ABJS. How did it go?

Overall, ABJS is nice because it feels like you’re in control. And all developers like to feel in control. It’s also nice because if you want to go set up a test, you can! It’s easy!

Where ABJS loses out is in the pile of lovely features that the big products out there can offer. Creating, managing, scheduling and analysing are all tasks that have been made a lot easier by some off-the-shelf products. But if you can’t afford that budget, or really rather enjoy thinking things through (or indeed love a bit of statistics!) then this is your lot - and it works well enough.

Later on in the series we'll be playing with Google Analytics' A/B testing suite and seeing how it compares. Stay tuned!