AB testing in PHP with namshi/ab

AB testing is a powerful tecnique that lets you gather metrics about different versions of a feature: it basically consist into displaying a number of different variations of it to your users and tracking the results to see which variation performed better.

An example? In an e-commerce system, you usually have an “Add to cart” button: have you ever though about the impact that single sentence has on your customers? What would sound better, between “Add to cart” and “Buy now”, for example? Copywriters away, you want data to tell you that!

This is why AB testing is important: you serve different versions of something, and track the results to improve the experience users have while using your application: for example, Google benchmarked 40 different shades of blue to find out how the rate of clickthrough would be altered.

At Namshi we decided to ease AB testing by creating a very simple library that would let you generate and manage tests in a very easy and practical way: that’s how Namshi/AB was born.

Installation

You can install the library via composer, as it’s available on packagist.

Then include it, specifying a major and minor version, in your composer.json:

1

"namshi/ab": "1.0.*"

Creating and running tests

The library is very small, and it comes bundled with 2 classes, Test and Container: as you can probably guess, the first is a representation of an AB test and the 2nd serves as a convenient container for all of your test instances.

Here’s how you can create a test:

1
2
3
4
5
6
7
8
9
10
11
12
13

<?php

use Namshi\AB\Test;
use Namshi\AB\Container;

$cssTest = new Test('css', array(
  'default.css'   => 2,
  'new.css'       => 1,
));

$abContainer = new Container(array(
  $cssTest
));

At this point, for example, you can start AB testing your website by changing the CSS in the view:

1
2
3
4
5

<html>
  <head>
      <link rel="stylesheet" type="text/css" href="<?php echo $abContainer['css']->getVariation(); ?>"  />
      ...
      ...

getVariation() will calculate the variation (default.css or new.css) according to the odds of each variation (66% for the first one, 33% for the second one) and will return a string representing the variation.

Persisting the variations through an entire session

Of course, you want to display variations but be consistent with each user, so that if a user gets a variation, it will continue getting the same variation throughout his entire session: to do so, just calculate a random integer (seed), store it in session and pass it to each test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<?php

session_start();

if (!isset($_SESSION['seed_for_example_test'])) {
    $_SESSION['seed_for_example_test'] = mt_rand();
}

$test = new Test('example', array(
  'a' => 1,
  'b' => 1,
));

$test->setSeed($_SESSION['seed_for_example_test']);

// as long as the seed doesn't change
// getVariation() will always return the
// same variation
$test->getVariation();

Soon, you will realize that having a per-test seed is not efficient at all, that’s why you can create a global seed and pass it to the container: from that seed, the container will take care of generating a seed for each test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<?php

session_start();

if (!isset($_SESSION['seed'])) {
    $_SESSION['seed'] = mt_rand();
}

// pass the seed into the constructor
$abContainer = new Container(array(
    new Test('greet', array(
        'Hey dude!' => 1,
        'Welcome'   => 1,
    )),
    new Test('background-color', array(
        'yellow'    => 1,
        'white'     => 1,
    )),
), $_SESSION['seed']);

// or with a setter
$abContainer->setSeed($_SESSION['seed']);

Disabling the tests

Sometimes you might want to disable tests for different purposes, for example if the user agent who is visiting the page is a bot:

1
2
3
4
5
6
7
8
9
10

<?php

$test = new Test('my_ab_test', array(
    'a' => 0,
    'b' => 1,
));

$test->disable();

$test->getVariation(); // will return 'a'!

Once you disable the test and run it, it will always return the first variation, no matter what its odds are, even if it’s zero.

An example

I would recommend you to have a look at the example provided under the examples directory: it’s pretty silly, but it gives you an idea of how easy is to create and run AB tests with this library.

If you look at the code, you will soon realize that it’s very simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

<?php

require __DIR__ . '/../vendor/autoload.php';

use Namshi\AB\Test;
use Namshi\AB\Container;

session_start();

if (!isset($_SESSION['seed'])) {
    $_SESSION['seed'] = mt_rand();
}

$abt = new Container(array(
    new Test('greet', array(
        'Hey dude!' => 1,
        'Welcome'   => 1,
    )),
    new Test('background-color', array(
        'yellow'    => 1,
        'white'     => 1,
    )),
), $_SESSION['seed']);

?>

<html>
    <head>
        <style>
            * {
                background-color: <?php echo $abt['background-color']->getVariation(); ?>;
            }
        </style>
    </head>
    <body>
        <h1>
            <?php echo $abt['greet']->getVariation(); ?>
        </h1>
        
        <div>
            Your seed is <?php echo $_SESSION['seed']; ?>
        </div>
    </body>
</html>

Of course, never write an application like this ;–) this serves just as an example.

Additional features

We tried to extensively cover the available features of the library in its README, so I will just sum them up here:

Why not choosing an existing library

Of course we checked out what the market was offering, but weren’t able to find out a very good, generic-purpose, library in order to generate AB tests:

At the end of the day, namshi/ab is a 1 man-day effort, so we spiked for a bit and decided that it was worth it.

Testing this library

We added a few PHPUnit tests, so you just have to:

1
2
3

cd /path/to/namshi/ab

phpunit

The funny thing is that we also added some test to check that the library correctly generates variations according to their odds.

FOSS

The library is available on Github: please let us know if you would like to see something different, have a suggestion or whatsoever: even better than that, feel free to open a pull request if we screwed up with anything!

In the mood for some more reading?

...or check the archives.