Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jun 15 2015
Jun 15

Then I found out about coding standards and profilers! Guidelines that told me how to document my code, inline with my code. No more maintaining class diagrams, flowcharts and what not. It could all be created, based on the little blocks of text above my classes and methods.

To start with, creating and maintaining those blocks of text could prolong the process a bit. But today, I don't really realise that I do it. I just happen to start every file, function, class and method by writing a little one-liner about what I expect from this piece of code.

Sometimes I don't even write the code. I just write a DocBlock, as it's called, making the process of writing the code much easier. The next person who has to fix that infinite redirect, or what ever silly bug I might have introduced, can quickly – by using the profiler – see where the code is stalling, and by looking at the DocBlock see what the code expects as input, how it processes the input, and what it is supposed to do. The clients were happy. Developers were happy. Everyone was happy. And we all went off to have cake.

Three years ago, this blog post would most likely have ended here.

But then someone introduced me to automated software testing. And boy was that annoying. Back to square one, with another "you should really do this" voice in my head, that only made sense to me, and not the client, and was therefore hard to sell.

Clients didn't like this because it would add extra cost to the development, and they don't see a tangible benefit.

What do we get from automated software testing then?

I'm glad you asked. Automated testing is capable of running all sorts of tests (called scenarios) against your code, making sure that your code responds in the way it's supposed to. The great advantage is that you can check what code is passing and what is failing very quickly (and then fix it). If new functionality is created for your website, you can quickly see if it breaks any other functionality.

The disadvantage is that it does take time to create the tests in the first place and then to maintain them. Because of this, we need to be careful about what tests we write, in what manner, and if the cost is justified.

You don't sell it very well. I think I'll skip on this line item.

Hold on a second. There are several types of automated tests. They all have their special purposes, pros and cons. One of them sticks out for me. It's called "smoke testing". It sounds awesome but, from the name alone, no one has any idea what it does. Though once you know what it does, you'll want it!

Why name something that does not explain what it does?

Well, I've tried to figure out where the name comes from, since it's pretty clear that it does not originate from the software development industry. Several suggestions turned up, but the one I like the best is the "plumbing" version. This version describes how plumbers use smoke testing to test if a pipe is leaking by pumping it full of smoke. If you could smell/see the smoke, there must be a leak somewhere.

It tests all aspects of the small pieces of work completed on the pipe. And that is exactly what it's supposed to do here too. It tests all the different layers that an application is depending upon. For example, if a URL fails, it will tell you that it failed (it doesn't tell where or why).

What does it do then?

It tests a URL and sees if it returns a reasonable response. That's all. Nothing more, nothing less.

In the most simple form the test-code pretends it's a user, browses to a specific URL, and sees if the server returns a status code 200, meaning that the request was processed without any problems.

Wait, I can do that. I do it all the time when I develop a website.

Of course you do. But I'm guessing that you don't visit all the other key feature pages on the web site. And, from experience, I can tell you a bug fix in a donation report for example might mess up the payment page, costing you lots of lost revenue.

This is where smoke testing might be able to assist you!

Every time you create a page on a website that contains key (or new) features, you add that page to a list of URLs and let the test framework attempt to visit it. Whenever you want, you can run a test and see if you unintentionally broke something elsewhere.

But all sort of issues could turn up that makes the web server return a status code 200.

Yes, and in case you need to check if different elements on the page are working as designed, you need another type of test.

This may not sound that useful, but given how little effort this takes, it's worth having it in your application. And in Annertech, we have smoke testing setup as part of our installation profile, so from there it's just a matter of adding another URL to a list, to get that URL tested.

In effect, we smoke test as a matter of course with almost no cost to our clients.

That sounds fair enough. How do I use it?

Well, that's something that might not be as simple to answer, because it very much depends on how the rest of your application works. And in Drupal 7 it's even less simple. Sorry for the promising introduction, but Drupal 7 has some issues regarding this, that we will cover later on. In the meantime, let's dig into some actual examples.

First, we need a testing framework PHPUnit happens to be the one that we use in this example. We get the framework via composer, to make things easier to set up. So we'll start by creating the composer.json file.

#composer.json
{
  "require-dev": {
    "phpunit/phpunit": "4.6.*",
    "guzzle/guzzle": "3.9.3"
  }
}

By simply running composer install, we should get all we need for this test to work.

Next, we create a new folder called tests. This is where we put all our relevant test code. Let set up PHPUnit, by creating phpunit.xml.dist (dist extension of the file name makes it possible to have it in a VCS repository, while still being able to be overridden by individual developers).

#tests/phpunit.xml.dist

        xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.1/phpunit.xsd"
        backupGlobals="false"
        colors="true"
        bootstrap="src/bootstrap.php">
    
        
            src/*
        
    

This file tells PHPUnit to look for test cases in the src folder, and that it should load tests/src/bootstrap.php before running the tests. bootstrap.php is where we can prepare the environment if we need anything special done prior to a test. So, let's write that bootstrap.php next. Create a src folder, inside the test folder, and add the following content:

#tests/src/bootstrap.php
/**
 * @file
 * Bootstrap file for unit tests.
 */

require_once __DIR__ . '/../settings.php';
// In this example this file resides in a Drupal installation profile, and if we
// want access to the Drupal environment, we include this file. Change, or
// remove, base on your needs.
require_once __DIR__ . '/../../../../includes/bootstrap.inc';
require_once __DIR__ . '/../../vendor/autoload.php';

We also need a file that can contain all the variables that are most likely to change, not only in different projects, but also in different environments. We call this settings.php

#tests/settings.php

/**
 * @file
 * Testing variables.
 */

// The HTTP address for the webserver to contact.
$GLOBALS['webserver'] = 'http://localhost/smoke-testing';

// Credentials, if needed for any of the URLs.
$GLOBALS['username'] = 'admin';
$GLOBALS['password'] = 'admin';
$GLOBALS['test_urls_for_anonymous_users'] = array(
  array('user'),
  array('user/password'),
  array('user/register'),
);
$GLOBALS['test_urls_for_authenticated_users'] = array(
  array('admin'),
  array('admin/content'),
);

And now we are ready to write the actual test. Inside tests/src, create new folder called smoketest. Inside that, create a new file called SmokeTest.php

#tests/src/smoketest/SmokeTest.php
/**
 * @file
 * Test if the user URLS are available.
 *
 * This is a demo of how smoke testing of URLs could work in Drupal.
 */

use Guzzle\Http\Client;
use Guzzle\Plugin\Cookie\CookiePlugin;
use Guzzle\Plugin\Cookie\CookieJar\ArrayCookieJar;

/**
 * Smoke testing.
 */
class SmokeTest extends \PHPUnit_Framework_TestCase {

  /**
   * A list of URLs to test, that doesn't require the user to log in.
   */
  public function urlProvider() {
    return $GLOBALS['test_urls_for_anonymous_users'];
  }

  /**
   * A list of URLs to test on pages that requires the user to be logged in.
   */
  public function urlProviderWithCredentials() {
    return $GLOBALS['test_urls_for_authenticated_users'];
  }

  /**
   * Test URLs without authentication.
   *
   * @dataProvider urlProvider
   */
  public function testPageIsSuccessful($url) {
    $webserver = $GLOBALS['webserver'];

    $client = new Client($webserver, array('request.options' => array('exceptions' => FALSE)));
    $request = $client->get($url);
    $response = $request->send();
    $this->assertEquals(200, $response->getStatusCode());
  }

  /**
   * Test URLs that requires the user to login.
   *
   * @dataProvider urlProviderWithCredentials
   */
  public function testPageIsSuccessfulWithCredentials($url) {
    $webserver = $GLOBALS['webserver'];

    $client = $this->getAuthenticatedClient();
    $request = $client->get($url);
    $response = $request->send();
    $this->assertEquals(200, $response->getStatusCode());
  }

  /**
   * Authenticates the user and attach the cookie to the $client object.
   */
  private function getAuthenticatedClient() {
    $webserver = $GLOBALS['webserver'];
    $username = $GLOBALS['username'];
    $password = $GLOBALS['password'];

    $client = new Client($webserver, array('request.options' => array('exceptions' => FALSE, 'allow_redirects' => FALSE)));

    // The CookieJar plugin helps us attache the Drupal cookie to the client
    // object.
    $cookie_plugin = new CookiePlugin(new ArrayCookieJar());
    $client->addSubscriber($cookie_plugin);

    // Send a request to the login page to get the required form_build_id.
    // After we receive the HTML, we traverse the DOM for the input field that
    // has the name 'form_build_id'.
    $body = (string) $client->get('user/login')->send()->getBody();
    $dom = new DOMDocument();
    $dom->loadHTML($body);
    $input_elements = $dom->getElementsByTagName('input');
    $form_build_id = '';
    foreach ($input_elements as $element) {
      if ($element->getAttribute('name') == 'form_build_id') {
        $form_build_id = $element->getAttribute('value');
      }
    }

    // These are the required values that we need to send to be able to log in.
    $post_data = array(
      'name' => $username,
      'pass' => $password,
      'form_id' => 'user_login',
      'form_build_id' => $form_build_id,
    );
    $client->post('user/login', array(), $post_data)->Send();
    return $client;
  }
}

In this file, two methods are worth some extra attention. testPageIsSuccessful() and testPageIsSuccessfulWithCredentials() creates the request, using Guzzle, fires the requests, and then we use PHPUnit to check if we are happy with the response. testPageIsSuccessfulWithCredentials() tests the URL as an authenticated user, which is why it's calling the getAuthenticatedClient() method, that takes care of logging the user in, before sending the request.

So far, so good. Let's try to run a test. Go back to the folder where your composer.json is and run this command:

vendor/bin/phpunit -c tests

If everything is set up correctly you should see something like this:

PHPUnit 4.6.9 by Sebastian Bergmann and contributors. Configuration read from /home/user/websites/smoketesting/profiles/smokeprofile/tests/phpunit.xml.dist ..... Time: 1.02 seconds, Memory: 7.75Mb OK (5 tests, 5 assertions)

In that case, you are free to add more URLs to the URL arrays in tests/settings.php.

You said something about Drupal 7 being a special flower in the smoke testing garden?

Yes, in Drupal 7 there are some special cases that you must think through, before you start relying on these tests. You might get what is called false positives on tests that you run, meaning that even if a test fails, Drupal 7 might return a response that makes the webserver think that everything is fine and dandy, and returns a status code 200. I have yet to figure out how to get around this.

Consider this case. You add a page at admin/reports/donations and add the URL into the array, so that it will get tested. Next week, another developer merges his changes with yours, gets a merge conflict, and, by accident, removes your URL from the menu hook. Now, the URL no longer exists. But Drupal 7 will still return a status code 200, because if Drupal cannot find the last piece of the URL (/donations), it tries to find a URL that matches, without that piece. This means it now tries to load admin/reports which exists. It calls the function that corresponds to that URL, and sends the last bit (donations) as a parameter. So, if the URL is missing, the test will still return positive.

So whenever you add a new URL, test the different cases, where it should fail, by hand, to make sure that it actually does.

That's all for today. Now, about that cake...

Apr 20 2015
Apr 20

Building great (Drupal) websites can often be made more difficult than it needs to be when your site builders, developers and themers haven't got the same content as each other.

Creating dummy content is fine up to a point: it lets you test your work to make sure you are going in the right direction. At Annertech we use devel generate all the time and it's a fantastic tool but as it generates content randomly, results can be very varied - for example, a phone number field could end up with 60 integers for example (fine for the developer, but a hindrance to the themer).

The best solution is to have the final site content prior to beginning development, but this rarely happens. If final content is not available, it's beneficial to at least have meaningful content that is relevant to the site. If you build this content into your development plan, it can be used right up to and post site deployment - saving you time and your clients money in the long run.

The Glossary

Bean: A Drupal module to makes blocks fieldable - it stands for "block entities aren't nodes"
Devel Generate: A sub-module of the devel module, used for generating dummy content, users, menus, etc.
Field collection: A module that allows you to group collections of fields and make them repeatable on Drupal entities such as nodes
Features: A module that allows you to export your configuration to code files
Drush: A Drupal shell command tool - the Swiss Army Knife of Drupal tools

The Problem

I'm using the excellent bean module to build a slideshow; its got a field collection for slides and each slide has a number of text fields and an image. We are using features to store the configuration. Each time we rebuild the site we would need to repopulate our bean with our lovely content which after 2 rebuilds could expose a problem in the form of RSI (Really Sore I) or something along those lines. Not an ideal workflow!

The Solution

So, here we are going to build a small PHP script that we can call via drush which will populate a bean with some content programatically, and here's how it looks:

This array contains the 'content' for our bean slideshow. We will loop through this to create each bean.

We will use something like this for each field collection: We are creating an instance of the entity we wish to create, in this case a field collection:

We then attach it to our bean entity. And the whole picture looks like so:

There is a bit more logic added to deal with image files and this script could be improved with a little bit of error checking, but it's a really simple approach and can be easily adapted to suit a number of scenarios.

To call the script with drush simply `drush scr sites/all/scripts/create_bean_slideshow.php` or what ever the path to the script is. For bonus points you can call this through a shell script or build tool like Phing.

In terms of a decent continuous integration workflow, if you're adding new features to a website, this method means:

  • Streamline deployments

  • Verbatim content across environments

  • Contexts, rules and other site elements can use your beans.

  • Less chance of human error

  • A repeatable and reusable process

Also it cuts down on the needless exposure to Really Sore I. And has improved our workflow for new website builds and migrations.

I'd love hear your thoughts in the comments below on how you've solved similar problems for site building.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web