Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Feb 06 2019
Feb 06

Mass.gov dev team releases open source project

Moshe Weitzman

The Mass.gov development team is proud to release a new open source project, Drupal Test Traits (DTT). DTT enables you to run PHPUnit tests against your Drupal web site, without wiping your database after each test class. That is, you test with your usual content-filled database, not an empty one. We hope lots of Drupal sites will use DTT and contribute back their improvements. Thanks to PreviousNext and Phase2 for being early adopters.

Mass.gov is a large, content-centric site. Most of our tests click around and assert that content is laid out properly, the corresponding icons are showing, etc. In order to best verify this, we need the Mass.gov database; testing on an empty site won’t suffice. The traditional tool for testing a site using an existing database is Behat. So we used Behat for over a year and found it getting more and more awkward. Behat is great for facilitating conversations between business managers and developers. Those are useful conversations, but many organizations are like ours — we don’t write product specs in Gherkin. In fact, we don’t do anything in Gherkin beside Behat.

Meanwhile, the test framework inside Drupal core improved a lot in the last couple of years (mea culpa). Before Drupal Test Traits, this framework was impossible to use without wiping the site’s database after each test. DTT lets you keep your database and still test using the features of Drupal’s BrowserTestBase and friends. See DrupalTrait::setUp() for details (the bootstrap is inspired by Drush, a different open source project that I maintain).

Zakim Bridge at Night, North End Boston. Photo by David Fox.
  • Our test cases extend ExistingSiteBase, a convenience class from DTT that imports all the test traits. We will eventually create our own base class and import the traits there.
  • Notice calls to $this->createNode(). This convenience method wraps Drupal’s method of the same name. DTT deletes each created node during tearDown().
  • Note how we call Vocabulary::load(). This is an important point — the full Drupal and Mink APIs are available during a test. The abstraction of Behat is happily removed. Writing test classes more resembles writing module code.
  • Typically, one does not run tests against a live web site. Tests can fail and leave sites in a “dirty” state so it’s helpful to occasionally refresh to a pristine database.

If you have questions or comments about DTT, please comment below or submit issues/PRs in our repository.

More from Moshe: Our modern development environment at Mass.gov

Interested in a career in civic tech? Find job openings at Digital Services.
Follow us on Twitter | Collaborate with us on GitHub | Visit our site

Jun 18 2017
Jun 18
Recently I set out to make a simple instrument for running simpletest tests without having LAMP stack installed on your local environment. I needed this for two reasons:
  1. for running tests locally
  2. for running tests on CI server
I've decided to use Docker and create monolith container with Drupal and all the LAMP stuff inside and here what I've got: docker-tester.

How to use

Before running container you have to setup next ennvironment variables:
  1. KEEP_RUNNING - specify  yes  if you want to keep container running when tests will be executed. Use for debugging purposes only. Default value is  no .
  2. DRUPAL_VERSION - specific version of Drupal. Supported Drupal 7 and Drupal 8. Example:  8.3.2 .
  3. MODULES_DOWNLOAD - a list of modules to download (by Drush) separated by comma. Example:  module_name-module_version,[...] .
  4. MODULES_ENABLE a list of modules to enable (by Drush) separated by comma. Example:  module_name,[...] .
  5. SIMPLETEST_GROUPS - a list of simpletest groups to run separated by comma. Example:  Group 1,[...] .
  6. SIMPLETEST_CONCURRENCY - amount of test runners to test code in parallel. Default value is  1 .
Then you need to build an image for container:
docker build -t drupal-tester .
Next you have two options: either run container with docker-compose tool or run container manualy with docker command.
For local usage I prefere to use docker-compose because it's easier than write all the CLI docker command manualy. Just specify what module you want to test inside of docker-compose.yml file and run:
docker-compose up && docker-compose down
It will run the container, install Drupal inside of it and run tests. That's all.

For running tests on CI server I use docker command and specify all the needed environment variables manualy:

docker run -v $(pwd)/test_results:/var/www/html/test_results -v $(pwd)/custom_scripts:/var/www/html/custom_scripts -e KEEP_RUNNING=no -e DRUPAL_VERSION=8.3.2 -e MODULES_DOWNLOAD=module-version -e MODULES_ENABLE=module -e SIMPLETEST_GROUPS=module_test_group -e SIMPLETEST_CONCURRENCY=1 drupal-tester
It allows you to override environment variables and volumes that you want to mount inside of the container. So you can setup different jobs on your CI server to test different modules on different Drupal versions with the help of this one container.

When docker finished the process all test results by default will be placed into test_results directory but you can easily override this by mounting some other directory inside of a container.

Setup and customization

Sometimes you need to do something before running tests. For example override some module specific settings or setup some Drupal variables etc. You can get it done with custom *.sh scripts. Just write sh file with all needed actions/commands and put it inside custom_scripts folder. All the files inside of this directory will be executed before running tests.
Dec 01 2016
Dec 01

dec16The twelfth issue of 2016 is now available! This month we look at how to write good tests with Behat and using Test Driven Development. This issue also includes articles on using HTTPlug to decouple your HTTP Client, Decoupled Blocks with Drupal and JavaScript. Our columnists have articles on writing a Chat bot, advice on securing your application’s secrets, making better bug reports, respecting diversity, and a look back at 2016.

Download your issue and read a FREE article today.

Oscar still remembers downloading an early version of the Apache HTTP server at the end of 1995, and promptly asking "Ok, what's this good for?" He started learning PHP in 2000 and hasn't stopped since. He's worked with Drupal, WordPress, Zend Framework, and bespoke PHP, to name a few. Follow him on Google+.
Aug 22 2016
Aug 22

As you'd be aware by now - Drupal 8 features lots of refactoring of form procedural code to object-oriented.

One such refactoring was the way forms are build, validated and executed.

One cool side-effect of this is that you can now build and test a form with a single class.

Yep that's right, the form and the test are one and the same - read on to find out more.


Firstly kudos here to Tim Plunkett who pointed this out to me, and to all of those who championed much of the refactoring that made this even possible.

Testing forms and elements in Drupal 7

In Drupal 7 to test a form, or an element you need the following:

  • A test module with:
    • A hook_menu entry
    • A form callback
    • (optional) A validate callback
    • (optional) A submit callback
  • A web-test (a test that extends from DrupalWebTestCase)

Drupal 8 forms

As you're probably aware from all the example code and posts out there, forms in Drupal 8 are objects that implement Drupal\Core\Form\FormInterface.

Luckily, you can write a test that both extends from KernelTestBase (Drupal\KernelTests\KernelTestBase) that also implements FormInterface. This means you don't need all of the additional routing plumbing you needed in Drupal 7.

Let's look at an example in Drupal - PathElementFormTest (\Drupal\KernelTests\Core\Element\PathElementFormTest). This test is to test core's PathElement (\Drupal\Core\Render\Element\PathElement) - a plugin that provides a form element where users can enter a path that can be optionally validated and stored as either a \Drupal\Core\Url value object or a array containing a route name and route parameters pair. So in terms of testing, the bulk of the logic in the element plugin is contained in the #element_validate callback - PathElement::validateMatchedPath.

There are several different combinations of configuration for the path element as follows:

  • Required and validated with no conversion
  • Required and non validated with no conversion
  • Optional and validated with no conversion
  • Optional, validated and converted into a route name/parameter pair
  • Required, validated and converted into a route name/parameter pair
  • Required, validated and converted into a Url object

So we need to set up several instance of the element on a test form.

So because our test extends from KernelTestBase, but also implements FormInterface, we just build a normal form array with all of these configurations in our implementation of FormInterface's ::buildForm method - see \Drupal\KernelTests\Core\Element\PathElementFormTest::buildForm to see how this is done. We're not interested in doing any additional validation or submission, so our implementation of FormInterface's ::submitForm and ::validateForm can be blank.

Testing the element behaviour

So to test the element validate works as expected for each of the fields, we need to trigger submission of the form. Now in a web-test, we'd use the internal test browser to visit the form on a route and then use a method like BrowserTestBase::submitForm to actually submit the form. But as we're using a kernel test here, there is no internal browser - so instead we can submit directly through the form_builder service (\Drupal\Core\Form\FormBuilderInterface). The code in PathElementFormTest looks something like this:

$form_state = (new FormState())
    'required_validate' => 'user/' . $this->testUser->id(),
    'required_non_validate' => 'magic-ponies',
    'required_validate_route' => 'user/' . $this->testUser->id(),
    'required_validate_url' => 'user/' . $this->testUser->id(),
$form_builder = $this->container->get('form_builder');
$form_builder->submitForm($this, $form_state);

So firstly we're building a new FormState object and setting the submitted values on it - this is just a key-value pair of form values. Then we're getting the form builder service from the container and submitting the form.

From here, if there were any errors, they'll be present on the form state object. So we can do things likes check for expected errors, or check for expected values.

For example, to check that there were no errors.

$this->assertEquals(count($form_state->getErrors()), 0);

Or to check that the conversion occurred (i.e. the input path was upcast to a route name/parameter pair or Url object).

$this->assertEquals($form_state->getValue('required_validate_route'), array(
  'route_name' => 'entity.user.canonical',
  'route_parameters' => array(
    'user' => $this->testUser->id(),

Or to check for a particular error.

$this->assertEquals($errors, array('required_validate' => t('@name field is required.', array('@name' => 'required_validate'))));

Summing up

So why would you want to use this approach?

Well for one, the test is damn fast. Kernel tests don't do a full site install, and because there is no HTTP to fetch and submit the form, you get fast feedback. And when you get fast feedback, you're more likely to practice good test driven development.

So if you're building an element plugin for a contrib or client project, I encourage you to start with a test, or rather a form, or rather both. Specify the various configurations of your element and test the expected behaviour.

I'm sure you agree, this is another clear case of Drupal 8 for the win.

Drupal 8 Drupal Development Testing
Aug 09 2016
Aug 09
Screenshot of Behat tests results

If automated testing is not already part of your development workflow, then it’s time to get started. Testing helps reduce uncertainty by ensuring that new features you add to your application do not break older features. Having confidence that your not breaking existing functionality reduces time spent hunting bugs or getting reports from clients by catching them earlier.

Unfortunately, testing still does not get the time and attention it needs when you’re under pressure to make a deadline or release a feature your clients have been asking for. But—like using a version control system and having proper development, staging, and production environments—it should be a routine part of how you do your work. We are professionals, after all. After reading all the theory, I only recently took the plunge myself. In this post, I’ll show you how to use Behat to test that your Drupal site is working properly.

Before we dive in, the Behat documentation describes the project as:

[…] an open source Behavior Driven Development framework for PHP 5.3+. What’s behavior driven development, you ask? It’s a way to develop software through a constant communication with stakeholders in form of examples; examples of how this software should help them, and you, to achieve your goals.

Basically, it helps developers, clients, and others communicate and document how an application should behave. We’ll see shortly how Behat tests are very easy to read and how you can extend them for your own needs.

Mink is an extension that allows testing a web site by simulating interacting with it through a browser to fill out form fields, click on links, and so forth. Mink lets you test via Goutte, which makes requests and parses the contents but can’t execute JavaScript. It can also use Selenium, which controls a real browser and can thus test JS and Ajax interactions, but Selenium requires more configuration.


To get started, you’ll need to have Composer on your machine. If you don’t already, head over to the Composer Website. Once installed, you can add Behat, Mink, and Mink drivers to your project by running the following in your project root:

composer require behat/behat
composer require behat/mink
composer require behat/mink-selenium2-driver
composer require behat/mink-extension

Once eveything runs, you’ll have a composer.json file with:

    "require": {
        "behat/behat": "^3.1",
        "behat/mink": "^1.7",
        "behat/mink-selenium2-driver": "^1.3",
        "behat/mink-extension": "^2.2"

This will download Behat and it’s dependencies into your vendor/ folder. To check that it works do:

vendor/bin/behat -V

There are other ways to install Behat, outlined in the quick introduction.

The Drupal community has a contrib project, Behat Drupal Extension, that is an integration for Behat, Mink, and Drupal. You can install it with the requre command below. I had to specify the ~3.0 version, otherwise composer couldn’t satisfy dependencies.

composer require drupal/drupal-extension:~3.0

And you’ll have the following in your composer.json:

       "drupal/drupal-extension": "~3.0",

Configuring Behat

When you run Behat, it’ll look for a file named behat.yml. Like Drupal 8, Behat uses YAML for configuration. The file tells Behat what contexts to use. Contexts provide the tests that you can run to validate behavior. The file configures the web drivers for Mink. You can also configure a region_map which the Drupal extension uses to map identifiers (left of the :) to CSS selectors to identify theme regions. These come in very handy when testing Drupal theme output.

The one I use looks like:

        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MarkupContext
        - Drupal\DrupalExtension\Context\MessageContext
        - FeatureContext
      goutte: ~
      javascript_session: selenium2
        wd_host: http://local.dev:4444/wd/hub
        capabilities: {"browser": "firefox", "version": "44"}
      base_url: http://local.dev
      blackbox: ~
        breadcrumb: '#breadcrumb'
        branding: '#region-branding'
        branding_second: '#region-branding-second'
        content: '#region-content'
        content_zone: '#zone-content'
        footer_first: '#region-footer-first'
        footer_second: '#region-footer-second'
        footer_fourth: '#region-footer-fourth'
        menu: '#region-menu'
        page_bottom: '#region-page-bottom'
        page_top: '#region-page-top'
        sidebar_first: '#region-sidebar-first'
        sidebar_second: '#region-sidebar-second'

Writing a Simple Feature

Now comes the fun part. Let’s look at writing a feature and how to test that what we expect is on the page. The first time we run it, we need to initialize Behat to generate a FeatureContext class. Do so with:

vendor/bin/behat --init

That should also create a features/ directory, where we will save the features that we write. To behat, a feature is test suite. Each test in a feature evaluates specific functionality on your site. A feature is a text file that ends in .feature. You can have more than one: for example, you might have a blog.feature, members.feature, and resources.feature if your site has those areas available.

Of course, don’t confuse what Behat calls a feature—a set of tests—with the Features module that bundles and exports related functionality into a Drupal module.

For my current project, I created a global.feature file that checks if the blocks I expect to have in my header and footer are present. The contents of that file are:

Feature: Global Elements

  Scenario: Homepage Contact Us Link
    Given I am on the homepage
    Then I should see the link "Contact Us" in the "branding_second" region
    Then I should see the "Search" button in the "branding_second" region
    Then I should see the "div#block-system-main-menu" element in the "menu" region

As you can see, the tests is very readable even though it isn’t purely parsing natural language. Indents help organize Scenarios (a group of tests) and the conditions needed for each scenario to pass.

You can set up some conditions for the test, starting with “Given”. In this case, given that we’re on the homepage. The Drupal Extension adds ways to specify that you are a specific user, or have a specific role, and more.

Next, we list what we expect to see on the webpage. You can also tell Behat to interact with the page by specifying a link to click, form field to fill out, or a button to press. Again here, the Drupal extension (by extending the MinkExtension), provides ways to test if a link or button are in one of our configured regions. The third test above uses a CSS selector, like in jQuery, to check that the main menu block is in the menu region.

Testing user authentication

If you’re testing a site that is not local, you can use the drush api driver to test user authentication, node creation, and more. First, setup a drush alias for your site (in this example, I’m using local.dev. Then add the following are in your behat.yml:

      api_driver: 'drush'
        alias: "local.dev"

You can then create a scenario to test the user login’s work without having to specify a test username or password by tagging them with @api

  Scenario: Admin login
    Given I am on the homepage
    Given I am logged in as a user with the "admin" role
    Then I should see the heading "Welcome" in the "content" region

If you’ve customized the username text for login, your test will fail. Don’t worry! Just add the following to your behat.yml file so that the test knows what text to look for. In this case, the username field label is just E-mail.

        username_field: "E-mail"

Custom Testing by Extending Contexts

When you initialized Behat, it created a features/bootstraps/FeatureContext.php file. This can be a handy class for writing custom tests for unique features on your site. You can add custom tests by using the Drupal Extension’s own sub-contexts. I changed my Feature Context to extend the Mink Context like this:

class FeatureContext extends MinkContext implements SnippetAcceptingContext {

Note that if you do that, you’ll need to remove MinkContext from the explicit list of default context in behat.yml.

No matter how you organize them, you can then write custom tests as methods. For example, the following will test that a link appears in the breadcrumb trail of a page. You can use CSS selectors to find items on the page, such as the ‘#breadcrumb’ div in a theme. You can also re-use other tests defined by the MinkContext like findLink.

 * @Then I should see the breadcrumb link :arg1
public function iShouldSeeTheBreadcrumbLink($arg1)
   // get the breadcrumb
     * @var Behat\Mink\Element\NodeElement $breadcrumb
   $breadcrumb = $this->getSession()->getPage()->find('css', 'div#breadcrumb');

   // this does not work for URLs
   $link = $breadcrumb->findLink($arg1);
   if ($link) {

   // filter by url
   $link = $breadcrumb->findAll('css', "a[href=\"{$arg1}\"]");
   if ($link) {

   throw new \Exception(
       sprintf("Expected link %s not found in breadcrumb on page %s",

If your context implements the SnippetAwareContext, behat will generate the Docblock and method signature when it encounters an unknown test. If you’re feature has the following:

    Then I should see "foo-logo.png" as the header logo.

When you run your tests, behat will output the error message below that you can copy and paste to your context. Anything in quotes becomes a parameter. The DocBlock contains the annotation Behat uses to find your test when it’s used in a scenario.

* @Then I should see :arg1 as the header logo.
public function iShouldSeeAsTheHeaderLogo($arg1)
   throw new PendingException();


Follow the Behat docs to install selenium: http://mink.behat.org/en/latest/drivers/selenium2.html. When you’re testing you’ll need to have it running via:

java -jar /path/to/selenium-server-standalone-2.53.0.jar 

To tell Behat how to use selenium your behat.yml file should have:

        wd_host: http://local.dev:4444/wd/hub
        capabilities: {"browser": "firefox"}

You’ll also need to have Firefox installed. OF course, at the time of this writing, Firefox is asking people to transition from use Webdriver to Marionette for automating browser usage. I have Firefox 47 and it’s still working with Webdriver as far as I can tell. I have not found clear, concise instructions for using Marionette with Selenium. Another option is to use Phantom.JS instead of Selenium for any features that need a real browser.

Once everything is working—you’ll know it locally because a Firefox instance will pop up—you can create a scenario like the following one. Use the @javascript tag to tell Behat to use Selenium to test it.

  Scenario: Modal Popup on click
    Given I am at "/some/page"
    When I click "View More Details"
    Then I wait for AJAX to finish
    Then I should see an "#modal-content" element
    Then I should see text matching "This is a Modal"


If you don’t have tests for your site, I urge you to push for adding them as part of your ongoing work. I’ve slowly added them to my main Drupal client project over the last few months and it’s really started to pay off. For one, I’ve captured many requirements and expectations about how pages on the site work that were only in my or the project manager’s heads, if not lost in a closed ticket somewhere. Second, whenever I merge new work in and before any deploy I can run tests. If they are all green, I can be confident that new code and bug fixes haven’t caused a regression. At the same time, I now have a way to test the site that makes it less risky to re-factor or reorganize code. I didn’t spend a lot of time building tests, but as I work on a new feature or fix a bug, writing a test is now just part of confirming that everything works as expected. For complicated features, it’s also become a time saver to have a test that automates a complicated interactions—like testing a 3-page web form, since Behat can run that scenario much faster than I can manually.

The benefits from investing in automated testing outweigh any initial cost in time and effort to set them up. What are you waiting for?

Jun 30 2016
Jun 30

I feel really excited to have cleared the mid-Term requirement for my project in Google Summer of Code (GSoC). The results of the mid-Term evaluations were announced June 28, 00:30 IST. This was the evaluation for the first phase of GSoC. In this evaluation process, set up by GSoC organisers, students and mentors have to share their feedback about the current progress of the project. Mentors need to give a pass/ fail grade. Students can continue coding once they clear the evaluations successfully.

I have been working on Porting Search Configuration module to Drupal 8. Please go through my previous posts if you would like to have a look into the past activities in this port process.

Last week I worked on testing some of the units of this module using the Php unit tests framework. Testing is an important process when it comes to any software development process. It plays a crucial role for any software. It helps us to understand the improve our software to the required level by making use of various test cases. We input various values and check whether the tests are passed according to the requirement. If any condition fails to our expectations, we need to make the required changes to suit the application needs.

Php unit tests are generally used to test some units of an application. To check whether the functions implemented gives the expected output, behaviour of the functions in various test cases, giving different types of arguments as inputs to check the errors or flaws for improving the application.

We need to install the Php unit for this process. You could follow this documentation for this process. Furthermore, they give a detailed analysis of the Php Unit Tests.

Once the installation is completed, we can start writing the unit tests for the functionalities we have implemented. The tests are generally stored in the tests/src/Unit directory of the module. The name of the unit test file will be of the format xyzTest.php. All tests are suffixed by ‘Test’. ‘xyz’ can be replaced by the functionality you are going to test.

The following is a simple test to check the sum of two numbers: sumTest.php

class SampleTest extends PHPUnit_Framework_TestCase
  public function testSum()
    $this->assertEquals(2+2, 4);

As mentioned in this above code snippet, we need to create a class, with class name suffixed by ‘Test’ which is an extension of PHPUnit_Framework_TestCase. Now, we need to write the tests inside as member functions. The functions starting with the name test are executed. Here we are checking the sum of the two numbers. This is a very simple demonstration.

The tests are run by using the command PHPUnit. i.e,

$ phpunit tests/src/Unit/sumTest.php

The output generated on running the above test is:

PHPUnit 5.4.6 by Sebastian Bergmann and contributors.

. 1 / 1 (100%)

Time: 252 ms, Memory: 13.25MB

OK (1 test, 1 assertion)

Stay tuned for future updates on this module port.

Jun 17 2016
Jun 17

PHPStorm has a lot of built in test runners, but it doesn't support Drupal's Simpletest test runner. In this blog post we'll see how we can execute Drupal tests inside PHPStorm using Drupal test runner.


As a Drupal 8 core developer and a back-end developer, I have to write tests all the time. PHPStorm is very handy tool when you are writing tests, because it supports a lot native test runners.

Native PHPStorm PHPUnit Test Runner

Whenever I’m writing PHPUnit test inside or outside Drupal it is very convenient to run the tests and debug the fails with just one click. If you are not familiar with this then have a look at running PHPUnit tests within PHPStorm on drupal.org or Run/Debug Configuration: PHPUnit - JetBrains.

Drupal Test Runner

Drupal uses its own test runner located under 'core/scripts/runtest.sh'. There is no native support in PHPStrom to run Drupal tests. You can execute runtest.sh as PHP script but to get the configuration right is bit tricky.

Native PHPStorm Drupal Test Runner

I recently had a chat with Matt Glaman (mglaman) on twitter about native PHPStorm Drupal Test runner when he tweeted about his one click run test configuration for Drupal 8.

My @phpstorm run configs to run PhantomJS, PHP built-in server, and PHPUnit with a click of a button https://t.co/2hPGPr78QN

— Matt Glaman (@nmdmatt) May 28, 2016

After our conversation Matt created a PHPStorm plugin which executes Drupal's test runner. PHPStorm Drupal's test runner allows to run Simpletest, Unit, Kernel, Functional, and FunctionalJavascript tests with just one single click.


To install the plugin go to File → Settings → Plugin → Browse repositories and search Drupal Test Runner and install

After restarting the PHPStorm a new option “Drupal run test” will be available.


After clicking “Drupal run test” you’ll see following configuration form

You can configure it to run single class.

Or you can configure it for a module/group.


Now you can run this class and when the test is finished you can view the results inside the PHPStrom.


Yes, you can also debug tests by enabling PHP CLI scripts debugging with PhpStorm.

Why use this plugin? Simpletest is deperecated in Drupal 8 anyway.

This is a fair question, since Symfony components were added to Drupal 8. There are several attempts to convert core scripts to symfony console component commands. There is also an issue in which simpletest is going to be replaced by PHPUnit. Now, all the core kernel tests are executed using PHPUnit test runner and core is focusing on converting all the web tests to browser tests, web tests are executed using runtest.sh script in core and browser tests are using PHPUnit test runner for execution. One of the pre-requisites for Drupal 9 is to remove simpletest module from core but this means simpletest module will remain a part of Drupal 8 codebase, so this plugin will be useful until then.

There is also a Drupal 7 version of this plugin.

What's next?

PHPStrom Drupal Test Runner is in a beta release and we need community help for a stable release so please test the plugin and create issues.

I'd like to thank Matt Glaman for considering my idea and building this amazing plugin.

PHPStorm Testing
Apr 25 2016
Apr 25
teaser image for blog post

Over the last couple of years, we have been using Codeception at Ixis for running automated acceptance tests during development work. Over this time we attempted to distil some of the ideas and abstract custom code into Codeception modules, which are all available on GitHub.

Adopting Codeception for automated acceptance testing has been quite the learning process: getting used to the Codeception system, best practices and underlying code; facing challenges when writing tests for unusual or difficult situations; moving from the cURL-based PhpBrowser to the more advanced and complex WebDriver; problems with underlying components such as Selenium and PhantomJS; lack of integration with Drupal... phew! And that's not before we've considered processes, methods and best practices for running automating testing itself!

The latter, however, is a story for perhaps another time. This post summarises the Codeception modules we've worked on, why they might be useful and some lessons learned.

Drupal Content Type Registry

Drupal Content Type Registry is a module to provide a set of classes that encapsulate Drupal content types. This makes it much easier to quickly test standard Drupal functionality relating to content types, taking into account how they exist on your site. It enables testing of things such as the content types admin page, the 'manage fields' page for each content type, and provides createNode() and deleteNode() methods that can be used to quickly create test nodes where you can provide the test data using specific values, random values, or a range of values where one is picked at random.

Once enabled, a contentTypes.yml file should be created which defines:

  • GlobalFields - fields that will be used across all of the content types on the site. This is useful for things like title and body fields, to save you having to redefine the exact same field on every content type.
  • ContentTypes - content types on the site and the fields they have (global or otherwise).

Here's the example from the README:

        machineName:    body
        label:          Body
        type:           Long text and summary
        selector:       "#edit-body-und-0-value"
        widget:         Text area with a summary
        required:       true
        machineName:    title
        label:          Title
        type:           Node module element
        selector:       "#edit-title"
        humanName:    News
        machineName:  news
                - title
                - body
                machineName:    field_image
                label:          Image
                type:           Image
                selector:       "#edit-field-image"
                widget:         Media file selector
                required:       true
                testData:       "image1.png"
                machineName:    field_icon
                label:          Icon
                type:           Text
                selector:       "#edit-field-icon"
                widget:         Text field
                    - editor
                    - publisher
                    - smiley
                    - grumpy
                    - happy
                    - wacky
                    - ["click", ["#button"]]
                    - ["fillField", ["#the-field", "the-value"]]
                    - ["waitForJs", ["return jQuery.active == 0;"]]
        submit: "#edit-submit-me-please"

The definition of fields comprises the machine name, label, field type, widget and CSS ID or selector used when filling in node edit forms. Similarly, the definition of content types comprises the machine name, human-readable name and a list of field definitions. The module also provides a set of classes representing most field widgets provided by Drupal 7 core.

There are other features too, such as: preSteps and postSteps, which allow optional steps to run before and after filling the field; testData, which allows specific test data (literals or random text) to be used for each field; skippedRoles, which skips certain fields for specific user roles when creating nodes; and "Extras", which simulate the user clicking things on the node edit form that are not actually fields, like set the sticky status or the publication status of a node. For more detailed information on the module's configuration, see the README file.

This module (in combination with Drupal User Registry) also provides enough to create close to generic tests that will check your Drupal 7 site for all expected content types and fields. See this example Gist.

Credit must go to Chris Cohen for his original work on the Content Type Registry.

Drupal Drush

Drupal Drush allows the running of Drush commands in acceptance tests. It also allows the use of the following statements in tests:

// Execute "drush cc all"
$I->getDrush("cc", array("all"))->mustRun();

The getDrush() method returns and instance of Symfony\Component\Process\Process so you can read stdout, get the exit code, etc. The ->mustRun() method is useful as a Symfony\Component\Process\Exception\ProcessFailedException exception will be thrown if the command fails, meaning your test will automatically fail.

Drupal Mail System

Drupal Mail System allows the testing of the Drupal mail system.

// Test to see expected number of emails sent.
// Clear emails from queue.
// Check email fields contains text
    "body" => "body contains this text",
    "subject" => "subject contains this text",

Relies on TestingMailSystem class which stores the emails in a Drupal system variable.

Drupal Pages

Drupal Pages is a dependency of some other modules listed here and contains several PageObjects for "generic" Drupal 7 pages, based on "vanilla Drupal 7' and the default Bartik front-end or Seven administration themes. These PageObject classes can be extended to override any static properties as required for the site being tested.

Drupal User Registry

Drupal User Registry is a Codeception module for managing test users when running acceptance tests with Codeception. Once configured, it can automatically create and delete test Drupal users at the beginning and end of a test suite run. These users can then be used during acceptance tests to login and test elements of the project specific to that role.

The module is configured in the suite configuration:

class_name: AcceptanceTester
        - PhpBrowser
        - DrupalUserRegistry
            url: 'http://localhost/myapp/'
            defaultPass: "foobar"
                    name: administrator
                    email: admin@example.com
                    pass: "foo%^&&"
                    roles: [ administrator, editor ]
                    root: true
                    name: editor
                    email: editor@example.com
                    roles: [ editor, sub-editor ]
                "sub editor":
                    name: "sub editor"
                    email: "[email protected]"
                    roles: [ sub-editor ]
                    name: authenticated
                    email: authenticated@example.com
                    roles: [ "authenticated user" ]
            create: true
            delete: true
            drush-alias: '@mysite.local'
  • defaultPass - use this password for all created user accounts, unless they have one individually specified.
  • users - a list of test user accounts to create, complete with username. email, password and a list of roles.
  • create and delete - whether to create and delete users at the start and end of a run.
  • drush-alias - the Drush alias to use when managing users via DrushTestUserManager.

Once configured, the module also allows the use of the following statements in tests:

// Returns a DrupalTestUser object representing the test user available for
// this role.
$user = $I->getUserByRole($roleName);
// Returns a DrupalTestUser object representing the test user available for
// exactly these roles.
$user = $I->getUserByRole([$roleName1, $roleName2]);
// Returns a DrupalTestUser object representing the user, or false if no users
// were found. Note this will only return a user defined and managed by this
// module, it will not return information about arbitrary accounts on the site
// being tested.
$user = $I->getUser($userName);
// Returns an indexed array of configured roles, for example:
//   array(
//     0 => 'administrator',
//     1 => 'editor',
//     2 => ...
//   );
$roles = $I->getRoles();
// Returns a DrupalTestUser object representing the "root" user (account with
// uid 1), if credentials are configured:
$rootUser = $I->getRootUser();

This module is used in any acceptance test suite we create in order to test specific elements where being logged in as a user with specific roles is necessary. However, there are some limitations and improvements that could be made.

The user registry can be used in combination with codeception Step Objects and Drupal Pages to provide login and logout functions to enable testing as an authenticated user. For example:
    $ php codecept.phar generate:stepobject acceptance AuthenticatedSteps
    class AuthenticatedSteps extends \AcceptanceTester
         * Log in.
         * @param DrupalTestUser $person
         *   The Drupal Person to log in.
        public function login(DrupalTestUser $person)
            $I = $this;
            $I->expectTo('not be redirected due to already being logged in');
            $I->seeCurrentUrlEquals('/' . UserAccountPage::route('login'));
            $I->expectTo('see various elements of the login page');
            $I->amGoingTo('fill in the login form');
            $I->fillField(UserAccountPage::$loginFormUsernameSelector, $person->name);
            $I->fillField(UserAccountPage::$loginFormPasswordSelector, $person->pass);
            $I->expectTo('log in successfully');
            $I->dontSee(UserAccountPage::$loginFormCredentialsErrorMessage, Page::$drupalErrorMessageSelector);
         * Log out.
        public function logout()
            $I = $this;
            $I->amGoingTo('log out');
            $I->expectTo('be redirected to the front page');
Then in a test:
     * Test pages.
     * @guy AcceptanceTester\AuthenticatedSteps
    class ThePagesCest
         * Test the page as an authenticated user.
         * @param AuthenticatedSteps $I
        public function testThePage(AuthenticatedSteps $I)
            // ...

There's potential for integrating the login(), logout() and other related functionality into the Drupal User Registry module itself, but still allow the test suite author to override or specify these procedures themselves.

Drupal Variable

Drupal Variable allows us to test Drupal system variables, for example:

// Assert that the target site has variable "clean_url" set to 1
$I->seeVariable("clean_url", 1);
// Set a variable.
$I->haveVariable("clean_url", 0);
// Delete a variable.
// Retrieve a variable value.
$value = $I->getVariable("clean_url");

This module was the key to working out different "connections" to Drupal itself. Previously, we were restricted by the assumption that everything needed to be done via the browser in acceptance testing (see the lessons learned around creating test content via the test browser, in Drupal Content Type Registry above).

Drupal Variable can connect to a Drupal site using one of three methods:

  • Bootstrapped - if the site is accessible on a locally mounted file system, fully bootstrapping the site is possible. In this case we use the Codeception\Module\Drupal\Variable\VariableStorage\Bootstrapped class and ensure we set the drupal_root configuration setting.
  • Direct connection - if the site is remote but we have access via an open MySQL connection, we use the Codeception\Module\Drupal\Variable\VariableStorage\DirectConnection class and set a dsn, user and password.
  • Drush - if the site is remote but access via a Drush alias is available, we use the Codeception\Module\Drupal\Variable\VariableStorage\Drush class and ensure we set the drush_alias setting in the module's configuration.

In review

Problems with creating test content in the browser

Creating lots of test content in the browser with Drupal Content Type Registry has proved problematic in places and in hindsight not the best way to achieve our goal. Many of our development projects and supported sites have additional elements that make things that little more difficult too, such as alternative publishing workflows with Workbench or WYSIWYG editing and media management with CKEditor and the Media suite of modules.

Whilst these are problems that must be tackled when explicitly testing the UI and user flow to verify a particular feature, it's not necessary to effectively repeatedly run this test in order to set up test data and content. Not only is it in places complex, but running a full suite (in multiple environments, i.e. in different browsers) is slow and a lot of that time is (obvious facepalm incoming) setting up test data.

Once we had a more 'useful' connection to Drupal with Drush, database or a fully bootstrapped connection as implemented in the Drupal Variable module, it became clear that there could be a better (or at least faster) way of setting up test content. We're not 100% what this will involve yet, perhaps custom code, or integration with modules such as Migrate or Node export.

Refactoring and relating the modules

Most of these module sprang from solving different problems whilst testing different projects. Whilst we have one or two suites using all if not most of them, the modules function together but may benefit from refactoring any shared code. For example, Drupal User Registry will currently check for, create and delete test users using Drush and aliases and the original intention was to have other ways of doing so via other connections. As mentioned above, Drupal Variable implements a different method but already has three types of connection, including Drush, which Drupal User Registry could utilise.


Fairly recently Codeception introduced Gherkin and natural language features which is up there at the top of a list of advantages that Behat may have over Codeception - second only to it's much more extensive integration with Drupal, of course! Behat is being considered as an alternative or supplemental method for testing our Drupal sites.

Nov 04 2014
Nov 04

I am proud to announce the release of the Behat Drupal Extension 3.0! Since 2012, the Drupal Extension has been used for Behaviour Driven Development of thousands of Drupal sites all over the world. The project began as part of the Drupal.org upgrade, and was quickly generalized to enable the testing of any Drupal site.

Version 3 has many exciting new features, and is compatible with Behat 3. Note version 2 does not exist, wanting to avoid version soup.

Behavior Driven Development at BADCamp

Before I dive into the new features, it is important to clarify the difference between testing an application, and Behavior Driven Development (BDD).

As everzet pointed out at Drupalcon no business-critical feature starts with

When browsing an article User Should be able to see related articles 1 2 3 When browsing an article User Should be able to see related articles

rather, it starts with a conversation

In order to read more interesting articles As a reader I need to see related articles to the one I'm reading 1 2 3 In order to read more interesting articles As a reader I need to see related articles to the one I'm reading

While both are testing something, only the latter is truly describing a behavior that matters to a stakeholder. This is the subject of a blog post, or series of posts, in and of itself, so more on this subject later.

What’s new


Documentation for this project has been integrated into the repository, and is automatically building on readthedocs.org. Thanks to Melissa Anderson for this awesome work!

A starter context

A starter Drupal context that makes no assumptions around the language used for each test, but still provides all the previous functionality to interact directly with Drupal.

Users are encouraged to start tests from this context which will allow them to use truly ubiquitous language that is specific to each project.

Drupal Drivers

The Drupal Drivers now exist in a separate project, allowing for non-Behat applications to interact with Drupal (e.g., calling directly from Mink, or Codeception).

Note that the Drupal 6 driver has been removed, but since drivers are now separate projects, it will be easy to port that over to the Drupal Extension 3, should somebody want.

It should also be noted that Drupal 8 support will require ongoing work as the code base there evolves towards release.

More granular pre-defined step-definitions

Existing step definitions have been split into 4 indepentent contexts:

  • DrupalContext – This contains steps for working with content, users, and taxonomies.
  • MinkContext – This is an extension to the Mink Extension, providing additional steps for working with regions and forms.
  • MessageContext – Provides steps for working with Drupal success/warning/error messages.
  • DrushContext – Provides steps for calling drush commands directly from scenarios

This allows for the use of some pre-definied step-definitions, rather than the previous all-or-none approach.

No more regex!

The pre-definied steps now use the new turnip syntax introduced in Behat 3:

Given I am viewing a/an :type (content )with the title :title 1 Given I am viewing a/an :type (content )with the title :title

rather than

Given /^(?:a|an) "(?P[^"]*)" node with the title "(?P[^"]*)"$/ 1 Given /^(?:a|an) "(?P[^"]*)" node with the title "(?P[^"]*)"$/

What’s a ‘node’?!

The term node has been removed from steps and replaced with content in all pre-defined steps.

What’s next?

BDD Mini Summit

I will be attending the Behat mini-summit at BADCamp along with several other folks from Phase 2. I hope to highlight the Drupal Extension 3, and discuss best-practices for the wide variety of testing needs. I also hope to continue discussions around Behat, Mink and Drupal core.

Oct 02 2012
Oct 02

Posted Oct 2, 2012 // 2 comments

While working on the SPS Module we became very well acquainted with Drupal's Unit Test Class. I will have a follow up Post on the lessons we learned about writing test, but here I will outline the basic steps for adding unit test to a module. It is worth noting that I will be talking about the Drupal Unit test not the Drupal Web Test.

Step 1 - Tell Drupal about your tests

Drupal will look for classes that define tests ,so all we need to do is make sure that those classes can be found. To do this we just add a files line to the .info file, so that Drupal knows which files to investigate for test classes.


  name = MODULE
  core = 7.x
  files[] = tests/*.test

Now we can put all of our tests into the tests folder of the module, as long as we add .test to the file name.

Step 2 - Test Base Class

While a base class is not needed, it can be very helpful, as there are things (such as enabling your module) that can be done in a setup method.

abstract class MODULEBaseUnitTest extends DrupalUnitTestCase {
   * One using of this function is to enable the module used for testing, any dependencies
   * or anything else that might be universal for all tests
  public function setUp() {
    //enable module
    // enable dep and other thing for all tests
   * Fake enables a module for the purpose of a unit test
   * @param $name
   *  The module's machine name (i.e. ctools not Chaos Tools)
  protected function enableModule($name) {
    $modules = module_list();
    $modules[$name] = $name;
    module_list(TRUE, FALSE, FALSE, $modules);

In the code sample, we simply create a base class that extends DrupalUnitTestCase, and add a setup method, to take care of anything that needs to be done before all of our tests. We also include an enableModule method that fakes enabling a module (the DrupalUnitTestCase, does not have access to a database, so enabling a module the normal way is not available).

Other things we can add to the base case are new methods to support our tests (such as asserts) that one might want to use over and over. For example in SPS module we add an assertThrows, which tests to ensure the correct exception is thrown (note this assert only works in PHP 5.3)

abstract class MODULEBaseUnitTest extends DrupalUnitTestCase {
   * One can also add helper assert functions that might get used in tests
   * This one test if the correct Exceptions is thrown (5.3 only)
  protected function assertThrows(Closure $closure, $type, $error_message = NULL, $message) {
    try {
    catch (Exception $e) {
      if (!($e instanceof $type)) {
        throw $e;
      if (isset($error_message)) {
        if ($e->getMessage() != $error_message) {
          $this->fail($message, "SPS");
      $this->pass($message, "SPS");
    $this->fail($message, "SPS");
   * One can also add helper assert functions that might get used in tests
   * Test that an object is an instance of a class
   * @param $class
   * @param $object
   * @param $message
  protected function assertIsInstance($class, $object, $message) {
    if ($object instanceof $class) {
      $this->pass($message, "SPS");
    else {
      $this->fail($message, "SPS");

After we have set up our base class, now we can start making our tests.

Step 3 - Write Tests

OK, now we get to write tests! The structure here gets a little bit confusing, one can have as many test classes as one wants, and each test class can have as many test methods, and each test method can have many assertions.

In the SPS module we did a test class for each class provided by the SPS module, with a test method for each method provided by the class and then a test for each method. We found this to be an effective way to structure the test, but there is no required structure.

I also used one file for each test class, each of which extended the base class defined earlier. Each test class should define a getInfo method, to tell us about the test.

The getInfo method returns a array with keys of name, description, and group (I use the module name for this).

class MODULETestnameUnitTest extends MODULEBaseUnitTest {
  static function getInfo() {
    return array(
      'name' => 'MODULE Testname ',
      'description' => 'Test the public interface to the Testname of the MODULE module',
      'group' => 'MODULE',

Now for the test methods. Each method that starts with the word 'test' will be run as a test. Each test should include one or more asserts. One can look at the Drupal Unit Test methods to see all of the asserts. AssertTrue and AssertEqual will be the most used asserts.

class MODULETestnameUnitTest extends MODULEBaseUnitTest {
   * All test start with test in lowercase letters
   * One can use any of the asserts in this space
  public function testContruction() {
    $value = $this->getCondition('baseball');
      'Description of what is being checked in this assertion'
    $expect = new ModuleCondition('baseball', 'other');
      'Description of what is being checked in this assertion'

Other methods (those that do not start with 'test') can be used in the test for doing tasks that might be needed by multiple test methods.

class MODULETestnameUnitTest extends MODULEBaseUnitTest {
   * Methods that do not start with test, are ignored as test, but can be used for function
   * that might be reused on multiple test.
  protected function getCondition($name) {
    return MODULE_get_condition($name, TRUE);

Running Tests

The last item is to run these tests; that can be done by using the /scripts/run-tests.sh script (this is off of the drupal root). One must have the simpletest module enable to run these tests.

To run only the tests in a specific group (remember I set it to the name of the module earlier) pass the group name as the first argument to the run-tests.sh script.

 php scripts/run-tests.sh GROUPNAME
While one is developing one's tests, or if obe has to drill down on a failing test, one can use the verbose and class flags. This will give a more verbose report on only one class' tests.
 php scripts/run-tests.sh --verbose --class CLASSNAME

One can also use the Testing admin area for running the test (they are grouped by group name), but I find that the script is much more conducive if one is doing Test Driven Development.

When Erik Summerfield joined our software development team, we knew that his natural talents in math and economics would be an asset to our team and clients alike. Plus, his experiences in various programming languages including .NET/C#, perl, ...

Sep 05 2012
Sep 05

In this video we will walk through the process of testing existing patches in the Drupal.org issue queue. We will read an issue, download and apply the patch with Git, and then test it to see if it works. This video follows the instructions on the Test patches lesson on learndrupal.org.

The lines that I have in my .bash_profile (shown at the end of this video) to add the Git branch to my command line prompt is:

function parse_git_branch {
git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
export PS1="\h:\W \u\$(parse_git_branch)\$ "

I'm using a Mac, so your mileage may vary on other systems.

May 09 2012
May 09

Posted May 9, 2012 // 0 comments

As a programmer, one of the things that I enjoy most is solving problems by creating new solutions. While I enjoy solving complex larger issues, I also love fixing an existing problem in the form of a bug. Bugs usually have a distinct problem that needs to be solved. When I am working on a bug fix, my mind usually cannot rest until I have come up with a sufficient solution. Once I have come up with a solution for a bug, there is a personal satisfaction that comes knowing the bug is resolved.

One of the most difficult things in fixing bugs is insuring that I have all the details to reproduce a particular bug. Being able to reproduce an issue is important in bug fixing. This provides a developer or programmer a test case to assist with debugging and testing the final solution. As a developer on many projects, I spend a considerable amount of time communicating with bug reports to insure I have all the necessary information to reproduce the bug. Often times, a bug report contains some information to get started in working on a solution, but not all of the information needed. The less time I spend deciphering the issue means I will spend less time fixing the bug and providing a solution.

This post will highlight some best practices and tips for filing a bug report.

Anatomy of a bug report

A bug report usually contains a set of fields for tracking specific information, a description field for describing the task, the ability to move the bug through different states, and provide additional comments as a solution is being worked out. At a minimum, most bug tracking application include the following fields:

  • Priority
  • Status
  • Assigned to
  • Description
  • Reporter

Along with tracking these fields some basics are usually captured as well such as date and time. These fields insure all the necessary information can be referenced when addressing an issue. The status field is important because it is used to communicate the stage or state that a bug is in. A bug typically goes through the following stages:

  • Open
  • Assigned
  • Fixed
  • Verified
  • Closed

Guidelines for reporting bugs

The most important part of a bug report is to provide the steps necessary to reproduce the issue. If you include all the steps and the details associated with reproducing the bug, then all of the important information should be captured. Screenshots and URL's are also important so that the developer can quickly locate and try to reproduce the problem. When writing a bug report, try to put yourself in the developers shoes and ask "What would I want to know to be able to investigate and fix this problem?". Here are some key items to think about and include with your bug report:

  • Title: The title should be concise, clear, and informative
  • Do not use vague language such as "crashed" or "failed" instead include an error message, or exactly what happened
  • Separate issues as best you can into individual bug reports. It is often tempting to clump several bugs into one bug report. This can be confusing as they all will have different test cases and some parts of the bug may get resolved before others.
  • Include details steps on how to reproduce the bug including the URL. If you are logged into a system include the username and/or role that you are logged in as.
  • Include browser version and operating system
  • Be specific and verbose in documenting your bug entry. The more detail that is included in the report will expedite the process of solving the bug.
  • Annotated screen shots are extremely helpful. If possible, include the location bar that shows the URL in the screenshot

Questions to ask?

There are a number of questions you should answer in your bug report.

  • Can you reproduce this bug?
  • Can you reproduce this bug in another environment or on another machine?
  • Does the bug occur in only one browser or in multiple browsers?
  • Has this ever worked before?
  • Does the bug report contain enough information for someone to reproduce the bug?


Writing a detailed bug report will expedite the bug solving process. Having a clear and concise title will insure that bug reviewers can have an idea at a glance on the issue. Separating out issues into individual bug reports will allow each issue to maintain its own status and be resolved independently of each issue. Once you have written your bug report, you should review it one last time to insure that you have included all the necessary information. It may take a little longer to file the bug report initially, but I guarantee you it will reduce the amount of back and forth communication required to solve the bug.

As one of our esteemed Web Developers, Tracy Smith affords us his vast experience in programming, database design, and project management. He has been developing web applications since 1999 and has used various languages and technologies ...

May 01 2012
May 01

Devopsdays Mountainview sold out in a short 3 hours .. but there's other events that will breath devops this summer.
DrupalCon in Munich will be one of them ..

Some of you might have noticed that I`m cochairing the devops track for DrupalCon Munich,
The CFP is open till the 11th of this month and we are still actively looking for speakers.

We're trying to bridge the gap between drupal developers and the people that put their code to production, at scale.
But also enhancing the knowledge of infrastructure components Drupal developers depend on.

We're looking for talks both on culture (both success stories and failure) , automation,
specifically looking for people talking about drupal deployments , eg using tools like Capistrano, Chef, Puppet,
We want to hear where Continuous Integration fits in your deployment , do you do Continuous Delivery of a drupal environment.
And how do you test ... yes we like to hear a lot about testing , performance tests, security tests, application tests and so on.
... Or have you solved the content vs code vs config deployment problem yet ?

How are you measuring and monitoring these deployments and adding metrics to them so you can get good visibility on both
system and user actions of your platform. Have you build fancy dashboards showing your whole organisation the current state of your deployment ?

We're also looking for people talking about introducing different data backends, nosql, scaling different search backends , building your own cdn using smart filesystem setups.
Or making smart use of existing backends, such as tuning and scaling MySQL, memcached and others.

So lets make it clear to the community that drupal people do care about their code after they committed it in source control !

Please submit your talks here

Nov 08 2011
Nov 08

Whether we like it or not, testing takes a huge amount of time. Often a third of each web development project is spent testing, but it's not usually something that gets a lot of attention.

SimpleTest and Selenium testing are now relatively common topics of discussion in the Drupal community, but the actual testing process is less widely discussed. Figuring out what to test for, how to track the results of testing, and techniques for testing Drupal projects are difficult problems to solve.

Whatever testing or project management methodologies are used, improving the testing process can save valuable development hours and increase the chances of finishing projects on schedule. There are lots of challenges particular to testing Drupal projects, and at DrupalCamp New Hampshire in October, I got a chance to share some of this at a session about Improving Your Drupal Testing Process.

Benefits of Testing

Besides the fact that testing a website will find bugs that need to get fixed, testing can have other positive outcomes. Testing is a great opportunity to write technical documentation and documentation for site administrators. Bug reports can lead to documentation of known issues and help unearth unforeseen requirements.

Challenges of Testing Drupal Websites

Testing Content

Most Drupal sites are full of content, and testing a site with real-world content is one of the main tasks we're faced with when testing Drupal projects. Variations in content that can cause issues to appear include:

  • Quantity of content
  • Length of individual pieces of content
  • Interlinking between content
  • Different types of media
  • Language
  • Formatting

If a website includes a content import component, doing this at the beginning of a project is a great way to ensure that real content is tested as the site's functionality is built. For user-generated content, creating a suite of realistic test content that includes a variety of what users might enter is a great investment of time at the beginning of the project.

Testing Configuration

In Drupal, configuration is functionality and the configuration settings that you're delivering need to be tested thoroughly.

Since contributed modules often introduce functionality through the admin UI that is not relevant to the project at hand, documenting which functionality has been tested is important. A good way to clarify this is to use Features as a form of documentation, including certain confiuration settings in 'configuration' features which only developers can update. This can clarify what configuration is intentional and key to the functionality of the website.

Things that are Assumed to Work

When you're writing requirements for a Drupal project, a lot of things get glossed over or left over because it's assumed that Drupal core will handle them, or that they will 'just work'. It's important to remember these during the testing process:

  • Core Drupal Features
  • Security
  • Performance
  • Accessibility
  • Browser Compatibility
  • Integration with APIs
  • Import Scripts
  • Admin UI
  • Admin Workflows
  • Mail Servers

Using an Issue Tracker to Improve Testing

My presentation included a series of tips for improving bug reporting by taking advantage of issue tracking features. Over the past few months at Evolving Web, we've been working to improve how we use Redmine to manage projects and track issues, and this plays a big role in the testing process.

Since Redmine is an open source tool, we've written some custom plugins to extend what it does and customize it to fit our needs. You can read more about our Redmine Google Docs Plugin.

Regardless of which issue tracker you use, you can improve your issue tracking process by making sure that testing results in better bug reports:

  • Review issues before they're closed or resolved
  • Assign issues to the right person
  • Link to related issues
  • Don't make duplicate issues
  • Categorize and prioritize issues
  • Integrate specifications with the issue queue
  • Use pictures (i.e. screenshots)
  • Add lots of links (to the issue, to the config settings for the issue, to related Drupal.org issues etc.)
  • Include 'Steps to reproduce', or even a script to reproduce

Another key to making effective use of your issue tracker is making sure that there's an issue for everything. This means that these changes are documented and, hopefully, tested over the course of the development process.

  • Configuration changes
  • Adding a content type
  • Adding a user role
  • Installing a module
  • Installing a new language
  • Importing content
  • Writing custom code

Methods of Testing

Different situations call for different methods of testing. Some features, like validation or e-commerce transactions, are critical and very specific. These are well-suited to specific, scripted tests that can be repeated and tested frequently. At the other end of the spectrum, usability and design features are very difficult to test in a scripted fashion. Exploratory testing, in which tests can be adjusted on the fly by the tester, are better suited for these types of features.

Automated Testing

One of the major benefits of scripted tests is that they can be automated. Tests that are automated can be run frequently. Automation can also serve as documentation in the issue queue. Using a tool like Selenium IDE, non-technical testers can record the 'steps to reproduce' when creating a bug report.

What is Selenium IDE?

Selenium IDE is a Firefox add-on that makes it easy to record tests. It records your actions in the browser and then replays them when you run each test. Tests consist of a set of command, which you can edit. You can also 'verify' that an element appears on the page in a particular place. A simple example of a Selenium test for Drupal would be filling in the fields for a particular content type and then verifying that they appear in the correct place on the node page. The Software Testing Club has a great intro to Selenium series if you're looking for help getting started.

Sharing Tests

Automated tests can be hard to maintain, since they need to be updated as the functionality of the site changes. Sharing tests through version control or on a testing server can help reduce efforts in writing automated tests. Selenium tests include a 'base URL', which lets you easily change the base URL of tests so they can be run on different environments.

Managing Test Content

One of the first things you'll notice when running Selenium tests is how quickly you can create a lot of test content. Writing tests that create, verify, and then remove content is a good way to manage this. It will also avoid overlapping namespaces if you're testing items that have to be unique, like adding custom fields to a content type or creating users.

Test Suites

Besides writing individual tests, Selenium allows you to create test suites which group tests together and allows you to run them sequentially. This is really for a set of tests for a particular user role. Rather than including a login step at the beginning of each test, this means you can login once before the tests are run.

Beyond the Basics

Selenium also has a Web Driver which can run tests server-side and allow you to test in other browsers. This is useful if you want to extend what Selenium can do, run your tests automatically, or write tests by hand.

We've come a long way since the early days of tracking issues on a white board, but our quest to turn Redmine into the ideal issue tracker is far from over. We're also still figuring out the best way to integrate unit tests and automated tests into our development workflow. As Drupal projects become more complex, testing becomes more important to each project's success. We hope to share more about the evolution of our testing process in the near future, so stay tuned!

Mar 02 2010
Mar 02

Drucumber is a new module that converts native-language like text into a Drupal Simpletests.

The idea is not entirely new, other projects like Cucumber have worked to solve this problem, but none of them does so for Drupal (at the time being). Drucumber also doesn't use Cucumber's syntaxis, it uses YAML instead. YAML has a lot of existing tools for processing and has the structural properties of XML while being more accessible for end users. Using YAML makes it also possible to later implement a backend for RDF or RDFa which will open some really exciting possiblities (integration with the semantic markup editor, for instance - remember Spezzle?).

The goal of this module is to allow end-users and project teams to implement behavior driven development: specify behaviors or features of a Drupal site in a test and never again worry about checking it. Here's an example:

After processing the mighty black box of my module delivers the following:

The indenting is a bit messy, but you can see that this is an actual test.

Currently, the module only works with drush (same way as a regular compiler), but the node support and services integration is on it's way. As mentioned before, possible implementations for RDF integration are also being evaluated. The API is ready for further implementations, and there is plenty of documentation in docs.php.

Dec 01 2009
Dec 01

Here at the Worx, we often build Drupal websites as intranets. These have on many occasions, become full office automation centers and allow for faster work and better communications. But, they are also VERY custom and require a large degree of testing by the customer as we move towards production.

There are a variety of 'tests' that an application must go through before it becomes production. Most of this is done behind the scenes by the developers as they work their way through the development of your new application.

There are also a variety of 'tests' that only the client can perform. If the website being developed is more of an 'application' then just content for general consumption, the client responsibility goes up dramatically.  Like any other application that is written, this needs extensive testing. This usually includes at least beta testing and often, parallel testing. In many cases, this is done one section at a time.

Beta testing really means that you are aware that errors WILL occur. This is the 'dry run' time where you try every function of the application multiple times. Does it do what is expected? Is it consistent? Is the user interface acceptable? All of this testing should be done with a notebook by your side and extensive notes taken to feed back to the developers any problems you may encounter.

Parallel testing is used whenever you are replacing an older application with something new. In this instance you will do all your processes TWICE. Once in each system. The basic question here is simple. Do you get the same results out of both systems? If not, is that what you expected? Part of the reason for the replacement might be improvement and correction. But in most of the application, you should get the same results or be able to explain why based on new functionality.

Any time an application is being replaced, it is absolutely imperative that a parallel test occur. To go directly into production with an untested application is suicide. If you take a new application and begin using it for production WITHOUT parallel, then you simply failed to test and will get problems. Usually lots of problems. 

So, in summary… If you have a new application with new functionality, you will go through a Beta phase. If you are replacing an old application with a new application, then you will go through Beta AND Parallel. Don't skip the Parallel or you may never know what the problems are in the new application until it is much too late!

To truly enjoy your new application, remember that testing is a very important part of the client responsibility.

Apr 10 2009
Apr 10

Is writing Drupal tests boring? Maybe. Is running tests boring? Not anymore!

There is a patch for Drupal 7 pending that introduces a couple of new hooks into the simpletest module. These hooks will allow you to listen to the progress of a test run and react to tests that pass or fail. I have already created a new testlistener module that uses these new hooks to execute shell commands.

Some examples:

I have a Dell XPS laptop with built-in LEDs that can be controlled using the dellLEDCtl shell command (Linux). Using these new hooks, I can turn the LEDs green at the start of a test run, make them red when a test fails and turn them off when the test run has completed. This is very similar to the functionality of my Eclipse XPS plugin that shows JUnit results in Eclipse.

Another (typical) example is to have two lava lamps, a red one and a green one. You connect both of them to an X10 device (I have an X10 starterkit from IntelliHome) so you can turn them on/off with the heyu shell command (Linux). The testlistener module allows you to let the lamps reflect the status of the Drupal code. Other people are using this setup in their (non-Drupal) continuous integration environment.

If lamps are too soft for you, you might consider shooting missiles instead.

Drupal testing will never be boring again ;)

Nov 13 2008
Nov 13
By Leslie Hawthorn, Open Source Team

You may recall that Charlie Gordon and Jimmy Berry were named two of Drupal's runner ups for the Google Highly Open Participation Contest (GHOP). What you may not know is that both of these gentleman are highly skilled testing aficionados, both of them continuing to improve Drupal long after the contest has ended. When we heard that our Drupal friends were organizing a Testing Sprint in Paris, we were excited to hear that years of desire for better testing had gelled into plans for a weekend of concerted community effort in this area.

We were even more excited when we heard that Charlie and Jimmy were counted among the key participants for the sprint, and we were happy we could assist them with their attendance at the sprint. While we wish we could have joined them in the City of Light, the most exciting part of all is hearing from Charlie and Jimmy about all the great things they managed to accomplish in just two days, in addition to all of the great work they've been doing for Drupal since GHOP. Both of them were kind enough to send us along updates on all things Drupal, testing and sprinting.

Charlie writes:

A few weeks ago, Google sponsored a trip for me and Jimmy Berry (18) to Drupal's Code Sprint in Paris. It is safe to say that without Google's sponsorship, I would not have been going. This opportunity was incredible, both for myself, Jimmy, and the Drupal community as a whole. Drupal is a Content Management Platform which has recently decided to make the switch to test driven development. Once all of us were there in Paris, we did our best to make this dream a reality. The results were superb— Drupal now has an automated testing system in its core, and we have functional tests written for nearly all of the Drupal core. This is truly an amazing achievement that Drupal has been striving for for nearly three years, but this sprint has made it into a reality. It is truly awesome to be involved in such a vibrant open source community with so much support. I'm only fifteen years old, but my involvement in open source has led me to go places and do things I would never have dreamed I could do. If someone had told me a year and a half ago, before I started working on open source, that in less than two years I would be sponsored by Google to go to Paris to help improve an automated testing framework in order to get it into the core a content management system called Drupal, I wouldn't have thought it possible.

Drupal Testing Sprinters
(clockwise from top left: Jimmy Berry, Dries Buytaert, Charlie Gordon, former Google Summer of Code™ student Rok Zlender, Douglas Hubler and Miglius Alaburda)

Jimmy shares these thoughts on graduation from GHOP, life after the contest, and moving on to become a Google Summer of Code student:

Since the end of GHOP I have become increasingly involved in the Drupal project. After much encouragement and generous donations I was able to attend Drupalcon Boston 2008. At the conference I spoke in two sessions, one about GHOP and the other regarding automated testing of Drupal, an area where I'd become particuarly involved. My contributions from GHOP had placed me at the forefront of the push to improve automated testing of Drupal, and it was quite rewarding to see Dries deliver his State of Drupal keynote in which he detailed plans to have 100% test coverage of Drupal 7, especially since his talk occurred right after our testing session.

The newly increased priority and accelerated timeline meant that much work would need to be done in the next several months. Upon returning home I got started and was soon given the opportunity to become the maintainer of the SimpleTest module, the focus of automated testing development. I was excited by the opportunity to play an influential role in the development of Drupal 7 and accepted heartily. Given the authority to make code changes and incorporate contributed patches, I accelerated the rate of development. SimpleTest soon saw an enormous amount of change and inched ever closer to inclusion in Drupal core.

The proposed Paris testing sprint started to become a reality and I began raising funds and further preparing SimpleTest for the sprint. Work continued furiously over the next several weeks. Thanks to generous donations from individuals and Google I received the necessary funding to attend and the sprint took place in Paris, France from April 19th to 21st. We worked for two days before resolving the outstanding issues that blocked SimpleTest's entrance into the core, but our efforts paid off once SimpleTest was committed. It was a great experience to work with a group of dedicated developers to accomplish goals.

I plan to continue my contributions to Drupal through the Google Summer of Code. My Usability Testing Suite was accepted as a project for the 2008 summer. I am looking forward to working on the project and seeing it put to good use. None of this would have happened without the GHOP initiative and all those involved, so my thanks to all of you.

Many thanks to Charlie and Jimmy for sharing their experiences. As we like to say here at Google, debugging sucks, but testing rocks! We're glad we could help Charlie, Jimmy and the whole Drupal community rock even more than they already do.

Apr 02 2008
Apr 02

ImageMagick support has been a release blocker on this module for me. My attempts at using the Imagick extension failed, and I finally broke down and wrote the imageapi_imagemagick module, and it needs testing.

If you have a working imagecache 2.x installation or use imagecache 1.x with imagemagick I would appreciate some testing. I suggest using imagecache HEAD with the imageAPI 1.0 RC 2.

Once I get stable 2.x releases of my image* modules out the door I can start on 6.x ports.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web