Jun 18 2017
Jun 18
Recently I set out to make a simple instrument for running simpletest tests without having LAMP stack installed on your local environment. I needed this for two reasons:
  1. for running tests locally
  2. for running tests on CI server
I've decided to use Docker and create monolith container with Drupal and all the LAMP stuff inside and here what I've got: docker-tester.


How to use


Before running container you have to setup next ennvironment variables:
  1. KEEP_RUNNING - specify  yes  if you want to keep container running when tests will be executed. Use for debugging purposes only. Default value is  no .
  2. DRUPAL_VERSION - specific version of Drupal. Supported Drupal 7 and Drupal 8. Example:  8.3.2 .
  3. MODULES_DOWNLOAD - a list of modules to download (by Drush) separated by comma. Example:  module_name-module_version,[...] .
  4. MODULES_ENABLE a list of modules to enable (by Drush) separated by comma. Example:  module_name,[...] .
  5. SIMPLETEST_GROUPS - a list of simpletest groups to run separated by comma. Example:  Group 1,[...] .
  6. SIMPLETEST_CONCURRENCY - amount of test runners to test code in parallel. Default value is  1 .
Then you need to build an image for container:
docker build -t drupal-tester .
Next you have two options: either run container with docker-compose tool or run container manualy with docker command.
For local usage I prefere to use docker-compose because it's easier than write all the CLI docker command manualy. Just specify what module you want to test inside of docker-compose.yml file and run:
docker-compose up && docker-compose down
It will run the container, install Drupal inside of it and run tests. That's all.

For running tests on CI server I use docker command and specify all the needed environment variables manualy:

docker run -v $(pwd)/test_results:/var/www/html/test_results -v $(pwd)/custom_scripts:/var/www/html/custom_scripts -e KEEP_RUNNING=no -e DRUPAL_VERSION=8.3.2 -e MODULES_DOWNLOAD=module-version -e MODULES_ENABLE=module -e SIMPLETEST_GROUPS=module_test_group -e SIMPLETEST_CONCURRENCY=1 drupal-tester
It allows you to override environment variables and volumes that you want to mount inside of the container. So you can setup different jobs on your CI server to test different modules on different Drupal versions with the help of this one container.

When docker finished the process all test results by default will be placed into test_results directory but you can easily override this by mounting some other directory inside of a container.

Setup and customization


Sometimes you need to do something before running tests. For example override some module specific settings or setup some Drupal variables etc. You can get it done with custom *.sh scripts. Just write sh file with all needed actions/commands and put it inside custom_scripts folder. All the files inside of this directory will be executed before running tests.
May 01 2015
May 01

In this article we are going to look at automated testing in Drupal 8. More specifically, we are going to write a few integration tests for some of the business logic we wrote in the previous Sitepoint articles on Drupal 8 module development. You can find the latest version of that code in this repository along with the tests we write today.

Drupal 8 logo

But before doing that, we will talk a bit about what kinds of tests we can write in Drupal 8 and how they actually work.

Simpletest (Testing)

Simpletest is the Drupal specific testing framework. For Drupal 6 it was a contributed module but since Drupal 7 it has been part of the core package. Simpletest is now an integral part of Drupal core development, allowing for safe API modifications due to an extensive codebase test coverage.

Right off the bat I will mention the authoritative documentation page for Drupal testing with Simpletest. There you can find a hub of information related to how Simpletest works, how you can write tests for it, what API methods you can use, etc.

By default, the Simpletest module that comes with Drupal core is not enabled so we will have to do that ourselves if we want to run tests. It can be found on the Extend page named as Testing.

Once that is done, we can head to admin/config/development/testing and see all the tests currently available for the site. These include both core and contrib module tests. At the very bottom, there is also the Clean environment button that we can use if any of our tests quit unexpectedly and there are some remaining test tables in your database.

How does Simpletest work?

When we run a test written for Simpletest, the latter uses the existing codebase and instructions found in the test to create a separate Drupal environment in which the test can run. This means adding additional tables to the database (prefixed by simpletest_) and test data that are used to replicate the site instance.

Depending on the type of test we are running and what it contains, the nature of this replication can differ. In other words, the environment can have different data and core functionality depending on the test itself.

What kinds of tests are there in Drupal 8?

There are two main types of tests that we can write for Drupal 8: unit tests using PHPUnit (which is in core now) and functional tests (using Simpletest). However, the latter can also be split into two different kinds: web tests (which require web output) and kernel tests (which do not require web output). In this article we will practically cover only web tests because most of the functionality we wrote in the previous articles is manifested through output so that’s how we need to test it as well.

Writing any type of test starts by implementing a specific class and placing it inside the src/Tests folder of the module it tests. I also encourage you to read this documentation page that contains some more information on this topic as I do not want to duplicate it here.

Our tests

As I mentioned, in this article we will focus on providing test coverage for some of the business logic we created in the series on Drupal 8 module development. Although there is nothing complicated happening there, the demo module we built offers a good example for starting out our testing process as well. So let’s get started by first determining what we will test.

By looking at the demo module, we can delineate the following aspects we can test:

That’s pretty much it. The custom menu link we defined inside the demo.links.menu.yml could also be tested but that should already work out of the box so I prefer not to.

For the sake of brevity and the fact that we don’t have too much we need to test, I will include all of our testing methods into one single class. However, you should probably group yours into multiple classes depending on what they are actually responsible for.

Inside a file called DemoTest.php located in the src/Tests/ folder, we can start by adding the following:

<?php

namespace Drupal\demo\Tests;

use Drupal\simpletest\WebTestBase;

/**
 * Tests the Drupal 8 demo module functionality
 *
 * @group demo
 */
class DemoTest extends WebTestBase {

  /**
   * Modules to install.
   *
   * @var array
   */
  public static $modules = array('demo', 'node', 'block');

  /**
   * A simple user with 'access content' permission
   */
  private $user;

  /**
   * Perform any initial set up tasks that run before every test method
   */
  public function setUp() {
    parent::setUp();
    $this->user = $this->drupalCreateUser(array('access content'));
  }
}

Here we have a simple test class which for every test it runs, will enable the modules in the $modules property and create a new user stored inside the $user property (by virtue of running the setUp() method).

For our purposes, we need to enable the demo module because that is what we are testing, the block module because we have a custom block plugin we need to test and the node module because our logic uses the access content permission defined by this module. Additionally, the user is created just so we can make sure this permission is respected.

For the three bullet points we identified above, we will now create three test methods. Keep in mind that each needs to start with the prefix test in order for Simpletest to run them automatically.

Testing the page

We can start by testing the custom page callback:

/**
 * Tests that the 'demo/' path returns the right content
 */
public function testCustomPageExists() {
  $this->drupalLogin($this->user);

  $this->drupalGet('demo');
  $this->assertResponse(200);

  $demo_service = \Drupal::service('demo.demo_service');
  $this->assertText(sprintf('Hello %s!', $demo_service->getDemoValue()), 'Correct message is shown.');
}

And here is the code that does it.

First, we log in with the user we created in the setUp() method and then navigate to the demo path. Simpletest handles this navigation using its own internal browser. Next, we assert that the response of the last accessed page is 200. This validates that the page exists. However, this is not enough because we need to make sure the text rendered on the page is the one loaded from our service.

For this, we statically access the \Drupal class and load our service. Then we assert that the page outputs the hello message composed of the hardcoded string and the return value of the service’s getDemoValue() method. It’s probably a good idea to write a unit test for whatever logic happens inside the service but for our case this would be quite redundant.

And that’s it with the page related logic. We can go to the testing page on our site, find the newly created DemoTest and run it. If all is well, we should have all green and no fails.

drupal 8 automatated tests

Testing the form

For the form we have another method, albeit more meaty, that tests all the necessary logic:

/**
 * Tests the custom form
 */
public function testCustomFormWorks() {
  $this->drupalLogin($this->user);
  $this->drupalGet('demo/form');
  $this->assertResponse(200);

  $config = $this->config('demo.settings');
  $this->assertFieldByName('email', $config->get('demo.email_address'), 'The field was found with the correct value.');

  $this->drupalPostForm(NULL, array(
    'email' => '[email protected]'
  ), t('Save configuration'));
  $this->assertText('The configuration options have been saved.', 'The form was saved correctly.');

  $this->drupalGet('demo/form');
  $this->assertResponse(200);
  $this->assertFieldByName('email', '[email protected]', 'The field was found with the correct value.');

  $this->drupalPostForm('demo/form', array(
    'email' => '[email protected]'
  ), t('Save configuration'));
  $this->assertText('This is not a .com email address.', 'The form validation correctly failed.');

  $this->drupalGet('demo/form');
  $this->assertResponse(200);
  $this->assertNoFieldByName('email', '[email protected]', 'The field was found with the correct value.');
}

The first step is like before. We go to the form page and assert a successful response. Next, we want to test that the email form element exists and that its default value is the value found inside the default module configuration. For this we use the assertFieldByName() assertion.

Another aspect we need to test is that saving the form with a correct email address does what it is supposed to: save the email to configuration. So we use the drupalPostForm() method on the parent class to submit the form with a correct email and assert that a successful status message is printed on the page as a result. This proves that the form saved successfully but not necessarily that the new email was saved. So we redo the step we did earlier but this time assert that the default value of the email field is the new email address.

Finally, we need to also test that the form doesn’t submit with an incorrect email address. We do so again in two steps: test a form validation failure when submitting the form and that loading the form again will not have the incorrect email as the default value of the email field.

Testing the block

/**
 * Tests the functionality of the Demo block
 */
public function testDemoBlock() {
  $user = $this->drupalCreateUser(array('access content', 'administer blocks'));
  $this->drupalLogin($user);

  $block = array();
  $block['id'] = 'demo_block';
  $block['settings[label]'] = $this->randomMachineName(8);
  $block['theme'] = $this->config('system.theme')->get('default');
  $block['region'] = 'header';
  $edit = array(
    'settings[label]' => $block['settings[label]'],
    'id' => $block['id'],
    'region' => $block['region']
  );
  $this->drupalPostForm('admin/structure/block/add/' . $block['id'] . '/' . $block['theme'], $edit, t('Save block'));
  $this->assertText(t('The block configuration has been saved.'), 'Demo block created.');

  $this->drupalGet('');
  $this->assertText('Hello to no one', 'Default text is printed by the block.');

  $edit = array('settings[demo_block_settings]' => 'Test name');
  $this->drupalPostForm('admin/structure/block/manage/' . $block['id'], $edit, t('Save block'));
  $this->assertText(t('The block configuration has been saved.'), 'Demo block saved.');

  $this->drupalGet('');
  $this->assertText('Hello Test name!', 'Configured text is printed by the block.');
}

For this test we need another user that also has the permission to administer blocks. Then we create a new instance of our custom demo_block with no value inside the Who field and assert that a successful confirmation message is printed as a result. Next, we navigate to the front page and assert that our block shows up and displays the correct text: Hello to no one.

Lastly, we edit the block and specify a Test name inside the Who field and assert that saving the block configuration resulted in the presence of a successful confirmation message. And we close off by navigating back to the home page to assert that the block renders the correct text.

Conclusion

In this article, we’ve seen how simple it is to write some basic integration tests for our Drupal 8 business logic. It involves creating one or multiple class files which simply make use of a large collection of API methods and assertions to test the correct behavior of our code. I strongly recommend you give this a try and start testing your custom code early as possible in order to make it more stable and less prone to being broken later on when changes are made.

Additionally, don’t let yourself get discouraged by the slow process of writing tests. This is mostly only in the beginning until you are used to the APIs and you become as fluent as you are with the actual logic you are testing. I feel it’s important to also mention that this article presented a very high level overview of the testing ecosystem in Drupal 8 as well as kept the tests quite simple. I recommend a more in depth look into the topic going forward.

Dec 14 2011
ben
Dec 14

Unit testing in Drupal using the standard SimpleTest approach has long been one of my pain points with Drupal apps. The main obstacle was setting up a realistic test "sandbox": The SimpleTest module builds a virtual site with a temporary database (within the existing database), from scratch, for every test suite. To accurately test the complex interactions of a real application, you need dozens of modules enabled in the sandbox, and installing all their database schemas takes a long time. If your site's components are exported to Features, the tests gain another level of complexity. You could have the test turn on every module that's enabled on the real site, but then each suite takes 10 minutes to run. And that still isn't enough; you also need settings from the variables table, content types real nodes and users, etc.

So until recently, it came down to the choice: make simple but unrealistic sandboxes that tested minutia but not the big-picture interactions; or build massive sandboxes for each test that made the testing workflow impossible. After weeks of trying to get a SimpleTest environment working on a Drupal 6 application with a lot of custom code, and dozens of hours debugging the tests or the sandbox setups rather than building new functionality, I couldn't justify the time investment, and shelved the whole effort.

Then Moshe Weizman pointed me to his alternate upal project, which aims to bring the PHPUnit testing framework to Drupal, with backwards compatibility for SimpleTest assertions, but not the baggage of SimpleTest's Drupal implementation. Moshe recently introduced upal as a proposed testing framework for Drupal 8, especially for core. Separately, a few weeks ago, I started using upal for a different purpose: as a unit testing framework for custom applications in Drupal 6.

I forked the Github repo, started a backport to D6 (copying from SimpleTest-6 where upal was identical to SimpleTest-7), and fixed some of the holes. More importantly, I'm taking a very different approach to the testing sandbox: I've set up an entirely separate test site, copied wholesale from the dev site (which itself is copied from the production site). This means:

  • I can visually check the test sandbox at any time, because it runs as a virtualhost just like the dev site.
  • All the modules, settings, users, and content are in place for each test, and don't need to be created or torn down.
  • Rebuilding the sandbox is a single operation (with shell scripts to sync MySql, MongoDB, and files, manually triggered in Jenkins)
  • Cleanup of test-created objects occurs (if desired) on a piecemeal basis in tearDown() - drupalCreateNode() (modified) and drupalVariableSet() (added) optionally undo their changes when the test ends.
  • setUp() is not needed for most tests at all.
  • dumpContentsToFile() (added) replicates SimpleTest's ability to save curl'd files, but on a piecemeal basis in the test code.
  • Tests run fast, and accurately reflect the entirety of the site with all its actual interactions.
  • Tests are run by the Jenkins continuous-integration tool and the results are visible in Jenkins using the JUnit xml format.

How to set it up (with Jenkins, aka Hudson)

(Note: the following are not comprehensive instructions, and assume familiarity with shell scripting and an existing installation of Jenkins.)

  1. Install upal from Moshe's repo (D7) or mine (D6). (Some of the details below I added recently, and apply only to the D6 fork.)
  2. Install PHPUnit. The pear approach is easiest.
  3. Upgrade drush: the notes say, "You currently need 'master' branch of drush after 2011.07.21. Drush 4.6 will be OK - http://drupal.org/node/1105514" - this seems to correspond to the HEAD of the 7.x-4.x branch in the Drush repository.
  4. Set up a webroot, database, virtualhost, DNS, etc for your test sandbox, and any scripts you need to build/sync it.
  5. Configure phpunit.xml. Start with upal's readme, then (D6/fork only) add DUMP_DIR (if wanted), and if HTTP authentication to the test site is needed, UPAL_HTTP_USER and UPAL_HTTP_PASS. In my version I've split the DrupalTestCase class to its own file, and renamed drupal_test_case.php to upal.php, so rename the "bootstrap" parameter accordingly. ** (note: the upal notes say it must run at a URL ending in /upal - this is no longer necessary with this approach.)
  6. PHPUnit expects the files to be named .php rather than .test - however if you explicitly call an individual .test file (rather than traversing a directory, the approach I took), it might work. You can also remove the getInfo() functions from your SimpleTests, as they don't do anything anymore.
  7. If Jenkins is on a different server than the test site (as in my case), make sure Jenkins can SSH over.
  8. To use dumpContentsToFile() or the XML results, you'll want a dump directory (set in phpunit.xml), and your test script should wipe the directory before each run, and rsync the files to the build workspace afterwards.
  9. To convert PHPUnit's JUnit output to the format Jenkins understands, you'll need the xUnit plugin for Jenkins. Then point the Jenkins job to read the XML file (after rsync'ing if running remotely). [Note: the last 3 steps have to be done with SimpleTest and Jenkins too.]
  10. Code any wrapper scripts around the above steps as needed.
  11. Run the tests!

Some issues I ran into (which you might also run into)

  1. PHPUnit, unlike SimpleTest, stops a test function after the first failure. This isn't a bug, it's expected behavior, even with --stop-on-failure disabled. I'd prefer it the other way, but that's how it is.
  2. Make sure your test site - like any dev site - does not send any outbound mail to customers, run unnecessary feed imports, or otherwise perform operations not meant for a non-production site.
  3. In my case, Jenkins takes 15 minutes to restart (after installing xUnit for example). I don't know why, but keep an eye on the Jenkins log if it's taking you a while too.
  4. Also in my case, Jenkins runs behind an Apache reverse-proxy; in that case when Jenkins restarts, it's usually necessary to restart Apache, or else it gets stuck thinking the proxy endpoint is down.
  5. I ran into a bug with Jenkins stopping its shell script commands arbitrarily before the end. I worked around it by moving the whole job to a shell script on the Jenkins server (which in turn delegates to a script on the test/dev server).

There is a pending pull request to pull some of the fixes and changes I made back into the original repo. In the pull request I've tried to separate what are merely fixes from what goes with the different test-site approach I've taken, but it's still a tricky merge. Feel free to help there, or make your own fork with a separate test site for D7.

I now have a working test environment with PHPUnit and upal, with all of the tests I wrote months ago working again (minus their enormous setUp() functions), and I've started writing tests for new code going forward. Success!

(If you are looking for a professional implementation of any of the above, please contact me.)

Dec 13 2011
Dec 13

I have been meaning to make this post for quite some time. I have posted pieces before and had many discussions, but I have yet to write it out formally. Moshe's recent post about Upal motivated me to finally write the post.

Background

The SimpleTest code for mimicking browser behavior, specifically the form handling, has required a fair amount of upkeep, improvement, and generally wasted resources that could be better applied elsewhere. I attempted to clean things up with the external browser component, but that effort ended up dying. Overtime I became more and more convinced that we should revisit the basis for our testing system and try to rebuild things on an existing framework.

You may be wondering why we ever decided to create our own framework in the first place, which is indeed a good question. Back before we had a testing framework in core the concern with adding the SimpleTest.org library to core was its size. At the time SimpleTest was not ready to depend on PHP 5 and specifically SimpleXML. It soon became clear that we could develop our own internal browser using a combination of PHP 5 tools that ended up being MUCH smaller then SimpleTest.org's implementation. This process was led by the poor assumption that we should commit the testing framework to Drupal core. In hindsight that was probably not the best idea.

Rather then commit a third-party library to core it should simply be included using a build script (like many other projects) and Drupal has a system tailored to do exactly that, Drush make. We could even use this approach for jQuery instead of committing the entire library into the core repository. Drupal.org already supports invoking Drush make scripts during release building so the general public wouldn't even notice the difference.

In addition to the size problem is the issue of bandwidth, which myself and many others have discussed many times before. Keeping the Drupal testing integration (with testing library) in contrib would allow it to be maintained and more easily developed while removing an unnecessary burden from Drupal core.

Combination of tools

It is great to see Moshe's post and plan for building atop PHPUnit. Using PHPUnit definitely seems like a great start as it provides integration with many other tools, a familiar API, and using drush removes the "two Drupal sites" problem, but PHPUnit doesn't replace the browser component nor provide a JavaScript testing platform. At this point it seems prudent to build the functional tests atop Selenium which allows the tests to be written in PHP while elevating the need for the custom browser component and allowing for JavaScript testing.

I was part of the initial effort to get QUnit into core along with cwgordon7. We had a working version integrated with the test runner, but things were derailed for various reasons which has since spawned the QUnit project on Drupal.org.

Yeah, ideally we use PHPUnit and QUnit for PHP/JS unit testing, respectively, and Selenium for functional testing. Those are each the best tools for the job.

Selenium has also received some love in the form of SimpleTest integration project on Drupal.org. We have the makings of a great testing setup, but we need to put the pieces together.

Battle plan

Moving forward it seems prudent to continue maintaining each of the pieces in their respective locations and outside of core.

  • Selenium project needs to refactor to build on Upal
  • Rebuild DrupalWebTestCase as a compatibility layer on top of Selenium
  • Integrate QUnit project with Upal in a similiar fashion to Selenium
  • Provide PHPUnit test output parser for testbot
  • Provide a drush make script for testing in core or in a central hub repository in contrib
  • Inventory and refactor Drupal 8 tests to use new system while removing duplicity and waste

Lets make this happen!

Dec 23 2010
Dec 23

One of the challenges of testing views or custom Drupal code is generating the right kind of data. Devel can generate hundreds of random nodes, but you might need more custom data than that. For example, on our project, we need to have test users, their content profiles, and nodes that follow a certain node reference structure. By creating a class that extends DrupalWebTestCase and provides convenience functions on top of drupalCreateNode, we can easily create test data as part of our test cases. Copying the code from drupalCreateUser and making our own version that uses roles and content profiles helps us set up the right users, too.

We wanted our tests and changes to use the same database tables used by the web interface, so we overrode the setUp methods to use the actual database. This not only makes the tests faster, it also makes them more useful for the web testing and demos.

Many of our test cases create the data they need. However, some test cases need even more complex structures that are similar from one test to another. Instead of creating and recreating them on each test, I’ve written another test case for populating the data. For example, PopulateTestUsers sets up about 30 users with different characteristics. I can then write other tests that assume PopulateTestUsers has been run and the sample users and nodes are available.

How do we generate the users and nodes without getting tangled in lots of PHP? Here’s a technique I picked up from Stuart Robertson, an IT architect with lots of good ideas. He fills in a spreadsheet with the values he wants test data to have. He then uses other columns to generate PHP that set the individual fields, and another column that generates PHP based on the generated PHP. For example, a formula to set a CCK value might look like this:

=IF(B3<>"",CONCATENATE("'field_widget_doohickey' => array(array('value' => '",B3,"')),"),"")

which turns a value of “foo” in B3 to

'field_widget_doohickey' => array(array('value' => 'foo'))

which is then something you can pass to the node creation function. To figure out the syntax for other node attributes, use var_dump or your favourite debugging tool to look at a node that has the information you want.

You might have the final generation like this:

=CONCATENATE("$this->createWidget('",A3,"',array(",E3,"));")

where createWidget is a function you’ve defined to make things more readable. It would be a wrapper around drupalCreateNode that sets the type and does other things.

This spreadsheet makes it so much easier for us to work with our test data because we can refer to it to find test data matching criteria when designing our tests or trying things out using the web interface. Adding new test items is easy: just fill in the rows, copy the equations, and then copy the generated code and paste it into the test case.

Naming tip: Using a different naming convention makes it easy for me to use our custom-coded Drush testre command (run tests matching a regular expression) to run just the tests that populate data, or just the tests that assume data is there. Likewise, test cases that use SimpleTest’s web client ($this->get, etc.) have a different naming convention so that I can avoid running these slower tests when I just want to do a quick check.

Simpletest is a powerful tool. I’ve used it on every Drupal project I’ve worked on, even when I was the only one writing tests. Combined with a spreadsheet for generating structured test data, it’s a great help for development and demonstration, because we can set up or refresh complicated sets of users and nodes in little time. Well worth investing time to learn and use.

SCHEDULED: 2010-12-23 Thu 08:00

Jun 24 2010
Jun 24

As you may have seen in my previous post I have been getting back into the swing of things after some time away. I have a number of projects that I need to attend to, but as always not enough time to work on them all. Bellow you will find a list of my most important projects that need attention. If you have time available and an interest in helping out I would appreciate any extra hands. A number of the items do not have links since I have yet to spec them out completely, but I will update this post as I fill them in.

  • Parse project .info files: present module list and dependency information - needed in order for contrib testing to work 100% of the time and to move it out of beta so anyone can enable testing of their project.
  • PIFT 2.3 (drupal.org side of qa.drupal.org) - In order to make the most of the qa framework a number of improvements need to be made, mainly options and information display on drupal.org so everyone can take advantage of all the features the system already provides.
  • PIFR 2.3 (qa.drupal.org and clients) - A fair amount of testing of the current development code base, a few more bug fixes, and features are needed to round out the next release.
  • A few remaining cleanup items for SimpleTest in Drupal 7 core.
  • SimpleTest 7.x-2.x - The branch will be used for continued development during Drupal 7 life cycle and already contains a number of improvements over the base Drupal 7 testing system. It was designed so that it can coincide with the Drupal cores framework and allow for tests to use either framework (one framework per module). The switch module, which no longer works (need to look into), allows you to switch between the contrib and core versions with a single click. The switch module also could be used by other projects attempting to replace core modules and can do so with a small amount of abstraction. With the next PIFT/PIFR release I plan to support projects that wish to use the 7.x-2.x API for testing.
  • Code coverage - Reports for all tests (patches or at least core/contrib commits).
  • Refactor/Rewrite SimpleTest for Drupal 8 - I have an extremely detailed roadmap in my head (and some paper notes) and a small code base that needs to be made public.
  • PIFR 3.x - Same as above, lots of plans, but not very much publicly available yet.

Feel free to read through the issues, information, and code and let me know if you need any help getting started. Thanks!

Nov 13 2008
Nov 13
By Leslie Hawthorn, Open Source Team

You may recall that Charlie Gordon and Jimmy Berry were named two of Drupal's runner ups for the Google Highly Open Participation Contest (GHOP). What you may not know is that both of these gentleman are highly skilled testing aficionados, both of them continuing to improve Drupal long after the contest has ended. When we heard that our Drupal friends were organizing a Testing Sprint in Paris, we were excited to hear that years of desire for better testing had gelled into plans for a weekend of concerted community effort in this area.

We were even more excited when we heard that Charlie and Jimmy were counted among the key participants for the sprint, and we were happy we could assist them with their attendance at the sprint. While we wish we could have joined them in the City of Light, the most exciting part of all is hearing from Charlie and Jimmy about all the great things they managed to accomplish in just two days, in addition to all of the great work they've been doing for Drupal since GHOP. Both of them were kind enough to send us along updates on all things Drupal, testing and sprinting.

Charlie writes:


A few weeks ago, Google sponsored a trip for me and Jimmy Berry (18) to Drupal's Code Sprint in Paris. It is safe to say that without Google's sponsorship, I would not have been going. This opportunity was incredible, both for myself, Jimmy, and the Drupal community as a whole. Drupal is a Content Management Platform which has recently decided to make the switch to test driven development. Once all of us were there in Paris, we did our best to make this dream a reality. The results were superb— Drupal now has an automated testing system in its core, and we have functional tests written for nearly all of the Drupal core. This is truly an amazing achievement that Drupal has been striving for for nearly three years, but this sprint has made it into a reality. It is truly awesome to be involved in such a vibrant open source community with so much support. I'm only fifteen years old, but my involvement in open source has led me to go places and do things I would never have dreamed I could do. If someone had told me a year and a half ago, before I started working on open source, that in less than two years I would be sponsored by Google to go to Paris to help improve an automated testing framework in order to get it into the core a content management system called Drupal, I wouldn't have thought it possible.


Drupal Testing Sprinters
(clockwise from top left: Jimmy Berry, Dries Buytaert, Charlie Gordon, former Google Summer of Code™ student Rok Zlender, Douglas Hubler and Miglius Alaburda)

Jimmy shares these thoughts on graduation from GHOP, life after the contest, and moving on to become a Google Summer of Code student:


Since the end of GHOP I have become increasingly involved in the Drupal project. After much encouragement and generous donations I was able to attend Drupalcon Boston 2008. At the conference I spoke in two sessions, one about GHOP and the other regarding automated testing of Drupal, an area where I'd become particuarly involved. My contributions from GHOP had placed me at the forefront of the push to improve automated testing of Drupal, and it was quite rewarding to see Dries deliver his State of Drupal keynote in which he detailed plans to have 100% test coverage of Drupal 7, especially since his talk occurred right after our testing session.

The newly increased priority and accelerated timeline meant that much work would need to be done in the next several months. Upon returning home I got started and was soon given the opportunity to become the maintainer of the SimpleTest module, the focus of automated testing development. I was excited by the opportunity to play an influential role in the development of Drupal 7 and accepted heartily. Given the authority to make code changes and incorporate contributed patches, I accelerated the rate of development. SimpleTest soon saw an enormous amount of change and inched ever closer to inclusion in Drupal core.

The proposed Paris testing sprint started to become a reality and I began raising funds and further preparing SimpleTest for the sprint. Work continued furiously over the next several weeks. Thanks to generous donations from individuals and Google I received the necessary funding to attend and the sprint took place in Paris, France from April 19th to 21st. We worked for two days before resolving the outstanding issues that blocked SimpleTest's entrance into the core, but our efforts paid off once SimpleTest was committed. It was a great experience to work with a group of dedicated developers to accomplish goals.

I plan to continue my contributions to Drupal through the Google Summer of Code. My Usability Testing Suite was accepted as a project for the 2008 summer. I am looking forward to working on the project and seeing it put to good use. None of this would have happened without the GHOP initiative and all those involved, so my thanks to all of you.

Many thanks to Charlie and Jimmy for sharing their experiences. As we like to say here at Google, debugging sucks, but testing rocks! We're glad we could help Charlie, Jimmy and the whole Drupal community rock even more than they already do.

Jul 14 2008
Jul 14

With an automated testing framework in core, Drupal is now far along the road to a practice of Test-driven development. But there's one thing missing: a complete, in-depth, up-to-date tutorial on how to actually write these tests. Although there is the SimpleTest Automator module to help you create these tests through the user interface, it is currently imperfect and under development, so this post will be a tutorial on writing these tests manually.

Testing resources

Here's just a general list of resources we should keep handy while writing our test:

  • Testing API functions - This is a quick cheat sheet of some functions that are very helpful during testing. These include controlling the internal testing browser, creating users with specific permissions, and simulating clicking on links that appear in the page.
  • Available assertions - Assertions are how you determine whether your code is working or not - you assert that it should be working, and let the testing framework handle the rest for you. This is a library on the assertions that are available to use in our testing framework.

Know what you're testing

In this example, I will be testing Drupal's ability to change the site-wide theme. I know how to test this manually - I would go to the admin/build/themes page, and select the radio button of a different default theme, and then make sure the theme had changed when I reload the page. In order to automate this test, I will simply write code to repeat the same actions I would have done manually.

Start writing your test case

Ok, now we get to the code. First of all, every test case will be a class that extends the DrupalWebTestCase class. So we'll start out with this code:

<?php
// $Id$

class ThemeChangingTestCase extends DrupalWebTestCase {

}
?>

Now, we have to tell the testing framework a little bit about our test. We give it three pieces of information - the name of the test, a brief description of the test, and the group of tests this test is a part of. We will return this information in our implementation of getInfo() in our test class:

<?php
// $Id$

class ThemeChangingTestCase extends DrupalWebTestCase {

  /**
   * Implementation of getInfo().
   */
 
function getInfo() {
    return array(
     
'name' => t('Changing the site theme.'),
     
'description' => t('Tests the changing of the site-wide theme through the administration pages.'),
     
'group' => t('Theming'),
    );
  }

}
?>

Now, let's add our test function. The most thorough way to test this would be to cycle through all the themes, setting each in turn as the default, and that's what we'll do. So we'll get this:

<?php
// $Id$

class ThemeChangingTestCase extends DrupalWebTestCase {
 
// We need a list of themes to cycle through.
 
var $themes = array('bluemarine', 'chameleon', 'garland', 'marvin', 'minnelli', 'pushbutton');

  /**
   * Implementation of getInfo().
   */
 
function getInfo() {
    return array(
     
'name' => t('Changing the site theme.'),
     
'description' => t('Tests the changing of both the site-wide theme and the user-specific theme, and makes sure that each continues to work properly.'),
     
'group' => t('Theming'),
    );
  }

  /**
   * This is the function where we will actually do the testing.
   * It will be automatically called, so long as it starts with the lowercase 'test'.
   */
 
function testSitewideThemeChanging() {
    foreach (
$this->themes as $theme) {
     
// We need a test user with permission to change the site-wide theme.
     
$user = $this->drupalCreateUser(array('administer site configuration'));
     
$this->drupalLogin($user);
     
// Now, make a POST request (submit the form) on the admin/build/themes page.
     
$edit = array();
      foreach (
$this->themes as $theme_option) {
       
$edit["status[$theme_option]"] = FALSE;
      }
     
$edit["status[$theme]"] = TRUE;
     
$edit['theme_default'] = $theme;
     
$this->drupalPost('admin/build/themes', $edit, t('Save configuration'));
    }
  }
}
?>

Whoa, that's a lot of code. Let's go through what I've done step by step:

  1. I've declared a class variable that lists the themes that we're going to test.
  2. In my test function, I'm cycling through each theme, in turn setting each one as the default.
  3. I create a user with enough permissions to change the site-wide theme.
  4. After logging this user in, I proceed to make a POST request to the admin/build/themes page, enabling only the theme we're testing, and setting that theme to be the theme default.
  5. I then submit the form by clicking on the 'Save configuration' button.

Now this works very well, but one thing is still missing - how am I sure that the theme has changed? If I were doing this manually I could tell by just looking - but when I'm automating this, I will test for the theme's css files in the page source of the reloaded admin/build/themes page upon submission. To do this, I will use the assertRaw() function, which makes sure that some text is found in the raw HTML of the current page in the internal browser:

<?php
// $Id$

class ThemeChangingTestCase extends DrupalWebTestCase {
 
// We need a list of themes to cycle through.
 
var $themes = array('bluemarine', 'chameleon', 'garland', 'marvin', 'minnelli', 'pushbutton');

  /**
   * Implementation of getInfo().
   */
 
function getInfo() {
    return array(
     
'name' => t('Changing the site theme.'),
     
'description' => t('Tests the changing of both the site-wide theme and the user-specific theme, and makes sure that each continues to work properly.'),
     
'group' => t('Theming'),
    );
  }

  /**
   * This is the function where we will actually do the testing.
   * It will be automatically called, so long as it starts with the lowercase 'test'.
   */
 
function testSitewideThemeChanging() {
    foreach (
$this->themes as $theme) {
     
// We need a test user with permission to change the site-wide theme.
     
$user = $this->drupalCreateUser(array('administer site configuration'));
     
$this->drupalLogin($user);
     
// Now, make a POST request (submit the form) on the admin/build/themes page.
     
$edit = array();
      foreach (
$this->themes as $theme_option) {
       
$edit["status[$theme_option]"] = FALSE;
      }
     
$edit["status[$theme]"] = TRUE;
     
$edit['theme_default'] = $theme;
     
$this->drupalPost('admin/build/themes', $edit, t('Save configuration'));
     
// Make sure we've actually changed themes.
     
$this->assertCSS($theme);
    }
  }

  /**
   * Custom assert method - make sure we actually have the right theme enabled.
   *
   * @param $theme
   *   The theme to check for the css of.
   * @return
   *   None.
   */
 
function assertCSS($theme) {
   
// Minnelli is the only core theme without a style.css file, so we'll use
    // minnelli.css as an indicator instead.
   
$file = $theme == 'minnelli' ? 'minnelli.css' : 'style.css';
   
$this->assertRaw(drupal_get_path('theme', $theme) . "/$file", t("Make sure the @theme theme's css files are properly added.", array('@theme' => $theme)));
  }
}
?>

Note that I've added my own assertCSS() function in the above code. You're perfectly free to add whatever functions you may desire—just keep in mind that if they start with the lowercase 'test', they will be automatically called as a test function!

That concludes the basic tutorial. Read on if you're interested in going beyond the basics! :)

Advanced testing techniques

Here I'll go over a few techniques for better, easier, simpler, and just overall awesomer testing.

  • Using setUp() and/or tearDown() methods.

    Sometimes it can be useful to run some code before and/or after our test methods are finished running. To do this, we can implement setUp() and/or tearDown(), which run before and after the test methods, respectively:

    <?php
    // $Id$

    class ThemeChangingTestCase extends DrupalWebTestCase {

      /**
       * Implementation of setUp().
       */
     
    function setUp() {
       
    // Invoke setUp() in our parent class, to set up the testing environment.
       
    parent::setUp();
       
    // Create and log in our test user.
       
    $user = $this->drupalCreateUser(array('administer site configuration'));
       
    $this->drupalLogin($user);
      }
    }
    ?>

    In the above example, we've created and logged in our test user in our setUp() method. As a result, that user will be logged in for our test methods; this can be considered somewhat cleaner code.

    We can also use setUp() to enable additional modules we may need:

    <?php
    // $Id$

    class ThemeChangingTestCase extends DrupalWebTestCase {

      /**
       * Implementation of setUp().
       */
     
    function setUp() {
       
    // Invoke setUp() in our parent class, to set up the testing environment.
        // We're going to test the theming of the search box and the tracker page,
        // so we need those modules enabled.
       
    parent::setUp('search', 'tracker');
       
    // Create and log in our test user.
       
    $user = $this->drupalCreateUser(array('administer site configuration'));
       
    $this->drupalLogin($user);
      }
    }
    ?>

    Note: Make sure to always call parent::setUp() and parent::tearDown() if you override them! If you don't, the testing framework will either fail to be set up, or fail to be teared down, successfully.

  • Dealing with HTML content using SimpleXML.

    Our assertCSS() function in the basic example is far from ideal. The path to the theme's style.css could appear on the page as plain text (not part of a css link), and the test would still pass.

    To get around this weakness, we can handle the HTML content of the fetched page using SimpleXML. For example:

    <?php
    // $Id$

    class ThemeChangingTestCase extends DrupalWebTestCase {

      /**
       * Custom assert method - make sure we actually have the right theme enabled.
       *
       * @param $theme
       *   The theme to check for the css of.
       * @return
       *   None.
       */
     
    function assertCSS($theme) {
       
    // Minnelli is the only core theme without a style.css file, so we use
        // minnelli.css as an indicator instead.
       
    $file = $theme == 'minnelli' ? 'minnelli.css' : 'style.css';
        if (
    $this->parse()) {
         
    $links = $this->elements->xpath('//link');
          foreach (
    $links as $link) {
            if (
    strpos($link['href'], base_path() . drupal_get_path('theme', $theme) . "/$file") === 0) {
             
    $this->pass(t("Make sure the @theme theme's css files are properly added.", array('@theme' => $theme)));
              return;
            }
          }
         
    $this->fail(t("Make sure the @theme theme's css files are properly added.", array('@theme' => $theme)));
        }
      }
    }
    ?>

    The parse() method must be called in order to populate $this->elements. It is not called automatically on every page load because this would lead to a significant performance drain.

  • Creating an unused base class, and then extending it.

    Sometimes, it would make sense for two test cases to share API functions, or even setUp and tearDown() functions. In order to do this, we'll set up one base test case that extends DrupalWebTestCase, and then create several children test cases that extend our base test case. For example:

    <?php
    // $Id$

    class ThemeChangingTestCase extends DrupalWebTestCase {

      function assertCSS($theme) {
       
    // ...
     
    }
    }

    class SiteWideThemeChangingTestCase extends ThemeChangingTestCase {

      /**
       * Implementation of getInfo().
       */
     
    function getInfo() {
       
    // ...
     
    }
    }

    class UserSpecificThemeChangingTestCase extends ThemeChangingTestCase {

      /**
       * Implementation of getInfo().
       */
     
    function getInfo() {
       
    // ...
     
    }
    }
    ?>

    Note that in order for a test case to not actually be run itself (or even show up on the administration interface), all it has to do is not implement getInfo().

Now you are officially qualified to start writing tests! If you're looking for a place to get started, check out the issue queue. Also, the code coverage reports are a great place to see what needs to be tested in core.

May 10 2008
May 10

Writing simpletests for imagecache_create_url and file_create_url today, I had to change a few of Drupal's persistent variables, namely clean_url and file_downloads. I was kind of frustrated by the need to save the old values, set the values I needed for the test case, run the test, then restore the old variables.

The process itself is a little dangerous in the case you hit runtime fatal errors and the variables never get restored.. Also during the course of the testing all other requests to the same drupal site are affected by this variable change, so there are potential concurrency errors as well.

I was thinking.. wouldn't it be awesome if I could set variables for just the current request... aka just set $conf['variable_name']...

since there have been movements to eliminate globals and I'm just generally scared of them... and I want non-persistant variable_set I though changing variable_set by adding a persistent argument would make testing a little easier and maybe enable some kinds of strange hacker for others...

<?php
function variable_set($name$value$persistent true) {
  global 
$conf;  if ($persistent) {
   
$serialized_value serialize($value);
   
db_query("UPDATE {variable} SET value = '%s' WHERE name = '%s'"$serialized_value$name);
    if (!
db_affected_rows()) {
      @
db_query("INSERT INTO {variable} (name, value) VALUES ('%s', '%s')"$name$serialized_value);
    }
   
cache_clear_all('variables''cache');
  }
 
$conf[$name] = $value;
}
?>

Just an idea since testing seems to be in the air... maybe the problem is already solved elsewhere...

Apr 20 2008
Apr 20


Only hours ago, after over three years of work on the SimpleTest module, Dries committed the SimpleTest module to core. This is an amazing achievement. Drupal is looking at getting 100% core test coverage, both unit and functional. A lot of work went into the SimpleTest module today before it was committed. We (the Drupal SimpleTest Sprinters) set up an SVN repository containing a version of Drupal with SimpleTest there as a module, and opened it up for all of us to commit. It was chaos (in a good way). Without the roadblock of patches, code was rapidly sucked into the SimpleTest module; finally Dries had to give us a SimpleTest "code freeze" so as to actually start to get stuff to work.

Current SimpleTest Status

SimpleTest is in core! What can be left to do? Actually, a lot, but first, let's go over what we've done today:

And then, there are the things that are left to do:

  • (Update: ) Make the SimpleTest module use the Batch API. This will give us that nice little progress bar that shows us our progress, which is one of the few feasible ways of running so much code.
  • (Update: ) Make the SimpleTest module REALLY comply with coding standards. Not only do we have all the old coding standards SimpleTest issues, but we also now have to make our own SimpleTest-Light library coding standard compliant, as well as alter our code to fit the new string concatenation rule.
  • We need to write tests! Our estimated test coverage at the moment is between 30 and 60 percent, unacceptably low levels. Tests are now easy to write, SimpleTests are in core; everyone, answer this call for help! Write SimpleTests for both core and contributed modules! Make test-driven development a reality!

For more and/or updated ways to help out, please visit http://groups.drupal.org/node/10099.

Apr 19 2008
Apr 19

Today we began the SimpleTest code sprint! This is an area of huge importance to Drupal, as one of our weaknesses has proven to be the number of testers we have, particularly how small that number is. Automated testing will provide an easy way to maintain the quality of core— each time a patch is created for core, it will be sent off to a testing server where the patch will be applied to a copy of the Drupal core and all the SimpleTests run on it. If there are failures, the patch would be marked as such and whoever submitted the patch would be forced to hang their head in shame.

What we accomplished today

Today, Dries Buytaert, Rok Žlender, Károly Négyesi, Jimmy Berry, Kevin Bridges, Douglas Hubler, Miglius Alaburda and I got together for 10 hours for the sole purpose of improving automated testing in core. It was great. We bounced ideas off one another, and came up with a great plan for unit testing with mock functions and classes without runkit. We made our test coverage tests work, although they are still running :). We gave a presentation to the people at Drupal Camp Paris on how to write basic functional tests for both core and contributed modules. We talked to Dries about the possibility of getting SimpleTest into core.

Unit testing with mock functions and objects

Here's the difference between functional testing and unit testing: functional testing makes sure the code functions the way you expect it to, e.g. if you load a page, is it the page you expected to get? Most of our current tests our functional tests. However, there is another kind of test that is valuable as well: unit testing. Unit tests know about certain internal or API functions that exist in Drupal, and make sure those functions work as expected.

The problem with efficient unit testing is that there are so many conditions that have to be accounted for, and in many cases, these conditions are hard to achieve. For example, if I were calling cache_get(), I have to account for this line:

$cache_flush = variable_get('cache_flush', 0);

which is used in further if() statements. In order to fully test it, I should call it both when this for all possible values of $cache_flush. In order to do this, I need to override the variable_get function which is used to generate this variable's value. Solutions such as runkit would work, but we want to have tests that all Drupal users can use, not just the small percentage with the ability to set up an environment with runkit.

How can we override functions? Once they are defined in their files, they cannot be undefined. However, that is just the key: don't include the files at all. Only include the specific function you are testing, and everything else can be represented by mock functions, used to return a certain value that can be set within the SimpleTest test case. In essence you're plucking a function out of its nest, making it think it's safe at home, and making sure it performs backflips properly. This awesome idea can be attributed to chx and Jimmy Berry. Basically you'd read all of the Drupal code base without parsing it, have PHP tokenize it, find all the functions (and classes and interfaces), cycle through and define them as:

function $function_name() {
  \$args = func_get_args();
  return simpletest_handle_mock_function($function_name, \$args);
}

This is an amazing code snippet. You basically make every single function in the Drupal code base call one centralized function from which you make all your mock-function settings.

We hope to work on this more on the next few days and hopefully get it into a working condition. This will also, hopefully, be made easier with the addition of the registry.

Drupal SimpleTest test coverage

In his blog, Dries promised that we could have three extra months of development time with complete core test coverage. Well, this raises the question: how do we determine how much of the Drupal core we've actually covered with our tests? We're looking on a way to figure this out.

There are many coverage-testing tools, but nearly all of them are based on the same principle: they look at your code while you run your tests, count the physical number of lines that were hit and not hit, and return that result. A tool we've been looking at called Spike PHPCoverage provides a means of doing this in PHP. Relying on XDebug for the heavy lifting, Spike PHPCoverage is a relatively lightweight utility which generates a nice html report of our current test coverage. A demo report generated by an early attempt at determining code coverage can be found here. (Note: this data is not accurate, but gives an idea of what the test coverage reports will look like.)

We have gotten the test coverage tests to work, but due to the massive amounts of time it takes to run all tests— plus the fact that adding test coverage makes it about twice as slow— and you can see that this is going to take a while not only to run all tests, but to record which lines of code are called and then generate a huge series of html files on our test coverage. But it should be awesome once we're finished, as not only will we know what are status is in regard to overall SimpleTest testing, but also where, exactly, our weaknesses are.

Presentation on Simpletest, and what you can do to help

Our testing sprint was going on parallel to Drupal Camp Paris, and we decided to give a short session on SimpleTest. However, come time for our session, we were nowhere near ready to present on our new unit tests. So we decided that I would do a repeat of my dojo lesson on writing functional SimpleTests for the Talk module. Not only did this go amazingly well, but there were a lot of questions— people seemed genuinely interested, and some even volunteered to help us out tomorrow by actually writing some of the tests we were making easier to write. This is completely and insanely AWESOME. And the best part is, you can help too, even if you're not in Paris! We have a list of tests that need to be written, and now we need people like you to write them! This is your opportunity to push back the dreadful day of the Drupal 7 code freeze. Let's get to it!

Apr 10 2008
Apr 10


Thanks to the amazing Leslie Hawthorn and the awesome Chris DiBona, we can add an additional two sprinters to the list for SimpleTest Sprint, Paris! Both Charlie Gordon (me) and Jimmy Berry (aka boombatower) are now going to be at the testing sprint! Let's all thank Google for this awesome act of kindness towards the Drupal community!

SimpleTest is critical to Drupal, not only for quality assurance, but also to postpone the code freeze so we can add more features to Drupal 7 and make it as awesome as possible.


As this beautiful graph clearly shows, after Drupal has made the switch to complete SimpleTest coverage for core, the quality of Drupal will skyrocket, as we will be able to ensure that each patch committed to core does not break any of the SimpleTests, therefore less bugs and more features! When we have complete core SimpleTest coverage for Drupal, we will be free to explore awesome new features instead of spending precious time on bug hunting and bug fixing.

Let's make that Drupal 7 release a killer! :)

Thanks to Google, Leslie Hawthorn, and Chris DiBona for speeding up this process. We promise to save the world for you!

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web