Feb 09 2015
Feb 09

February 09, 2015

To get the most of this blog post, please read and understand Getting Started with Docker (Servers for Hackers, 2014/03/20). Also, all the steps outlined here have been done on a Vagrant CoreOS virtual machine (VM).

I recently needed a really simple non-production Drupal Docker image on which I could run tests. b7alt/drupal (which you can find by typing docker search drupal, or on GitHub) worked for my needs, except that it did not have the cUrl php library installed, so drush en simpletest -y was throwing an error.

Therefore, I decided to create a new Docker image which is based on b7alt/drupal, but with the php5-curl library installed.

I started by creating a new local directory (on my CoreOS VM), which I called docker-drupal:

mkdir docker-drupal

In that directory, I created Dockerfile which takes b7alt/drupal as its base, and runs apt-get install curl.

FROM b7alt/drupal

RUN apt-get update
RUN apt-get -y install curl

(You can find this code at my GitHub account at alberto56/docker-drupal.)

When you run this you will get:

docker build .
...
Successfully built 55a8c8999520

That hash is a Docker image ID, and your hash might be different. You can run it and see if it works as expected:

docker run -d 55a8c8999520
c9a98bdcab4e027e8571bde71ee92b4380247a44ef9314749ef5680864de2928

In the above, we are telling Docker to create a container based on the image we just created (55a8c8999520). The resulting container hash is displayed (yours might be different). We are using -d so that our containers runs in the background. You can see that the container is actually running by typing:

docker ps
CONTAINER ID        IMAGE               COMMAND...
c9a98bdcab4e        55a8c8999520        "/usr/bin/supervisor...

This tells you that there is a running container (c9a98bdcab4e) based on the image 55a8c8999520. Again, your hases will be different. Let’s log into that container now:

docker exec -it c9a98bdcab4e bash
[email protected]:/#

To make sure that cUrl is successfully installed, I will figure out where Drupal resides on this container, and then try to enable Simpletest. If that works, I will consider my image a success, and exit from my container:

[email protected]:/# find / -name 'index.php'
/srv/drupal/www/index.php
[email protected]:/# cd /srv/drupal/www
[email protected]:/srv/drupal/www# drush en simpletest -y
The following extensions will be enabled: simpletest
Do you really want to continue? (y/n): y
simpletest was enabled successfully.                   [ok]
[email protected]:/srv/drupal/www# exit
exit

Now I know that my 55a8c8999520 image is good for now and for my purposes; I can create an account on Docker.com and push it to my account for later use:

Docker build -t alberto56/docker-drupal .
docker push alberto56/docker-drupal

Anyone can now run this Docker image by simply typing:

docker run alberto56/docker-drupal

One thing I had a hard time getting my head around was having a GitHub project and Docker project, and both are different but linked. The GitHub project is the the recipe for creating an image, whereas the Docker project is the image itself.

One we start thinking of our environments like this (as entities which should be versioned and shared), the risk of differences between environments is greatly reduced. I was used to running simpletests for my projects on an environment which is managed by hand; when I got a strange permissions error on the test environment, I decided to start using Docker and version control to manage the container where tests are run.

Please enable JavaScript to view the comments powered by Disqus.

Feb 06 2015
Feb 06

February 06, 2015

I have been using Simpletest on Drupal 7 for several years, and, used well, it can greatly enhance the quality of your code. I like to practice test-driven development: writing a failing test first, then run it multiple times, each time tweaking the code, until the test passes.

Simpletest works by spawning a completely new Drupal site (ignoring your current database), running tests, and destroying the database. Sometimes, a test will fail and you’re not quite sure why. Here are two tips to help you debug why your tests are failing:

Tip #1: debug()

The Drupal debug() function can be placed anywhere in your test or your source code, and the result will appear on the test results page in the GUI.

For example, if when you are playing around with the dev version of your site, things work fine, but in the test, a specific node contains invalid data, you can add this line anywhere in your test or source code which is being called during your test:

...
debug($node);
...

This will provide formatted output of your $node variable, alongside your test results.

Tip #2: die()

Sometimes the temporary test environment’s behaviour seems to make no sense. And it can be frustrating to not be able to simply log into it and play around with it, because it is destroyed after the test is over.

To understand this technique, here is quick primer on how Simpletest works:

  • In Drupal 7, running a test requires a host site and database. This is basically an installed Drupal site with Simpletest enabled, and your module somewhere in the modules directory (the module you are testing does not have to be enabled).
  • When you run a test, Simpletest creates a brand-new installation of Drupal using a special prefix simpletest123456 where 123456 is a random number. This allows Simpletest to have an isolated environment where to run tests, but on the same database and with the same credentials as the host.
  • When your test does something, like call a function, or load a page with, for example, $this->drupalGet('user'), the host environment is ignored and temporary environment (which uses the prefixed database tables) is used. In the previous example, the test loads the “user” page using a real HTTP calls. Simpletest knows to use the temporary environment because the call is made using a specially-crafted user agent.
  • When the test is over, all tables with the prefix simpletest123456 are destroyed.

If you have ever tried to run a test on a host environment which already contains a prefix, you will understand why you can get “table name too long” errors in certain cases: Simpletest is trying to add a prefix to another prefix. That’s one reason to avoid prefixes when you can, but I digress.

Now you can try this: somewhere in your test code, add die(), this will kill Simpletest, leaving the temporary database intact.

Here is an example: a colleague recently was testing a feature which exported a view. In the dev environment, the view was available to users with the role manager, as was expected. However when the test logged in as a manager user and attempted to access the view, the result was an “Access denied” page.

Because we couldn’t easily figure it out, I suggested adding die() to play around in the environment:

...
$this->drupalLogin($manager);
$this->drupalGet('inventory');
die();
$this->assertNoText('denied', 'A manager accessing the inventory page does not see "access denied"');
...

Now, when the test was run, we could:

  • wait for it to crash,
  • then examine our database to figure out which prefix the test was using,
  • change the database prefix in sites/default/settings.php from '' to (for example) 'simpletest73845'.
  • run drush uli to get a one-time login.

Now, it was easier to debug the source of the problem by visiting the views configuration for inventory: it turns out that features exports views with access by role using the role ID, not the role name (the role ID can be different for each environment). Simply changing the access method for the view from “by role” to “by permission” made the test pass, and prevented a potential security flaw in the code.

(Another reason to avoid “by role” access in views is that User 1 often does not have the role required, and it is often disconcerting to be user 1 and have “access denied” to a view.)

So in conclusion, Simpletest is great when it works as expected and when you understand what it does, but when you don’t, it is always good to know a few techniques for further investigation.

Please enable JavaScript to view the comments powered by Disqus.

Jan 28 2015
Jan 28

As recently I became the webmaster on the drupal.org and few people started asking me about how to become a webmaster and where to start. So I write a blog post so that others can benefit.
In simple words: "Start Contributing". Webmaster privileges aren't granted lightly for obvious reasons. A solid history of consistent contributions on drupal.org is a must to get consideration for an elevated role.

How to start contributing & where you can contribute :

  1. Help out in the content queue, especially reviewing Service Listing requests .Marketing guidelines for reviewing the service listing requests.
  2. Training Listing requests, Marketing guidelines to review training listing requests.
  3. Help out in the webmasters queue, Textual improvements requests.
  4. Depending on your skills, you can also contribute to drupal customizations as well.
  5. if you found any problem while contributing to above sections,just comment on the below post/ if you need immediate answer you can try and find one of the webmasters on IRC - #drupalorg IRC channel on Freenode.
If you need guidance, join the #drupalorg IRC channel and find dddave, lizzjoy, tvn or myselfA general overview about the various ways to contribute to Drupal.org can be found in the documentation.
If you have good coding skills, you are more than welcome to help reviewing project applications. I encourage you to learn more about that process and join the group of reviewers.
Next article:  We will look at how to be a git administrator on drupal.org

Jan 23 2015
Jan 23

Was looking out some good English Speaking training course website for my cousinto become a fluent English speaker.


On opening the homepage I found the section "Free trial".

I did the same thing as all guys do.Click on the Free trial link,created my account.I personally feeling happy to get the 5 free audio English lessons.
I clicked on one of my lessons link and come to the page, found the download link of the audio file.



Copied and pasted the url in the browser and the downloaded the file. https://www.dailystep.com/en/download/file/fid/13531 Just change the fid in the url and tried to download some random file with file id. I thought what the Shit!   I then look around which drupal module has provided this route.After checking I got to know that this is the download file module which provides this route.After checking I assume that it might be permission stuff problem.
I found the similar permission problem on the drupal.org as well https://www.drupal.org/node/2394993
Trust is the Major Strategy in Security of any web application.We should be sure which user role we should assign which permissions.
Note: Published the post after suggesting the fix to the site owner Jane
Jan 20 2015
Jan 20

January 20, 2015

When building a Drupal 7 site, one oft-used technique is to keep the entire Drupal root under git (for Drupal 8 sites, I favor having the Drupal root one level up).

Starting a new project can be done by downloading an unversioned copy of D7, and initializing a git repo, like this:

Approach #1

drush dl
cd drupal*
git init
git add .
git commit -am 'initial project commit'
git remote add origin ssh://[email protected]/myproject

Another trick I learned from my colleagues at the Linux Foundation is to get Drupal via git and have two origins, like this:

Approach #2

git clone --branch 7.x http://git.drupal.org/project/drupal.git drupal
cd drupal
git remote rename origin drupal
git remote add origin ssh://[email protected]/myproject

This second approach lets you push changes to your own repo, and pull changes from the Drupal git repo. This has the advantage of keeping track of Drupal project commits, and your own project commits, in a unified git history.

git push origin 7.x
git pull drupal 7.x

If you are tight for space though, there might be one inconvenience: Approach #2 keeps track of the entire Drupal 7.x commit history, for example we are now tracking in our own repo commit e829881 by natrak, on June 2, 2000:

git log |grep e829881 --after-context=4
commit e8298816587f79e090cb6e78ea17b00fae705deb
Author: natrak <>
Date:   Fri Jun 2 18:43:11 2000 +0000

    CVS drives me nuts *G*

All of this information takes disk space: Approach #2 takes 156Mb, vs. 23Mb for approach #1. This may add up if you are working on several projects, and especially if for each project you have several environments for feature branches. If you have a continuous integration server tracking multiple projects and spawning new environments for each feature branch, several gigs of disk space can be used.

If you want to streamline the size of your git repos, you might want to try the --depth option of git clone, like this:

Approach #3

git clone --branch 7.x --depth 1 http://git.drupal.org/project/drupal.git drupal
cd drupal
git remote rename origin drupal
git remote add origin ssh://[email protected]/myproject

Adding the --depth parameter here reduces the initial size of your repo to 18Mb in my test, which interestingly is even less than approach #1. Even though your repo is now linked to the Drupal git repo, by running git log you will see that the entire history is not being stored.

Please enable JavaScript to view the comments powered by Disqus.

Dec 03 2014
Dec 03

December 03, 2014

What is content? What is configuration? At first glance, the question seems simple, almost quaint, the kind one finds oneself patiently answering for the benefit of Drupal novices: content is usually information like nodes and taxonomy terms, while content types, views and taxonomy vocabularies are usually configuration.

Content lives in the database of each environment, we say, while configuration is exportable via Features or other mechanisms and should live in the Git repo (this has been called code-driven development).

Still, a definition of content and configuration is naggingly elusive: why “usually”? Why are there so many edge cases? We’re engineers, we need precision! I often feel like I’m trying to define what a bird is: every child knows what a bird is, but it’s hard to define it. Ostriches can’t fly; platypuses lay eggs but aren’t birds.

Why the distinction?

I recently saw an interesting comment titled “A heretic speaks” on a blog post about code-driven development. It sums up some of the uneasiness about the place of configuration in Drupal: “Drupal was built primarily with site builders in mind, and this is one reason [configuration] is in the database”.

In effect, the primary distinction in Drupal is between code (Drupal core and config), and the database, which contains content types, nodes, and everything else.

As more complex sites were being built, a new distinction had to be made between two types of information in the database: configuration and content. This was required to allow development in a dev-stage-production workflow where features being developed outside of a production site could be deployed to production without squashing the database (and existing comments, nodes, and the like). We needed to move those features into code and we called them “configuration”.

Thus the features module was born, allowing views, content types, and vocabularies (but not nodes and taxonomy terms) to be developed outside of the database, and then deployed into production.

Drupal 8’s config management system takes that one step further by providing a mature, central API to deal with this.

The devil is in the details

This is all fine and good, but edge cases soon begin to arise:

  • What about an “About us” page? It’s a menu item (deployable) linking to a node (content). Is it config? Is it content?
  • What about a “Social media” menu and its menu items? We want a Facebook link to be deployable, but we don’t want to hard-code the actual link to our client’s Facebook page (which feels like content) – we probably don’t even know what that link is during development.
  • What about a block whose placement is known, but whose content is not? Is this content? Is it configuration?
  • What about a view which references a taxonomy term id in a hard-coded filter. We can export the view, but the taxonomy term has an incremental ID ans is not guaranteed to work on all environments.

The wrong answer to any of these questions can lead to a misguided development approach which will come back to haunt you afterward. You might wind up using incremental IDs in your code or deploying something as configuration which is, in fact, content.

Defining our terms

At the risk of irking you, dear reader, I will suggest doing away with the terms “content” and “configuration” for our purposes: they are just too vague. Because we want a formal definition with no edge cases, I propose that we use these terms instead (we’ll look at each in detail a bit further on):

  • Code: this is what our deliverable is for a given project. It should be testable, versioned, and deployable to any number of environments.
  • Data: this is whatever is potentially different on each environment to which our code is deployed. One example is comments: On a dev environment, we might generate thousands of dummy comments for theming purposes, but on prod there might be a few dozen only.
  • Placeholder content: this is any data which should be created as part of the installation process, meant to be changed later on.

Code

This is what our deliverable is for a given project. This is important. There is no single answer. Let’s take the following examples:

  • If I am a contributor to the Views contrib project, my deliverable is a system which allows users to create views in the database. In this case I will not export many particular views.

  • For another project, my deliverable may be a website which contains a set number of lists (views). In this case I may use features (D7) or config management (D8) to export all the views my client asked for. Furthermore, I may enable views_ui (the Views User interface) only on my development box, and disable it on production.

  • For a third project, my deliverable may a website with a number of set views, plus the ability for the client to add new ones. In this only certain views will be in code, and I will enable the views UI as a dependency of my site deployment module. The views my client creates on production will be data.

Data

A few years ago, I took a step back from my day-to-day Drupal work and thought about what my main pain points were and how to do away with them. After consulting with colleagues, looking at bugs which took longest to fix, and looking at major sources of regressions, I realized that the one thing all major pain points had in common were our deployment techniques.

It struck me that cloning the database from production to development was wrong. Relying on production data to do development is sloppy and will cause problems. It is better to invest in realistic dummy content and a good site deployment module, allowing the standardized deployment of an environment in a few minutes from any commit.

Once we remove data from the development equation in this way, it is easier to define what data is: anything which can differ from one environment to the next without overriding a feature.

Furthermore, I like to think of production as just another environment, there is nothing special about it.

A new view or content type created on production outside of our development cycle resides on the database, is never used during the course of development, and is therefore data.

Nodes and taxonomy terms are data.

What about a view which is deployed through features and later changed on another environment? That’s a tough one, I’ll get to it (See Overriden features, below).

Placeholder content

Let’s get back to our “About us” page. Three components are involved here:

  • The menu which contains the “About us” menu item. These types of menus are generally deployable, so let’s call them code.
  • The “About us” node itself which has an incremental nid which can be different on each environment. On some environments it might not even exist.
  • The “About us” menu item, which should link to the node.

Remember: we are not cloning the production database, so the “About us” does not exist anywhere. For situations such as this, I will suggest the use of Placeholder content.

For sake of argument, let’s define our deliverable for this sample project as follows:

"Define an _About us_ page which is modifiable".

We might be tempted to figure out a way to assign a unique ID to our “About us” node to make it deployable, and devise all kinds of techniques to make sure it cannot be deleted or overridden.

I have an approach which I consider more logical for these situations:

First, in my site deployment module’s hook_update_N(), create the node and the menu item, bypassing features entirely. Something like:

function mysite_deploy_update_7023() {
  $node = new stdClass();
  $node->title = 'About us';
  $node->body[LANGUAGE_NONE][0]['format'] = 'filtered_html';
  $node->body[LANGUAGE_NONE][0]['value'] = 'Lorem ipsum...';
  $node->type = 'page';
  node_object_prepare($node);
  $node->uid = 1;
  $node->status = 1;
  $node->promote = 0;
  node_save($node);

  $menu_item = array(
    'link_path' => 'node/' . $node->nid,
    'link_title' => 'About us',
    'menu_name' => 'my-existing-menu-exported-via-features',
  );

  menu_link_save($item);
}

If you wish, you can also implement hook_requirements() in your custom module, to check that the About us page has not been accidentally deleted, that the menu item exists and points to a valid path.

What are the advantages of placeholder content?

  • It is deployable in a standard manner: any environment can simply run drush updb -y and the placeholder content will be deployed.
  • It can be changed without rendering your features (D7) or configuration (D8) overriden. This is a good thing: if our incremental deployment script calls features_revert() or drush fra -y (D7) or drush cim -y (D8), all changes to features are deleted. We do not want changes made to our placeholder content to be deleted.
  • It can be easily tested. All we need to do is make sure our site deployment module’s hook_install() calls all hook_update_N()s; then we can enable our site deployment module within our simpletest, and run any tests we want against a known good starting point.

Overriden features

Although it is easy to override features on production, I would not recommend it. It is important to define with your client and your team what is code and what is data. Again, this depends on the project.

When a feature gets overridden, it is a symptom that someone does not understand the process. Here are a few ways to mitigate this:

  • Make sure your features are reverted (D7) or your configuration is imported (D8) as part of your deployment process, and automate that process with a continuous integration server. That way, if anyone overrides a feature on a production, it won’t stay overridden long.
  • Limit administrator permissions so that only user 1 can override features (this can be more trouble than it’s worth though).
  • Implement hook_requirements() to check for overridden features, warning you on the environment’s dashboard if a feature has been overridden.

Some edge cases

Now, with our more rigorous approach, how do our edge cases fare?

Social media menu and items: Our deliverable here is the existence of a social media menu with two items (twitter and facebook), but whose links can be changed at any time on production without triggering an overridden feature. For this I would use placeholder content. Still, we need to theme each button separately, and our css does not know the incremental IDs of the menu items we are creating. I have successfully used the menu attributes module to associate classes to menu items, allowing easy theming. Here is an example, assuming menu_attributes exists and menu-social has been exported as a feature.

/**
 * Add facebook and twitter menu items
 */
function mysite_deploy_update_7117() {
  $item = array(
    'link_path' => 'http://twitter.com',
    'link_title' => 'Twitter',
    'menu_name' => 'menu-social',
    'options' => array(
      'attributes' => array(
        'class' => 'twitter',
      )
    )
  );
  menu_link_save($item);
  $item = array(
    'link_path' => 'http://facebook.com',
    'link_title' => 'Facebook',
    'menu_name' => 'menu-social',
    'options' => array(
      'attributes' => array(
        'class' => 'facebook',
      )
    )
  );
  menu_link_save($item);
}

The above code creates the menu items linking to Facebook and Twitter home pages, so that content editors can put in the correct links directly on production when they have them.

Placeholder content is just like regular data but it’s created as part of the deployment process, as a service to the webmaster.

A block whose placement is known, but whose content is not. It may be tempting to use the box module which makes blocks exportable with feature. But in this case the block is more like placeholder content, so it should be deployed outside of features. And if you create your block programmatically, its id is incremental and it cannot be deployed with context, but should be placed in a region directly, again, programmatically in a hook_update_N().

Another approach here is to create a content type and a view with a block display, fetching the last published node of that content type and displaying it at the right place. If you go that route (which seems a bit overengineered to me), you can then place your block with the context module and export it via features.

A view which references a taxonomy term id in its filter: If a view requires access to a taxonomy term nid, then perhaps taxonomy is the wrong tool here. Taxonomy terms are data, they can be deleted, their names can be changed. It is not a good idea for a view to reference a specific taxonomy term. (Your view can use taxonomy terms for contextual filters without a problem, but we don’t want to hard-code a specific term in a non-contextual filter – See this issue for an example of how I learned this the hard way, I’ll get around to fixing that soon…).

For this problem I would suggest rethinking our use of a taxonomy term. Rather I would define a select field with a set number of options (with defined keys and values). These are deployable and guaranteed to not change without triggering a features override. Thus, our views can safely use them. If you are implementing this change on an existing site, you will need to update all nodes from the old to the new technique in a hook_update_N() – and probably add an automated test to make sure you’re updating the data correctly. This is one more reason to think things through properly at the onset of your project, not midway through.

In conclusion

Content and configuration are hard to define, I prefer the following definitions:

  • Code: deployable, deliverable, versioned, tested piece of software.
  • Data: anything which can differ from one environment to the next.
  • Placeholder content: any data which should be created as part of the deployment process.

In my experience, what fits in each category depends on each project. Defining these with your team as part of your sprint planning will allow you create a system with less edge cases.

Please enable JavaScript to view the comments powered by Disqus.

Nov 10 2014
Nov 10

I have recently joined QE42 pune, As the drupal 8 module port code sprint will happen in
QED42 on 8 November,2014 https://groups.drupal.org/node/448748 and It was postponed later and I was unaware of that.
So I decided to port a module, so I picked a random project from the code reviews done by me from project applications.
So I picked Anonymous suggestion box and ported the module code to drupal 8 and created its meta in the module issue queue https://www.drupal.org/node/2371803

After that I have the time left for code writing so I decided to pick another one jquery carousel and ported half of its code to drupal 8 https://www.drupal.org/node/2371855
If anyone wants to join hands while porting this module then tweet me over twitter lets finish this before this week end and will take another one for next week end.

Waiting for the postponed sprint to meet #pune drupalers over there.

Sep 10 2014
Sep 10

September 10, 2014

What is code-driven development and why is it done?

Code-driven development is the practice of placing all development in code. How can development not be in code?, you ask.

In Drupal, what makes your site unique is often configuration which resides in the database: the current theme, active modules, module-specific configuration, content types, and so on.

For the purpose of this article, our goal will be for all configuration (the current theme, the content types, module-specific config, the active module list…) to be in code, and only content to be in the database. There are several advantages to this approach:

  • Because all our configuration is in code, we can package all of it into a single module, which we’ll call a site deployment module. When enabled, this module should provide a fully workable site without any content.
  • When a site deployment module is combined with generated content, it becomes possible to create new instances of a website without cloning the database. Devel’s devel_generate module, and Realistic Dummy Content can be used to create realistic dummy content. This makes on-ramping new developers easy and consistent.
  • Because unversioned databases are not required to be cloned to set up new environments, your continuous integration server can set up new instances of your site based on a known good starting point, making tests more robust.

Code-driven development for Drupal 7

Before moving on to D8, let’s look at a typical D7 workflow: The technique I use for developing in Drupal 7 is making sure I have one or more features with my content types, views, contexts, and so on; as well as a site deployment module which contains, in its .install file, update hooks which revert my features when needed, enable new modules, and programmatically set configuration which can’t be exported via features. That way,

  • incrementally deploying sites is as simple as calling drush updb -y (to run new update hooks).
  • deploying a site for the first time (or redeploying it from scratch) requires creating the database, enabling our site deployment module (which runs all or update hooks), and optionally generating dummy content if required. For example: drush si -y && drush en mysite_deploy -y && drush en devel_generate && drush generate-content 50.

I have been using this technique for a few years on all my D7 projects and, in this article, I will explore how something similar can be done in D8.

New in Drupal 8: configuration management

If, like me, you are using features exclusively to deploy websites (as opposed to using it to bundle generic functionality, for example having a “blog” feature, or a “calendar” feature you can add to any site), config management will replace features in D8. In D7, context is used to provide the ability to export block placement to features, and strongarm exports variables. In D8, variables no longer exist, and block placement is now exportable. All of these modules are thus no longer needed.

They are replaced by the concept of configuration management, a central API for importing and exporting configuration as yml files.

Configuration management and site UUIDs

In Drupal 8, sites are now assigned a UUID on install and configuration can only be synchronized between sites having the same UUID. This is fine if the site has been cloned at some point from one environment to another, but as mentioned above, we are avoiding database cloning: we want it to be possible to install a brand new instance of a site at any time.

We thus need a mechanism to assign the same UUID to all instances of our site, but still allow us to reinstall it without cloning the database.

The solution I am using is to assign a site UUID in the site deployment module. Thus, in Drupal 8, my site deployment module’s .module file looks like this:

/**
 * @file
 * site deployment functions
 */
use Drupal\Core\Extension\InfoParser;

/**
 * Updates dependencies based on the site deployment's info file.
 *
 * If during the course of development, you add a dependency to your
 * site deployment module's .info file, increment the update hook
 * (see the .install module) and this function will be called, making
 * sure dependencies are enabled.
 */
function mysite_deploy_update_dependencies() {
  $parser = new InfoParser;
  $info_file = $parser->parse(drupal_get_path('module', 'mysite_deploy') . '/mysite_deploy.info.yml');
  if (isset($info_file['dependencies'])) {
    \Drupal::service('module_installer')->install($info_file['dependencies'], TRUE);
  }
}

/**
 * Set the UUID of this website.
 *
 * By default, reinstalling a site will assign it a new random UUID, making
 * it impossible to sync configuration with other instances. This function
 * is called by site deployment module's .install hook.
 *
 * @param $uuid
 *   A uuid string, for example 'e732b460-add4-47a7-8c00-e4dedbb42900'.
 */
function mysite_deploy_set_uuid($uuid) {
  \Drupal::configFactory() ->getEditable('system.site')
    ->set('uuid', $uuid)
    ->save();
}    

And the site deployment module’s .install file looks like this:

/**
 * @file
 * site deployment install functions
 */

/**
 * Implements hook_install().
 */
function mysite_deploy_install() {
  // This module is designed to be enabled on a brand new instance of
  // Drupal. Settings its uuid here will tell this instance that it is
  // in fact the same site as any other instance. Therefore, all local
  // instances, continuous integration, testing, dev, and production
  // instances of a codebase will have the same uuid, enabling us to
  // sync these instances via the config management system.
  // See also https://www.drupal.org/node/2133325
  mysite_deploy_set_uuid('e732b460-add4-47a7-8c00-e4dedbb42900');
  for ($i = 8001; $i < 9000; $i++) {
    $candidate = 'mysite_deploy_update_' . $i;
    if (function_exists($candidate)) {
      $candidate();
    }
  }
}

/**
 * Update dependencies and revert features
 */
function mysite_deploy_update_8003() {
  // If you add a new dependency during your development:
  // (1) add your dependency to your .info file
  // (2) increment the number in this function name (example: change
  //     change 8003 to 8004)
  // (3) now, on each target environment, running drush updb -y
  //     will call the mysite_deploy_update_dependencies() function
  //     which in turn will enable all new dependencies.
  mysite_deploy_update_dependencies();
}

The only real difference between a site deployment module for D7 and D8, thus, is that the D8 version must define a UUID common to all instances of a website (local, dev, prod, testing…).

Configuration management directories: active, staging, deploy

Out of the box, there are two directories which can contain config management yml files:

  • The active directory, which is always empty and unused. It used to be there to store your active configuration, and it is still possible to do so, but I’m not sure how. We can ignore this directory for our purposes.
  • The staging directory, which can contain .yml files to be imported into a target site. (For this to work, as mentioned above, the .yml files will need to have been generated by a site having the same UUID as the target site, or else you will get an error message – on the GUI the error message makes sense, but on the command line you will get the cryptic “There were errors validating the config synchronization.”).

I will propose a workflow which ignores the staging directory as well, for the following reasons:

  • First, the staging directory is placed in sites/default/files/, a directory which contains user data and is explicitly ignored in Drupal’s example.gitignore file (which makes sense). In our case, we want this information to reside in our git directory.
  • Second, my team has come to rely heavily on reinstalling Drupal and our site deployment module when things get corrupted locally. When you reinstall Drupal using drush si, the staging directory is deleted, so even if we did have the staging directory in git, we would be prevented from running drush si -y && drush en mysite_deploy -y, which we don’t want.
  • Finally, you might want your config directory to be outside of your Drupal root, for security reasons.

For all of these reasons, we will add a new “deploy” configuration directory and put it in our git repo, but outside of our Drupal root.

Our directory hierarchy will now look like this:

mysite
  .git
  deploy
    README.txt
    ...
  drupal_root
    CHANGELOG.txt
    core
    ...

You can also have your deploy directory inside your Drupal root, but keep in mind that certain configuration information are sensitive, containing email addresses and the like. We’ll see later on how to tell Drupal how it can find your “deploy” directory.

Getting started: creating your Drupal instance

Let’s get started. Make sure you have version 7.x of Drush (compatible with Drupal 8), and create your git repo:

mkdir mysite
cd mysite
mkdir deploy
echo "Contains config meant to be deployed, see http://dcycleproject.org/blog/68" >> deploy/README.txt
drush dl drupal-8.0.x
mv drupal* drupal_root
cp drupal_root/example.gitignore drupal_root/.gitignore
git init
git add .
git commit -am 'initial commit'

Now let’s install our first instance of the site:

cd drupal_root
echo 'create database mysite'|mysql -uroot -proot
drush si --db-url=mysql://root:[email protected]/mysite -y

Now create a site deployment module: here is the code that works for me. We’ll set the correct site UUID in mysite_deploy.install later. Add this to git:

git add drupal_root/modules/custom
git commit -am 'added site deployment module'

Now let’s tell Drupal where our “deploy” config directory is:

  • Open sites/default/settings.php
  • Find the lines beginning with $config_directories
  • Add $config_directories['deploy'] = '../deploy';

Edit: using a config directory name other than ‘sync’ will cause an issue Config Split at the time of this writing.

We can now perform our first export of our site configuration:

cd drupal_root
drush config-export deploy -y

You will now notice that your “deploy” directory is filled with your site’s configuration files, and you can add them to git.

git add .
git commit -am 'added config files'

Now we need to sync the site UUID from the database to the code, to make sure all subsequent instances of this site have the same UUID. Open deploy/system.site.yml and find UUID property, for example:

uuid: 03821007-701a-4231-8107-7abac53907b1
...

Now add this same value to your site deployment module’s .install file, for example:

...
function mysite_deploy_install() {
  mysite_deploy_set_uuid('03821007-701a-4231-8107-7abac53907b1');
...

Let’s create a view! A content type! Position a block!

To see how to export configuration, create some views and content types, position some blocks, and change the default theme.

Now let’s export our changes

cd drupal_root
drush config-export deploy -y

Your git repo will be changed accordingly

cd ..
git status
git add .
git commit -am 'changed theme, blocks, content types, views'

Deploying your Drupal 8 site

At this point you can push your code to a git server, and clone it to a dev server. For testing purposes, we will simply clone it directly

cd ../
git clone mysite mysite_destination
cd mysite_destination/drupal_root
echo 'create database mysite_destination'|mysql -uroot -proot
drush si --db-url=mysql://root:[email protected]/mysite_destination -y

If you visit mysite_destination/drupal_root with a browser, you will see a plain new Drupal 8 site.

Before continuing, we need to open sites/default/settings.php on mysite_destination and add $config_directories['deploy'] = '../deploy';, as we did on the source site.

Now let the magic happen. Let’s enable our site deployment module (to make sure our instance UUID is synched with our source site), and import our configuration from our “deploy” directory:

drush en mysite_deploy -y
drush config-import deploy -y

Now, on your destination site, you will see all your views, content types, block placements, and the default theme.

This deployment technique, which can be combined with generated dummy content, allows one to create new instances very quickly for new developers, testing, demos, continuous integration, and for production.

Incrementally deploying your Drupal 8 site

What about changes you make to the codebase once everything is already deployed. Let’s change a view and run:

cd drupal_root
drush config-export deploy -y
cd ..
git commit -am 'more fields in view'

Let’s deploy this now:

cd ../mysite_destination
git pull origin master
cd drupal_root
drush config-import deploy -y

As you can see, incremental deployments are as easy and standardized as initial deployments, reducing the risk of errors, and allowing incremental deployments to be run automatically by a continuous integration server.

Next steps and conclusion

Some aspects of your site’s configuration (what makes your site unique) still can’t be exported via the config management system, for example enabling new modules; for that we’ll use update hooks as in Drupal 7. As of this writing Drupal 8 update hooks can’t be run with Drush on the command line due to this issue.

Also, although a great GUI exists for importing and exporting configuration, I chose to do it on the command line so that I could easily create a Jenkins continuous integration job to deploy code to dev and run tests on each push.

For Drupal projects developed with a dev-stage-prod continuous integration workflow, the new config management system is a great productivity boost.

Please enable JavaScript to view the comments powered by Disqus.

Jul 30 2014
Jul 30

July 30, 2014

I had this checklist documented internally, but I keep referring back to it so I’ll make it available here in case anyone else needs it. The idea here is to document a minimum (not an ideal) set of modules and tasks which I do for almost all projects.

Questions to ask of a client at the project launch

  • Is your site bilingual? If so is there more than one domain? (if so, and you are exporting your languages as Features, your domain is exported with it. If your domains are different on different environments, you might want to use language_domain to override the domains per environment)
  • What type of compatibility do you need: tablet, mobile, which versions of IE?
  • How do you see your post-launch support and core/module update contract?
  • Do you need SSL support?
  • What is your hosting arrangement?
  • Do you have a contact form?
  • What is your anti-spam method? Note that CAPTCHA is no longer useful; I like Mollom, but it’s giving me more and more false positives with time. Honeypot has given me good results as well.
  • Is WYSIWYG required? I strongly suggest using Markdown instead.
  • Confirm that all emails are sent in plain text, not HTML. If you’re sending out HTML mail, do it right.
  • Do you need an on-site search utility? If so, some thought, and resources, need to go into it or it will be frustrating.
  • What kind of load do you expect on your site (anonymous and admin users)? This information can be used for load testing.
  • If you already have a site, should old paths of critical content map to paths on the new site?
  • Should users be allowed to create accounts (with spam considerations, and see if an admin should approve them).

Here is what should get done in the first Agile sprint, aka Sprint Zero:

  • If you are using continuous integration, a Jenkins job for tracking the master branch: this job should fail if any test fails on the codebase, or if quality metrics (code review, for example, or pdepend metrics) reach predefined thresholds.
  • A Jenkins job for pushing to dev. This is triggered by the first job if tests pass. It pushed the new code to the dev environment, and updates the dev environment’s database. The database is never cloned; rather, a site deployment module is used.
  • An issue queue is set up and the client is given access to it, and training on how to use it.
  • A wiki is set up.
  • A dev environment is set up. This is where the code gets pushed automatically if all tests pass.
  • A prod environment is set up. This environment is normally updated manually after each end of sprint demo.
  • A git repo is set up with a basic Drupal site.
  • A custom module is set up in sites/*/modules/custom: this is where custom function go.
  • A site deployment module in sites/all/modules/custom. All deployment-related code and dependencies go here. A .test file and an .install should be included.
  • A site development module is set up in sites/*/modules/custom, which is meant to contain all modules required or useful for development, as dependencies.
  • A custom theme is created.
  • An initial feature is created in sites/*/modules/features. This is where all your features will be added.
  • A “sites/*/modules/patches” folder is created (with a README.txt file, to make sure it goes into git). This is where core and contrib patches should go. Your site’s maintainers should apply these patches when core or contrib modules are updated. Patch names here should include the node id and comment number on Drupal.org.

Basic module list (always used)

Development modules (not enabled on production)

I normally create a custom development module with these as dependencies:

I make sure this module is in my repo but it is not enabled unless used:

Experimental modules

  • dcycle, this is a module that is in active development, not ready for prime yet, but where I try to add all my code to help with testing, etc.

Multilingual modules

  • i18n
  • potx
  • l10n_update
  • entity_translation if you need the same node id to display in several languages. This is useful if you have references to nodes which should be translated.
  • title if you are using entity translations and your titles can be multilingual.

Launch checklist

  • Design a custom 404, error and maintenance page.
  • Path, alias and permalink strategy. (Might require pathauto.)
  • Think of adding revisions to content types to avoid clients losing their data.
  • Don’t display errors on production.
  • Optimize CSS, JS and page caching.
  • Views should be cached.
  • System messages are properly themed.
  • Prevent very simple passwords.
  • Using syslog instead of dblog on prod

In conclusion

Most shops, and most developers, have some sort of checklist like this. Mine is not any better or worse than most, but can be a good starting point. Another note: I’ve seen at least three Drupal teams try, and fail, to implement a “Drupal Starter kit for Company XYZ” and keep it under version control. The problem with that approach, as opposed to a checklist, is that it’s not lightweight enough: it is a software product which needs maintenance, and after a while no one maintains it.

Please enable JavaScript to view the comments powered by Disqus.

May 23 2014
May 23

May 23, 2014

One of the techniques I use to make sure I write tests is to write them before I do anything else, which is known as test-driven development. If you develop your functionality before writing a test, in most cases you will never write the test to go with it, because you will be pressured to move on to new features.

I have found, though, that when writing tests, our team tends to think only about the happy path: what happens if everything goes according to plan.

Let me give an quick example: let’s say you are developing a donation system for anonymous users to make donations on your site. The user story calls for a form where a donation amount can be entered before redirecting the user to the payment form. Using test-driven development and Drupal’s Simpletest framework, we might start by writing something like this in our site deployment module’s .test file:

// @file mysite_deploy.test

class MysiteDonate extends DrupalWebTestCase {

  ...

  public function testSite() {

    $edit = array(
      'amount' => 420,
    );
    ...
    $this->drupalPost('donate', $edit, 'Donate now!');
    ...
    $this->assertText('You are about to donate $420', 'The donation amount has been recorded');
  }

  ...
}

When you first run this test it will fail, and your job as a developer will be to make this test pass. That’s test-driven development.

The problem with this approach is that it only defines the happy path: what should happen when all goes according to plan. It makes no provision for the sad path: what happens if a user puts something other than a number? What happens if 0 is entered? These are known as sad paths, and most teams never think about them until they occur (human nature, I guess).

To make sure we think about the sad path, I start by making sure the right questions are asked during our Agile sprint planning sessions. In the case of the “donation” user story mentioned above, the following business questions should be asked during sprint planning:

  • What’s the minimum donation? Obviously it should not be possible to donate $0, but is $0.01 OK?
  • Is there a maximum donation? Should the system bring you to the checkout page if you enter 1 billion dollars in the donation box?

Often, the client will not have thought of that, and will answer something like: sure there should be a minimum and a maximum, and we also want site administrators to be able to edit those. Let’s say the team agrees on this (and the extra work it entails), the admin interface too should be tested.

Once the sprint planning session is over, I will start by writing the test based on business considerations above, and also integrating other sad paths I can think of, into my test.

Here is what our test might look like now, assuming we have a setUp() function which enables our site deployment module and dependent features (including roles); and we are using the loginAsRole() method, documented here:

// @file mysite_deploy.test

class MysiteDonate extends DrupalWebTestCase {

  ...

  public function testSite() {

    // Manage minimum and maximum donation amounts.
    $this->drupalGet('admin/options');
    $this->assertText('Access denied', 'Non-admin users cannot access the configuration page');
    $this->loginAsRole('administrator');
    $edit = array(
      'minimum' => '50',
      'maximum' => $this->randomName(),
    );
    $this->drupalPost('admin/option', $edit, 'Save');
    $this->assertText('Minimum and maximum donation amounts must be numeric');
    $edit['maximum'] = '40';
    $this->drupalPost('admin/option', $edit, 'Save');
    $this->assertText('Minimum amount must be equal to or less than maximum donation amount');
    $edit['maximum'] = '30';
    $this->drupalPost('admin/option', $edit, 'Save');
    $this->assertText('Minimum maximum donation amounts have been saved');
    $this->drupalLogout();

    // Make a donation, sad path
    $edit = array(
      'amount' => '<script>alert("hello!")</script>',
    );
    $this->drupalPost('donate', $edit, 'Donate now!');
    $this->assertText('Donation amount must be numeric', 'Intercept non-numeric input.');
    $edit['amount'] = 29;
    $this->drupalPost('donate', $edit, 'Donate now!');
    $this->assertText('Thanks for your generosity, but we do not accept donations below $30.');
    $edit['amount'] = 41;
    $this->drupalPost('donate', $edit, 'Donate now!');
    $this->assertText('Wow, $41! Do not do this through our website, please contact us and we will discuss this over the phone.');

    // Make a donation, happy path
    $edit['amount'] = 30;
    $this->drupalPost('donate', $edit, 'Donate now!');
    $this->assertText('You are about to donate $30', 'The donation amount has been recorded');
  }

  ...
}

The above example is a much more complete portrait of what your site should do, and documenting everything in a failing test even before you or someone else starts coding ensures you don’t forget validations and the like.

One interesting thing to note about our complete test is that sad paths actually take up a lot more effort than the happy path. There are many advantages to thinking of them first:

  • The client can be involved in making business decisions which can affect the sad path.
  • The entire team (including the client) is made aware as early as possible about sad path considerations, and the extra work they entail.
  • Nothing is taken for granted as obvious: time is set aside for sad path development.
  • The sad path becomes an integral part of your user story which can be part of the demo. Often in Agile sprint reviews, if no one has ever thought of the sad path, only the happy path is demonstrated.
  • There is less technical debt associated with sad path development: you are less likely to get a panicked call from your client once your site goes live about getting dozens of 50 cent donations when the payment processor is taking a dollar in fees.
  • Your code will be more secure: you will think about how your system can be hacked and integrate hacking attempts (and the appropriate response) directly into your test.
  • You will be more confident putting a failing test on a feature branch and handing it to junior developers: they will be less likely to forget something.
  • Thinking of the sad path can make you reconsider how to define your features: a contact form or commenting system can seem trivial when you only think of the happy path. However, when you take into account how to deal with spam, you might decide to not allow comments at all, or to allow only authenticated users to post comments or use the contact form.

Note that as in all test-driven development, your test is not set in stone. It is like any other code: developers can modify it as long as they follow the spirit of your test. For example, maybe your config page is not admin/option but something else. Developers should feel that they own the test and can change it to fit the real system.

Please enable JavaScript to view the comments powered by Disqus.

Apr 28 2014
Apr 28

Sometimes you want to license files without people needing to purchase them. Even using coupon codes to make products free still requires them to be purchased through the Commerce Checkout system.

This is fine for physical products where you still want email and address details of potential future clients.

However when it comes to files, users require an account to access their files, so chances are you have all the details for them already. And there is no shipping required so why make them go through the checkout process just to get a license for a free file? (Seriously if you have reasons comment!)

Here is a snippet of how to generate a file license for a user:

Unrelated

Grammar Lesson:

Today I learnt the difference between 'license' and 'licence'. Unless you are American (in which case just ignore the existence of 'licence') read this.

Apr 22 2014
Apr 22

April 22, 2014

My development team is using a site deployment module which, when enabled, deploys our entire website (with translations, views, content types, the default theme, etc.).

We defined about 30 tests (and counting) which are linked to Agile user stories and confirm that the site is doing what it’s supposed to do. These tests are defined in Drupal’s own Simpletest framework, and works as follows: for every test, our site deployment module is enabled on a new database (the database is never cloned), which can take about two minutes; the test is run, and then the temporary database is destroyed.

This created the following problem: because we were deploying our site 30 times during our test run, a single test run was taking over 90 minutes. Furthermore, we are halfway into the project, and we anticipate doubling, perhaps tripling our test coverage, which would mean our tests would take over four hours to run.

Now, we have a Jenkins server which performs all the tests every time a change is detected in Git, but even so, when several people are pushing to the git repo, test results which are 90 minutes old tend to be harder to debug, and developers tend to ignore, subvert and resent the whole testing process.

We could combine tests so the site would be deployed less often during the testing process, but this causes another problem: tests which are hundreds of lines long, and which validate unrelated functionality, are harder to debug than short tests, so it is not a satisfactory solution.

When we look at what is taking so long, we notice that a majority of the processing power goes to install (deploy) our testing environment for each test, which is then destroyed after a very short test.

Enter Simpletest Turbo, which provides very simple code to cache your database once the setUp() function is run, so the next test can simply reuse the same database starting point rather than recreate everything from scratch.

Although Simpletest Turbo is in early stages of development, I have used it to almost quadruple the speed of my tests, as you can see from this Jenkins trend chart:

I know: my tests are failing more than I would like them to, but now I’m getting feedback every 25 minutes instead of every 95 minutes, so failures are easier to pinpoint and fix.

Furthermore, fairly little time is spent deploying the site: this is done once, and the following tests use a cached deployment, so we are not merely speeding up our tests (as we would if we were adding hardware): we are streamlining duplicate effort. It thus becomes relatively cheap to add new independent tests, because they are using a cached site setup.

Please enable JavaScript to view the comments powered by Disqus.

Apr 15 2014
Apr 15

Super Site Deployment with ctools exportable revert snippets

Sometimes when you are deploying new code to a production site you want to update views, panels, etc. with new code exports, but for one reason or another the defaults are overriden by the database.

Well with the following scripts you can stop worrying about that and just have an update hook take care of reverting (or deleting) the overriding database entries.

Improvements appreciated and feel free to comment!

Apr 14 2014
Apr 14

Alias Directory

You can place the aliases.drushrc file either in the 'sites/all/drush' directory or your global environment drush folder (eg. /home/username/.drush) and the naming convention is 'group.aliases.drushrc.php' so I normally use 'project.aliases.drushrc.php' or 'client.aliases.drushrc.php' to group related sites.

Dev (/local)

Create an alias array defining your local development site:

$aliases['dev'] = array(
  'uri' => 'sitename.dev',      // The uri as configured in you apache hosts
  'root' => '/path/to/web/root',
  'path-aliases' => array(
    '%files' => 'sites/default/files',
   ),
);

You can now (if you placed the alias file in your global drush directory) use drush from any directory, using:

drush @project.dev status

or

drush @project.dev cc all

Did you say any directory?!

Yep! Since you have defined you webroot in the global drush aliases file, you don't have to be in your webroot when running drush, and really, you don't even have to be on the same server...

Production (/remote)

To get the alias details for a remote machine, the easiest place to start would be to just ssh into it and run:

drush sa @self --with-db --show-passwords --with-optional

The result looks like this:

$aliases['self'] = array (
  'root' => '/path/to/drupal/root',
  'uri' => 'http://default',
  'path-aliases' => array(
    '%drush' => '/path/to/drush',
    '%site' => 'sites/default/', 
  ),
  'databases' => array(
    'default' => array(
      'default' => array(
        'database' => 'site_db',
        'username' => 'site_user',
        'password' => 'site_pass',
        'host' => 'localhost',
        'port' => '',
        'driver' => 'mysql',
        'prefix' => '',
      ),
    ),
  ),
);

You can just copy this directly into your local drush alias file and add remote details liek this:

$aliases['live'] = array (
...
  'uri' => 'http://mysite.com',
  'remote-host' => 'ip.or.domain',
  'remote-user' => 'ssh_user',
...
  'path-aliases' => array(
    '%files' => 'sites/default/files',
...

The result allows you to run drush commands locally and have them acting on a remote site.

Jiminy Cricket!

  'remote-port' => 3201,

If you have a seperate port for mysql

  'ssh-options' => '-o PasswordAuthentication=yes',

If you can't use an ssh key

Syncing files

You can sync the files directory between sites:

drush rsync -y @project.live:%files @project.dev:sites/default

or

drush -r /path/to/web/root rsync -y @project.live:%files @self:sites/default

This post is mainly snippets and tips for me to remember drushrc tools in my day to day work.
Other (/better) blog posts are as follows:

Apr 07 2014
Apr 07

The first revision control system I ever used was called RCS. It was the pre-cursor to CVS and stored all revision data locally. It was nifty but very limited and not suited for group development. CVS was the first shared revisioning system I used. It was rock solid, IMHO. But it had a few big problems, like the inability to rename or move files. Everything had to be deleted and re-added. 

Since those days, I've used several other revisioning systems: Perforce, Bitkeeper, Clearcase, Subversion and GIT.

I'm tired of learning yet another system. I just want to know which horse is going to win the race for the forseeable future and go all in.

That's where Google Trends comes in very handy. It quickly reveals that I need to bet on GIT. 

I just hope I can make it through the next 5 years or more before having to learn the next greatest solution to our shared problem of tracking code revisions.

Feb 26 2014
Feb 26

February 26, 2014

Many Drupal projects now under maintenance suffer from technical debt: a lot of the functionality is in the database and outside of git, and the code lacks automated testing. Furthermore, the functionality is often brittle: a change to one feature breaks something seemingly unrelated.

As our community and our industry mature, teams are increasingly interested in automated testing. Having worked on several Drupal projects with and without automated testing, I’ve come to the conclusion that any line of code which is not subject to automated testing is legacy code; and I agree with Michael Feathers who stated in his book Working Effectively with Legacy Code[1] that a site with zero automated tests is a legacy site from the moment you deliver it.

But the road to automatic testing for Drupal is, as I’ve learned the hard way, strewn with obstacles, and first-time implementations of automated testing tend to fail. Here are a few tips to keep in mind if your team is willing to implement automated testing.

Tip #1: Use a continuous integration server

Tests are only useful if someone actually runs them. If you don’t automate running the test suite on each push to your git repo, no one will run your tests, however good their intentions are.

The absolute first thing you need to do is set up a continuous integration (CI) server which runs a script every time your git repo changes. To make this easier I’ve set up a project on GitHub which uses Vagrant and Puppet to set up a quick Jenkins server tailored for use with Drupal.

Even before starting to write tests, make sure your continuous integration job actually runs on your master branch. When your project passes tests (which is easy at first because you won’t have tests), your project will be marked as stable.

Notice that I mentioned the master branch: although git has advanced branching features, the only branch you should track in your CI server is your stable branch (often master, although for projects with more than one stable release, like Drupal itself, you may have two or three stable branches).

It is important at this point to get the team (including the client) used to seeing the continuous integration dashboard, ideally by having a monitor in a visible place (this team even plugged Jenkins into a stop light, which really grabs attention in case of a failure). If your code is flagged as failed by your CI server, you want it to be known as soon as possible, and you want the entire team to have responsibility for fixing it immediately. Your main enemy here is failure fatigue: if your master branch is broken, and no one is working at fixing it, you will get used to seeing failures and you will fail at implementing automated testing.

Eventually, you will want to add value to your continuous integration job by running Code Review tests, and other code analysis tools like Pdepend. With these kinds of tools, you can get a historical perspective on metrics like adherance to Drupal coding standards, the number of lines of code per function, code abstraction, and the like. I even like to have my Jenkins job take a screenshot of my site on every push (using PhantomJS), and comparing the latest screenshot to the previous one ImageMagick’s compare utility.

Basically, any testing and analysis you can do on the command line should be done within your continuous integration job.

If done right, and if you have high confidence in your test suite, you can eventually use your CI server to deploy continuously to preproduction, but let’s not get ahead of ourselves.

Tip #2: Test your code, not the database

Most Drupal developers I’ve talked to create their local development environment by bringing their git repo up to date, and cloning the production database.

They also tend to clone the production or preproduction database back to Jenkins in their continuous integration.

For me, this is the wrong approach, as I’ve documented in this blog post.

Basically, any tests you write should reside in your git repo and be limited to testing what’s in the git repo. If you try to test the production database, here is a typical scenario:

  • Someone will do something to your database which will break a test.

  • Your Jenkins job will clone the database, run the test, and fail.

  • Another person will make another change to the database, and your test will now pass.

You will now see a history of failures which will indicate problems outside of your code. These will be very hard to reproduce and fix.

Keep in mind that the tests you write should depend on a known good starting point: you should be able to consistently reproduce an environment leading to a success or a failure. Drupal’s Simpletests completely ignore the current host database and create a new database from scratch just for testing, then destroy that database.

How to do this? First, I always use a site deployment module whose job it is to populate the database with everything that makes your site unique: enabling the site deployment module should enable all modules used by your site, and, using Features and related modules, deploy all views, content types, and the like, set all variables and set the default theme. The site deployment module can then be used by new developers on your team who need a development environment, and also by the CI server, all without cloning the database. If you need dummy content for development, you can use Devel’s devel_generate utility, along with this trick to make your generated content more realistic.

When a bug is reported on your production site, you should reproduce it consistently in your dummy content, and then run your test against the simulation, not the real data. An example of this is the use of Wysiwyg: often, lorem ipsum works fine, but once the client starts copy-pasting from Word, all kinds of problems arise. Simulated word-generated markup is the kind of thing your test should set up, and then test against.

If you are involved in a highly-critical project, you might eventually want to run certain tests on a clone of your production database, but this, in my opinion, should not be attempted until you have proper test coverage and metrics for your code itself. If you do test a clone of your production database and a bug is found, reproduce the bug in a simulation, add a test to confirm the bug, and fix your code. Fixing your code to deal with a problem in production without simulating the problem first, and testing the simulation, just results in more legacy code.

Tip #3: Understand the effort involved

Testing is time-consuming. If your client or employer asks for it, that desire needs to come with the appropriate resources. Near the beginning of a project, you can easily double all time estimates, and the payoff will come later on.

Stakeholders cannot expect the same velocity for a project with and without automated testing: if you are implementing testing correctly, your end-of-sprint demos will contain less features. On the other hand, once you have reached your sweet spot (see chart, above), the more manageable number of bugs will mean you can continue working on features.

Tip #4: Start gradually

Don’t try to test everything at once. If your team is called upon to “implement automated testing” on a project, you are very likely to succumb to test paralysis if you try to implement it all at once.

When working with legacy sites, or even new sites for which there is pressure to deliver fast, I have seen many teams never deliver a single test, instead delivering excuses such as “it’s really simple, we don’t need to test it”, or “we absolutely had to deliver it this week”. In reality, we tend to see “automated testing” as insurmountable and try to weasel our way of it.

To overcome this, I often start a project with a single test: find a function in your code which you can run against a unit test (no database required), and write your first test. In Drupal, you can use a Simpletest Unit test (as in this example) and then run it straight from the browser.

Once you’re satisfied, add this line to your CI job so the test is run on every push:

drush test-run mytestgroup

Once that is done, it becomes easier for developers to write their own tests by adding it to the test file already present.

Tip #5: Don’t overestimate how good a developer you are

We all think we’re good developers, and really we can’t imagine anything ever going wrong with our code, I mean, it’s so elegant! Well, we’re wrong.

I’ve seen really intelligent people write code which looks really elegant, but still breaks.

I’ve seen developers never write tests for the simple stuff because it’s too simple, and never write tests for the more complex stuff because they never practiced with the simple stuff.

Even though you’re positive your code is so robust it will never break, just test it.

Tip #6: Start with the low-hanging fruit

This is an error I made myself and which proved very painful. Consider a system with three possible use cases for the end user. Each use case uses the same underlying calls to the database, and the same underlying pure functions.

Now, let’s say you are using a high-level testing framework like Behat and Selenium to test the rich user interface and you write three tests, one for each use case. You think (wrongly, as we’ll see) that you don’t need unit tests, because whatever it is you want to test with your unit tests is already tested by your high-level rich user interface tests.

Don’t forget, your specs also call for you to support IE8, IE9, Webkit (Safari) and Firefox. You can set up Jenkins to run the rich GUI tests via Selenium Grid on a Windows VM, and other fancy stuff.

This approach is wrong, because when you start having 5, 8, 10, 20 use cases, you will be tempted to continue just implement dozens of new, expensive rich GUI tests, and your tests will end up taking hours.

In my experience, if your entire test suite takes more than two hours to run, developers will start resenting the process and ignoring the test results, and you are back to square one.

In his book Succeeding with Agile, Mike Cohn came up with the idea of a test pyramid, as shown in the diagram below (you can learn more about the concept in this blog post).

Based on this concept, we quickly realize that:

  • Several steps are redundant among the GUI use cases.
  • The exact same underlying functionality is tested several times over.

Thinking of this from a different angle, we can start by testing our pure functions using unit tests. This will make for lightning-fast tests, and will get the team into the habit of not mixing UI functions, database functions and pure functions (for an example of what not to do, see Drupal’s own block_admin_display_form_submit).

Once you have built up a suite of unit tests which actually has value, move on to the next step: tests which require the database. This requires some variation of a site deployment module or another technique to bring the database to a known-good starting point before you run the test; it is harder to grasp and setting up a CI job for these types of tests is difficult too. However, your team will more likely be willing to work hard to overcome these obstacles because of the success they achieved with unit tests.

All of the above can be done with Drupal’s core simpletest.

Finally, when you are satisfied with your unit test suites and your database tests, you can move outside of Drupal and on targeted tests (not all usecases, only a few to make sure your widgets work) with Behat, Mink, Selenium, Windows/IE VMs. If you start with the fancy stuff, though, or have too much of it, the risk of failure is much greater.

Tip #7: Don’t underestimate developers’ ability to avoid writing tests

If you implement all the tips you’ve seen until now in this article, something curious will happen: no one will write any tests. Not even you.

Here’s the psychology behind not writing tests:

  • You really have the intention of writing tests, you just want to get your feature working first.
  • You work hard at getting your feature ready for the end-of-sprint demo.
  • You show off your feature to the team and they like it.
  • You don’t write any tests.

The above will happen to you. And keep in mind, you’re actually very interested in automated testing (enough to have read this article until now!). Now imagine your teammates, who are less interested in automated testing. They don’t stand a chance.

These are some techniques to get people to write tests:

The first is used by the Drupal project itself and is based on peer review of patches. If you submit a patch to core and it does not contain tests, it will not make it in. This requires that all code be reviewed before making it into your git repo’s stable branch. There are tools for this, like Phabricator, but I’ve never successfully implemented this approach (if you have, let me know!).

The second approach is to write your tests before writing a new feature or fixing a bug. This is known as test-driven development (TDD) and it generally requires people to see things from a different angle. Here is a typical scenario of TDD:

  • A bug comes in for project xyz, and you are assigned to it.

  • You write a test for it. If you don’t know something (no function exists yet, so you don’t know what it’s called; no field exists yet, so you don’t know how to target it), just put something feasible. If you’re dealing with the body field in your test, just use body. Try to test all conceivable happy paths and sad paths.

  • Now switch modes: your goal is to make the test pass. This is an iterative process which entails writing code and changing your test as well (your test is code too, don’t forget!). For example, perhaps the body field’s machine name is not body but something like field_body[und][0]. If such is the case, change the test, as long as the spirit of the test remains.

The above techniques, and code coverage tools like code_coverage or the experimental cover, which I like, will help you write tests, but changing a team’s approach can only be achieved through hard work, evangelizing, presentations, blogging, and the like.

Tip #8: Don’t subvert your process

When it becomes challenging to write tests, you might figure that, just this once, you’ll not test something. A typical example I’ve seen of this, in project after project, is communication with outside systems and outside APIs. Because we’re not controlling the outside system, it’s hard to test it, right? True, but not impossible. If you’ve set aside enough time in your estimates to do things right, you will be able to implement mock objects, making sure you test everything.

For example, in this blog post, I demonstrate how I used the Mockable module to define mock objects to test integration between Drupal and a content deployment system.

You will come across situations where implementing testing seems very hard, but however much effort I put into implementing automated testing for something, I have never regretted it.

Bonus tip: the entire team should own the tests

Your tests cannot be imposed by any one member of the team if they are to succeed. Instead, agree on what should be tested during your sprint planning.

For example, some developers (myself included) like to have close to zero Drupal styling errors. Others don’t really see the point of using two spaces instead of a tab. Unless you agree on what defines a failure (more than 100 minor styling errors? 1000? No threshold at all?), developers will feel resentful of having to fix it.

Because in Agile, your client is part of team as well, it is a good idea to involve them in defining what you are testing, providing them with the costs and benefits of each test. Perhaps your client doesn’t know what a MySQL query is, but if told that keeping the number of queries to less than 100 on the home page (something that can be tracked automatically) will keep performance up, they will be more likely to accept the extra cost associated.

Conclusion

Automated testing is about much more than tools (often the tools are quite simple to set up). The human aspect and the methodology are much more important to get your automated testing project off the ground.

[1] See Jez Humble and David Farley’s Continuous Delivery, Addison Wesley.

Please enable JavaScript to view the comments powered by Disqus.

Jan 20 2014
Jan 20

January 20, 2014

Drupal uses incremental IDs for such data as taxonomy terms and nodes, but not content types or vocabularies. If, like me, you believe your site’s codebase should work with different environments and different databases, your incremental IDs can be different on each environment, causing your code to break.

But wait, you are thinking, I have only one environment: my production environment.

Even if such is the case, there are advantages to be able to spawn new environments independently of the production environment without cloning the database upstream:

  • Everything you need to create your website, minus the content, is under version control. The production database, being outside version control, should not be needed to install a new environment. See also “what is a deployment module?”.
  • New developers can be up and running with a predictable environment and dummy content.
  • Your automated tests, using Drupal’s Simpletest, by default deploy a new environment without cloning the database.
  • For predictable results in your continuous integration server, it is best to deploy a new envrionment. The production database is unpredictable and unversioned. If you test it, your test results will be unpredictable as well.
  • Maybe in the future you’ll need a separate version of your site with different data (for a new market, perhaps).

Even if you choose to clone the database upstream for development, testing and continuous integration, it is still a good idea to avoid referencing incremental IDs of a particular database, because at some point you might decide that it is important to be able to have environments with different databases.

Example #1: using node IDs in CSS and in template files

I have often seen this: particular pages (say, nodes 56 and 400) require particular markup, so we see template files like page--node--56.tpl.php and css like this:

.page-node-56 #content,
.page-node-400 #content {
   ...
}

When, as developers, we decide to use this type of code on a website, we are tightly coupling our code, which is under version control, to our database, which is not under version control. In other words our project as a whole can no longer be said to be versioned as it requires a database clone to work correctly.

Also, this creates all sorts of problems: if, for example, a new node needs to be created which has the same characteristics as nodes 56 and 400, one must fiddle with the database (to create the node) and the code. Also, creating automatic tests for something like this is hard because the approach is not based on underlying logic.

A better approach to this problem might be to figure out why nodes 56 and 400 are somehow different than the others. The solution will depend on your answer to that question, and maybe these nodes need to be of a different content type; or maybe some other mechanism should be used. In all cases, though, their ID should be irrelevant to their specificity.

Example #2: filtering a view by taxonomy tag

You might have a website which uses Drupal’s default implementation of articles, with a tag taxonomy field. You might decide that all articles tagged with “blog” should appear in your blog, and you might create a new view, filtered to display all articles with the “blog” tag.

Now, you might export your view into a feature and, perhaps, make your feature a dependency of a site deployment module (so that enabling this module on a new environment will deploy your blog feature, and do everything else necessary to make your site unique, such as enabling the default theme, etc.).

It is important to understand that with this approach, you are in effect putting an incremental ID into code. You view is in fact filtering by the ID of the “blog” taxonomy term as it happens to exist on the site used to create the view. When creating the view, we have no idea what this ID is, but we are saying that in order for our view to work, the “blog” taxonomy term needs to be identical on all environments.

Here is an example of how this bug will play out:

  • This being the most important feature of your site, when creating new environments, the “blog” taxonomy term might always have the ID 1 because it is the first taxonomy term created; you might also be in the habit of cloning your database for new environments, in which case the problem will remain latent.
  • You might decide that such a feature is too “simple” to warrant automated testing; but even if you do define an automated test, your test will run on a new database and will need to create the “blog” taxonomy term in order to validate. Because your tests are separate and simple, the “blog” taxonomy term is probably the only term created during testing, so it, too will have ID 1, and thus your test will pass.
  • Your continuous integration server which monitors changes to your versioned code will run tests against every push, but, again, on a new database, so your tests will pass and your code will be fine.

This might go on for quite some time until, on a given environment, someone decides to create another term before creating the “blog” term. Now the “blog” term will have ID #2 which will break your feature.

Consider, furthermore, that your client decides to create a new view for “jobs” and use the same tag mechanism as for the blog; and perhaps other tags as well. Before long, your entire development cycle becomes dependent on database cloning to work properly.

To come up with a better approach, it is important to understand what we are trying to accomplish; and what taxonomy terms are meant to be used for:

  • The “blog” category here is somehow, logically, immutable and means something very specific. Furthermore, the existence of the blog category is required for our site. Even if its name changes, the key (or underlying identity) of the blog category should always be the same.
  • Taxonomy terms are referenced with incremental IDs (like nodes) and thus, when writing our code, their IDs (and even their existence) cannot be counted upon.

In this case, we are using taxonomy terms for the wrong purpose. Taxonomy terms, like nodes, are meant to be potentially different for each environment: our code should not depend on them.

A potential solution in this case would be to create a new field for articles, perhaps a multiple selection field, with “blog” as one of the possible values. Now, when we create a view filtered by the value “blog” in our new field, we are no longer referencing an incremental ID in our code.

I myself made this very mistake with my own website code without realizing it. The code for this website (the one you are reading) is available on Github and the issue for this problem is documented here (I’ll try to get around to fixing it soon!).

Deploying a fix to an existing site

If you apply these practices from the start of a project, it is relatively straightforward. However, what if a site is already in production with several articles already labelled “blog” (as is the case on the Dcycle website itself)? In this case we need to incrementally deploy the fix. For this, a site deployment module can be of use: in your site deployment module’s .install file, you can add a new update hook to update all your existing articles labelled “blog”, something like:

/**
 * Use a machine name rather than an incremental ID to display blog items.
 */
function mysite_deploy_update_7010() {
  // deploy the new version of the view to the target site
  features_revert(array('mysite_feature' => array('views_view')));
  ...
  // cycle through your nodes and add "blog" to your new field for any
  // content labelled "blog".
}

Of course, you need to test this first with a clone of your production site, perhaps even adding an automatic test to make sure your function works as expected. Also, if you have a lot of nodes, you might need to use the “sandbox” feature of hook_update_n(), to avoid timeouts.

Once all is tested, all that needs to be done, on each environment (production, every developer’s laptop, etc.), is run drush updb -y on the command line.

Conclusion

Drupal makes it very easy to mix incremental IDs into views and code, and this will work well if you always use the same database on every environment. However, you will quickly run into problems if you want to write automated tests or deploy new sites without cloning the database. Being aware of this can help you write more logical, consistent and predictable code.

Please enable JavaScript to view the comments powered by Disqus.

Jan 07 2014
Jan 07

January 07, 2014

It is generally agreed that cloning the database downstream (that is, from development toward production) is a bad idea, if only because by doing so all production content is lost; most developers use Features, Context, some variation on a site deployment module, or a rudimentary written procedure to move new configuration downstream.

However, in a dev-stage-production workflow, the database is often still periodically cloned back upstream:

In such an approach, anything not in Features or a site deployment module exists solely in the database. For example: any content, your default theme, and other information (such as variables not exported with Strongarm or block placement information not exported with Context) are defined only in your database and not in code. Therefore, to create a realistic development environment, it is tempting to clone your database.

I’ll explain why I think database cloning is the wrong approach, and then look at other ways to achieve the same goals. Finally, I’ll look at some situations where cloning the database is a good idea.

Why is cloning the database the wrong approach?

Cloning the database is wrong for the following reasons:

  • The database is not under version control.
  • The database is not a known-good starting point.
  • Database cloning makes automated testing harder.
  • Database cloning makes continuous integration harder.
  • What if there is more than one “production” site?
  • Your production database may be very large.
  • Your production database may contain sensitive data.
  • Fixes to a cloned database will “work on my machine”, but not elsewhere.

The database is not under version control

In Drupal, the database contains all configuration, content types, variables, views, and content; and none of this is under version control.

A good development practice is to put everything except content into code, and into version control, via Features, Context, Strongarm, and a site deployment module. These are code and can be kept under version control.

The database is not a known-good starting point

One important aspect of writing modern software is the importance of automated testing, and the importance of knowing that our test will always yield the same result. This is the concept of a known good starting point, discussed in the book Continuous Delivery. The production database changes continually, for example when new comments or content are added. If your tests, either manual or automated, depend on a cloned production database, there is always a chance that different versions of the database will be yield different test results.

Database cloning makes automated testing harder

Because of the importance of having a known-good starting point, Drupal automated tests which require the database always work in the following manner:

  • Build a brand-new temporary (throw-away) database from scratch.
  • Perform a plain installation.
  • Create the required content and set the required configuration.
  • Perform the test.
  • Discard the throw-away database.

For example, let’s say you have a block appear when there are more than 20 registered users on your site. The only way to accurately test this is to have your test control the number of users, and test the presence or absence of your block. If the only way to deploy a new environment with your site is to clone the database, the test has no real way of creating the conditions (active theme, block placement, active modules) to run this test.

However, if you are using Features and a site deployment module, all your tests needs to do for the above example is to:

  • (1) Enable your site deployment module.
  • (2) Make sure the special block does not appear.
  • (3) Create the 20 users.
  • (4) Make sure the block does appear.

Database cloning makes continuous integration harder

Continuous integration (CI) and continuous deployment are quite popular these days, with good reason, because without CI, automated testing is not that useful (because developers tend to ignore tests).

Basically, CI runs a script on every push to version control. So: every time there is a change to the code base, the tests can be run and either pass or fail.

I have seen many shops experiment with continuous integration, and in many cases the Drupal site is recreated by cloning the production database. Therefore, the CI server’s test site is always in an unknown, unversioned state. So when a test fails, it is impossible to say whether a change to the database caused the fail, or a change to the code did.

In my experience this causes frustration and confusion, and eventually will cause your CI server to be worthless, and hence abandoned.

What if there is more than one “production” site?

When we are cloning the production site’s database, what do we mean exactly? Take the following example: we are developing a code base for a university with dozens of faculties. Each faculty uses the same code base but a different theme, and some slightly different settings.

It doesn’t make sense for new developers to clone one production database rather than another for development, so often a random choice is made, leading to uncertainty.

Consider your codebase to be a software product which can be deployed on any number of sites, just as any software. Would it make sense for developers of a word processor to clone the computer of one of their clients during routine development?

Your production database may be very large

Beyond the logical considerations, cloning production databases can be unwieldy, requiring one to remove cache tables, finding a mechanism to either copy all files and images, ignore them, or use placeholder files and images (that does not feel right, no?). Still, you can quickly find yourself with very large databases.

Your production database may contain sensitive data

Once your production site actually starts being used, you end up with much sensitive data there: email addresses, hashed passwords, order history, addresses, or worse. Consider the consequences if you dump this database on a developer’s laptop (which will eventually be stolen or lost).

Fixes to a cloned database will “work on my machine”, but not elsewhere

So you’ve cloned a database on your laptop, and you changed some configuration on administration pages, and now the problem seems fixed, you’ve made a demo for your team and your client. The next part is messy though: a list of admin screens to click through on the production site to reproduce the fix (ugh!), or, as I’ve already seen, cloning the development database downstream (double-ugh!). Both methods are error-prone and do not record the fix in version control, so a month from now you’ll forget how it was done. In fact, you will find yourself in a sysyphian effort of repeatedly fixing the same problem over and over, and explaining to your clients and your team, with the help of out-of-date wiki pages, email exchanges and undecipherable comments on issue queues, that you are not an incompetent oaf.

What are the alternatives to database cloning?

We generally clone the database to have a realistic development environment. Among other things, during development, we need to have:

  • The same configuration and features.
  • Realistic content.
  • Some exact problem-causing content.

This is possible without cloning the database. Here are some tips and techniques.

Getting the same configuration and features as production

In an ideal world any Drupal site should be deployable without cloning the database, by getting the code from git and enabling the site deployment module.

You are most likely, however, to inherit a site which is a mess: no site deployment module, no tests, with Features, if they exist at all, likely to be overridden on the production site. On some projects you’d be lucky to even have a git repo.

One might think that for such sites, which we’ll call legacy sites for the purpose of this article, cloning the production database is the only viable option. Unfortunately, that is true, but it should only be a temporary solution, to give you time to extract the important configuration into code, and to create a site deployment module.

Let’s say, for example, I get a work request to “fix a little bug on a site which is almost ready”. The first thing I do is to clone the entire site to my laptop, with the database and all, and and determine which configurations, features, and variables are affected by the bug. Let’s say the site in question has 20 content types, 20 views, 50 enabled modules, three languages and a custom theme.

But the bug in question only affects 2 content types, one view, 3 modules and does not require the custom theme or i18n. I would start by generating a feature (if one does not exist) with the required views and content types, and a site deployment module with the feature as a dependency and a basic automated test. Now I can use test-driven development to fix my bug, push everything back to version control and to my continuous integration server, and deploy to production using drush.

Thus, every time an issue is being worked on, a site gradually moves from being a legacy site to a modern, tested site with continuous integration (don’t do it all at once as you will get discouraged).

Realistic content

For developers, Devel’s devel_generate module is great for generating lorem ipsum content with dummy images, so even if you don’t clone your database, you can still get, say, 50 (or 1000) blog posts with 5 (or 50) comments each.

During automated testing, several DrupalWebTestCase API functions allow you to create as much dummy content as you want, being as specific as you want.

Some exact problem-causing content

I have recently had to deal with a bug where the a site’s “layout was periodically going berserk”. That was the exact issue title, and I was lucky because my client was thoughtful enough to provide a screenshot and even the source code.

This problem could be tracked down to a often-seen misconfiguration of views and marked-up content: views would trim all body fields to 100 characters, which works fine with standard lorem ipsum, but in the real world the client was using markup in the content, so if a <div> tag would appear before the 100 character mark, but end after it, the ending tag would be omitted, screwing up the html.

Several colleagues who are used to cloning the database concluded that this a limitation of generated content.

I see this situation as more of an opportunity, and have come up with a way of altering generated lorum ipsum to suit your needs. So, when starting to work on such an issue, first make sure that your generated content better reflects real content, both for developers and for the automated tests.

When is it OK to clone the database?

“Don’t clone the database” is a good rule of thumb, but in some cases cloning the database is good idea, for example in the following cases:

  • For backups and restores.
  • For hard-to-debug “production-only” problems.
  • As a temporary measure to update a legacy site.
  • For proproduction environments.

For backups and restores

Code is not everything. The database contains your content, so you need to have a strategy to clone your database somewhere nightly, test it often, and make sure you can restore it. This is mot easily done by cloning the database.

For hard-to-debug “production-only” problems

Once in a while, you will have a problem which only manifests itself on a production site. Reproducing this type of problem systematically can be best achieved by cloning your production database to figure out what the problem is (never work directly on production, of course).

As a temporary measure to update a legacy site

As mentioned in “Getting the same configuration and features as production”, above, most projects are a complete mess once you get your hands on them. We’ll call these legacy sites. The only way to move important configuration information into code is often to clone these sites temporarily until you have working Features and a site deployment module.

For proproduction environments

For some critical projects, you might decide to continually deploy, but not directly to production. In such circumstances, you might have your Jenkins projects continually deploy to a preproduction site (cloned from production before each deployment), to give the team, and the client, a few hours or a day to walk through the changes before approving them for deployment to production.

Conclusion

Since being interested in Drupal dev-stage-prod, deployment and testing, I have often come across colleagues who systematically cloned the database, and have always felt uneasy about it, and in writing this post I have set out to explain why. The post turned out a lot longer than I thought, and the main take-away is that we should all consider our sites as software products, not single-use sites.

As software products, we need standardized deployment methods, both initial and incremental, via a site deployment module.

As software products, we also need to implement modern testing and continuous integration techniques.

As software products, we need to be able to deploy anywhere, with no environment dependant on any other.

Such a focus on reproducibility will hopefully pave the way to more dependable tests, a better understanding of what is content and what is configuration, and faster, more efficient and more consistent development.

Please enable JavaScript to view the comments powered by Disqus.

Dec 13 2013
Dec 13

December 13, 2013

Edit (2016-10-03): This website is no longer Drupal-based.

Deployments are often one of the most pain-inducing aspects of the Drupal development cycle. I have talked to Drupal developers in several shops, and have found that best practices are often ignored in favor of cloning databases downstream, manually reproducing content on prod environments, following a series of error-prone manual steps on each environment, and other bad practices, all of which should be thrown out the door.

In this article I am referring to deployment of site configuration (not content) on a Drupal 7 site. Configuration refers such aspects of your site as the default theme, CSS aggregation status, content types, views, vocabularies, and the like.

Taking best practices to the extreme, it is possible to deploy continually, dozens of times a day, automatically. The following procedure is a proof of concept, and you will probably want to adapt it to your needs, introducing a manual step perhaps, if only to make sure your deployments happen on fixed schedule.

Still, I have started using the exact procedure discussed herein to deploy the website you are currently reading, dcycleproject.org. Furthermore, the code is on Github, so anyone can reproduce the dcycle project site, without the content (I’ll detail how later on).

In an effort to demonstrate that the principles of the Dcycle manifesto work well for a real — albeit simple — website, I have started to deploy changes to the Dcycle website itself via a site deployment module with automatic testing, a continuous integration server (Jenkins), and continuous deployment. In fact, if you visit the dcycle website, you might come across a maintenance page. This is a deployment in action, and chances are it’s happening automatically.

So what is continuous deployment? For our purposes, it is a site development method which follows these principles:

  • The site’s code is under version control (we are using Git).
  • Our site is deployed, initially and incrementally, via a site deployment module.
  • Automatic testing confirms that the features we have developed actually work.
  • Two branches exist: master, on which development occurs, and prod, considered stable, on which all tests have passed.
  • A continuous integration server (in our case Jenkins) monitors the master branch, and moves code to the prod branch only if automated tests pass.
  • The production site is never changed directly, but via a job in the continuous integration server.
  • Once the prod branch is updated, so is the production site.
  • Databases are never cloned, except to move legacy sites to your local development environment. (A legacy site, in the context of this article, is a site which you can’t deploy (minus the content) without cloning the database).

A typical workflow happens like this:

  • Code is committed and pushed to the master branch in the git repo.
  • Jenkins picks up on the change, runs the tests, and they fail. The prod branch and the production site are untouched.
  • The problem leading to the failing test is fixed, and the code is pushed, again to the master branch in the git repo.
  • Jenkins picks up on the change, runs the tests, and this time they pass. The prod branch is updated automatically, and the production site itself is updated. Automatically.

Before getting started, make sure you have the following.

  • A continuous integration server, which can be on your laptop if you wish. Jenkins is easy to set up, and took me five minutes to install on Mac OS, and another five minutes on CentOS. Just follow the instructions.
  • A central git repo. You can fork the code for the dcycleproject.org website if you like.
  • A webserver on your laptop. I am using MAMP.
  • Access to your production website on the command line via SSH.
  • SSH public-private key access to the production server, to avoid being asked for passwords. This is important for Jenkins to modify the production server automatically.

We won’t be using git hooks or Drupal’s GUI.

More often than not, we are working on existing Drupal sites, not new ones, and we don’t have the luxury of redeveloping everything with best practices. So we’ll start with a single issue, either a bug or feature request. Here is a real-life example for the Dcycle website:

I like the idea of each article having its ID reflected in the URL, as is the case with Stack Overflow. I want the path of my articles to be in the format blog/12345/title-of-the-post. I also want it to be possible to shorten the path and have it redirect the full path, so for example blog/12345 redirects to blog/12345/title-of-the-post, as is the case on Stack Overflow.

So, I started out with the goal of implementing this feature using continuous deployment and automated tests.

If your site has a site deployment module or something like it, download your code from git and deploy the site locally, using these commands, substituting your own site deployment module name and database credentials for those in the example:

echo 'create database example' | mysql -uroot -proot
drush si --db-url=mysql://root:[email protected]/example --account-name=root --account-pass=root
drush en example_deploy -y

If you want to try this at home and create a local version of the Dcycle website, make sure you have a webserver, PHP and MySQL installed, and run the following commands (if you want to actually modify the code, fork it first and use your project URL instead of mine). This example uses MAMP.

cd /Applications/MAMP/htdocs
git clone https://github.com/alberto56/dcyclesite.git dcyclesample
cd dcyclesample
echo 'create database dcyclesample' | mysql -uroot -proot
drush si --db-url=mysql://root:[email protected]/dcyclesample --account-name=root --account-pass=root
drush en dcycle_deploy -y

The above will yield an empty website. Adding some generated content will make development easier:

drush en devel_generate -y
drush generate-content 50

If there is no site deployment site, you can clone the database, but don’t make a habit of it!

To work well with continuous deployment, your site needs to have a consistent way of being initially and incrementally deployed. To achieve this, I recommend the use of a site deployment module.

Create one for your site (one already exists for the Dcycle website code, if you are using that), and make sure the .install file contains everything necessary to deploy your site. To make sure initial deployment and incremental deployment result in the same state, I just call all my update hooks from my install hook, and that has worked fine for me. Your .install file might look something like:

/**
 * @file
 * sites/default/modules/custom/dcycle_deploy/dcycle_deploy.install
 * Initial and incremental deployment of this website.
 */

/**
 * Implements hook_install().
 */
function dcycle_deploy_install() {
  for ($i = 7001; $i < 8000; $i++) {
    $candidate = 'dcycle_deploy_update_' . $i;
    if (function_exists($candidate)) {
      $candidate();
    }
  }
}

/**
 * Admin menu
 */
function dcycle_deploy_update_7007() {
  module_enable(array('admin_menu_toolbar'));
  module_disable(array('toolbar'));
}

...

/**
 * Set dark_elegant as theme
 */
function dcycle_deploy_update_7015() {
  theme_enable(array('dark_elegant'));
  variable_set('theme_default', 'dark_elegant');
}

The above code sets the default theme and changes the toolbar to admin_menu_toolbar, which I prefer.

Because these features were deployed at different times (the theme was changed after the toolbar was changed), numbered update hooks are used.

Notice how the install hook cycles through all the update hooks, ensuring that our initial deployment and incremental deployments result in the same state. For any given environment, now, regardless of the previous state, bringing it up to date is simply a matter of updating the database. The following script can now be used on any environment:

drush vset maintenance_mode 1
drush updb -y
drush cc all
drush vset maintenance_mode 0

The above puts the site in maintenance mode, runs all the update hooks which have not yet been run, clears the caches and takes the site out of maintenance mode.

We now have standardized deployment, both initial and incremental.

For continuous deployment to be of any use, we need to have very high confidence in our tests. A good first step to that end is for our tests to actually exist. And a good way to ensure that your tests exist is to write them before anything else. This is Test-driven development.

If you cloned my git repo for this site, the “short path” feature, introduced above, has already been implemented and tested, so the test passes. Still, here is the code I had written, which initially was failing. You might want to write something similar, or add a test...() function to your .test file, for another feature.

/**
 * @file
 * sites/default/modules/custom/dcycle_deploy/dcycle_deploy.test
 * This file contains the testing code for this module
 */

// Test should run with this number of blog posts.
define('DCYCLE_DEPLOY_TEST_BLOG_COUNT', 5);

/**
 * The test case
 */
class dcyclesiteTestCase extends DrupalWebTestCase {
  /**
   * Info for this test case.
   */
  public static function getInfo() {
    return array(
      'name' => t('dcyclesite: basic test'),
      'description' => t('describe test.'),
      'group' => 'dcyclesite',
    );
  }

  /*
   * Enable your module
   */
  public function setUp() {
    // set up a new site with default core modules, dcyclesite, and
    // dependencies.
    parent::setUp('dcycle_deploy');
  }

  /*
   * Test case for dcyclesite.
   */
  public function testModule() {
    $this->loginAsRole('administrator');
    $blogs = array();
    for ($i = 1; $i <= DCYCLE_DEPLOY_TEST_BLOG_COUNT; $i++) {
      $this->drupalCreateNode(array('type' => 'article', 'title' => 'É' . $blogs[$i] = $this->randomName()));
      foreach (array('blog', 'node') as $base) {
        // passing alias => TRUE, otherwise, the test converts our call
        // to the alias before the query.
        $this->drupalGet($base . '/' . $i, array('alias' => TRUE));
        // assertUrl() does not work here, because the current url (node/1)
        // equals node/1 and equals also its alias. We want it to equal its
        // alias only.
        $url = $this->getUrl();
        global $base_url;
        $expected = $base_url . '/blog/' . $i . '/e' . strtolower($blogs[$i]);
        $this->assertEqual($url, $expected , format_string('Blog can be accessed using @base/x and will redirect correctly because the end url (@url) is equal to @expected.', array('@base' => $base, '@url' => $url, '@expected' => $expected)));
      }
    }
  }

  /*
   * Login as administrator role.
   *
   * This can be a useful for tests in your deployment module, especially
   * if you create several roles in a Feature dependency.
   *
   * @param $role = 'administrator'
   *   Log in as any role, or administrator by default.
   */
  public function loginAsRole($role = 'administrator') {
    // Get all of the roles in the system.
    $roles = user_roles();
    // Find the index for the role we want to assign to the user.
    $index = array_search($role, $roles);
    // Get the permissions for the role.
    $permissions = user_role_permissions(array(array_search($role, $roles) => $role));
    // Create the user with the permissions.
    $user = $this->drupalCreateUser(array_keys($permissions[$index]));
    // Assign the role.
    $user->roles[$index] = $role;
    // Log in as this user
    if (!($user = user_save($user))) {
      throw new Exception(format_string('cannot save user with role @r', array('@r' => $role)));
    }
    $this->drupalLogin($user);
  }

}

Don’t forget to reference your test in your .info file:

...
files[] = dcycle_deploy.test
...

What are we doing in the automatic test, above?

Take a look at the setUp() function, which does everything required to create a new environment of this website. Because we have used a site deployment module, “everything” is simply a matter of enabling that module.

The key here is that whether the scenario works or not on any given environment (local, prod, etc.) is irrelevant: it needs to work based on a known good starting point. Databases are moving targets and it is thus irrelevant to test your code against an existing database (except if you want to monitor your production environment, which is a more advanced use case and outside the scope of this article). Therefore, we need to bring the throw-away testing database to a point where we can test run a test, and know that our test will always yield the same result. The concept of a known good starting point is discussed in the book Continuous Delivery.

Given a new throw-away environment, testModule() now runs the test, defining a scenario which should work: we are logging in as an administrator, creating new blog posts (making sure to use foreign characters in the title), and then making sure the foreign characters are transliterated to ASCII characters and that our content redirects correctly when using only the ID. Let’s enable Simpletest now and make sure our test is visible and fails:

drush en simpletest -y

Now log into your site, and visit admin/config/development/testing, and run your test. If the desired functionality has not yet been developed, your test should fail.

At this point let’s switch gears and focus our energy on making our test pass. This normally involves several code iterations, and running the test dozens of times, until it passes.

An important note for test-driven development: the initial test is an approximation, and may have to be modified during coding. The spirit of the test, as opposed to the letter of the test, should be conserved.

Test-driven development has the interesting side effect that it makes it easier for teams to collaborate: if I am working with a team in a different time zone, it is less error-prone for me to instruct them to “make the test work on branch xyz and then merge it to master”, rather than explain everything I have in mind.

In the case of the task at hand, I wrote some custom code in a new module, dcyclesite, and then enabled some new modules and configuration. Don’t forget, all operations which modify the database have to be done in update hooks. Here is a partial example of how my site deployment module’s .install file looks after I made the test pass:

/**
 * Enable some custom code
 */
function dcycle_deploy_update_7022() {
  # this is where my custom code is; check my github code if you
  # are curious.
  module_enable(array('dcyclesite'));
}

/**
 * Pattern for articles
 */
function dcycle_deploy_update_7023() {
  variable_set('pathauto_node_article_pattern', 'blog/[node:nid]/[node:title]');
}

/**
 * Enable transliteration
 */
function dcycle_deploy_update_7024() {
  module_enable(array('transliteration'));
}

Coming back to continuous deployment, we need our tests to be run every time code is pushed to our master branch, and running our tests will eventually be done by our Jenkins server.

The idea behind Jenkins is quite simple: run a script as a reaction to an event. The event is a change in the master branch. Jenkins, though, does not know how to fiddle around in a GUI. Therefore we must make it possible to run the tests in the command line. Fortunately this is quite easy:

drush test-run dcyclesite

The above runs all tests in the test group “dcyclesite”. Change “dcyclesite” to whatever your group name is. For tests to run correctly from the command line, you must make sure you base_url is set correctly in your sites/default/settings.php file. This depends on your environment but must reflect the URL at which your site is accessible, for example:

$base_url = 'http://localhost:8888/mywebsite';

Now, running drush test-run dcyclesite should fail if there is a failing test. I normally develop code using Simpletest in the GUI, and use the command-line for regression testing in Jenkins.

Now the fun starts: create a Jenkins job to continuously deploy. Simply put, we need our jenkins server to monitor the master branch of git, and if everything passes, move our code to the production branch, and, if we are feeling particularly confident in our tests, deploy to the production site.

Jenkins is one of the easiest pieces of software I have ever installed. You can probably get a CI server up and running in a matter of minutes by downloading the appropriate package on the Jenkins website.

Once that is done, set up Jenkins:

  • Make sure the jenkins user on your Jenkins server has an SSH key pair, and has public-private SSH key access to both the git repo and the production server. Note that this will allow anyone with access to your Jenkins server to access your production server and your git repo, so apply security best practices to your Jenkins server!
  • In Jenkins’s plugin manager page, install the Git plugin and the post-build task plugin, which allows you to add a second script if the first script succeeds.

Now create a single job with the following attributes:

  • Source code management: git.
  • Source code repository URL: the complete URL to your git repo. If you get an error here, make sure you can access it via the command line (you might need to accept the server’s fingerprint). In the “Advanced…” section, set the name of your repo to origin.
  • Branches to build: master.
  • Build triggers: Poll SCM every minute (type “* * * * *”).
  • Add build step: execute shell: drush test-run dcyclesite. If you are on Mac OS X, you might have to add [ $? = 0 ] || exit $? as explained here, otherwise your job will never fail.
  • Add post-build action “git publisher”. Push only if build succeeds to the prod branch of your origin.
  • Add another action, “post-build task”, selecting “Run script only if all previous steps were successful”, and “Escalate script execution status to job status”. This is a script to actually deploy your site.

In the last script, Jenkins will log into your remote site, pull the prod branch, and update your database. You might also want to backup your database here. In my case I have a separate job which periodically backs up my database. Here is some sample deployment code.

# set the site to maintenance mode
ssh [email protected] "cd /path/to/drupal && drush vset maintenance_mode 1 &&
# get the latest version of the code
git pull origin prod &&
# update the database
drush updb -y &&
# set maintenance mode to off
drush vset maintenance_mode 0 &&
# finally clear the cache
drush cc all"

Note that you can also use rsync if you don’t want to have git on your production server. Whatever you use, the trick is for deployments to production to happen through Jenkins, not through a human.

Now save your job and run it. It won’t work yet; don’t worry, this is normal, we haven’t finished yet.

To run tests, Jenkins will need a database, but we haven’t yet set one up. It will also need HTTP access to its workspace. Let’s do all this now.

Return to configure your Jenkins job, and in “Advanced project options”, click “Advanced…”. Click “Use custom workspace” and set a path which will be available via an URL. For example, if your Jenkins server is on your Mac and you are using MAMP, you can set this to /Applications/MAMP/htdocs/mysite.jenkins. This workspace will be available, for example, via http://localhost:8888/mysite.jenkins/.

Switch to your jenkins user and install a plain Drupal site with the Simpletest module:

sudo su jenkins
echo 'create database mysitejenkins' | mysql -uroot -proot
drush si --db-url=mysql://root:[email protected]/mysitejenkins --account-name=root --account-pass=root

Note that you don’t need to deploy your site here using drush en example_deploy -y: all this site really is needed for is hosting tests. So we just need a plain Drupal site, with simpletest enabled:

drush en simpletest -y

Now set the base url of your workspace in sites/default/settings.php:

$base_url = 'http://localhost:8888/mysite.jenkins';

That’s basically all there is to it! However, because of the sheer number of steps involved, it is probable that you will run into a problem and need to debug something or other. I will appreciate hearing from you, noting any pitfalls and comments you may have. With the above steps in place, you will be able to make any change to the codebase, adding tests and function, test locally, and push to master. Then sit back and look at your Jenkins dashboard and your production site. If all goes well, you will see your job kick off, and after a few minutes, if you refresh your production site, you will see it is in maintenance mode. Some time later, your Jenkins job will end in success and your production site, with your new code, will be live again! Now bring your colleagues into the fold: this technique scales very well.

Please enable JavaScript to view the comments powered by Disqus.

Nov 22 2013
Nov 22

November 22, 2013

In a Drupal development-staging-production workflow, the best practice is for new features and bug fixes to be developed locally, then moved downstream to the staging environment, and later to production.

Just how changes are pushed downstream varies, but typically the process includes Features, manual changes to the production user interface, drush commands, and written procedures.

Some examples include:

  • A view which is part of a Feature called xyz_feature is modified; the feature is updated and pushed to the git repo; and then the feature is reverted using drush fr xyz_feature on the production site.
  • A new default theme is added to the development site and tested, and pushed to the git repo; and then the new theme is selected as default on the production site’s admin/appearance page.
  • Javascript aggregation is set on the dev site’s admin/config/development/performance page, and once everything works locally, it is set on the production via the user interface.

This approach is characterized by the following properties:

  • Each incremental deployment is different and must be documented as such.
  • If there exist several environments, one must keep track manually of what “remains to be done” on each environment.
  • The production database is regularly cloned downstream to a staging environment, but it is impossible to tell when was the last time it was cloned.
  • If an environment is out of date and does not contain any important data, it can be deleted and the staging environment can be re-cloned.
  • Many features (for example javascript aggregation) are never in version control, at best only documented in an out-of-date wiki, at worst in the memory of a long-gone developer.
  • New developers clone the staging database to create a local development environment.
  • Automated functional testing by a continuous integration server, if done at all, uses a clone of the staging database.

The main issue I have with this approach is that it overly relies on the database to store important configuration, and the database is not under version control. There is no way to tell who did what, and when.

Using a deployment module aims to meet the following goals:

  • Everything except content should be in version control: views, the default theme, settings like Javascript aggregation, etc.
  • Incremental deployments should always be performed following the same procedure.
  • Initial deployments (for example for a new developer or for a throwaway environment during an automated test) should be possible without cloning the database.
  • Tests should be run agains a known-good starting point, not a clone of a database.
  • New developers should be up and running without having to clone a database.

Essentially, anything not in version control is unreliable, and cloning the database today can yield a bug which won’t be present if you clone the database tomorrow. So we’ll avoid cloning the database in most cases.

The Dcycle manifesto states that each site should have a deployment module whose job it is to keep track of deployment-related configuration. Once you have settled on a namespace for your project, for example example, by convention your deployment module should reside in sites/*/modules/custom/example_deploy.

Let’s now say that we are starting a project, and our first order of business is to create a specific view: we will create the view, export it as a feature, and make the feature a dependency of our deployment module. Starting now, if all your code is under version control, all new environments (production, continuous integration, testing, new local sites) are deployed the same way, simply by creating a database and enabling the deployment module. Using Drush, you would call something like:

echo 'create database example' | mysql -uroot -proot
drush si --db-url=mysql://root:[email protected]/example --account-name=root --account-pass=root
drush en example_deploy -y

The first line creates the database; the second line is the equivalent of clicking though Drupal’s installation procedure; and the third line activates the deployment module.

Because you have set your feature to be a dependency of your deployment module, it is activated and your view is deployed.

We want the incremental deployment procedure to always be the same. Also, we don’t want it to involve cloning the database, because the database is in an unknown state (it is not under version control). Another reason we don’t want to clone the database is because we want to practice our incremental deployment procedure as must as possible, ideally several times a day, to catch any problems before we apply it to the production site.

My incremental deployment procedure, for all my Drupal projects, uses Drush and Registry rebuild, and goes as follows once the new code has been fetched via git:

drush rr
drush vset maintenance_mode 1
drush updb -y
drush cc all
drush cron
drush vset maintenance_mode 0

The first line (drush rr) rebuild the registry in case we moved module files since the last deployment. A typical example is moving contrib modules from sites/all/modules/ to sites/all/modules/contrib/: without rebuilding the registry, your site will be broken and all following commands will fail.

drush vset maintenance_mode 1 sets the site to maintenance mode during the update.

drush updb -y runs all update hooks for contrib modules, core, and, importantly, your deployment module (we’ll get back to that in a second).

drush cc all clears all caches, which can fix some problems during deployment.

On some projects, I have found that running drush cron at this point helps avoid hard-to-diagnose problems.

Finally, move your site out of maintenance mode: drush vset maintenance_mode 0.

Our goal is for all our deployments (features, bug fixes, new modules…) to be channelled though hook_update_N()s, so that the incremental deployment procedure introduced above will trigger them. Simply, hook_update_N() are functions which are called only once for each environment.

Each environment tracks the last hook_update_N() called, and when drush updb -y is called, it checks the code for new hook_update_N() and runs them if necessary. (drush updb -y is the equivalent of visiting the update.php page, but the latter method is unsupported by the Dcycle procedure, because it requires managing PHP timeouts, which we don’t want to do).

hook_update_N()s is the same tried-and-true mechanism used to update database schemas for Drupal core and contrib modules, so we are not introducing anything new.

Now let’s see how a few common tasks can be accomplished the Dcycle way:

Instead of fiddling with the production environment, leaving no trace of what you’ve done, here is an ideal workflow for enabling Javascript aggregation:

First, in your issue tracker, create an issue explaining why you want to enable aggregation, and take note of the issue number (for example #12345).

Next, figure out how to enable aggregation in code. In this case, a little reverse-engineering is required: on your local site, visit admin/config/development/performance and inspect the “Aggregate JavaScript files” checkbox, noting its name property: preprocess_js. This is likely to be a variable. You can confirm that it works by calling drush vset preprocess_js 1 and reloading admin/config/development/performance. Call drush vset preprocess_js 0 to turn it back off again. Many configuration pages work this way, but in some cases you’ll need to work a bit more in order to figure out how to affect a change programmatically, which has the neat side effect of providing you a better understanding of how Drupal works.

Now, simply add the following code to a hook_update_N() in your deployment module’s .install file:

/**
 * #12345: Enable javascript aggregation
 */
function example_deploy_update_7001() {
  variable_set('preprocess_js', 1);
  // you can also do this with Features and the Strongarm module.
}

Now, calling drush updb -y on any environment, including your local environment, should enable Javascript aggregation.

It is important to realize that hook_update_N()s are only called on environments where the deployment module is already in place, and not on new deployments. To make sure that new deployments and incremental deployments behave similarly, I call all my update hooks from my hook_install, as described in a previous post:

/**
 * Implements hook_install().
 *
 * See http://dcycleproject.org/node/43
 */
function example_deploy_install() {
  for ($i = 7001; $i < 8000; $i++) {
    $candidate = 'example_deploy_update_' . $i;
    if (function_exists($candidate)) {
      $candidate();
    }
  }
}

Once you are satisfied with your work, commit it to version control:

git add sites/all/modules/custom/example_deploy/example_deploy.install
git commit -am '#12345 Enabled javascript aggregation'
git push origin master

Now you can deploy this functionality to any other environment using the standard incremental deployment procedure, ideally after your continuous integration server has given you the green (or in the case of Jenkins, blue) light.

If we already have a feature which is a dependency of our deployment module, we can modify our view; update our features using the Features interface at admin/structure/features or using drush fu xyz_feature -y; then adding a new hook_update_N() to our deployment module:

/**
 * #12346: Change view to remove html tags from trimmed body
 */
function example_deploy_update_7002() {
  features_revert(array('xyz_feature' => array('views_view')));
}

In the above example, views_view is the machine name of the Features component affecting views. If you want to revert other components, make sure you’re using the 2.x branch of Features, visit the page at admin/structure/features/xyz_feature/recreate (where xyz_feature is the machine name of your feature), and you’ll find the machine names of each component next to its human name (for example node for content types, filter for text formats, etc.).

Say we create a new default theme xyz and want to enable it:

/**
 * #12347: New theme for the site
 */
function example_deploy_update_7003() {
  theme_enable(array('xyz'));
  variable_set('theme_default', 'xyz');
}

I normally remove toolbar on all my sites and put admin_menu’s admin_menu_toolbar instead. To deploy the change, add admin_menu to sites/*/modules/contrib and add the following code to your deployment module:

/**
 * #12348: Add a drop-down menu instead of the default menu for admins
 */
function example_deploy_update_7004() {
  // make sure admin_menu has been downloaded and added to your git repo,
  // or this will fail.
  module_enable(array('admin_menu_toolbar'));
  module_disable(array('toolbar'));
}

Of course, nothing prevents clueless users from modifying views, modules and settings on the production site directly, so I like to add hook_requirements() to perform certain checks on each environment: for example, if Javascript aggregation is turned off, you might see a red line on admin/reports/status saying “This site is designed to use Javascript aggregation, please turn it back on”. You might also check that all your Features are not overridden, that the right theme is on etc. If this technique is used correctly, when a bug is reported on the production site, the admin/reports/status page will let you know if any settings on the production site are not what you intended, and what your automated tests expect.

Now that everything we do is in version control, we no longer need to clone databases, except in some very limited circumstances. We can always fire up a new environment and add dummy content for development or testing; and, provided we’re using the same commit and the same operating system and version of PHP, etc., we’re sure to always get the same result (which is not the case with database cloning).

Specifically, I normally add a .test file in my deployment module which enables the deployment module on a test environment, and runs tests to make sure things are working as expected.

Once that is done, it becomes easy to create a Jenkins continuous integration job to monitor the master branch, and confirm that a new environment can be created and simpletests pass.

Please enable JavaScript to view the comments powered by Disqus.

Nov 13 2013
Nov 13

November 13, 2013

I recently inherited a Drupal project which periodically imported content from a Nuxeo server, synchronizing it with Drupal nodes, thus creating, updating and deleting nodes as need be. Nuxeo content was in no case modified by Drupal.

The Nuxeo server was set up by a third-party provider with whom I had no contact.

The site was not using mock objects or automated testing. A custom Drupal module was used, which leveraged the CMIS module.

A series of problems were occurring with the setup, among them:

  • Nuxeo Categories were supposed to map to Drupal taxonomy terms, but if a category was deleted from Nuxeo, the corresponding taxonomy term was not removed from the node in Drupal
  • If more than one category was added to a Nuxeo content, only the first was imported to Drupal
  • The site used what seemed like a custom implementation of the Nuxeo API, so it was hard to get help from the community. The custom implementation returns Nuxeo content IDs for some contents and Nuxeo revision IDs for others. After some testing, I did not manage to figure out in which circumstances content IDs or revision IDs were used.
  • The Nuxeo server’s clock was a few minutes late, and the custom module was comparing timestamps rather than revision numbers. As a result, if a Nuxeo content was modified less than five minutes after its previous modification, synchronisation did not occur correctly.

These problems had a common symptom from the client’s perspective: “synchronisation does not happen correctly”. They also caused a common emotion: frustration.

In order to implement a robust fix, trial and error was not enough. Here are the steps I followed to reproduce the problems, implement testing, and finally fix them.

Making sure I had access to a “staging” nuxeo folder allowed me do do testing without messing up production data; I could also control the number of content items, making testing that much faster. This was a convenient stop-gap measure until I could set up mock objects and internal testing.

This might go without saying, but of course you should have a local environment before attempting to modify Drupal. We use git for version control, and I just grabbed a copy of the production database to my local dev site. Because Drupal is only reading Nuxeo data, not modifying it, this is not too risky.

Before setting up a true mock object for interaction with Nuxeo, I first had to set up a level of abstraction, in effect a sort of switch, between Drupal and Nuxeo. Later on I would plug in a mock object.

The CMIS module is not using automated testing, and does not seem to allow for any form of mocking or simulation, from what I can tell (a search of the strings ‘mock’, ‘simulate’, or ‘simulation’ came up empty for that project’s code).

So now I had to decide between these two approaches:

  • provide a patch for CMIS providing mock object functionality.
  • provide a level of abstraction between the custom code and CMIS itself.

Because there is no real standard way (yet) to provide mock object functionality in Drupal modules, I have been using my own solution for a few projects: the Mockable module. Although I have released a beta version, the module is not widely used and I would rather wait for this or some other solution to be more accepted before submitting patches for third-party modules. I therefore decided to use Mockable between my own module and CMIS.

The Mockable module is not meant to be active on production sites. Here is how I used it:

First, I downloaded Mockable and activate the mockable module (but not the other modules in the Mockable project) on the local development site.

Then, I identified the lines of code in my custom module which interacted with an external system (in this case the CMIS module). Here is one example:

...
$object = cmisapi_getProperties('default',$document_id);
...

cmisapi_getProperties() is defined in CMIS and I did not want to modify that module, so I am going to mock it instead.

I started by installing Devel, and calling my custom code from the devel/php page with different sets of data on my Nuxeo staging environment. Adding a var dump or dpm() call helped me figure out the structure of the response from cmisapi_getProperties() in different circumstances:

...
$object = cmisapi_getProperties('default',$document_id);
// this should be removed after testing. dpm() is defined in the devel
// module.
dpm($object);
...

Once I had a good idea of how this function works, I defined a new set of functions instead of calling cmisapi_getProperties() directly:

...
$object = cmis_ms_get_properties('default',$document_id);
...

/**
 * Mockable version of CMIS's cmisapi_getProperties().
 */
function cmis_ms_get_properties($type, $document_id) {
  if (module_exists('mockable')) {
    // if the Mockable function is active, as it might be on testing
    // and dev environments (not on prod), then call cmisapi_getProperties()
    // or cmisapi_getProperties_mock() (if it exists) depending on whether
    // mocking is turned on or not.
    $return = mockable('cmisapi_getProperties', $address, $document_id);
  }
  else {
    $return = cmisapi_getProperties($address, $document_id);
  }
  return $return;
}

Now that my abstraction layer was in place, all I had to do was define some functions and objects to replace, in my developement and continuous environments, those used in production.

/**
 * Mock version of CMIS's cmisapi_getProperties(), which will be called
 * instead of cmisapi_getProperties() if the Mockable version is installed
 * and mocking is turned on (using drush mockable-set).
 */
function cmis_ms_get_properties_mockable_mock($type, $document_id) {
  $return = new stdClass;
  ...

  // Do whatever you want here to best simulate all possible responses of
  // the real cmisapi_getProperties()

  return $return;
}

cmisapi_getProperties() was not the only function which interacted with the third-party system. new SoapClient($address, $options) and other such calls were scattered across my code. I had to figure out how each of these worked and mock them appropriately.

Your mock objects or mock functions can be as simple or complex as you need them to be. In my case I am using variables to switch between different simulated behaviours. For example, I can easily simulate a timeout or 500 error on my remote system.

Now that my mock function is in place, I can turn on mocking. With the Mockable module, this can be done with a GUI or with Drush:

drush mockable-set

Starting now, I need to understand and reproduce exactly, in my mock object, how each error occurs.

In the case of taxonomy import problems, I can set cmis_ms_get_properties_mockable_mock() to always return two categories, and confirm that they don’t get imported into Drupal.

Reproducing the problem manually with a mock object is a step in the right direction, but we need to make sure that once it’s fixed, it stays fixed. To do that I added the following .test file to my custom module (and linked to it in my .info file).

/**
 * The test case
 */
class mymoduleTestCase extends DrupalWebTestCase {
  /**
   * Info for this test case.
   */
  public static function getInfo() {
    return array(
      'name' => t('mymodule: basic test'),
      'description' => t('describe test.'),
      'group' => 'mymodule',
    );
  }

  /*
   * Enable your module
   */
  public function setUp() {
    // set up a new site with default core modules, mymodule, and
    // dependencies.
    parent::setUp('mymodule', 'mockable');
  }

  /*
   * Test case for mymodule.
   */
  public function testModule() {
    // start using mock objects
    mockable_set();
  
    // sync with our mock version of Nuxeo, in which documents all have
    // two categories. Note that before adding this function to a test,
    // I had to modify it to use mockable functions and objects instead
    // of always interacting with external servers. Because we called
    // mockable_set(), above, if we have correctly defined mock objects,
    // the external server should not be hit at this point. A good way
    // of making sure is to deactivate internet access during the local
    // test.
    mymodule_sync_nuxeo();
  
    $node = node_load(1);
  
    $taxonomy_count = count($node->field_tags[LANGUAGE_NONE]);
    $this->assertTrue($taxonomy_count == 2, format_string('We were expecting 2 taxonomy terms and we have obtained @count', array('@count' => $taxonomy_count)));
  }
}

When I ran this test, I could confirm that the test failed because even though my mock object was defining two categories, only the first ended up as a taxonomy term on my node.

Now that I had a failing test, my job was to make sure the test passed. Now the trial and error phase can really being:

  1. Try something in code (note: both your test and your logic are “code”).
  2. Run the test.
  3. If the test still fails, make sure your test’s logic makes sense and go back to step 1.
  4. Do a manual test. If it fails, go back to step 1.
  5. If the test passes, do a manual test, commit your code and push to master.

To avoid this test being broken by another change in the future, you can set up a Continous integration server (Jenkins, for example), and set it up so that it runs your test, and indeed all tests for your project, on each commit:

drush test-run mymodule

Only once all of this is done, can we be confident to show our fix to the client, and mark it as fixed. A bug should be marked as fixed, or a new feature marked as done, when:

  • A test exists.
  • Mock objects are used to define external system behaviour.
  • The test passes.
  • Ideally, the test is checked with every new commit to avoid regressions.

Please enable JavaScript to view the comments powered by Disqus.

Nov 13 2013
Nov 13

November 13, 2013

Let’s say you are working locally and you need to add a new module to the site. Here is an example with Login Toboggan:

Let’s start by downloading the module to our local dev site

drush dl logintoboggan

Instead of enabling it outright, we’ll want to add it as a dependency to our deploy module, and define a hook_update_n() to enable it on existing environments.

; in your deploy module's .info file
dependencies[] = logintoboggan

/**
 * Enable logintoboggan in our deploy module's .install file
 */
function mysite_deploy_update_7001() {
  module_enable(array('logintoboggan'));
}

Deployable module settings need to be in code as well, so we can have the same configuration on all environments (local, development, stage, production). Here is an example with Login Toboggan’s user login form on Access Denied:

/**
 * Set logintoboggan to put login form on 403 pages
 */
function mysite_deploy_update_7002() {
  variable_set('site_403', 'toboggan/denied');
}

Make sure the above is also called during initial deployment, in your deployment module’s .install file (don’t forget: your continuous integration server and new members of the team will need to deploy new environments for your site):

/**
 * Implements hook_install().
 */
function mysite_deploy_install() {
  mysite_deploy_update_7002();
}

Now, you can enjoy this new functionality either on a new site environment by enabling mysite_deploy, or an an existing environment by updating your database:

drush updb -y

However, if you are running a multilingual site, the newly-added Login Toboggan module will not be translated. I avoid fetching the translations from localize.drupal.org in the update hook, because you can’t be sure it will work on all environments (for example, some clients disable internet access on their web servers for security reasons).

Rather, I prefer to download the translations myself and import them from the local repo in the update hook. Here’s how:

  • Start by visiting http://localize.drupal.org and finding your project
  • In the export tab, I like to check the Add Suggestions box, so, in the case of strings without “official” translations, at least I have something.
  • Click Export Gettext file, which will save a .po or .po.txt file to your computer, for example, for Login Toboggan in French, the file is logintoboggan-all.fr.po.txt
  • I put this file in my deployment module, for example sites/all/modules/custom/mysite_deploy/translations/ (if you have a .txt extension you can remove it)

Next, import in an install hook

/**
 * Import translations for Login Toboggan.
 */
function mysite_deploy_update_7003() {
  _mysite_deploy_import_po('logintoboggan-all.fr.po');
}

/**
 * Import translation po file
 *
 * @param $filename
 *   A filename in mysite_deploy/translations/
 */
function _mysite_deploy_import_po($file) {
  try {
    // Figure out the full filepath
    $filepath = DRUPAL_ROOT . '/' . drupal_get_path('module', 'mysite_deploy') . '/translations/' . $file;
    // move the contents of the file to the public stream. I can't get
    // _locale_import_po() to work with the file directly.
    $contents = file_get_contents($filepath);
    // In some cases the destination file might already exist, so I'll create a
    // random name; there probably is a better way of doing this...
    $random = md5($filepath) . rand(1000000000, 9999999999);
    $file = file_save_data($contents, 'public://po-import' . $random . '.po');
    // finally import the file from the public stream.
    _locale_import_po($file, 'fr', LOCALE_IMPORT_OVERWRITE, 'default');
  }
  catch (Exception $e) {
    // don't break the update process for translations.
    drupal_set_message(t('Oops, could not import the translation @f. Other updates will still take place (@r).', array('@f' => $filename, '@r' => $e->getMessage())));
  }
}

Don’t forget to call mysite_deploy_update_7003() from mysite_install(), to make sure that the translations are available to new deployments as well.

There is room for improvement in the above code, I admit, but I like this approach because it’s testable and does not depend on internet access during updates. Also, the same approach can be used to translate your custom modules.

Update: thanks to mikran for implementing a module (for now in sandbox mode) which expands on this idea.

Please enable JavaScript to view the comments powered by Disqus.

Nov 11 2013
Nov 11

November 11, 2013

Examples like these are rampant throughout Drupal 7, in block_admin_display_form_submit(), for example:

/**
 * Form submission handler for block_admin_display_form().
 *
 * @see block_admin_display_form()
 */
function block_admin_display_form_submit($form, &$form_state) {
  $transaction = db_transaction();
  try {
    foreach ($form_state['values']['blocks'] as $block) {
      $block['status'] = (int) ($block['region'] != BLOCK_REGION_NONE);
      $block['region'] = $block['status'] ? $block['region'] : '';
      db_update('block')
        ->fields(array(
          'status' => $block['status'],
          'weight' => $block['weight'],
          'region' => $block['region'],
        ))
        ->condition('module', $block['module'])
        ->condition('delta', $block['delta'])
        ->condition('theme', $block['theme'])
        ->execute();
    }
  }
  catch (Exception $e) {
    $transaction->rollback();
    watchdog_exception('block', $e);
    throw $e;
  }
  drupal_set_message(t('The block settings have been updated.'));
  cache_clear_all();
}

What’s wrong with this is that the logic for performing the desired action (moving a block to a different region) is tied to the use of a form, in the GUI. Let’s say you want a third-party module to move the navigation and powered-by block out of the theme xyz, one would expect there to exist, in the API, a function resembling block_move($blocks, $region, $theme), but there is no such function.

On a recent project where I needed to do exactly that, I installed and enabled devel, and put dpm($form); dpm($form_state); at the top of the block_admin_display_form_submit() function then went through the actions in the GUI, to figure out what Drupal is doing, finally coming up with this code.

foreach (array('navigation', 'powered-by') as $block) {
  $num_updated = db_update('block') // Table name no longer needs {}
    ->fields(array(
      'region' => '-1',
    ))
    ->condition('delta', $block, '=')
    ->condition('theme', 'xyz', '=')
    ->execute();
}

That’s reverse engineering, and it’s time-consuming, error-prone, and developer-unfriendly.

Recently in one of my own modules I realized I had made the same mistake.

The correct approach is to define an api (for example a function like block_move() in the above example, and call that API function from your form- and GUI-related functions like block_admin_display_form_submit()).

The result will be that developers will have as easy a time interacting with your module as human users. This will open the door to third-party interaction, adding value to your module.

Also, it will allow you to run more automated tests without actually loading pages, which is faster.

Please enable JavaScript to view the comments powered by Disqus.

Nov 01 2013
Nov 01

Disclaimer: I have much more experience USING Solr with Drupal than setting up a Solr service so please use the comments to correct me.

Having Solr index your content has loads of benefits, but best of all in my humble opinion is the beauty of have facetted filtered search. Of course you could do it with database indexes, but the performance win of using Solr is very noticeable.

Prerequisites:

  • A working LAMP stack.
    • I am using Ubuntu 12.04.
  • Java version 1.6 or higher.
    • Use 'java -version' to check.

Download Solr

Download a copy of the Solr tarball to your computer and extract it to a directory of your choice. Make sure you have root privileges, otherwise you will need to prepend 'sudo' to the following:

cd /usr/share
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/lucene/solr/4.5.1/solr-4.5.1.tgz
tar zxvf solr-4.5.1.tgz
rm solr-4.5.1.tgz
mv solr-4.5.1 solr

And now you have Solr downloaded. You can test it by going to the example directory and executing the start.jar

cd /usr/share/solr/example
sudo java -jar start.jar

Now if you navigate to the example.com:8983/solr you should get this lovely screen:

Now it is a bit of hassle always having to run a command from terminal when you want to start solr so here is a little trick to automate it.

cd /etc/init
vim start-solr.conf

Enter the following into the file and save it.

# start-solr

start on startup

script
    cd /usr/share/solr/example
    java -jar start.jar
end script

Now solr will start whenever the machine you are running starts up.

Configure Solr for Drupal

Most of what you need to know for this can be found on d.o here but I will take you through it anyway.

Download the newest release of the Drupal search_api_solr module. There are some files that we need to to our solr settings.

cd /path/to/search_api_solr/solr-conf
cp 4.x/*  /usr/share/solr/example/solr/collection1/conf/

And that is pretty much it. Simples!

Configure Drupal for Solr

Now install Drupal and Search API Solr (including dependencies) as you would normally. Go to the Search API configuration page (example.com/admin/config/search/search_api) add a new server (choosing Solr as the service class) and if you get this wonderful message:

you are done!

You can now index Drupal to your heart's delight.

Oct 29 2013
Oct 29
Here are some of the Performance tips for the Drupal site.

  1. APC : APC (Alternative PHP Cache) is a PHP OP code cache. It is a very quick win when working with PHP and can offer a great performance boost when using Drupal. It is very much a “set it and forget it” type of application which can just be installed, enabled and left to do it’s thing.
  2. Memcache : Drupal's support for memcache is really good and easy to implement.There is a memcache drupal module(https://drupal.org/project/memcache) to integerate the memcache on the site.
  3. Varnish : When you have a lot of anonymous users reverse proxy cache can save you a lot of server load. Varnish is one of the more popular solutions within the Drupal world. Varnish sits in front of your web server application, for example Apache, Nginx or lighttpd, and can run on the same server or a remote server.Use the varnish module to integrate the varnish on your site https://drupal.org/project/varnish
  4. Boost : Boost provides static page caching for Drupal enabling a very significant performance and scalability boost for sites that receive mostly anonymous traffic. For shared hosting this is your best option in terms of improving performance. On dedicated servers, you may want to consider Varnish instead. When the page is then requested it is loaded quickly, because it is coming straight from the disk and no PHP or MySQL processing is needed.See the Boost module here https://drupal.org/project/boost
  5. CDN : A CDN is used to distribute static assets such as images, documents, CSS and JavaScript across many locations.The goal of a CDN is to serve content to end-users with high availability and high performance.There is a CDN drupal module(https://drupal.org/project/cdn) to use the Content delivery Network.
  6. Disable Database logging module : This module logs the action performed on the site to the database.Use syslog module which is also in drupal core.Using syslog you can also write the more technical log entires to the server's standard log on the file system and save the database queries.
  7. Enable Page & Block Cache : Enable Drupal caching (Administer > Configuration > Performance). When enabled, Drupal will render the page and associated blocks once, and then save that result in the database. This can drastically reduce the number of database calls run on a page since the results are pre-rendered. Drupal’s caching engine is most effective for anonymous visitors – if your site is mostly “read only” and doesn’t have visitors logging in, caching can make a dramatic improvement in site load speed.
  8. Increase Cache Lifetime : An option for some sites may be to increase the cache lifetime. This determines how long Drupal will hold onto a cached result before it will re-generate the page. If you have frequently changing content, you may want to set the cache lifetime to only 5 minutes, but if your content doesn’t change often, an acceptable value may be several hours.The cache lifetime depends on your site usage.
  9. Optimize JavaScript and CSS Files : Enable the option of optimize javascript and CSS files in the performance settings. When enabled, Drupal will consolidate all CSS and JS files included on each page into a minimum files, and compress the code by removing whitespace. This will reduce the overall file size and improve page load speeds.if you need more aggeration of css and js files then use Advanced CSS/JS Aggregation module https://drupal.org/project/advagg
  10. Disable Un-used Modules from the site .

if you have done all the performance tips written above and you are still getting the performance problems then either get the suggestions from the High performance drupal group https://groups.drupal.org/high-performance

There are also some of the related performance related articles.See there links below

  1. http://www.creativebloq.com/web-design/drupal-performance-tips-9122837
Oct 29 2013
Oct 29
There are many service providers in the world that provide drupal related services.As per the drupal marketplace some of the providers are Featured(those which have an exceptional community contributions and show continued support of the Drupal project) and others are not Featured.See the Top 10 Featured service providers here https://drupal.org/drupal-services.you can filter them according to the Services,location and sectors.

When hiring a Drupal site developer it's important to understand what you need and where you can find the service provider with that set of skills.you can find the detail here https://drupal.org/node/51169

if you are getting some problem while hiring a Drupal Developer then feel free to contact me using my contact form.

Jul 11 2013
Jul 11
How to create the node programmatically in drupal 7 with Reference field modules either References or Entity Reference(that is in drupal 8 core now).

Also find the below example of creating a simple node with page type with Title,Description and a Reference(References or Entity Reference) fields.

$node = new stdClass();
$node->uid = 1;
$node->title = 'Title of the Field';
$node->status = 1;
$node->comment = 1; //Hidden the Comment on the node
$node->promote = 1;
$node->sticky = 0;
$node->type = 'article';
$node->created = '';
$node->changed = '';
$node->timestamp = '';
$node->body['und'][0]['value'] = 'Full Story';
$node->body['und'][0]['summary'] = 'Summary';
$node->body['und'][0]['format'] = 'full_html';
$node->revision = 0;
$node->field_entity_reference['und'][0]['target_id'] = 50;
//Here field_entity_reference is the name of the Entity Reference Field and 50 is the nid of the reference node.
$nid = node_save($node); Here field_entity_reference is the name of the Entity Reference Field and 50 is the nid of the reference node.
you will get the $nid after saving the node
Feb 21 2013
Feb 21

Going from static design files to a completed Drupal theme can be a daunting task. I often get asked "strategic" questions by companies who are looking to hire a designer/themer for an upcoming redesign and aren't quite sure what skills they should be looking for. And I often get asked by new-to-Drupal designers how they should approach the task of transforming their design into a Drupal theme. In these conversations the following question (almost) always comes up:

Should I create a static HTML page of my design and convert that into a Drupal theme?

Can you guess what my answer is?

Five years ago, when I was first writing Front End Drupal, the answer would have been "Yes!" Five years ago there were fewer moving parts to creating a Drupal theme. There were fewer contributed modules. Responsive design wasn't even an idea yet. Five years ago I don't think that "mobile" was even a thought for most businesses, let alone a concern.

But today, in 2013, my answer has definitely changed. Now when asked if people should create a static HTML page of their design before converting it into a Drupal theme, my answer is "No!" In fact, depending on your budget size for the entire build and the type of site you're building, you may actually want to skip "static" design files for each layout (and layout variant if you're building a responsive site) and have your designer supplement their work by creating "style tiles" instead of piles and piles of page-specific layouts.

Designing for a style, instead of a specific layout, is a different workflow though. You'll need to have a relatively experienced content strategist, designer *and* themer to help you navigate this workflow for theming Drupal. Pixel precise designers are much easier to find. But here's the problem with pixel perfect design: it relies on exact markup. Drupal is markup-heavy and can drive a themer crazy if every single tag, and every single class must be exactly perfect. These days, it's easier to skip the "static" step and get into Drupal as fast as possible. Sure, you'll still want to refactor some markup (or maybe just start with a different base theme), but the point is that you aren't starting from thin air.

Themers who want markup purity tend to be coming at their work from an Model-View-Controller (MVC) perspective...which isn't what Drupal does. We're a Presentation-abstraction-control (PAC) framework. The main difference between the two (in laymen's terms) is the fragmentation of the output/markup files. Our system has lots of little nested template fragements which are compiled, or collected, into a whole. This means if you want very, very precise markup, you have to alter a lot of little files. Some of us love it because you can get really fine grained control in very tiny places without having to touch all of the markup, others hate it for exactly the same reason.

Regardless of which camp you're in, there doesn't seem to be a lot of information about building themes that talk about these software design patterns. A quick search reveals the following tutorials and articles (most are quite old). If you've got others you'd like to add to the pile, please leave them in the comments. Things get very technical very quickly...but hey, this is a list of links about software design patterns. It's supposed to be technical.

Feb 09 2013
Feb 09

Yesterday I made a very bad assumption that's wasted a day of my time. Today, while I wait to hear back on where I went wrong, I thought I'd write up a little warning to my future self.

First, a bit of background: It's very unusual for me to start a project at the beginning. Most of the time when I'm working on a project it's doing little fix-its for a client as part of a larger training contract. Sometimes it's simplifying things to eliminate the need to train users on complex tasks; sometimes it's tweaking a theme to accommodate the way content is being placed into the site now that it's a Real Live Web Site. Regardless of what I'm doing, I seem to spend a lot of time fitting into someone else's work flow. I'm used to it, and usually I enjoy the discovery process.

Yesterday when I hopped onto a project, I was thinking about checklist of jobs (images in the sidebar needed to float: left, lightbox module needed to be installed for a gallery, verify backups are working properly)...but I wasn't thinking about the fact that this project had already had multiple rounds of developers working on it...potentially each set having their own preferred workflow. In retrospect, this is what I *should* have done right from the beginning (and all at once) instead of only doing little bits and pieces over the course of the day while pausing to bang my head against a wall.

Conditions:

  • The project has had more than one developer/development team working on the project before you start.
  • You only have FTP access to the server (yes, seriously).
  • The only way for you to pick up the files you need to be working on is by downloading them via FTP from the live server to your work station.

How I *should have* started this project:

  1. Look in the theme folder for documentation about how the theme was built. If there is none but there are preprocessor files (LESS or SASS), stop immediately, and ask to speak with the previous developers. (I assumed I could just muscle through it without docs...because I'm psychic and code is self-documenting. HINT: turns out this actually just means that I'm psychotic, because CSS preprocessors are totally NOT self-documenting....so you can probably tell that I skipped this step?)
  2. Download as many relevant files as necessary to run the site locally (in this case it was the entire site + a B&M snapshot of the database).
  3. Put everything into version control. Do not pass go. Do not make changes until everything is in version control. (Did this. /me pats self on the back.)
  4. (This is where I went wrong.) Create a new branch and recompile the CSS files from the source preprocessor files (e.g. SASS or LESS).
  5. Assuming you're using Git for your version control, run the command:
    $ git difftool mynewbranch..master
    Look for differences between the two branches.
  6. If there are any differences in Step 5, stop immediately. There is a problem. Either you are missing the source files (i.e. .scss or .less), OR someone has been editing the CSS files directly. Stop and find out which problem you're dealing with.

Possible scenario #1: If you're missing source files: easy! Once you've got the right files you can continue to take advantage of all the awesomeness you get with a CSS preprocessor. Of course once you get the source files, you should repeat steps 4-6 until you can get your output to match what's on the server.

Possible scenario #2: If one of the developers along the way has been editing the CSS files directly: easy! Delete all of the source files from the preprocessor and do your work like it was 1999. Well. Except with more awesome version control than what was possible back then. Of course if you love to punish yourself, you can carefully inspect the live CSS files and pull out the hand-edits and put them back into the preprocessor files where they belong. Chances are very good that your client won't want to pay for this step. Only you can decide if it's worth eating the time costs.

So there we have it: a note to my future self about making assumptions and jumping into the middle of a problem. Do you have your own process for dealing with inherited projects? I'd love to hear any tips you'd like to share.

PS @rupl over at Four Kitchens helps to solve this problem by including a WARNING file in his SASS source files. You can read it here. I think it's brilliant. and so does fubhy.

Feb 04 2013
Feb 04

On the back of an envelope, tucked into one of my office drawers, is something that comes very close to being my personal mission statement: your problem is already solved--you just haven't found the answer yet. You see I'm not much of an inventor. I'm a connector. I take stories out of context and apply them to new situations. I look for patterns and love helping others to make their own connections to make their own work flow better. My talk on base themes in Munich is a perfect example of the kind of work that comes naturally to me: sorting, examining, classifying, naming, and sharing the results with others.
 
It's probably because of my love of patterns that I rarely start a project from the very beginning. If it's possible: I like to find the middle of the problem and then work out.
 
Theming, by it's very nature, will always put you into the middle of the problem first. When you're lucky you're given static design files and a site full of content and your job, as a themer, is to start in the middle and work out until the design has been reincarnated as a fluid, interactive, adaptable colony of content. But where is the middle? Sometimes you can intuit where to begin. (Although it's probably because you used to know and simply forgot.) Leaving things up to intuition seems a bit risky though. Especially when, as I personally believe, the answer is already waiting for you.
 
When I start a new theme, I ask myself: "What is here that I have seen before?"
 
I look for patterns.
 
I don't mean background textures...I mean the shape of the content inside the design. You can do this too: you can teach yourself to look for patterns, and then use existing solutions to quickly solve your problems.
 
Here are a few of the bookmarks I've collected over the years on pattern used by front end developers. I hope you'll find them as useful in solving your own theming problems.
 
Let's start with a bit of inspiration:

  • User Interface Design Patterns. A community-generated collection of web site elements (aka design patterns). Includes a problem summary, example, and usage guidelines for each pattern.
  • Elements of Design. This one isn't so much a pattern library as an inspiration gallery. the elements are linked back to their site of origin so that you can see them in context, however, sites are often redesigned and it isn't always possible to see how the element fit into the bigger site design.
  • Yahoo! Design Pattern Library. A tightly curated gallery of patterns that Yahoo! uses across its sites. You'll recognize examples from Flickr, Yahoo! Mail, and Delicious (bookmarking).

Now let's get more technical:

  • Pears. Common patterns of markup and style. Don't be put off by the CMS. Look through the pages and you'll start to recognize the types of elements you probably need to build on your site too (e.g. Slats (thumbnails) could easily be a view with a resized image field). Your base theme may also give you a bunch of HTML selectors to help you target patterns (e.g. Fusion and Zen).
  • Modular CSS helps you "see" the patterns in a design file and convert the patterns into CSS.
  • Jesper's talk from DrupalCon Munich, Dominate The Theme Layer, expands on the ideas in Modular CSS, and applies them to the Drupal context. (note: I don't think Jesper's presentation is based on the other article, but as a reader I think you'll find they flow together nicely.)

So go ahead and jump right in with your theme. But don't think for a second that your design is a special snowflake that is nothing like anything ever built on the Web before now. Do the research: look for patterns. And remember: your problem is already solved--you just might not have found the answer yet.

Jan 30 2013
Jan 30

Sometimes I work with clients who don't have the internal capacity to set up a developer environment for me...but they still want to be able to track what I'm working on in a semi-private space. In fact I had one such project a couple of weeks ago. My go-to service for this is WebEnabled. It's a paid service (and I don't get paid to promote their service), but you could definitely replicate the concept in other environments. I use a similar process when I'm working with the Acquia Developer Desktop on my windows machine.

This process isn't ideal for working with teams. And it's not my typical setup. But I think it helps to get people thinking about how they can use version control even on very tiny projects.

  1. Install the base environment (Drupal+LAMP stack) with the basic D7 core package from the WebEnabled app library. This gives me Drush and the basic file structure for Drupal. I won't end up using much of the basic environment, but it's an easy starting point.
  2. Use Drush to add the module Backup and Migrate. In WebEnabled this can be done through their Web-based admin tool, or on the command line (through an SSH connection).
  3. With Backup and Migrate, create hourly schedule with 3 days of keeping backups. (Test the backups!)
  4. Add version control through the WebEnabled interface (you can choose SVN or Git, I use Git).
  5. If the client has provided a snapshot of their Drupal file system, I copy the relevant folders (especially sites/all, and profiles) into my dev environment.
  6. Perform an *environment* backup of WebEnabled in case the next step fails. This is not the same as the Backup and Migrate *database* backup.
  7. Import the database snapshot from the client. Depending on how it's been provided, this might happen through phpMyAdmin (WebEnabled sets this up as part of the originial provisioning of the environment), or through Drupal if it's a Backup and Migrate snapshot.
  8. Use WebEnabled to reset the user/1 drupal password.

The development environment is now set up and ready for me to use.

For this particular project I was *only* working on a simple theme. No extra modules. Nothing fancy. So I used my git repo on WebEnabled to store all my theme folder.

In my local work environment (not WebEnabled), I completed the following steps:

  1. Clone the git repo from WebEnabled. (It's empty right now.)
  2. Add the starter kit from my base theme of choice. (Or, in this case, my client's base theme of choice.)
  3. Tidy up the starter kit to match my theme name.
  4. Add "major" images (e.g. background images, logo) to give a quick win for decorating my new theme to match the design files.
  5. (all along I've been committing my little changes locally to git) upload my new theme to the central git repository that WebEnabled provides, by using the command git push.

Next, I log into the WebEnabled dev environment via SSH, and complete the following steps:

  1. Navigate to the appropriate theme folder (e.g. sites/all/themes).
  2. Clone the WebEnabled git repository for this project as a new theme folder.
  3. Use Drush to enable the new theme.

Now the real work begins! Most of the tasks I'll do in my local developer environment. When I'm ready to share a batch of changes, I'll complete the following steps:

  1. "git push" the files to the central repository on WebEnabled.
  2. Log into the WE server and "git pull" the updated files from the central repo into the semi-private development server's theme folder.
  3. Use Drush to clear all the caches (drush cc all).

That's it!

What are your favourite tools and services for quickly sharing your work? I'm especially interested in hearing from other small teams who may not have a lot of infrastructure support.

Nov 14 2012
Nov 14

 I've been working on a new talk, to go with a new workshop, on setting up a developer's environment for very small teams. It got some great questions today at DIG London, and I'm hoping you'll enjoy hearing it at DrupalCamp Toronto as well.

Interested in the full workshop? It starts next Monday. Details about it are available here.

Sep 30 2012
Sep 30

I recently worked on a project which required a updated version of the jQuery library. While there is the jQuery Update module, it only allows you to upgrade Drupal 6 to jQuery 1.3. If you really know what you're doing and want to upgrade beyond that version, you can either hack core or create your own simple module to do it. While hacking core is certainly the easier approach (simply overwriting misc/jquery.js with a newer version), it is very bad practice. You do not want to get yourself in the habit of altering Drupal core unless you want to kill your upgrade path and deal with a new slew of bugs and unpredictable behavior.

Let's start by creating an admin area for configuring the version of jQuery we want to use.

<?php
/**
* Module configuration admin form.
*
*/
function mymodule_admin_form() {
 
$form['mymodule_custom_jquery'] = array(
   
'#type' => 'checkbox',
   
'#title' => t('Override Jquery'),
   
'#description' => t('Replace the version of jQuery that ships with Drupal
      (jQuery 1.2.6) with the jQuery library specified below. You will need to
      flush your cache when turning this feature on or off.'
),
   
'#default_value' => variable_get('mymodule_custom_jquery', 0),
  );
 
$form['mymodule_custom_jquery_file'] = array(
   
'#type' => 'textfield',
   
'#title' => t('Jquery file location'),
   
'#description' => t('Specify the location of the updated jQuery library
      you would like to use. The location is relative to the docroot and
      so should probably begin with "sites/".'
),
   
'#size' => 128,
   
'#default_value' => variable_get('mymodule_custom_jquery_file', NULL),
  );
$form = system_settings_form($form);

  return

$form;
}
?>

Next, we're going to write a validation hook that simply makes sure the file exists.

<?php
/**
* Admin form validation callback.
*
*/
function mymodule_admin_form_validate($form, &$form_state) {
 
$file = trim($form_state['values']['mymodule_custom_jquery_file']);
 
$form_state['values']['mymodule_custom_jquery_file'] = $file; // Only validate this value if js override is turned on
 
if (variable_get('mymodule_custom_jquery', 0)) {
    if (!
file_exists($file)) {
     
form_set_error('mymodule_custom_jquery_file', t('The file you specified does not exist: !file.', array('!file' => $file)));
    }
  }
}
?>

And lastly, we use hook_preprocess_page() to safely replace the jQuery that ships with Drupal core with our own version.

<?php
/**
* Implementation of hook_preprocess_page().
*/
function mymodule_preprocess_page(&$variables) {
  if (
variable_get('mymodule_custom_jquery', 0)) {
   
$file = variable_get('mymodule_custom_jquery_file', 0);
   
$scripts = drupal_add_js(); // remove core jquery and add our own
   
unset($scripts['core']['misc/jquery.js']);
   
$add_scripts['core'][$file] = array(
     
'cache' => TRUE,
     
'defer' => FALSE,
     
'preprocess' => TRUE,
    );
   
$scripts['core'] = array_merge($add_scripts['core'], $scripts['core']);
   
$variables['scripts'] = drupal_get_js('header', $scripts);
  }
}
?>

Make note of the line:

<?php
    $scripts
['core'] = array_merge($add_scripts['core'], $scripts['core']);
?>

We are careful to add our new jQuery include at the beginning of the array (where the original jquery.js include was) in order to meet any jquery dependencies in the scripts that follow.

Sep 11 2012
Sep 11

Since Drupal 6 and Drupal 7 a lot of popular modules like ctools, rules and views (and probably many more) had some kind of way to extend it's functionality by writing so called plugins. The idea was to swap out certain behavior of your system by another plugin of the same type.
For example you can replace the output/style of a view to a fancy slideshow by choosing another style in the user interface.

There were multiple "standards" how to create such systems,
so for different modules you always had to learn a new way,
made different mistakes and basically wasted a lot of time.
Therefore in d8 one goal was to unify them and make it
much easier to use.

A plugin in d8 will be always some kind of php-class which are part of a namespace, which correlates with the folder/filename. This sounds maybe complicated but let's have a look at an example.

Assume you want to add a new style plugin to views you would in d8

  • Create a file called Example.php in the folder your_module/lib/Drupal/your_module/Plugin/views/style
  • At the top of the file you define the namespace:

    namespace Drupal\your_module\Plugin\views\style;

To be able to tell drupal of your plugin, you use Annotations,
which are basically comments in your php code.

They contain metainformation attached to
the plugin directly. You can compare this basically with the info hooks in d7:

// Add the below used views baseclass.
use Drupal\views\Plugin\views\style\StylePluginBase;
// These two lines are required to scan the code for the plugin.
use Drupal\Core\Annotation\Plugin;
use Drupal\Core\Annotation\Translation;

<?php
// The actual annotation starts:
/**
 * @Plugin(
 *   id = "style_example",
 *   title = @Translation("Just an example style plugin to explain the plugin system"),
 *   additional_info = "Some random information"
 * )
 */
class Example extends StylePluginBase {
}
?>

That's all you need to start writing your plugin. Just have a look at the attachment for everything that is needed.

Now we learned how to write a views plugin, but you will be able to apply this to nearly everything in drupal8 for example a block plugin (the d8 way to write custom blocks).
You add the file with a class of the right namespace and add the annotations, probably clear your caches and be happy ... see screenshot.

I neglegted some internal things of the plugin system which either will let you explode your brain or feel really excited.
A long documentation of the new system can be found under: http://drupal.org/node/1637614

Thanks for the scotch and wscci initiative for the implementation of such a flexible system.

AnhangGrösse 2.88 KB
Sep 04 2012
Sep 04

Are you a Drupal developer? How are your PHP skills? Are you ready to upgrade your modules for Drupal 8 with new OOP design patterns? Do you know what all the changes are in PHP 5.3? Are you excited, but thinking perhaps you might need to blow into a paper bag for a while to stop freaking out about all that's going to change? Yes? Read on, my friends.

In two short weeks PHP expert Lorna Jane and Drupal expert Emma Jane (that's me!) will be offering a brand new master class for Drupal developers. This course is specifically designed for Drupal developers who will need to master PHP in order to prepare themselves for Drupal 8.

This workshop will lead you through:

  • Best practices for object oriented programming in PHP.
  • Refactoring your modules for new language features in PHP 5.3+
  • Using Symfony components in Drupal 8.

Unlike other workshops, the seats for this class are being sold per company. We encourage you to participate with your co-workers. We already have a number of skilled companies signed up for this workshop. This is a class you're going to look forward to. All the smart kids will be there, and you should be too.

How do you sign-up? It's as easy as filling out the registration form. Once you've completed the form, Emma will be in touch to coordinate payment.

We look forward to seeing you in class. It's going to be amazing!

PS for mr.baileys: yes, the goal is to GPL the curriculum so that it is available for the entire community. You can read about the funding model here.

Sep 01 2012
Sep 01

I'm not sure how it happened, but today I noticed that Drupal's menus were behaving very oddly. After upgrading to Drupal 6 and installing several additional modules, I noticed duplicate menu entries as well as other disturbing oddities. Items I was placing into the menu were not showing up. Menu items that I moved around were apparently saved but they did not appear properly in a dropdown context. 

Looking further into it via the absolutely awesome SQLYog tool, I verified that there were dozens of duplicate entries. Some items were duplicated up to six times. It was a real mess.

The database tables involved here are menu_links and menu_router. These are two of the more mysterious tables in Drupal. I haven't had the need to spend much time with them in the past, and I know now that this is a good thing. Fortunately, you do not have to know anything about them to fix this problem. While I spent a couple hours carefully deleting items from this table, ultimately I gave up. I was able to remove all the duplicates, but the menus were still misbehaving. At this point, I just wanted to do a factory reset on the menus, but it's not so simple as flushing cache. However, that is not far from the solution.

This solution will do a 'factory reset' on your menus. You will lose any customizations you have made. However, all core and contrib module entries will be restored very nicely.

Please backup your entire database before doing any destructive database manipulation. 

Step one is to empty the corrupted tables:

In your favorite SQL client, run the following commands:

truncate menu_links;
truncate menu_router;

At this point, your site will be completely unusable. But not for long. To complete the final step, you will need to be comfortable with the Drupal admin's best friend, drush.

Simply run the following commands from your terminal (Viva tcsh!):

drush php-eval 'menu_router_build();'
drush cc menu

Now my menus are as fresh as the day they were installed.

Though I could not clearly identify the cause of this problem, I would suggest backing up your database before installing the Taxonomy Menu module

Aug 27 2012
Aug 27

So this was my first DrupalCon and i really don't know whether i am happy with it or not...

I am not the hardcore-coder like some others, my experience in drupal is a mixture of sitebuilding and developing mainly smaller modules. So i just left of for munich very open-minded as i knew several other Cons e.g. the Linux-Tage in Germany.

It was a big meeting indeed, lots of people you just know by their nicknames, you just know what they have done so far for the community. You had a lot of possibilities to get to know each other and after all the location was great - also the food. So a big "thumbs up" for the venue!

I visisted the KeyNote on the first day and yes, i really WAS disappointed by this one. Allthough the idea of the Interview-Mode was cool, there was nothing really cool about the talk itself.
Nothing new, nothing you could not read on the internet weeks and months before... If this should be kind of "kick off", it totally failed in my opinion. There was no big bang or something like that, seen better keynotes on other OSS-Meetups.

I attended some of the talks the next days, mainly to see that most people have the same challenges in project that we have every day. But i really liked to see how the others solved them, got to know a few modules i didn't know and saw a lot of
solutions i never thought of. Allthough i atteneded just sessions witch where labeled "intermediate" or above it was not possible to go too deep in the problems, they where mainly just scratched at the surface... But as mostly on every convention you
get new ideas and hints where to find new solutions, so here all i am really satisfied with the sessions.

The biggest problem of this con was that it was something "in between"... Too many business-people to meet developers and to get mostly technical details out of it, too many developers to make business. In my opinion it's much better for the
technical interested people to attend one of the various Drupal-Camps out there then the Con. We should leave that to business and sales as this seems to be the targeted audience. But this problem is not only drupal specific, it's noit really easy to meet
everbodys needs.

Aug 27 2012
Aug 27

From 20th to 24th to August 2012 the DrupalCon Europe came for the first time to Germany. During the week of this summer's hottest week so far 1800 Drupalistas from all over the world gathered under the motto "Open Up! Connecting systems and people" in the Grand Westin Hotel in Munich, Germany.

The DrupalCon started with a big surprise, announced on the opening session: Several European Drupal shops (NodeOne, Krimson, Mearra and Wunderkrau) fusion together to become the new, Captain Drupal powerd Wunderkraut - after Acquia, the next big elephant among Drupal companies. This surely will have a positive effect to the Drupal community as some people saw Acquia taking disproportional influence into Drupal development and having another big player in the community could diverse continued and strong development directions. For the smaller companies and start ups this will probably not cause disadvantages since it make Drupal generally stronger and more interesting to larger - industry size - companies, which would probably not hire a small shop, like Dries said on the opening session: "Elephants want to dance with elephants".


Personally I liked this DrupalCon a lot! As a developer I was positively surprised to see not as many sales persons but a lot of community members to talk to, exchanging knowledge, opinions and living the "momentum" about Drupal. The venue was awesome and not only the geek-friendly coffee stands all over the place made the attendees feel well. The food was remarkable excellent and with over €300k the biggest expense of all, but really, it is worth to keep everybody happy and good food achieves that more than anything else.


Although DrupalCons have become huge, it had the spirit of a perfectly sized conference since there were a lot of rooms with great sessions for everybody: Heavy programmers could come together for the core conversations track on a 5 minutes walking distance at the Sheraton hotel, which made it a cozy and productive place. The main sessions were nicely grouped into eight tracks with clear topics, this way, for all kinds of interests, good quality and interesting sessions were offered. It was definitively not a DrupalCon where you could follow all happenings, but rather everybody could enjoy exactly what one was looking for. You can find all the session's recordings on the schedule or the DrupalCon channel on blib.tv
Unfortunately I was a little bit disappointed by the keynotes: Dries' keynote was as a simple interview without any new information and Anke Domscheit-Berg's keynote was just a general summary of Open Data initiatives, an interested person could read together quickly through blog posts and general news. No innovative stuff here, but paired with some awkward invitations to participate. Yes, it is an important field, and we all need to demand actively and know about it, but somebody with more activist insight would have rocked way more.


For me, the motto had it's truth on the part connecting people: As the community lead for DrupalCon São Paulo I got to know personally and work with the staff and people from the Drupal Association, which outside of the US' Drupal community doesn't seem to have arrived yet. Surely, the Drupal Association is not working as open and community based as most of the community members would like, but they are really opening up, and it is the responsibility of the international community to get involved! I would like to point to the awesome session by Donna Benjamin Infiltration Day 853: Drupal Association Board. Confessions of a not-so-secret double agent, the only one of the non North-American board members. Where she did a great call to people to participate and form constructively the Drupal Association, which is important to all of us. A nice detail was that out of a quarter million people entitled to vote for members of the board only 650 actually made use of it. It is necessary that the community accepts the Drupal Association, not as a decision making instance rather than as a representative that forsters the Drupal community and for this reason the whole world and active groups have to participate in the election and forming process.


This DrupalCon in Munich was an important step for scaling Drupal (community) internationally: The German community, as they were cooking their own soup for a long time, now got deeply involved into the international scene, two next DrupalCons have been announced in new parts of the world (Latin America, Brazil, São Paulo and Australia, Sydney), a prosperously discussion about internationalizing the Drupal Association has begun and another huge Drupal company has arisen on the European Drupal horizon.

Drupal rocks! Drupal rockt! Drupal rockea!

Aug 27 2012
Aug 27

I've been putting it off for a few years, but I finally decided to upgrade devbee.com to Drupal 6. 

I didn't really need to, but it bothered me that I wasn't running supported code and I figured I might learn something. And I did. Mostly obvious things that I should be familiar with already. 

Drush

I've only ever played around with this. I don't like learning new things unless they are going to be truly useful to me. Drush is definitely something I shouldn't have ignored for so long. It comes in particularly handy when doing a site upgrade as you can download and install modules, clear cache, set variables, run DB updates and a lot more all from the command line. This tool is crazy good if you're comfortable in a terminal.

Themeing didn't change much

I was able to convert my dated theme to 6 with very little effort. I created a .info file, changed the names of the region variables in my page.tpl.php file and tweaked a small amount of CSS. Took me about 20 minutes. Pretty painless. I was able to preserve my table-based layout so all you markup nazis will have an example of bad design.

Minimalism

The whole process was made much easier by the fact that I don't like using a lot of modules. I tend to go with a handful of the more popular modules such as Views, CKeditor, CCK, Pathauto, etc... Because I use only popular modules, I'm almost certainly going to have no trouble upgrading them. At least that's my experience.

Key things to remember

  1. Start with a brand new install of D6 with no contrib modules. It's tempting to try to force an upgrade on your existing site, but it's just not going to work. You must create a clean install. 
  2. Import your old database into the clean D6 database. I tried to skip this and just point D6 to my old database, but then  you end up with missing tables and a whole lot of brokenness.
  3. Run DB updates and get your core Drupal site working before worrying about contrib modules.
  4. Lastly, and this is where drush will save you time, download and enable the contrib modules you need one at a time. 

Unfortunately, the upgrade process is no longer something that can be done by the average site owner. There are going to be snags, and you're going to need to know how to interact with your database to get things up and running. I think this is the reason so many users are still on D5 and D6. I'd really like to see an upgrade path that was more accessible to non-developers, but at the same time, I'm also grateful that Drupal is upgradable and there is no need to manually migrate data for major upgrades. 

At this rate, I should be upgrading to D7 sometime around the end of the decade.

Aug 25 2012
Aug 25

The day started with a other nice breakfast. Then for the first time this week I to ride my bike to the Con on my own because my friends want to start the day a bit later. I went totally lost and need to ask the way several times.
The last time I need to ask I asked a very nice girl who seems to work close to the Sheraton so we biked together to the Sheraton. She told me she was a graphic designer and did sometimes designed website. So we talked shop while biking to the Sheraton.

Today I did less sessions and spend more time talking to all kind of people, and getting information about topics I was interested in from people at there stands. So for example know I know when to use mongoDB.I got a mug with mongoDB on it that actually is a bit funny because my initials are DB

But first the day started with again a very inspiring keynote. This time from Fabien Potencier.
He explained why it is a job that does not work as normal day job where you work from nine to five and then forget about it until the next working day. You need to be passionate and constantly want to learn. And that is so true. We (DrupalCon visitors) all do that. We visit Drupal meetings participate in forums or chat on the IRC. Trying new stuff out in our own time because it is so ******* awesome. And (and this is what I think is so very cool about this community) help each other with the problems we have and learn from each others.

So I missed the next session I had planned because the great conversations I had with all kind of guys and girls. What to think about a professional Opera singer who does Drupal to pay his bills. And I even learned a bavarian bachelors party joke. I really feel sorry for the poor guys :- )

Meeting the amazing fubhy.
One of my hero's is Sebastian Siemssen (fubhy). He is the guy behind the code from the Omega base theme. The base theme that we in our company use as the default base theme. He was working on version 4 and showed how you can very easy and very quick make a sub theme with it that is not only very flexible but also amazing fast. Can't wait to try it.

The last session was from the kind people of Erdfisch who where so kind to give me a ticket to this amazing event.
But be sure that it will not influence my opinion on there session. I will be brutally honest.

So here is my brutally honest opion
After this session I was very proud to be their guest.
There use case about the work done for Greenpeace is so detailed so well worked out it was a pleasure to look. It shows that our job is moved from doing a nice branding site for clients in to making online solutions for clients. So I think from now on it is no longer called web design but online solution design.
Fabian did a great job when making this presentation It had a look and feel that fitted perfectly with the branding of this use case. What was extra hard to do because he was not allowed to show anything about the actual site the are building.

So that was my last DrupalCon session. My head is full information and need time to sort it all out.

My thanks go to
Pascal Grott
For not only helped me to get a drupalcon ticket but also made me have a least one question right at the trivia.

Frank Holldorff
For actually getting me the ticket and even let me blog about this event.

Floris van Geel
For taking me in his car to Munich, being such a good friend and teaching me so much.

Serge Christiaans
For the nice time we had.

And all the nice people at this great event that I have talked to and sometimes shared a beer or cola with. Hope to see you all again next year.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web