Nov 16 2016
Nov 16

The Entity Reference Display module allows editors of Drupal sites to select which view mode they would like to display a list of entities as.

The Paragraphs module has changed a lot of things in Drupal. The same can be said for Bean in Drupal 7 and Custom Blocks in Drupal 8. All of these entities open up a lot of freedom for editors as they allow for the embedding of components in a page. At Morpht we have pioneered these techniques and have written and presented on them extensively. The Paragraphs Demo website still stands as an example of what can be done with Paragraphs.

One of the key concepts we use for site building these days is that of the Node List, Media List and Item Block List. Each of these components is very simple, holding an unlimited entity reference allowing editors to add and sort entities as they wish. This approach is great for landing pages as items can be called out and placed as desired. You could think of it as a componentized version of Nodequeue and later Entityqueue.

But how are these items to be displayed? What view mode shall we use? How can editors select a teaser or a tile? That’s where Entity Reference Display comes in.

This module provides a select list interface to the Editor, allowing them to select the view mode they would like to display the list as. This is a very simple interface and makes content creation a breeze for editors. We have found this to be a more pleasurable way to build than say, adding items with IPE or embedding in the WYSIWYG, although that does have its place.

A final part of the recipe is to use the Entity Class Formatter module to also allow the editor to select the layout they would like to apply to the list. Gives these two modules a try and see how you go with building better editor experiences for your users.

Sep 14 2015
Sep 14

Each year the Sydney Drupal Community puts on a two day "DrupalCamp", generally held over a weekend at a location where attendees can get away from it all and get down to some serious Drupal talk. This year we held DrupalGlamp Sydney on Cockatoo Island, located in the middle of Sydney Harbour.

DrupalGlamp Sydney 2015 was held last weekend and it went off pretty well. We had all the important things covered: the venue, the programme, internet, a bar, cafe and people! The weather even managed to hold out for us.

Overall we had:

  • 50 attendees,
  • 11 speakers,
  • 2 Drupal 8 training sessions,
  • 5 lightning talks,
  • 2 BOFs,
  • 2 gold sponsors (PNX, Catalyst),
  • 6 silver sponsors (Morpht, Acquia, Linux Australia, Infinity, Demonz, This Little Duck),
  • 2 supporting partners (Drupal Association, Xyber Data Recovery),
  • 10 people camping,
  • 1 bicycle.


From my perspective there were a few highlights of the event which are worth mentioning:

We try hard to come up with an interesting venue. This year was no different and thanks to a suggestion from Chloe, we were very happy to find the venue at Cockatoo Island. I think that everyone enjoyed the ferry ride over to the island and getting away from it for a little while. We had a good number of people who were prepared to "rough it" and camp out on the Saturday night. A couple even made it for two. On the Saturday night the firepit was lit up and about 15 of us enjoyed a few roasted marshmallows around the fire. The fire may not have been as big as Ivan expected (sorry about that mate) but it did the trick. This was Sydney at its best

We got a good range of sessions this time around with frontenders representing more than usual. We had talks on styleguide driven development, require JS, CSS units, and beautiful accessible CSS. Who says that all camps are backend focussed? :)

The Drupal 8 training provided by Kim and Magda was very well received and these sessions were very well attended, packing out the training rooms we had booked. This goes to show that there is a lot of interest for Drupal 8 knowledge and that people are receptive to training materials at a Camp, especially when it is free :) I think that this could serve as a blueprint for future Camps we put on.

Some reflections

This was the third Camp the Sydney community has put on in the last 15 months - all have been around the same size of 40 to 50 people, which seems to be the long term size of the community. Interestingly, there are many new faces coming along to the events these days, indicating that the community is being renewed as time goes by. There is perhaps a slight increase in momentum as we continue to put on events which have a good programme and are well attended. We hope to continue this momentum with the monthly meetups and socials which are held. If you haven't come along to a Sydney meetup yet, we'd love to see you there.

This time around we decided to go with the Drupal Association for the backer of the event. The DA provided us with a bank account and insurance to cover the venue hire. One important feature of the offering from the DA was the ability to retain any leftover "profits" in the account for future use by the community, after the deduction of a small 10% fee. Whilst we did not have the intention of making money from the event, the contributions from sponsors meant that we did have some left over funds. This money can now be used to back future events, making it much easier to commit to securing a venue next time. This does help the stability of the community and will hopefully allow us to put on more ambitious events over time.

We were pleasantly surprised by the number of sponsors the event managed to attract. Our two gold sponsors (PNX, Catalyst) were quick to sign up at the $1500 level. This was very important for securing the venue and giving us the confidence to proceed. The pricepoint of $1000 proved too high for the silver sponsors and we decided to reduce it to $500. Dropping the price was a big hit with 6 sponsors coming on board pretty quickly at this level. This was a good lesson in how to plan out future sponsorsip levels. Its good to know that there is good support there at the grassroots level, with many smaller to medium agencies prepared to help out.

As usual, attendees are quite slow to sign up to events. We tried to minimise this effect through the use of an early bird scheme which had a 20% saving on the tickets. This worked somewhat, but it seems that there is no avoiding the last minute rush on tickets. In the end we were very happy with the numbers we got, filling the venue whilst avoiding overcrowding. 

It's my belief that attendees are swayed by the speaker list. They want to know what they can expect from the event. Other Camps, such as Melbourne take an unconference approach where the programme is decided on the day. We have some aspects of this, with BOFs and lightning talks, however, we do have a set programme of presentations. It appears that attendees to start to sign up a lot more readily when they can see some of the programme taking shape. I would say that sorting the venue and then a few speakers early is one way to get things moving.

We do try to keep things interesting by having a variety of spaces where people can interact. The venue was pretty good for this with a main presentation area, a coding and training room, a breakout area and a cafe just outside. This had all the bases covered and I would say that any future events we held should have a similar mix.

What is the effort in putting on a Camp? For DrupalCamp Katoomba 2014 I tracked around 40 hours of time to organise it. This time around it was a little more involved. Magda (sponsors, speakers, accounts, misc) and I (speakers, venue, wifi, food, bar, insurance, misc) would have put in around 70 hours into the leadup to the event. This is a fair amount of work but it's well worth it when we get a good number of people coming along.

Massive thanks to Colin (for the logo), Chloe (for the site), Plamen (for the wifi), Ivan (for all round help) and Ian (for working the door). Big shout out to all of the speakers and trainers who took the time to prepare and presents such great material.

We will no doubt be looking to put on a similar event next year. We would love for some new people to step up and bring their own energy and ideas into the mix, so if you feel up to it, please get in touch with any of the Sydney Organisers and let them know that you would like to be involved. Its not so hard putting one of these things on and there are lots of people around who can help out and give advice.

Thanks to everyone who came along - I'm sure we all had a great time and learnt a thing or two as well.

Some pics

The full photo gallery on the Meetup page for the event has a stack of photos of the event - with 65 pics and counting. Here are a few selected images copied from the gallery with the permission of the photographers involved.

May 09 2014
May 09

It's been a while since the Sydney Drupal community put on a Drupal camp. It's time for that to change.

A few of us were sitting around the airport bar at the conclusion of DrupalSouth and hatched a plan to put one on "sometime soon". We kicked around a few crazy locations such as Broken Hill and Bali - somewhere a bit different. We eventually got our act together and decided on a weekend in the Blue Mountains as the venue for the event.

Drupal Camp Sydney 2014 is being held at the Katoomba YHA on the weekend of June 28 to June 29. Katoomba YHA on June 28 to 29 2014

It's going to be an initimate event with 40 tickets up for sale over both the days. The venue is a large, single room with a few different spaces for collaboration. There will be presentation, chillout, mentoring and coding areas. We also have access to a kitchen and the room 24 hours. Basically we can make the space our own for the weekend.

We hoping to get a good range of people up there from those just starting out their Drupal path to old hands. There will be volunteers on hand to answer questions people may have.

To find out more about the event please visit the Drupal Camp Sydney 2014 microsite, the GDO announcement or the Eventbrite page to book your tickets.

We hope to see you there.

Feb 05 2014
Feb 05

Cyclone is a new Drupal module which enables users to deploy sites to a variety of platforms including Pantheon and Aegir.

The main aim of Cyclone is to put control into the hands of the end user, to give them the ability to spin up sites when they desire. A secondary aim is to make it easy for publishers, such as agencies, entrepreneurs and ditro builders, to publish site templates which can easily be browsed and selected by users.

The desired outcome is that we have more Drupal sites being created and more happy users of Drupal.

Drupal hosting APIs

The Drupal hosting market has come along in leaps and bunds over the last few years. Companies such as Pantheon, Aberdeen Cloud and Acquia have developed platforms which are very developer friendly, allowing for the easy deployment of code between environments: dev, stage and production. They augment the open source Aegir project which provides a powerful web front end to the provision project, allowing sites to be installed, cloned and migrated between platforms.

Each of these platforms offer APIs to control various part of the system. Aegir is perhaps the most direct, being built entrely around Provision and Drush. It is essentially just a front end to components designed to handle sites and platforms. Pantheon is another standout. Their Terminus CLI allows control of many aspects of sites including creating sites from Drush archives and installing for base "products. Acquia offer a RESTful CloudAPI which is able to control all aspects of installed sites. Aberdeen has command line tools and is soon to release a RESTful interface as well.

In order to faciitate user provisioned sites, these APIs must support basic site clone or install functionality. Once these methods are available, Cyclone is able to use them to create sites. At the time of writing only Aegir and Pantheon offered such methods. Cyclone therefore targets either of these two platforms.

Introducing Cyclone

Morpht will be presenting Cyclone and the companion project Cyclone Jenkins, at DrupalSouth, in Wellington. We will be demonstrating how it works as well as some of the code behind it. We will post the presentation here once it has been uploaded.

Feb 21 2013
Feb 21

Video of DrupalCon Sydney 2013: Information architecture for site builders

Information architecture for site builders video presentation presented at DrupalCon Sydney 2013. Find out about the horizontal aspects of Drupal, modelling multi typed objects, relationships and tips for building faceted classification.

There were heaps of great presentations and conversations. It came and went way to fast. Morpht was lucky enough to give one presentation for the site building track: Information architecture for site builders. Thanks to all those who came along and asked questions at the end.

Aug 04 2012
Aug 04

Video of CDNs made simple, fast & cheap - Murray Woodman

The time it takes for a page to be delivered to a user is very important. The shorter the better. Sites which suffer long download times will typically have higher bounce rates and lower conversions than sites which are faster. Google has recognised the importance of a snappy site by rewarding faster sites with better search results rankings.

Cdn great deliveryHaving a fast site is a no brainer. The question becomes, "how do I make sure I have a fast site"?

Hosting your site on an optimised platform with well placed caching is a big part of having a fast site. However, the majority of download time experienced by a user is in fact due to waiting for resources to be downloaded and rendered in the browser. What happens on the server plays only a small role in delivering a site promptly.

The use of a Content Delivery Network, or CDN for short, allows for "heavy" resources such as videos, audio and images to be downloaded quickly from a server which is close to the user. Instead of requesting a file which may reside on the other side of the planet, a CDN will attempt to serve a file which is much closer, resulting in faster download speeds.

Morpht can easily configure your website to use a CDN if your site requires it. This will be very helpful for sites which fall into one of the following categories: heavy use of media files or many overseas visitors.The end result will be pages which are served and rendered faster for your users.

Photo credit:

Feb 11 2012
Feb 11
Beer and developer conferences go hand in hand.

A few weeks ago I presented “CDNs made simple fast and cheap” at the Drupal Downunder conference in Melbourne Australia.

The talk covered:

  • the importance of good client side performance,
  • how A CDN works,
  • recommended CDN providers (from an Australian’s perspective),
  • a demonstration of how to set up a CDN and
  • a summary of the results (better YSlow score and page download times).

Setting up a CDN is very easy to do and cost effective. If you want you users to have the best online experience then there is nothing stopping you!

The CDN presentation is available as PDF slides and a video.

Thanks to my employer PreviousNext who kindly sponsored my trip to Melbourne. Hats off to Wim Leers for contributing the CDN module.

[embedded content]

Be Sociable, Share!
May 20 2011
May 20

Last night I made a presentation on the “Business of Drupal” to the Sydney Drupal users meetup. The talk covered the subject areas of scalable jobs and wild randomness, basic business models in the software industry, the GPL, eight business models for Drupal in increasing order of scalability, ways developers can deepen their skills and a round up of how various organisations in the Drupal community are structuring the way they do businesss. I have just uploaded the slides to the talk.

For those wanting a little bit more detail without going to the slides, I’ll reproduce some of the content here.

Drupal business models in increasing order of scalability

1. Employment

  • Employment at Drupal shop or company
  • Income limited by salary (skill, experience)
  • Non scalable
  • Very regular

2. Pure services

  • Contractors, Drupal shops, F2F training
  • eg. Cross Functional, Previous Next
  • Income limited by incoming jobs (supply) and staff
  • Non scalable due to staffing requirements
  • Variable regularity, no subscriptions

3. GPL products with services

  • Distribution owners, module authors
  • eg. Phase2, Ubercart
  • Income limited by product popularity and staff
  • Non scalable due to staffing requirements
  • Variable regularity

4. Drupal hosting platform

  • Drupal hosting
  • eg. Acquia Dev Cloud, Managed Cloud, Chapter Three Pantheon, Omega8cc Aegir
  • Overhead of maintaining platform – Aegir
  • Scalable
  • Regular

5. Drupal as a service (DaaS)

  • Drupal running as a SaaS
  • eg. Drupal Gardens, Buzzr,
  • Overhead of maintaining platform
  • Scalable
  • Regular

6. Software as a service (Saas)

  • Service accessed via bridge module.
  • eg. Mollom, Acquia Solr
  • Overhead of maintaining platform
  • Scalable
  • Regular

7. Products with some non GPL code

  • Themes
  • eg. Top Notch Themes
  • Overhead of deloping product
  • Scalable
  • Irregular
  • Problem: Is the main IP in the code or the images?

8. Products with all non GPL code

  • Online training, documentation, books
  • eg. Lullabot
  • Overhead of deloping product
  • Scalable (online training)
  • (Ir)regular

Possible areas of specialisation for service providers

  • Data migration: Data is like wine, code like fish
  • Theming: Where are the themers?
  • Custom module development
  • Project scoping
  • Verticals: distros
  • Server admin, deployment (?)
  • Performance (?)

The main takeaway idea from the talk was that working in non-scalable areas such as full time employment is a safe option which will yield good results so long as you have skill and apply yourself. However, exposing yourself a little to some “wild randomness” in the form of scalable ventures (startups, SaaS, distros) could be a worthwhile pursuit if you are successful.

Be Sociable, Share!
Jan 12 2011
Jan 12

The conclusive article on a four part series on mobile sites in Drupal. Previous articles have discussed the groundwork, setup and code required for setting up a mobile site in Drupal. It’s now time to reflect on a few of the challenges thrown up and the best way forward.


Given the above discussion there are a couple of missing pieces to the mobile jigsaw puzzle as I see it.

Device vs Theme

There should not necessarily be a one to one mapping from device (mobile, desktop) to a theme. This certainly is the pattern within much of Drupal (theme blocks, theme caching strategies). This pattern is achieved by making sure that theme_default conf variable is defined in settings.php. The theme is the switch for customisation. However, if this assumption holds we will never see single “responsive web design” themes developed for Drupal as they rely on a single theme to serve to multiple devices.

Global variable device

It’s important to have a device variable easily available to all code. The best approach would be to set a well known variable in conf in settings.php. This could be based on a header extension (x-device), a regex on the UA, a cookie, the URL or a combination of them. The important thing is that is is an agreed upon variable.This variable is then available for all conditional tests in modules as well as to Context. Both the mobile and desktop versions of the site could be driven by a single progressively enhanced theme.


The Blocks UI is dated and on the way out, to be replaced by Context or other solutions. Context works well for controlling Blocks but it does have troubles with supporting theme based configurations.

In the issue queue for Context there has been some discussion around using Theme for a condition. IMO it would be great if Context could support a “conf variable” condition so that it would be possible to add conditions to check for a “device” global variable set in settings.php. It would then be possible to trigger Block reactions based on this variable. This would free up the possibility of a single theme supporting both sites.

Module loading

Being able to control the loading of modules would be a helpful addition. This would allow for turning off whole slabs of functionality not needed for mobile, providing a much better solution than mopping up HTML, JS and CSS in the theme layer. This would require changes to core so I can’t see it happening. in the meantime we have Mobile Tools and permissions.

Better caching

Upgrading caching layers to handle a “device” prefix to the key would enable a single domain to be used for serving the site. Boost is heading down this path already. There are other solutions available for Varnish.

Progressive themes

And finally we need some themes to experiment with responsive web design. From a practical perspective, my project had some slight annoyances because I was using two very different base themes: Fusion and Nokia Mobile. Translating my mobile subtheme across from the Desktop version was beset with a number of issues mainly to do with regions and bespoke features within the themes. If theme designers could supply a mobile counterpart for their desktop themes life would be easier. Even better if some themes were designed with progressive enhancement in mind.

Write up guidelines for mobile themes A brief discussion on implementing a mobile theme for Zen.

Up and running

If you want to get a mobile site up and running today then my recommendations are:

  • Think about mobile first and the core actions and data for your users.
  • Setup two sites, either multisite or with a settings.php conditional. Caching should work well with this.
  • Put User Agent mod_rewrite testing in Apache to handle redirects.
  • Some reverse proxy foo can serve one canonical domain if you know how.
  • KISS and be frugal with your module choices. Mobile Tools is a good option.
  • Select your mobile theme carefully and ideally match it up with the desktop theme.
  • Spend time on tweaking content in a custom module. Be a firefighter for a while :( .
  • Test.

I think there is a kernel of another idea in these articles as well and that is for a full adoption of a mobile first strategy for building websites in Drupal. With some small changes in mindset and code outlined above it should be relatively easy to do. This would allow the development of progressively enhanced themes, served from a single domain. The information architecture of Drupal would be improved significantly because we need only one site, one theme, are more RESTful, just as scalable, with simpler CSS, simpler SSL and simpler DNS and no duplicate content issues. Nirvana.


Some bedtime grazing…

Mobile Group GDO group. Training video (217M) “The first part of the training, which is an overview of basic theory of building mobile-accessible websites, is followed by a practical, hands-on component that steps through the practice of making mobile websites with the popular Drupal framework.” Interesting discussion from Drupal Everywhere: Making Mobile Websites with Drupal but not all solutions would be considered best practice. mobiForge “The world’s largest independent mobile development community” Programming the Mobile Web [Maximiliano Firtman] Book from O’Reilly, 2010 Mobile Design and Development: Practical Concepts and Techniques for Creating Mobile Sites and Web Apps [Brian Fling] Another book from O’Reilly, 2009 Be Sociable, Share!
Jan 12 2011
Jan 12

Previous articles have discussed the conceptual groundwork and setup of mobile sites in Drupal. It’s now time to look at a number of themes and modules which will help you in your endeavours. We’ll also cover a number of custom fixes which can be made to the HTML, CSS and JS in your site.

Mobile themes

Funnily enough, the selection of the mobile theme is looking to be one of the least important technical consideration with the whole mobile site. It’s one area where I am not in the best position to comment on the various merits of themes as I haven’t really tested theme all. I went with Nokia Mobile because it seemed to have a solid base, being based on code developed by Nokia. That said, I did have to make a number of changes to it to get it to work in with my site. Be prepared to get your hands dirty with page.tpl etc. The Adaptivetheme Mobile theme looks quite promising, being a sub theme itself it would naturally fit well with a desktop theme derived from the same base.

Nokia Mobile “Provides different presentation layers to best serve basic devices and high-end smartphones.” Mobile Garland Garland inspired mobile optimized Drupal theme intended to be used with a mobile optimization module Mobile Plugin. Adaptivetheme Mobile Hurrah! A mobile sub theme. “Adaptivetheme Mobile is a subtheme for Adaptivetheme. It is designed purely to build Drupal mobile themes for mobile web devices (for mobile websites).” Mobile “intended to return only clean HTML with no styling (images and styling in content is maintained) .mobi “Display a mobile, portable format.”

Mobile modules

There are a lot of options available to you when it comes to deploying modules to help you with your task. I am very much of the mind that modules should only be deployed if they are fit for the task and don’t introduce too much overhead or code you don’t understand. My aim is to keep things as “pure” as possible. In many cases you may be better writing your own custom code if you feel comfortable doing that.

Many tutorials recommend going with Domain Access and Mobile Tools with Browscap. It is a combination which could work well for you. However, I ended up not deploying any of these modules, chosing to go my own way. I’ll walk through each of the modules, their main features and why I went the way I did. It basically boiled down to the fact that Apache, settings.php and my own custom tweaks got me most the way there.

Domain Access

Domain Access is a popular suite of modules which can be used to manage (sub) domains for a (mobile) site. It is exceedingly well documented and structured. It looks to be a high quality module which supports a lot of functionality. Many mobile tutorials speak highly of it and recommend it for mobile sites.

Knowing relatively little about the module I reviewed its main features to see what it had to offer a mobile installation. From my quick review I have been unable to find anything compelling for the problem set I was facing. That said, if your mobile site is to diverge significantly from the desktop site you may find that some of the customisation features quite useful. There may well be stuff that I am missing and would be happy to be enlightened. The relevant features are as follows:

  • Domain Access: The core module allows for (sub) domains to be registered. This is really just the basic setup for the modules. In order for this to work your DNS and VirtualHosts need to be set up as you normally would for a multisite. ie. each domain pointing to the IP of your Drupal installation.
  • Domain Alias: It is possible to define domain aliases for each registered (sub) domain. eg -> Alternatively, this result could be achieved by adding some aliases in you VirtualHost section in Apache.
  • Domain Theme: Allows you to define a theme for each (sub) domain. Alternatively, if you were using a multisite setup (or some conditional logic) you could set the default theme in settings.php.
  • Domain Config: Offers an extensive set of site configuration options including email, slogan, mission, footer, frontpage, anon user, admin theme, time, cache, menu mappings. Most of these tweaks can be achieved by traditional means. Conf variables can be overridden in settings.php. Custom menu blocks can be placed into regions.
  • Domain Source: Source domain for linking to content. This ensured that some links are rewritten to point to main site. In a mobile setup you would want the site to operate as normal (no rewriting). The duplicate content can be fixed with canonical URL link in the head.

Mobile Tools

Mobile Tools is a popular module which has a bunch of handy utility features most mobile sites could use.

  • Detection of UA and redirect based on Browscap and WURFL:Possible to map user agents to themes. More sophisticated if Browsercap or WURFL is used. This redirection should be taking place outside of PHP so I am a fan of doing this in Apache rewrite rules or maybe even a caching/reverse proxy layer. This alternative approach has been discussed above.
  • Block telling users of Mobile site: Helpful but easy to add manually.
  • Panels integration: No doubt very helpful if you are using Panels, as Panels own a path and that’s it. This could be a deal breaker so this could be essential. Personally, I stuck to very simple design so Panels wasn’t an issue for me.
  • Permissions integration: Mobile roles can be used to turn block aspect of the site based on permissions. This is a really good idea and a neat way to turn stuff off.
  • Change number of nodes on homepage: Helpful but could be done with a different view exposed as a block.
Drupal Support for Mobile Devices [Rachel Scott] Overview of the Mobile Tools module with screenshots. Mobilize Your Drupal Site with Mobile Tools Overview of the Mobile Tools module describing multisite setup.

Mobile Plugin

Wide range of features. Tackles some practical isses such as word breaks, scaling images, video embedding, filtering JS. Does device detection and provides own mobile theme. Unfortunately the doc specifies that “Page caching must be off to support multiple themes!”. This warning would put most people off. Does this apply even if two sites are being used?


A popular module which returns capabilities based on user agent. The module will fetch updates to a database of browser user agents. Integrates with Mobile Tools.


“The WURFL module helps you in detecting the device capabilities of the mobile device visiting your website.” Integrates with Mobile Tools. Knowing the capabilities of a device at a very fine level of granularity could be helpful if you are into eeking out every enhancement you can. the question is whether you need this level of control.

Module code vs Theme code

Adding a mobile version of your site will make you think about code duplication issues. If you have custom logic in your theme for the desktop site then there is a pretty good chance that a large chunk will be copied across to the mobile site. Bad news. Much of what makes it into themes is not 100% concerned with presentation. It’s hard to draw a line but if the code is required for mobile and desktop then it is a good candidate for being factored out into a more central place such as a module. Less code means less work for you in the future. If you do have custom code in template.php then take a look through it and see what can be moved.

Custom content

Not all changes can be made in the theming layer, it will be necessary to change and optimise the content served.

Custom blocks

Managing block configuration (region, order, title, paths, permissions, etc) is a right royal pain in the you know where, especially if you have a lot of blocks and you need to deploy across dev, staging and production. Going into the blocks admin interface and moving stuff around, editing, saving and repeating gets old real quick. Configuration concerns such as this have been overcome largely though code from DevelopmentSeed. Features to hold logic and configuration for grouped functionality. Features work nicely together with Context, which allows for Blocks to be positioned according to an overarching context. Cool. Context could be the answer we are looking for. It certainly is for a normal desktop site.

However, when it comes to configuring blocks for a mobile site, Context only knows about the current theme. This is a known issue for Context. There is another module, called Features Extra which possibly offers a way to capture config info for blocks, however it too suffers with themes. AFAICT it still isn’t possible to capture block config with multiple themes. Bummer. I’d be interested to know if there are solutions here.

In the meantime you can manually configure blocks the old school way but it really isn’t ideal.

Custom modules

This is one area I was unable to nail as well. In a few places it would have been very handy if I could have turned off a module dynamically to make the mobile site a bit simpler, eg. colorbox, admin menu. AFAICT there is no way to do this. Tracing the calls during bootstrap, I see that module_load_all() is called very late in the procedure at _drupal_bootstrap_full(). module_load_all() calls module_list() which gets all active modules. It would be great if module_list() could look to conf variables to respect a stop filter of modules. Not going to happen I know, but would be handy.

This is where the permissions integration in Mobile Tools could really shine. Instead of disabling a module you could control the operation of a module via permissions. Most modules should have permissions limitations on functionality and so can be turned off for the mobile site.

One way to work around this is to mop up the HTML/JS/CSS in the theme layer. This approach is ugly, error prone and brittle, but does hold some promise. You will find recipes similar to the following around the traps:

* Implementation of hook_preprocess_page().
function mw_mobile_preprocess_page(&$vars) {
if (!cuf_mobile_is_mobile()) { return; }
// Strips out JS and CSS for a path.; //
// WARNING: The code below messes up jQuery Update even when no scripts are
// replaced. Use at own risk.
$remove_csss = array(
//’colorbox’ => array(‘/styles/default/colorbox_default_style.css’),
$remove_jss = array(
//’colorbox’ => array(‘/js/colorbox.js’, ‘/styles/default/colorbox_default_style.js’),
// JS
$scripts = drupal_add_js();
if (!empty($vars['scripts'])) {
foreach($remove_csss as $module=>$paths) {
foreach($paths as $path) {
$module_path = drupal_get_path(‘module’, $module);
unset($scripts['module'][$module_path . $path]);
$vars['scripts'] = drupal_get_js(‘header’, $scripts);
// CSS
$css = drupal_add_css();
if (!empty($variables['css'])) {
foreach($remove_csss as $module=>$paths) {
foreach($paths as $path) {
$module_path = drupal_get_path(‘module’, $module);
unset($css['all']['module'][$module_path . $path]);
$vars['styles'] = drupal_get_css($css);

In the end I gave up on going down this path because I was running into a problem with jQuery not being updated, leading to JS errors on the page. It was too brittle for me to trust.

For me, the take away is that you are pretty much stuck with using the some modules if you are sharing the database. You just have to be aware of this when designing the site. The only way to solve this is to place some conditional login into you own custom modules which checck for the site being mobile. If you are using contrib then things will be a trickier.

You may desire have custom primary and secondary links for the mobile site. If you really have thought mobile first then maybe the menus will be the same :) but there’s a good chance they will be paired down for the mobile site. It’s not possible to easily define two sets of primary menuas, one for mobile and one for desktop. However, Mobile Tools offers a way to map primary/secondary menus to other menus. There are two other options though if you don’t want to install Mobile Tools.

  • Define different menus (eg. Primary Mobile) and drop them into desired region using Blocks. Comment out the primary links in page.tpl.
  • Programmatically set the links in a custom module

In the end I just programmed these menus in code in my mw_mobile module because the menus had some logic in them for login/logout links:

* Implementation of hook_preprocess_page().
function mw_mobile_preprocess_page(&$vars) {
if (!mw_mobile_is_mobile()) { return; }
// Completely hijack the primary menu and set it from code. This allows
// the primary menu to be managed in features for the desktop site. We just
// need to oveerride it here.
$vars['primary_links'] = array();
$vars['primary_links']['blah'] = Array (
;’title’ => t(‘Blah’),
‘attributes’ => Array(‘title’ => ‘Blah.’),
‘href’ => ‘blah’
// etc

Custom Views

This section really gets back to the “mobile first” and “responsive web design” concepts we discussed earlier. Views are very powerful and there is a strong temptation to make them as sexy as possible, displaying images, extra content, edit links, star ratngs and the like. Step back and take a look at what you are doing. It maybe possible to design a simple display which works well in mobile and desktop.

Often you really do want to display rich tabular information in the desktop version of the site. In these cases you shouldn’t compromise – you’ll need to create different versions. In these cases progressive enhancement doesn’t really cut it as you want to return more content, not just tweak the presentation.

If it is a View Block which is giving you grief then just make a mobile version and use that instead. Use the block system to place different blocks for the different themes.

If it is a View Page then you could be in trouble as the View takes hold of the path and it is difficult to customise that on a per site/theme basis. One solution is to expose the View as a Block (with a mobile version) and then place that block on a Page (node) or menu path you have made. In this case the page is acting like poor man’s Panels. A bit ugly but it works.

If you are lucky you might be able to define a custom formatter which just returns a blank (or a simple version) if the site is mobile.

A final alternative is to define View templates which have some conditional logic in them. This is possibly the purest way but I think it could become a maintenance issue. We are trying to minimise duplication and effort – creating new files with extra display logic is best avoided.

Custom Panels

I’ll come clean here and own up to not having caught the Panels bug just yet, being content to limit my degrees of freedom. Yes, I am that boring :) Anyway, Panels faces a similar problem as Views Pages in that they are tied to a path which isn’t scoped by a theme (as Blocks are). In this case, Mobile Tools could be quite helpful and showing different layouts for a Panel.

Custom Variables

Drupal can be configured with a whole bunch of variables, many of which are available for editing in the site configuration part of the site. Fire up phpMyAdmin and browse the variable table to get an idea of what is available. These variables are site wide and as such will apply equally to the mobile and desktop versions in our multisite setup. It is possible to override these variables for the mobile site by tweaking settings.php. We have already seem this in action for the default theme. You can do it for other things as well. Mobile Tools offers an interface for this but you can do it manually. I have found that only a small number of rarely changed variables need to be tweaked and so settings.php is a viable option.

$conf = array(
‘theme_default’ => ‘mw_nokia_mobile’,
‘googleanalytics_account’ => ‘UA-xxxxxxxx-2′,

Output tweaks

Mobile devices have special requirements, not all of which could be handled by the theme templates alone. The metadata and content of the sites may need some special work. The Rethinking the Mobile Web slideshow above noted that we need to adjust and compress content to make it palettable for mobile. This is where a lot of that nasty work happens. You’ll probably only run into these issues after testing the site for real. No doubt you will have your own special set of problems to deal with. The Plugin module plugs some of these holes.


You probably have a bunch of ImageCache presets defined for your site, which may or may not be captured in Features config. These presets may be outputting images at a size which is too big for the mobile site. Anything wider than 100px is probably too big. You are aiming to be frugal with bandwidth as well as screen real estate. Time to get shrinking those images. See hook_preprocess_page code below.

Secure Login

If you are using the Secure Login module, you may run into difficulties if you have the multisite setup. The way I had Secure Login configured was to specify the URL to redirect to. This URL is of course for the desktop version of the site and your mobile users will be routed to the desktop site after they log in. They may not notice it if URL redirecting is working for mobile users but we would want to minimise redirects such as this.

It is possible to leave the Secure Login URL blank and then it will apparently use the base_url defined in settings.php. This would be a sensible approach, however, I was having all sorts of path difficulties with ImageCache if I specified these URLs. Don’t know why. Anyway, the easiest solution for me was to stick with the hardcoded URL for Secure Login and then to fix it up in the module.

* Implementation of hook_preprocess_page.
function mw_nokia_mobile_preprocess_page(&$vars) {
// imagcache images are shrunk to mobile size
$fix_regions = array(‘content’, ‘right’, );
foreach($fix_regions as $fix_region) {
// secure login will target the url you entered into the site config.
// there might be a better way to fix this but we just string replace here

* Secure login hardcodes the URL entered in the config: securelogin_baseurl
* This will be the desktop version of the site. We need to change it to the
* mobile version. There isn’t an obvious way to do this via code, unless you
* write your own hook_form_alter but that would usurp the function of
* securelogin. So we just mop up afterwards. These login pages will be
* cached anyway.
* eg ->
* NB: It MIGHT be possible to leave out securelogin_baseurl in the config and
* manually set the base_url in the settings.pgp for the site. However, when
* I did settings like this in the past I ran into problems… can’t remember
* what they were now… So this might be a solution which would avoid the need
* for this code.
function _mw_nokia_mobile_fix_secure_login_form(&$text) {
if (!module_exists(‘securelogin’)) {
$sec_url = variable_get(‘securelogin_baseurl’, ”);
if (empty($sec_url )) {
$new_url = str_replace(‘https://’, ‘https://m.’, $sec_url);
$pre = ‘<form action=”‘;
$paths = array(‘/user/login’, ‘/user/register’);
foreach($paths as $path) {
$search = $pre . $sec_url . $path;
$replace = $pre . $new_url . $path;
$text = str_replace($search, $replace, $text);

* Map imagecache presets to smaller presets. This is VERY UGLY because you
* need to correct for the width and height as well. Sorry to have impinged
* upon your senses!
* Adapted from
function _mw_nokia_mobile_fix_imagecache(&$text) {
if (!module_exists(‘imagecache’)) {
// mappings. ignore: slider_perview, slider_thumbnail
// old ic, old width, old height, new ic, new width, new height
$mappings = array(
array(‘thumbnail_small’, ’83′, ’83′, ‘mobile_thumbnail’, ’75′, ’75′),
array(‘thumbnail’, ’100′, ’100′, ‘mobile_thumbnail’, ’75′, ’75′), // thumbnail last
// fix
$file_url = base_path().file_directory_path();
foreach($mappings as $mapping) {
list($old, $old_w, $old_h, $new, $new_w, $new_h) = $mapping;
$old_path = $file_url .’/imagecache/’ . $old;
$new_path = $file_url .’/imagecache/’ . $new;
$old_class_size = ‘imagecache-’ . $old . ‘” width=”‘ . $old_w . ‘” height=”‘ . $old_h . ‘”‘;
$new_class_size = ‘imagecache-’ . $new . ‘” width=”‘ . $new_w . ‘” height=”‘ . $new_w . ‘”‘;
$text = str_replace($old_path, $new_path, $text);
$text = str_replace($old_class_size, $new_class_size, $text);

Search snippets URLs

I’m not sure if the following applies to ordinary Drupal search but it certainly does with Apache Solr Search. The URLs for the individual search results were coming back with the fully qualified URL pointing to the desktop site. This was solved by a bit or mopping up in a base feature, mw_base.

function phptemplate_apachesolr_search_snippets($doc, $snippets) {
// mobile site?
$mobi = module_exists(‘cuf_mobile’) && cuf_mobile_is_mobile();
$url = $doc->url;
if($mobi) {
$url = str_replace(‘http://’, ‘http://m.’, $url);


The combination of Location and GMap is a very popular one on Drupal sites. GMap module is currently targetting version 2 of the Google Maps API. Version 3 offers a bunch of new features for mobile devices.

Google Maps JavaScript API V3 “The Google Maps Javascript API lets you embed Google Maps in your own web pages. Version 3 of this API is especially designed to be faster and more applicable to mobile devices, as well as traditional desktop browser applications.”

For now users of GMap are stuck on v2 but there is active development in GMap to bring the module up to support v3.


WYSIWYG textareas do not display properly on some mobile devices. You need to turn them off.

WYSIWYG on mobile devices Discussion on WYSIWYG issue queue regarding the difficulties faced on a variety of devices. End conclusion appears to be that you need to turn it off for best results.

How to turn of WYSIWYG? After a bit of poking around I worked out that setting the ‘format’ for the textarea to an empty array was the way to do it. The following code in your mobile module will do the trick for comment and node bodies. If you have other forms which need fixing then you’ll need to do a bit of debugging to suss out what the form_id and element id is.

* Implementation of hook_form_alter.
function mw_mobile_form_alter($form_id, &$form) {
if (!mw_mobile_is_mobile()) { return; }
// turn off wysiwyg forms for textareas. you need to manually find the form
// with wysiwyg and then work out its id and where ‘format’ is.
//print $form_id['#id'] . ‘ ‘;
$no_wysiwyg = array(
‘comment-form’ => ‘comment_filter’,
‘node-form’ => ‘body_field’
$id = $form_id['#id'];
if (array_key_exists($id, $no_wysiwyg)) {
$form_id[$no_wysiwyg[$id]]['format'] = array();


The primary and secondary tabs which appear on the top of the page tend to take up a fair amount of horizontal space and will be the element which causes horizontal scrolling. These tabs can be easily restyled to display as a traditional list. You can also update page.tpl and move the tabs to the bottom of the page so they don’t detract from the main content.


Flash is not going to work in iPhones, iPad and many other devices. It’s also heavy and resource intensive. As such it shouldn’t reallu be used for content or navigation on the mobile site. The exception might be to show video content, however, even in this case there might be better workarounds.


Suckerfish provides dropdown menus which can take up a lot of room. The hover metaphor doesn’t work for touch devices. Best avoided.

Make sure that links are big enough to clickable: large with enough whitespace around key navigation links.


Mobile devices such as the iPhone and iPad may have a special app to handle YouTube videos natively so clicking on a link is preferable than displaying an embedded video.


Yes – ads can be optimised for mobile as well.

Google Mobile Ads “In May 2010 Google acquired AdMob, a leading mobile advertising network that developed innovative mobile-specific ad units and solutions, to significantly enhance our mobile display ad services for publishers.”


Testing an afterthought? Never :) The fact is that a lot of the hard work is in getting over the config hurdles. once the mobile site is up and running you are going to uncover a bunch of things you never dreamed about. Here’s a quick checklist of things to look out for:

  • UA testing and redirect working.
  • Boost/Drupal Core page caching working.
  • SSL login working OK.
  • Basic functionality and navigation operational.
  • Theming is up to scratch.
  • Site fulfills its goals.
Be Sociable, Share!
Jan 12 2011
Jan 12

A previous article covered some basic groundwork for mobile sites in Drupal. This article goes on to look at different ways to setup a mobile site in Drupal. It covers single site, multisite, single site with settings.php tweak and the Domain Access module. Caching strategies, redirect rules and other server side settings are also discussed.

RESTful design of URLS

REST defines the architecture of the World Wide Web. One of the principles of REST is that a single URI represents a resource and that resource is conceptually different from the representations returned to the client.

Representational State Transfer “Representational State Transfer (REST) is a style of software architecture for distributed hypermedia systems such as the World Wide Web. The term Representational State Transfer was introduced and defined in 2000 by Roy Fielding in his doctoral dissertation.[1][2] Fielding is one of the principal authors of the Hypertext Transfer Protocol (HTTP) specification versions 1.0 and 1.1″

Here’s the passage from Roy Fielding’s thesis (emphasis added) which discusses the differences between resource and representation:

“This abstract definition of a resource enables key features of the Web architecture. First, it provides generality by encompassing many sources of information without artificially distinguishing them by type or implementation. Second, it allows late binding of the reference to a representation, enabling content negotiation to take place based on characteristics of the request. Finally, it allows an author to reference the concept rather than some singular representation of that concept, thus removing the need to change all existing links whenever the representation changes (assuming the author used the right identifier).”

A resource is named by a URI. The server chooses the best representation to provide to the client based on headers sent by the client. In this case we are looking at the User Agent.

If we were to follow RESTful principles then the mobile site should indeed be served from the some domain as the desktop site. ie. one resource, different representations. In this scenario the HTML returned to the mobile client is just a different respresentation to that provided to the desktop site. This is a natural way to design a web app as it means that there is only one “canonical” URI for the resource with no chance of nasty duplicate content issues. From an SEO point of view this is desireable. However…

Caching, the fly in the ointment

We’ve just seen that serving different representations from a single URI is a good thing from many perspectives: mobile first, progressive enhancement, REST and SEO. However, there is one reason why we may we may decide to go down the path of using two domains instead of one: caching.

Caching mechanisms, such as Drupal Core and Boost, used the fully qualified domain name of a URI to determine caching keys. This allows the cache to quickly serve content to different clients without knowing the criteria which decides the representation received by the client, ie. the cache just has to know about the URI, it doesn’t need to decipher the user agent. Currently, if different representations are served for the same resource then the cache will likely become populated with a mix of different representations, leading to chaos. For this reason it is generally accepted that having a separate mobile site on a sub domain is a good way to go. ie. we would have two sites:

Cache by theme for mobile sites mikeytown2 offering some great advice on Apache and Boost rules. .htaccess Mobile Browser Redirect User Agent processing in Apache to redirect to mobile.

Some users have solved the caching problem AND manage to serve different representations from the same URI. Going mobile with a news site that Just Works describes how browser detection can be done in the caching layer, in this case Squid, before redirecting the request invisibly to another domain. This is the perfect setup as RESTful principles are maintained and the site is scalable. Hats off. Unfortunately not everyone is running a reverse proxy which allows for this kind of setup. A request looks like this:

  1. mobile client does GET,
  2. Squid (port 80) looks at User Agent, determines device and sends to,
  3. Boost finds “about” in /cache/normal/ -> Static HTML returned OR,
  4. Drupal serves from multisite -> Dynamic HTML returned.

mikeytown2 claims that it should be easy enough to add some logic into the Boost rules based on user agent, he just needs to know what they are. So there is a good chance that Boost user’s will be able to server both mobile and desktop from two sites with one URI space. From my understanding of the proposed approach it looks like a single domain will be all that is required.

  1. mobile client does GET,
  2. Boost looks at User Agent, determines device and uses a different “mobile” device rather than “normal”,
  3. Boost finds “about” in /cache/mobile/ -> Static HTML returned OR,
  4. Drupal serves from single site -> Dynamic HTML returned.

A slightly different approach has been described in Mobile Detection with Varnish and Drupal where Varnish sets a header which can then be read in the webserver or Drupal. This is a neat approach as it means that device logic needn’t be repeated in settings.php. The flow described by Morten Fangel is as follows:

  1. mobile client does GET,
  2. Varnish also sets a X-Device header for the device
  3. Varnish looks at User Agent, determines the device and appends it to the hash for the key
  4. Varnish “about” in cache -> Static HTML returned OR,
  5. Drupal serves from single site -> Dynamic HTML returned.

Assuming you don’t have Squid, Varnish or a patched Boost to hand you will probably have a setup as follows:

  1. mobile client does GET,
  2. Apache rewrite looks at User Agent, determines device and redirects,
  3. Drupal Core or Boost finds “about” in cache -> Static HTML returned OR,
  4. Drupal serves from multisite -> Dynamic HTML returned.

Sub domain vs Different Domain

If you are going to use a separate site to host the mobile site then you are free to chose whatever domain you like. eg. However, it is generally recommended to stick with using a sub domain of the desktop site. This confuses users less and it is possible to share cookies across sites on the same domain.

Different theme

As discussed in the previous article, it is possible to serve the same default theme to both mobile and desktop sites and then progressively enhance the desktop site with some extra CSS. The method proposed in Rethinking the Mobile Web at slide 106:

<link href=’default.css’ type=’text/css’ rel=’stylesheet’
media=’screen’ />
<link href=’desktop.css’ type=’text/css’ rel=’stylesheet’
media=’screen and (min-device-width:1024px) and (max-width:989px)’ />

This is a very cool way to design a site as it keep things very simple. Mobile is first and then comes the progressive enhancement. However, this isn’t a pattern which is adopted by most Drupal themes where the presumption is for the desktop theme. If we did take this approach it would preclude us from using the majority of themes designed for Drupal so far. Given this, I would say that our Drupal site will support two separate themes, one for desktop and one for mobile. The general approach is to use a multisite setup. define a desktop theme as default in the GUI and then to override that theme via a tweak in settings.php for the mobile site.

Multisite setup

Assume we are using two domains due to caching requirements. How do we serve this content? Drupal does have a multisite feature built in where a single Drupal “platform” can support many different site instances. These sites can share all data, no data or partial data, depending on how they are setup in settings.php. In the case of a mobile site we would want to share all data between the sites.

One possible setup is to create a directory for the desktop and mobile versions under sites/


  • all/
    • modules/
      • contrib/
      • custom/
        • mw_mobile/
    • themes/
      • base_desktop/
      • base_mobile/
      • mw_desktop/
      • mw_mobile/
  • default/
    • settings.php
    • settings.php

The only trick to get this work is to manually set the default theme for the mobile site in the sites/ file. For every page request, the config in settings.php will override the default variables defined in the variables table in the database.

$conf = array(
‘theme_default’ => ‘mw_nokia_mobile’,

If you manually set a value like this you won’t be able to change it in the UI, naturally enough. Make sure the theme is active in the GUI.

Alternative 1: Single site with settings.php logic

The above multisite setup will work, however, there is something wrong with it. It will stop you from hosting a true multisite setup where the sites share code but have different databases. This may not worry you if you are only hosting a single site for the platform but it could be important if you want multisites. Imagine a site for Company X served on and Company Y is on You couldn’t use multisites with the above settup because of the reliance on shared files in default/files.

However, you can achieve a very similar effect with a single site by using a bit of conditional logic in settings.php for and The idea is to set the theme based on the domain leading to only needing a single site to support desktop and mobile. Add this to sites/

$parts = explode(‘.’, $_SERVER['HTTP_HOST']);
if ($parts[0] == ‘m’) {
$conf = array(
‘theme_default’ => ‘company_a_mobile’,

You could then support mobile with a pure sites setup with shared code and different databases/files. This is a good way to go.


  • all/
    • modules/
      • contrib/
      • custom/
    • themes/
      • base_desktop/
      • base_mobile/
  • default/
    • files/ -> empty
  • -> company A
    • files
    • modules
    • themes
      • company_a_mobile
      • company_a_desktop
    • settings.php -> with conditional setting of default theme
  • -> company B
    • files
    • modules
    • themes
      • company_b_mobile
      • company_b_desktop
    • settings.php -> with conditional setting of default theme
multi site a) standard theme, site b) mobile theme – same code and same tables? Discussion of multisite setups.

Alternative 2: Domain Access

The Domain Access module, discussed later, can set a lot of this up for you including sub domains, domain aliases and themes. You may prefer to use it for convenience, especially if you like configuring stuff in a GUI rather than settings.php or custom modules.

Mobile global variable

Modules are going to want to access a global variable which tells them the device accessing the site: mobile or desktop. There are a variety of ways to do this, some set the variable early, others late:

  1. Custom “X-Device” header set in a reverse proxy
  2. Conf variable set in settings.php
  3. Global variable set by a module during hook_init()
  4. API function offered by a module

It is possible to do this through the use of hook_init() in a custom module. I tried this but ran into problems with timing and module weight. Sometimes you will want the mobile module to be heavy, sometimes light :) In the end I went with an “api” function in my mw_mobile module which stored a static variable. It should be pretty fast and not to cumbersome. Other contrib modules take an approach similar to this.

* An API function in a custom module
* Efficiently returns whether the site is mobile.
* Other modules should call it as follows:
* $mobi = module_exists(‘mw_mobile’) && mw_mobile_is_mobile();
function mw_mobile_is_mobile(){
static $out;
if (isset($out)) {
return $out;
// set and return
if (substr($_SERVER["SERVER_NAME"], 0, 2) == ‘m.’) {
$out = TRUE;
} else {
$out = FALSE;
return $out;

This approach is perhaps not the best. It may be better to set a global variable very early in the bootstrap process, in settings.php, so that it could be reliably used by all other Drupal code.

Cross site authentication

It is possible to set cookies up so that they will be sent no matter what the sub domain. In settings.php uncomment the $cookie_domain variable and set it to the domain, excluding the sub domain. Please note that this will not work if you are using different domains.

$cookie_domain = ‘’;

Redirecting the user to mobile

When a mobile user hits the desktop version of the site you want them to be redirected to the mobile site. There’s at least three ways to do this:

  • PHP
  • JS
  • Apache

The first inclination maybe to go with PHP as afterall, we are PHP developers. However, this has the shortcoming of requiring that Drupal be bootstrapped before the PHP can be run, destroying the chance to safely cache the page for anonymous users. It’s slow and ineffective. Doing it in PHP therefore isn’t an option. This is the approach some of the mobile modules take but I think it’s something to be avoided.

You could of couse do a client side check in Javascript for the client’s user agent. This will allow for caching but has the downsides of forcing a full page download. Also, not every client will have JS enabled. Not really an option.

The final option of doing it in Apache (or your webserver) is the only viable alternative. I went with a recipe similar to the following in my .htaccess.

# Mobile: force mobile clients across to the mobile site
RewriteCond %{HTTP_HOST} !^m\.(.*)$
RewriteCond %{HTTP_USER_AGENT} !ipad [NC]
RewriteCond %{HTTP_ACCEPT} “text/vnd.wap.wml|application/vnd.wap.xhtml+xml” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “acs|alav|alca|amoi|audi|aste|avan|benq|bird|blac|blaz|brew|cell|cldc|cmd-” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “dang|doco|erics|hipt|inno|ipaq|java|jigs|kddi|keji|leno|lg-c|lg-d|lg-g|lge-” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “maui|maxo|midp|mits|mmef|mobi|mot-|moto|mwbp|nec-|newt|noki|opwv” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “palm|pana|pant|pdxg|phil|play|pluc|port|prox|qtek|qwap|sage|sams|sany” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “sch-|sec-|send|seri|sgh-|shar|sie-|siem|smal|smar|sony|sph-|symb|t-mo” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “teli|tim-|tosh|tsm-|upg1|upsi|vk-v|voda|w3cs|wap-|wapa|wapi” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “wapp|wapr|webc|winw|winw|xda|xda-” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “up.browser||windowssce|iemobile|mini|mmp” [NC,OR]
RewriteCond %{HTTP_USER_AGENT} “symbian|midp|wap|phone|pocket|mobile|pda|psp” [NC]
RewriteCond %{HTTP_USER_AGENT} !macintosh [NC]
RewriteRule ^(.*)$ http://m.%{HTTP_HOST}/$1 [L,R=302] .htaccess Mobile Browser Redirect Outlines the approach taken above.

SSL issues

If you are using SSL on the desktop version of your site then you have a couple of extra hurdles to jump in order to get it working on the mobile site.

Firstly, as it isn’t possible to set up SSL for two different domains on the same IP address, you will probably need to rent a new IP address for the mobile version of the site. Sort this out with your ISP. It should cost you between $1 and $3 a month for another IP. They may have instructions for setting up A records, static routing for your IP addresses, etc.

Secondly, you will also need to sort out another certificate for the mobile site. You could purchase a wildcard certificate for the domain and all sub domains. These cost a fair bit more and will save you from buying a new cert. However, it is probably cheapest to get another cert for the mobile site along with a new IP. You will then need to install the certificate on your server and tweak your site config for the mobile site. This certainly is one of the pains of having two separate sites.

PositiveSSL from Comodo $10 pa. Gandi Free first year then 12 Euro pa.

Custom version of robots.txt

A corollary of having a shared database and file system with a multisite install is that you can’t have custom versions of key files such as robots.txt which sits in the root of your Drupal platform. In the simple case I don’t believe that there is any need to have a different version, however, if you do need to support different versions then you can do it with a bit of .htaccess magic. Place the following code under the mobile redirect rule. Just be sure to add robots.txt to the /sites/%{HTTP_HOST}/files/robots.txt.

# robots.txt: solve multisite problem of only one robots.txt
# redirects to file in /sites/files/robots.txt
RewriteRule ^robots\.txt$ sites/%{HTTP_HOST}/files/robots.txt [L] multi-site robots.txt GDO discussion of this approach.

Duplicate content, Canonical URLs and robots.txt

You now have two sites where the mobile site replicates all the content of the desktop site. This is a major issue as search engines such as Google will treat is as duplicate content leading to declines in ranking. We need to sort this out. Google came up with the concept of a canonical URL which can be defined in a link element in the head of the HTML page. In our case the link points back to the desktop site.

<link rel=”canonical” href=”” /> Specify your canonical Google documentation on how to define a canonical URL.

We need every page in the mobile site to support this tag. This can be set in your mobile module:

* Implementation of hook_init().
function mw_mobile_init() {
if (!mw_mobile_is_mobile()) { return; }
// Add a canonical URL back to the main site. We just strip “m.” from the
// domain. We also change the https to http. This allows us to use a standard
// robots.txt. ie. no need to noindex the whole of the mobile site.
$atts = array(
‘rel’ => ‘canonical’,
‘href’ => str_replace(‘://m.’, ‘://’, _mw_mobile_url(FALSE)),

* Current URL, considers https.
function _mw_mobile_url($honour_https = TRUE) {
$u = ‘http’;
if ($_SERVER["HTTPS"] == “on” && $honour_https) {$u .= “s”;}
$u .= “://”;
if ($_SERVER["SERVER_PORT"] != “80″) {
} else {
return $u;

The final thing to resolve is whether to set “noindex” on the mobile site. This definitely an area where there is some confusion on the web. After sniffing around I came to the conclusion that it was OK to allow Google to index the mobile site, so long as the canonical links have been specified. This means that any page rank given to the mobile site will flow to the desktop site and you won’t be punished for duplicate content.

The outcome is that you can go with the same robots.txt for both sites, ie. robots are free to index the mobile site. There is no need to specify a different robots.txt for mobile. You want the same stuff indexed for the mobile as you do with the desktop.

The one exception to this would be the files/ directory. A recent core update to 6.20 allowed files/ to be indexed. Fair enough, you want your public images to be indexed. However, you could raise the case that files/ shouldn’t be indexed in the mobile site, given that there is no way to specify a canonical link for these binary files. So, you may well want to support a different robots.txt for each site by blocking access to files on the mobile site. This is a very minor issue and probably not worth worrying about.

Be Sociable, Share!
Jan 12 2011
Jan 12

Recently I was involved in a project to convert a site into a mobile site for Drupal. During the process I had to overcome a number of problems – problems which pretty much everyone designing a mobile site will have to solve. This series of articles covers a bit of theory, the practical problems and the various decisions I made. Hopefully it will help some of you out. The solutions offered may not be the best or only solution by any means – in many cases you need to decide what is best for your site or setup.

Get amped for mobile

In the past it has been easy to not even consider mobile when designing a site or an application. However, there are so many compelling reasons these days to treat your mobile site as a key component of your web strategy. Mobile devices are becoming increasingly popular, more of your users will be accessing your site through a phone or tablet. They will expect a good experience from your site or else will not return. Watching a user attempt to browse your unoptimised desktop app on a mobile device is an embarrassing experience for you and frustrating for the user.

Drupal in a tablet world [Dries Buytaert] Dries opens up a discussion about the future of Drupal for mobile and tablets. “This all begs the question: in a world of tablets and mobile handheld devices, what do we need to do so that Drupal will be a go-to platform? How can Drupal contribute so as to be a player in the ever-expanding ecosystem of tablets and mobile phones?” Everything you ever wanted to know about mobile, but were afraid to ask [Tomi T Ahonen] A long article which covers the myths and misconceptions about mobile. Very useful for getting your head around just what a remarkable medium it is.

There are many reasons for choosing to develop a mobile web site as opposed to a “native app”. This doesn’t have to be an either/or decision of course, however, developing a mobile site first is probaly a sensible decision. Firstly, the fractured nature (programming language, feature sets, application stores) of the “native apps” space across the various mobile platforms (iPhone, Android, Blackberry, etc, etc) makes development of separate apps an expensive and time consuming process. WebKit and HTML5 offer a lot of promise for developing richer clients which approach the features provided by native apps. It is interesting to note that Google and Apple continue to support WebKit even though they have their own native app platforms. Secondly, the chances are that you are going to have more visitors hitting your site after searching Google than will download a custom app. Optimising this user experience from Google makes sense.

Design philosophy

Before jumping into the nitty gritty of tweaking Drupal up to accommodate a mobile site, it’s worth putting in a bit of groundwork. Planning for and designing a mobile website should involve taking a fresh approach to your site or application. By focussing on mobile, and it’s inherent limitations and features, we can concentrate on what is truely important to users. I thoughly recommend that you take 10 minutes and flick through the following slideshow:

Rethinking the Mobile Web by Yiibu Slideshow covering the old and new ways of doing mobile. “Progressive enhancement” and “Responsive web design” techniques discussed.

Of particular interest to us is Slide 43 which covers a soup of considerations we face when designing a mobile site. These are some of the thorny issues which crop up:

  • device detection,
  • content adaption,
  • multiple sites,
  • device capabilities,
  • doctypes,
  • multiple templates.

There are a lot of issues to deal with and this article will deal with many of them. There is no silver bullet to solving a lot of this stuff. However, we will go into the exercise armed with some solid principles which should serve us well. Slide 125 lays it out for us:

  • mobile first,
  • well structured meaningful markup,
  • progessively enhance,
  • adapt content,
  • compress content.

Drupal as a platform is pretty good at well structured markup, adaptable content and compressed content. Two themes worth pondering a little further at this point are “mobile first” and “progressive enhancement” as they will help us think about the mobile site in a functional way rather than just as a cut down version of the desktop site.

Mobile first

Mobile has traditionally been considered an afterthought, if at all. However the landscape has shifted and it is now vitally important. Luke Wroblewski, a strong proponent of the approach, identifies three key reasons:

  1. Mobile is exploding
  2. Mobile forces you to focus
  3. Mobile extends your capabilities

Focussing on the mobile platform forces the developer to prioritize the important things. What are the key actions and data for your application? Strip everything away and focus on just that. The end result will be an app which is simpler, easier to understand, just as functional and much better. This brings development back from flashy extraneous eye candy to a functional site which is designed to serve the user.

Some questions you might ponder. Why are your users accessing the site? What are the likely actions they want to perform? What information do they want to access? What’s the simplest way to do it? Do I need that navigation? Are those icons distracting? Is the page fast to load? Do I need a better algorithm for finding nearest locations…

Mobile First [Luke Wroblewski] Three reasons why Web applications should be designed for mobile first instead. Mobile First [Luke Wroblewski] “In this presentation, Luke Wroblewski will dig into the three key reasons to consider mobile first: mobile is seeing explosive growth; mobile forces you to focus; and mobile extends your capabilities.” Barcelona: Mobile First Eric Schmidt (Google) talks mobile. Google encourages developers to work on mobile first before desktop. “The phone is no longer the phone, it’s your alter ego. It’s the extension of everything we are. It doesn’t think as well as we do, but it has a better memory…. This is the time for us, now is the time for us to get behind this. … We understand that the new rule is mobile first.”

Progressive enhancement

One of the big conceptual advancements in web development in the last seven years has been a move away from “graceful degradation” towards progressive enhancement. Coined by Steve Champeon (of hesketh mailing list fame), progressive enhancement describes a process of starting with the basics and then building up functionality for richer clients. That is, instead of designing for a rich client and taking away bits and pieces for simpler devices, we start with semantic HTML and the progressively add CSS, Javascript and other goodies which improve the content.

Drupal has been very good in this regard. There is a strong emphasis on producing semantic HTML with no inline styles or Javascript. Inline images for presentation are avoided. Drupal pages rock when viewed with no stylesheet whatsoever. They are pretty pure. The ultimate expression of this is the Stark theme in Drupal 7 which shows the quality of the HTML produced by Drupal.

However, there is a very strong bias towards designing for Desktop sites. Many recent developments in theming revolve around areas such as 960 grid designs. Further, the use of jQuery as a default Javascript library means that many modules and themes make use of it is presenting a page. This isn’t bad on its own but does present problems when it isn’t easy to turn off. In short, Drupal is in a good position as far as progressive enhancement is concerned but there are a few hurdles to jump.

Inclusive Web Design For the Future with Progressive Enhancement [Champeon and Finck] The original presentation at SXSW 2003. Graceful degradation versus progressive enhancement [Christian Heilmann] Article from Opera with practical examples.

Responsive Web Design

Another meme which has emerged recently is that of Responsive Web Design. Ethan Marcotte wrote a short article for A List Apart coining the phrase which basically describes fluid layouts which work well on a variety of screen sizes. By using floated divs, which are relatively narrow, layouts can adapt to the screen size. The secret sauce here is the use of media queries to serve progressively enhanced CSS to clients with more screen real estate. The “mobile” site is the “default” site suitable for the most basic of clients and the “desktop” site is the enhancement.

Fluid grids, flexible images, and media queries are the three technical ingredients for responsive web design, but it also requires a different way of thinking. Rather than quarantining our content into disparate, device-specific experiences, we can use media queries to progressively enhance our work within different viewing contexts.

This approach is counter to the traditional way of theming in Drupal where the presumption is for the desktop site. Currently there aren’t any themes that I am aware of which take this approach. Anyone?. It possibly is an area for experimentation in the future. So while we won’t necessarily be able to immediately make use of the mechanics of the technique (media queries) the basic principle of using floated divs in areas such as Views is certainly one we can make good use of.

Responsive Web Design [Ethan Marcotte] A Discusses fluid grids and media queries. Outlines a possible future where the default version of the site looks good in all browsers but can be progressively enhanced using extra stylesheets for bigger screens. Responsive Web Design Book [Ethan Marcotte] Learn how to think beyond the desktop and craft beautiful designs that anticipate and respond to your users’ needs. Media Queries “A media query consists of a media type and zero or more expressions that check for the conditions of particular media features. Among the media features that can be used in media queries are ‘width’, ‘height’, and ‘color’. By using media queries, presentations can be tailored to a specific range of output devices without changing the content itself. “ Responsive Web Layout – in anticipation of the Mobile Event [nodiac] One of the few mentions of responsive web design on GDO. Responsive web design – Drupal theming Slideshow which covers the details of media queries with some good advice on being practical within the confines of Drupal.

Zen of mobile

In the spirit of the Zen of Python here is my Zen of Mobile. Consider these things when sketching out a design for your mobile site. Import this :)

  • simple is better than complex,
  • fast is better than slow,
  • sparse is better than busy,
  • fewer options are better than more,
  • most important to least important,
  • every element must add,
  • every page to satisfy a single aim.

Some more practical pointers after looking around at a few decent mobile sites:

  • banner to be frugal on vertical space,
  • search is crucial gateway to site,
  • main content first,
  • single “focus” image for the page is OK,
  • other ancialliary images are distracting,
  • lists are good,
  • primary nav is available but not necesarily at top of page,
  • collapsing divs are good to compress blocks to be readily consumed on single screen,
  • minimise clicks by displaying simple form on that page if user likely to want to take action,
  • minimise typing if possible,
  • homepage gateway to user paths with focus on single key task.
Mobile Web Design Trends For 2009 Some good tips and screenshots from popular, well designed mobile sites. A Study of Trends in Mobile Design Stats on different approaches taken by various websites. [W3C] Best practices to follow when designing for mobile devices.

Location aware

Mobile devices have extra capabilities over traditional desktop environment making them more immediate and powerful. The Mobile First section mentioned that mobile devices are superior to normal desktop devices and that apps should take advantage of this. The most obvious is the ability to be location aware. The massive dollop of cream that can go on top of the mobile cake is knowing where your users are. This then enables you to pre-empt many of the tasks the user might wish to carry out at that point in time: where is the nearest store, best deals in the area, directions, etc. from a functional perspective we can better guess what they want to do.

Modern mobile clients offer APIs which allow developers to ask the user for access to their location. If the user agrees this data can be used to customise their experience. This may mean running custom queries on the DB using lon/lat or querying other APIs such as Google Places or other geo services. Want to carve out a niche for yourself? This could be just the area. I’ll leave that for another article :)

Supported OS Platforms & Widget Frameworks Outlines the various platforms and the plugins/frameworks/apis to access geo location info from the client.


As you will be testing the site initially in your web browser it is a good idea to set it up so as to resemble a mobile browser. If you are using Firefox to test the site there are a couple of add ons, both developed by Chris Pederick, which make your life easier.

User Agent Switcher Allows you to switch your user agent across to iPhone or any other agent you care to define. This is great for testing that your redirect rules are working. Web Developer Tools Has a “resize” feature where you can define widths for 240, 320 and 480 to see what the page looks like on narrow screens.

Finally, have a couple of target devices around so that you can give the designs a final once over in the wild.

Be Sociable, Share!
Aug 25 2010
Aug 25

Drupal 7 will ship with RDFa built into the core. This will enable thousands of Drupal websites to publish semantic markup alongside the usual HTML, enabling robots such as Google and other aggregators to extract machine understandable information from the page. This is a big leap forward for Drupal and promises to bring it to the forefront of semantic web endeavours.

This deck of slides was prepared in June 2010 and presented at Drupal Camp Sydney 2010 as well as the Semantic Web Meetup in Sydney.

Drupal and the Semantic Web slides

Be Sociable, Share!
Feb 11 2010
Feb 11

Uriverse, a Drupal based website, was released in January 2010. Much of the data in Uriverse is based upon a data import from DBpedia, a semantic version of Wikipedia. Uriverse contains over 13M nodes and contains 90 languages covering around 3M primary subjects. This article is a case study of how the import was done and the challenges faced.

Drupal proves itself to be a flexible system which can handle large amounts of data so long as some bottlenecks are worked around and the hardware, particularly RAM, is sufficient to handle database indexes. The large data set was tedious to load but in the end the value added by Drupal made it worth it.


DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data. We hope this will make it easier for the amazing amount of information in Wikipedia to be used in new and interesting ways, and that it might inspire new mechanisms for navigating, linking and improving the encyclopaedia itself.

Over the years I have had an interest in the semantic web and linked data in particular. The DBpedia project had always impressed me because it was an effort which allowed Wikipedia to be used as a hub of subjects for use in the semantic web. Wikipedia represents most of the commonly discussed subjects of our times, created and edited from the ground up by real people. It therefore forms a practical basis from which to build out relationships to other data sets. If you are looking for subjects to represent things you want to talk about the DBpedia is a good place to start. There is an increasing momentum around it as the linked data meme starts to spread.

The Linked Open Data Cloud

Not only has the DBpedia project formalized these subjects, it has extracted a large amount of information which was codified in the somewhat unstructured wikitext syntax used by Wikipedia. The knowledge within Wikipedia has been made explicit in a way that can be used (browsed, queried and linked) easily by other systems. The DBpedia data set therefore provides a convenient store for import into a CMS such as Drupal.

Choosing to import DBpedia into a content management system is not necessarily a natural thing to do. The RDF data model is very simple and flexible and allows for all kinds of data structures which may not fit well within a system such as Drupal. It may seem like I was attempting to get the worms back into the can after it had been opened. At times it did :) However, there was a good deal of regularity and structure within DBpedia. The provision of an ontology and a “strict” infobox mapping in version 3.4 made the importation process possible. Whilst not everything could be imported, most of DBpedia made it in. Concessions had to be made along the way, and the places where Drupal proved to be too inflexible are indicative of areas which could be improved in Drupal. More on that later.

I chose Drupal as the target platform because I had been impressed with the flexibility I had seen from the system. Unlike other popular blogging platforms/CMSs, Drupal has a decent way of specifying a schema through the Content Construction Kit (CCK). I believed that it was possible to mimic the basic subject-predicate-object structure of RDF. Drupal supports different content types with custom properties (strings, dates, integers, floats) and relationships between objects (node references, node referrer). It also has a relatively strong category system baked in which can be used for filtering. Drupal also offers a lot of other attractive features apart from the data modeling: users, permissions, themes, maps, ratings, timelines, data views, friendly URLs, SEO, etc. Utilizing these features was the carrot to get the data into the system.

The DBpedia data set

Version 3.4 of DBpedia is based upon a September 2009 dump from Wikipedia. Here’s a quick list of some of the data set‘s statistics:

  • 2.9 million things
  • 282,000 persons
  • 339,000 places
  • 88,000 music albums
  • 44,000 films
  • 15,000 video games
  • 119,000 organizations
  • 130,000 species
  • 4,400 diseases
  • 91 different languages
  • 807,000 links to images
  • 3,840,000 links to external web pages
  • 415,000 Wikipedia categories

The data set is provided in two main formats: N-Triples and CSV. The N-Triples format is suitable for loading into RDF stores with RDF software such as ARC. The CSV format was handy for quickly loading data into a relational database such as MySQL. In a data set as big as DBpedia it is essential that you do things the quickest way lest you will be spending weeks importing the data. Here are some very rough comparisons of import speed. NB: These are from memory and are very rough estimates.

Method Throughput (triples/sec) NTriples into ARC 100 MySQL inserts from a select on another table 1 000 CSV, LOAD DATA IN FILE (same disk) 10 000 CSV, LOAD DATA IN FILE (different disk) 100 000

Obviously if you are importing 100s of millions of triples these comparisons are important. (I can’t quite believe the last result but that is what I remember it to be.) Put your dumps and DB on different disks when doing imports! Mind you, adding the indexes after the data has been imported is still a very time consuming process requiring data to be copied to disk so the super fast results are a bit misleading.

Mapping to Drupal structures

The DBpedia data set can be divided into two parts: infobox and non-infobox. Infobox data includes all of the varied properties which are applied to each class as defined in the ontology. As such, it presents a substantial data modeling exercise to fit it into Drupal. Much of my time was spent analyzing the class hierarchy and properties to work out the sweet spot as to how content types and properties would be allocated. The non-infobox data is much more predictable and easier to import in standard ways.

DBpedia Drupal Comment Resources Nodes Natural mapping. Wikipedia article maps to a DBpedia subject maps to a Drupal node. Classes Taxonomy Class hierarchy in ontology maps cleanly to taxonomy. Classes Content Types Each DBpedia class belonged to a base super class which forms the Content type. eg. actors and politicians are both Persons. Person is the base class and therefore becomes the natural candidate for the Content Type in Drupal. Categories Nodes Wikipedia categories are messy (irregular, loops) and better suited to being a Node. Language Labels Translations A base English node with translations handles the subject + different language labels for DBpedia well, if not perfectly. URI URL Alias The URL in Wikipedia maps to DBpedia maps to Drupal URL Alias. Articles Labels Node title In various languages. Long Abstract Node content In various languages. Short Abstract Node teaser In various languages. Wikipage CCK Link Applied to all translations. Homepage CCK Link Applied to all translations. Image CCK Link Applied to English version only. Instance Type Applied to English version only. Redirect Node with CCK Node Ref Different names/spellings redirect to English version. Disambiguation Node with CCK Node Ref Points to various English versions. Page Links CCK Node ref Untyped links between Nodes External Links Ignored Too many to import. Geo CCK Location Applied to English version only. Person data – Names CCK Strings Applied to people and organizations. Infobox – strict Various CCK fields DBpedia ontology specifies which classes have which properties.

Content Types, CCK and Importation Concessions

There were a number of areas where there was not a clear mapping between DBpedia and the features offered by CCK. The notes below refer to Drupal 6, and do not consider the Fields in Core initiative in Drupal 7.

Property Explosion

Drupal handles object properties through a system known as the Content Construction Kit (CCK). CCK offers a handy interface for defining a schema for various content types. Each object (node) is a instance of a single class (content type). Each node therefore has the properties of its content type.

Those of you familiar with the inner workings of CCK in Drupal 6 will understand the idiosyncrasies of the way CCK works behind the scenes. On the backend things work as expected up to a certain level and then they get a bit complicated. Properties for a content type are grouped together in a single database table, as you would expect. There are two exceptions to this rule. Firstly, if the property can have multiple values, it is stored in a separate table. This too is natural enough. Secondly, if two content types share a property then it is split out into its own table. This is a bit strange and can catch you unaware if you aren’t expecting it. However, it is sensible enough as it allows easy queries across content types on a single property.

Things become tricky when you have (i) lots of “multi” properties or (ii) lots of “shared” properties. Drupal needs to issue a new query on a different table to get the data back for that property. This is alright for most sites but has the potential to be a worry in the case of DBpedia where there are a massive amount of different relationships and in some cases relatively few instances of those relationships. ie. the data is not very dense. We are potentially talking about 100s of properties which would need to be retrieved.

Unfortunately, in these cases it make sense to pick only the properties where you get he most bang for your buck. Which “shared” properties are shared amongst the most types? Which “multi” properties have the most instances? Which “single” properties have the most density down the rows? Along the way we had to be pragmatic and pick and choose the properties we would support. At the end of the day this wasn’t a limitation of what CCK, rather a sensible decisions were made to stop the database exploding out into thousands of tables with very little data in them.

I don’t see an easy way to solve the table “explosion” problem save from moving to an RDF datastore such as that implemented by ARC. For data sets which have demands similar to DBpedia it makes sense to have something such as an RDF store in the backend. This conclusion is incredibly ironic given the lengths I have gone to to get the data into DBpedia. All was not lost however as the majority of data made it in and is usable within Drupal using standard techniques.

Interestingly this very issue seems to have plagued the backend design process for Drupal 7. According to the DADS Final Report (4th Feb 2009), “CCK 2 for Drupal will drop its variable schema and use the multi-value schema style for all fields.” Hmmm. I haven’t checked out Drupal 7 in this much detail but if this is true then the table explosion problem is going to be worse in Drupal 7. I’m not abreast of current developments here so I can’t comment further.

No sub classing properties

Drupal doesn’t allow for for sub classing content types. Each content type exists as its own base class. This means that we have to define base content types with all of the properties of the contained subclasses. The ontology in DBpedia can be up to four levels deep: Eurovsion Song Contest Entry is the deepest, preceded by Song, Musical Work, Work and finally Thing (a catch all at the root). This of course leads to base content types with many properties which will be null for the instance in question. The database tables would become very wide and have relatively low density of information.

The Taxonomy system does allow us to partially work around the sub classing problem. A class hierarchy maps nicely to a hierarchy of Terms in a Taxonomy. Further, multiple Terms can be applied to a Node making it possible to specify different classes for a node. It doesn’t cover the difficulty of property storage however.

No Multiple inheritance

When it comes to data modeling there are different ways to handle typing. The most simplistic and limited way is to allow instances to have a single class. This is the way Drupal currently works with content types and CCK. Each node belongs to a single content type and has the properties defined by CCK for that node. A more flexible way of modeling data allows for multiple inheritance where an instance can have more than one class.

Where an instance did straddle two base classes it was impossible to carry data for both types. This query shows all “people” who are “works” as well. I think these cases can be put down to DBpedia being a bit too promiscuous in the infoboxes it processes for each article. This isn’t a strong argument for multiple inheritance because the data is probably erroneous, however, it does demonstrate an area where modeling could be more flexible.

Compound Types

In some cases a compound type was required. For example, images data had three components: thumbnail link, depiction link and copyright info link. All three of these should have been considered one unit, however this is not possible through the standard CCK interface which handles atomistic primitives. Because these image properties applied to multiple content types, the end outcome was that they were represented by different database tables. It was very frustrating to know three queries were being issues when one (or none) would suffice. It is possible to define your own custom datatypes through a module but this is a fairly high barrier to jump.

Interfaces: A possible solution

Freebase is a collaboratively edited system of open data which is similar to DBpedia in many respects. The main difference is that Freebase allows users to create and edit content according to certain schemas. One of the very impressive aspects of the Freebase system is its ability to support multiple types, or co-types, for an object. From the Freebase data modeling guide we have:

A novel aspect of the Metaweb system is that instances may have multiple types. A single topic such as “Kevin Bacon” may have multiple types such as a Person, Film Actor, TV Actor and Musical Artist. and others. Since no single type could encapsulate such diversity, multiple types are required to hold all properties to fully describe Kevin Bacon and his life.

This approach has the advantage of grouping properties in a table allowing for fast retrieval and querying, as well as allowing for flexibility and sensible design. I believe that Drupal could benefit from the Freebase approach. The current content type + CCK model could remain in place AND be augmented by an interface system which allowed for grouping of properties for various types. To take the Kevin Bacon example, “Kevin Bacon” would be a Person content type with the base properties of Birthday and Deathday. “Kevin Bacon” would then have the FilmActor, TVActor and MusicalArtist interfaces which could be represented by separate tables on the backend. I believe that this offers good flexibility for those desiring a powerful system whilst maintaining simplicity for those who just need base types. It also solves a lot of the hand wringing which goes with the way some CCK tables are formed.

Import into Drupal

As a new comer to Drupal, by far the most disappointing aspect of the system was the lack of a clear and easy to understand API. I assumed that there would be a nice object abstraction which I could use to populate the system. I gradually came to understand that most development was done by examining the excretions of print_r() to determine what data was available at that particular moment. Where was the interface to content types, CCK and nodes? How could I create these programatically? There were times where I stopped and paused and considered a framework which was cleaner and more lightweight. The rich functionality was the thing that kept me though.

The large size of the data set pretty much dictated that it needed to be imported in the most efficient way possible. If I accepted a 1 second overhead for a save/update I would be waiting four months at least for the data to load. So, notwithstanding the state of the API in Drupal 6, a straight database import was the order of the day. After a bit of reverse engineering mucking around I had a few PHP/SQL scripts which could insert content pretty quickly, with insertion rates of around 1000 rows a second.

The import process followed these simplified steps for the various pieces of DBpedia.

  • Import DBpedia data from CSV format into MySQL.
  • Create a staging_node table with columns: title, language, teaser, content, url_alias, nid.
  • DBpedia data populated into staging_node and cleaned.
  • staging_node data copied into Drupal database.

The process was therefore semi automated with a number of scripts. It still took a few weeks to run through from start to finish. Importing 75M page links was the final time consuming process. I could only get throughput of about <200 rows a second as a url_alias to id lookup was required. This part of the process took around 7 days. Not something I want to repeat.

During the import I realized a few things about MySQL techniques which may come in handy for other people.

  • Loading data into MySQL direct from file, with no index is very fast.
  • It’s even faster if the source file is on a different disk to the DB.
  • The overhead from very heavy/large select queries can be minimized by using mysql_unbuffered_query. The connection can be fragile though so be prepared with an internal counter so you know where the process got up to before it died. You can’t write to the table you are reading from and it is locked for other operations.
  • Sometimes pure SQL is a good way to go: INSERT INTO SELECT
  • Sometimes joining is no good and running a SELECT for each row is best (if caching and keys) are working.
  • SHOW FULL PROCESSLIST is your friend.
  • Sorting is your enemy.
  • Dumping temp tables to disk must be avoided. Try tweaking your conf. You want to hear the cooling fans humming rather than the disk ticking away.

Drupal Features

Characteristics of the rich DBpedia data set provided a good foundation for using the following Drupal features:

  • Image thumbnails pointing to Wikipedia display in the contents expanding to full depiction via thickbox.
  • Geo cordinates for Nodes displayed in Google Map via Geo module.
  • Geo Cordinates used with Flickr API to pull back Creative Commons images for places from Flickr.
  • FiveStar ratings used on People, Places, Organizations and Music Genres.
  • Views provided top rated lists of the most popular things.
  • Simile Timeline widget used to display chronology of events.
  • Solr used as search engine and filters on language and class.
  • Solr recommends “More like this” based on Node content.
  • Filtered Views provide lookups for titles, firstname, lastname and geo coordinates
  • Various node properties allow for sorted list Views of richest people etc.

Performance Shortcomings

OK. So the data is in Drupal – were there any problems when it came to running the site? Yes, I ran into a few challenges along the way. Some were fixed, others worked around and others still remain a thorn in our side.

Database Indexes

The database is the most pressing concern when it comes to performance on a big site. If simple queries run slowly then the site will not function acceptably even for low traffic. The most important area is to ensure that indexes for key tables have been loaded into the key buffer. Lets look at a couple of simple selects with and without a primed key buffer.

Query No key buffer Key buffer select sql_no_cache title from node where nid=1000000; 0.05s 0.00s select sql_no_cache dst from url_alias where src=’node/1000000′; 0.02s 0.00s

If these indexes aren’t in RAM then the most basic of lookups will take a long time, ie. 0.07s. If you are on a page with a view with 50 nodes to look up then just getting the title and path out will take (0.07s * 50) 3.5 seconds. Note that this doesn’t include all the other processing Drupal must do. This in completely unacceptable and so it is mandatory to get these indexes into RAM. I recommend putting the following SQL into a file and running it every time MySQL is started up using the init-file variable in my.cnf.

USE drupal6;
LOAD INDEX INTO CACHE node_revisions;

On massive sites you probably won’t be able to get all of the node indexes into the key buffer, even if you have been generous in its allocation (up to 50% of RAM). In this case I resorted to running a query which seems to get the node nids into the buffer whilst leaving out all the other indexes which aren’t used as much. It takes a while to run but does the trick.

select count(n.nid), count(n2.nid) from node n inner join node n2 on n.nid=n2.nid;

In the best case scenario we would all have RAM (money) to burn but unfortunately that’s generally not the case.

Core and Contributed modules

There are a few areas in Drupal which bogged down when handling huge data sets. As I developed the site I took note of problematic areas. Most of these areas are probably well known so we’ll just mention them briefly.

  • In general a node load is very heavy. Lazy loading of CCK properties would make the system much faster if CCK didn’t have to be loaded. It would also mean the API could be used when speed is an issue. ie. when processing millions of nodes at once. During import and update, the solution is to work directly with the database. Just for a laugh I tried setting the status to 0 for a large node but the 404 still tried to load the whole node and then died.
  • Editing a node with many multi properties it all but impossible. The edit page size is massive and RAM/CPU is hammered. Solution is not to edit! Real solution is to page multi properties in node edit.
  • Viewing a page with many multi properties requires that the properties be paged. Looking up all those nodes gets slow quickly even with fast database. Solution is to use a module such as CCK Pager.
  • Viewing list of content is not possible as SQL query relies on a join to user table. This query kills the database. Solution is to make a small hack to core to stop this join. If users were looked up with separate queries then this would be a better solution.
  • Search is impossible from many angles. Queries for indexing are very slow. Database not designed to handle such large amounts of data. Solution is to let Solr handle search.
  • Solr search design is generally good with data held in apachesolr_search_node. However, Solr indexing can be a drain if you exclude node types from search. The preparatory query to return the nodes will inner join to node and lead to a very slow query. It returns after a while (70s for me) so you can live with it. Definitely not something you want to be doing regularly on production server as CPU goes to 100%. Solution is (i) to replicate node type data in apachesolr_search_node or (ii) get the excluded nodes anyway and ignore them. First option is best.
  • Taxonomy pages fail when there are many nodes in a category. SQL query is very slow. Solution is to let Solr show Taxonomy pages.
  • Strangely, the Search module still was running even when Solr was handling search. Core had to be hacked to turn it off. There must be a better way but I couldn’t see it. Make sure the “search” tables aren’t populated when using Solr.
  • Views queries can be slow with no extra indexes on content tables.
  • Views queries use left joins which can be slow. Inner joins exclude a lot of rows you don’t need. Solution is to rewrite the Views SQL with a hook.
  • Displaying nodes or teasers in Views can be very RAM intensive if the displayed nodes are very heavy with CCK fields. Solution is to use fields.
  • The Voting API module stores data in an inefficient way leading to some hairy indexes on the table and some equally hairy queries in Views when trying to display ratings. Be careful.
  • CCK node referrers datatype has a mandatory sort order specified. This kills query performance for large result sets.
  • XML Sitemap has some queries which return massive data sets which need to be dumped to disk. I know this module is in the process of being reworked.

Solr: Some stats

Uriverse uses Solr as its search engine and it is a component which has performed remarkably well. The filtering capabilities of Solr are excellent and the speed at which results come back are very impressive, even for large corpuses. Since it runs as a service over HTTP it is possible to deploy it to a second server to reduce load on the web server/DB box.

It takes a while to index 10M articles (categories, redirects, disambiguations, photos and pages were excluded from the 13M) and it is a task not suited for the standard Drupal cron. A custom script was written to build the index so that it could hammer away almost constantly without upsetting the other tasks cron performs. Initially the script was written to be a long running process which would potentially run for months. However, a memory leak in Drupal meant that this was not possible. After 10 000 nodes RAM became to much for PHP and the script died. The solution was to limit the number of nodes processed each time. The script now processes around 8 000 nodes every 30 minutes. It therefore takes around a month to build the index.

On the web there are quite a few articles regarding memory usage with Solr. The JVM needs to be given enough room when started. These reports had me concerned because I am running 32 bit machines with only 3G at my disposal. Would an index of 10M articles run in a JVM limited to around 2G? What size would the index be on disk? These are the numbers for the 7883304 articles currently in the index:

Resource Total Per Article Disk 47.5 GB 6.0 KB RAM 1.5 GB 198 B

Obviously these numbers are dependent on the average size of the title, body, fields and taxonomy. RAM is also affected by number of facets, sorting and server load. I therefore I have been very conservative in what I have indexed and have turned of sorting in the interface. It looks like the current RAM allocation will be sufficient.


Most Drupal performance best practices (opcode cache, aggregation, compression, Boost, Expires, database indexes, views and block caches) have been followed to get the most out of the site. A CDN has not been deployed because Uriverse doesn’t serve many images from its own server. The images that are served have an expires header and all other images come from Wikipedia and Flickr which probably have their own CDN solutions in place. Further optimizations would include going with MPM Worker + fcgid or Nginx. This will be required if traffic picks up and MaxClients is reached.

Two problematic areas remain. The first is the amount of RAM available for the database indexes. It would be nice to be able to increase that one day. Ordinary node page build times do come in at respectable times considering so this is not such a big issue. The second problematic area is some of the queries in Views. A bit more research is required here but it is likely that some Views will have to be dumped if they are hitting the disk with temp tables. Sometimes its easiest to forgo some functionality to maintain the health of the server.


All up the project has taken longer than expected – more than a few months. Most of the time was spent wrangling the data rather than fighting Drupal, although there were quite a few issues to work through, this being my first serious Drupal project. If I knew of the pain I would suffer from having to massage and prepare the data as well as the patience required to babysit the import process over days and weeks then I probably wouldn’t have commenced the project. That said, I am pleased with the outcome now that it is all in. I am able to leverage the power of many contributed modules to bring the data to life. There is a great sense of satisfaction seeing Solr return results from 10M articles in 90 languages, as well as the pretty theme, Google Maps, Thickbox, Similie timelines and Five Star ratings. I am humbled by the efforts of all Drupal contributors over the years. What I am left with now is a good platform which will form the basis of future data aggregation efforts.


Class hierarchy

Be Sociable, Share!
Feb 08 2010
Feb 08
To do list

Time to revisit the different types of Drupal sites to see where gains can be made. What type of site do you have? This quick reference recaps the previous articles and lists the areas where different types of Drupal sites can improve performance.

All Sites

  • Get the best server for your budget and requirements.
  • Enable CSS and JS optimization in Drupal
  • Enable compression in Drupal
  • Enable Drupal page cache and consider Boost
  • Install APC if available
  • Ensure no slow queries from rouge modules
  • Tune MySQL for decent query cache and key buffer
  • Optimize file size where possible

Server: Low resources

  • Boost stops PHP load and Bootstrap
  • Sensible module selection
  • Avoid node load in views lists
  • Smaller JVMs possibly if running Solr
  • Nginx smaller than Apache
  • mod_fcgid has smaller footprint over mod_php

Server: Farm

  • Split off Solr
  • Split off DB server, watch the latency
  • With Cache Router select Memcache over APC for shared pools
  • Master + slaves for DB
  • Load balancing across web servers

Size: Many Nodes

  • Buy more RAM for database indexes
  • Index columns, especially for views
  • Thoroughly check slow queries
  • Warm up database
  • Swap in Solr for search
  • Solr to handle taxonomy pages

Activity: Many requests

  • Boost or
  • Pressflow and Varnish
  • Nginx over Apache
  • InnoDB on cache tables

Users: Mainly logged in

  • View/Block caching
  • CacheRouter (APC or Memcache)

Contention: Many Writes

  • InnoDB
  • Watchdog to file

Content: Heavy

  • Optimized files
  • Well positioned server
  • CDN

Functionality: Rich

  • Well behaved modules
  • Not too many modules
  • View/Block caching

Page browsing: Dispersed

  • Boost over Varnish if RAM is tight

Audience: Dispersed

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Be Sociable, Share!
Feb 07 2010
Feb 07

The time for a page to render in a user’s browser is comprised of two factors. The first is the time it takes to build a page on the server. The second is the time it takes to send and render the page with all the contained components. This guide has mainly been concerned with the former – how to get the most from your server, however, it is estimated that 80% to 90% of page rendering time is taken up during the rendering phase.

It’s no good to serve a cached page in the blink of an eye if there are countless included files which need to be requested and many large images which need to be transported across the globe. Optimizing page rendering time can make a noticeable difference to the user and is the cream on the cake of a well optimized site. It is therefore important to consider and optimize this final leg of the journey.

Improving Drupal’s page loading performance Wim Leers covers all the bases on how to improve loading performance. High Performance Web Sites: Essential Knowledge for Front-End Engineers Steve Souders, Chief Performance Yahoo! and author of YSlow extension, covers the Yahoo recommedations in this book. High Even Faster Web Sites: Performance Best Practices for Web Developers Another Steve Souders book covering Javascript (AJAX), Network (Image compression, chuncked encoding) and browser (CSS selectors, etc).

It is worthwhile reviewing Yahoo’s YSlow recommendations to see all of the optimizations which are possible. We cover selected areas where the default Drupal install can be improved upon.

Combined Files

The Out of The Box section covered the inbuilt CSS and JS aggregation and file compression. The use of “combined files” is a significant factor in Drupal’s relatively good score in the YSlow tests. Make sure you have this enabled.

All sites: Enable CSS and JS aggregation.

CSS Sprites

CSS Image Sprites are another method of cutting down the number of requests. This approach combines a number of smaller images into one large one which is then selectively displayed to the user through the use of background offset in CSS. It is a useful approach for thing such as small icons which can have a relatively large amount of HTTP overhead for each request. Something for the theme designers to consider.

Custom designs: Use CSS sprites if appropriate.

CSS Sprites: Image Slicing’s Kiss of Death Overview of how CSS sprites work and how they can be used. A lesson in the usefulness of CSS sprite generators Covers commonly used spite generators.

This is the number two recommended best practice.

A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content more efficiently to users. The server selected for delivering content to a specific user is typically based on a measure of network proximity. For example, the server with the fewest network hops or the server with the quickest response time is chosen.

Of all the CDN web services SimpleCDN seems to be getting positive press amongst Drupal folks as it is simple and cheap. It offers the “origin pull” Mirror Buckets service which will serve content from 3.9 cents to 1.9 cents per GB. At this price you will probably be saving money on your bandwidth costs as well as serving content faster.

The CDN integration module is the recommended module to use for integration with content delivery networks as it supports “origin pull” as well as push methods. It supports content delivery for a all CSS, JS, and image files (including ImageCache).

High traffic, geographically dispersed: use CDN

CDN integration module Wim Leers’ fully featured module which integrates with a wide range of CDN servers. SimpleCDN module Simple CDN re-writes the URL of certain website elements (which can be extended using plugins) for use with a CDN Mirror service. Drupal CDN integration: easier, more flexible and faster! Slides covering advantages of CDNs and possible implementations. mod_cdn Apache2 module which shows some promise but not much info available for it with regards to Drupal. Best Drupal CDN module? Drupal Groups discussion.

On a related note many sites can benefit from judicial placement of the server if traffic tends to come from one place and no CDN is being used. Sites based out of the US may find the proximity of a site hosted in their area worth the extra cost of hosting.

When a file is served by a web server an “Expires” header can be sent back to the client telling it that the content being sent will expire at a certain date in the future and that the content may be cached until that time. This speeds up page rendering because the client doesn’t have to send a GET request to see if the file has been modified.

By default the .htaccess file in the root of Drupal contains rules which sets a two week expiry for all files (CSS, JS, PNG, JPG, GIF) except for HTML which are considered to be dynamic and therefore not cachable.

# Requires mod_expires to be enabled.

# Enable expirations.
ExpiresActive On
# Cache all files for 2 weeks after access (A).
ExpiresDefault A1209600
# Do not cache dynamically generated pages.
ExpiresByType text/html A1

The Expires header will not be generated unless you have mod_expires enabled in Apache. To make sure it is enabled in Apache2 run the following as admin.

# a2enmod expires
# /etc/init.d/apache2 restart

Ensuring this is enabled will elevate your YSlow score by about 10 points or so.

All sites: Configue Apache correctly for fewer requests.

You can Gzip by enabling compression in the performance area of admin. Alternatively you could configure Apache to do it.

All Sites: Enable Gzip compression

Binary files do not shrink significantly after Gzip compression. Gains can be made by ensuring that rich media such as images, audio and video are (i) targeted for the correct display resolution and (ii) have an appropriate amount of lossy compression applied. Since these files will generally only be downloaded once they do not benefit from caching in the client and so care must be taken to ensure that they are as small as reasonably possible.

All Sites: Compress binary files

Pngcrush Pngcrush is an optimizer for PNG (Portable Network Graphics) files. It can be run from a commandline in an MSDOS window, or from a UNIX or LINUX commandline.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Be Sociable, Share!
Feb 06 2010
Feb 06
Blue Tape

Benchmarking a system is a reliable way to compare one setup with another and is particularly helpful when comparing different server configurations. We cover a few simple ways to benchmark a Drupal website.

A performant system is not just one which is fast for a single request. You also need to consider how the system performs under stress (many requests) and how stable the system is (memory). Bechmarking with tools such as ab allows you to stress the server with many concurrent requests to replicate traffic when a site is being slashdotted. With a more customised setup they can also be used in more sophisticated ways to mimic traffic across a whole site.

Documentation which covers tools of the trade including Apache Bench (ab) and SIEGE.

ab is the most commonly used benchmarking tool in the community. It shows you have many requests per second your site is capable of serving. Concurrency can be set to 1 to get end to end speed results or increased to get a more realistic load for your site. Look to the “failed requests” and “request per second” results.

In order to test the speed of a single page, turn off page caching and run ab with concurrency of one to get a baseline.

ab -n 1000 -c 1 http://drupal6/node/1

To check scalability turn on the page cache and ramp up concurrent connections (10 to 50) to see how much the server can handle. You should also make sure keep alives are turned (-k) on as this leads to a more realistic result for a typical web browser. At higher concurrency levels making new connections can be a bottleneck. Also, set compression headers (-H) as most clients will support this feature.

ab -n 1000 -c 10 -k -H 'Accept-Encoding: gzip,deflate' http://drupal6/node/1

Testing with ab and simple changes you can make within Drupal. Covers server side tools and walks through ab options and use. Demonstrates how to pull out current session id and how to pass that to ab so that authenticated users can be tested. Illustrative discussion where different Drupal setups are benchmarked with ab.

JMeter is a Java desktop app designed to test function and performance. It is the preferred testing tool of many administrators.

Perl script which runs a JMeter test on Drupal and provides graphs. Some scripts to get you started testing with JMeter.

Benchmarking is essential if you wish to have an objective comparison between different setups. However, it is not the final measurement with regards to performance. Remember that page rendering times are what are important for users and that too needs to be optimized. Also, benchmarks tend to be artificial in the sense that they often measure unrealistic situations. Will all of your requests be for one anonymous page only? Maybe in the Slashdot situation but there are other considerations obviously. Finally, it is easy to focus intently on the number, especially when it comes to caching scores, and forget that minor differences may not make so much of a difference to real life scenarios. Don’t forget the logged in user.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web