Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Dec 08 2009
Dec 08

Aegir is an excellent way to manage multi site drupal instances, but sometimes it can be a bit too heavy. For example if you have a handful of sites, it can be overkill to deploy aegir. If there is an urgent security fix and you have a lot of sites (I am talking 100s if not 1000s) to patch, waiting for aegir to migrate and verify all of your sites can be a little too slow.

For these situations I have a little script which I use to do the heavy lifting. I keep in ~/bin/update-all-sites and it has a single purpose, to update all of my drupal instances with a single command. Just like aegir, my script leverages drush, but unlike aegir there is no parachute, so if something breaks during the upgrade you get to keep all of the pieces. If you use this script, I would recommend always backing up all of your databases first - just in case.

I keep my “platforms” in svn, so before running the script I run a svn switch or svn update depending on how major the update is. If you are using git or bzr, you would do something similar first. If you aren’t using any form of version control - I feel sorry for your clients.

So here is the code, it should be pretty self explanatory - if not ask questions via the comments.

#!/bin/sh
# Update all drupal sites at once using drush - aka lazy person's aegir
#
# Written by Dave Hall
# Copyright (c) 2009 Dave Hall Consulting http://davehall.com.au
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
# Alternatively you may use and/or distribute it under the terms
# of the CC-BY-SA license http://creativecommons.org/licenses/by-sa/3.0/
# Change this to point to your instance of drush isn't in your path
DRUSH_CMD="drush"
if [ $# != 1 ]; then
    SCRIPT="`basename $0`"
    echo "Usage: $SCRIPT path-to-drupal-install"
    exit 1;
fi
SITES_PATH="$1"
PWD=$(pwd)
cd "$SITES_PATH/sites";
for site in `find ./ -maxdepth 1 -type d | cut -d/ -f2 | egrep -v '(.git|.bzr|.svn|all|^$)'`; do
    if [ -f "${site}/settings.php" ]; then
        echo updating $site
        $DRUSH_CMD updatedb -y -l $site
    fi
done
# Lets go back to where we started
cd "$PWD"

OK, so my script isn’t any where as awesome as aegir, but if you are lazy (or in a hurry) it can come in handy. Most of the time you will probably still want to use aegir.

Notes:

Make sure you make the script executable (hint run chmod +x /path/to/update-all-sites)

If you don’t have drush in your path, I would recommend you add it, but if you can’t then change DRUSH_CMD="drush" to point to your instance of drush - such as DRUSH_CMD="/opt/drush/drush".

Thanks to Peter Lieverdink (aka cafuego) for suggesting the improved regex.

Dec 03 2009
Dec 03

My blog is now syndicated on Planet Drupal. I am very excited about this - thanks Simon.

For the last 8 years or so I have been running my own IT consulting business, focusing on free/open source software and web application development. My clients have range from micro businesses up to well known geek brands like SGI. Until recently I lead the phpGroupWare project.

My Drupal profile doesn’t really give much of a hint about my involvement with Drupal. My biggest regret is not signing up for a d.o account sooner. I forget when I started using Drupal 4.7, but I liked it straight away. It was the first CMS which worked the way I thought a CMS should work.

Over time I have learned how to get Drupal to do what I want it to do. Due to the massive range of contrib modules I haven’t got my hands very dirty hacking on Drupal - yet.

This year I have been involved in a major Drupal project which involves hosting around 2100 sites. Aegir has made a lot of this painless, especially with our 3,000 line install profile. Over the Christmas period I hope to find the time to blog about the setup, parts of it are pretty crazy.

I’ll get around to upgrading my site to Drupal 6 one of these days when I get some time, that should coincide with a visual and content refresh. Feel free to check out some of my older Drupal related posts.

Nov 30 2009
Nov 30

I have just finished reading Matt Butcher’s latest book, Drupal 6 JavaScript and jQuery, published by Packt Publishing - ISBN 978-1-847196-16-3. It is a good read. It is one of those books that arrived at the right time and left me inspired.

I have always leaned towards Yahoo’s YUI toolkit when I need an Ajax framework, while the rest of the time I just bash out a bit of JS to get the job done. The more I use Drupal, the more I have been wanting to find time to get into jQuery. This book has got me motivated to play with jQuery - especially in combination with Drupal.

The book is logically structured and flows well from chapter to chapter. I find Matt’s writing style easy to read, he even brought a smile to my face a few times. Matt assumes a basic knowledge of JS and Drupal, but he also provides links so the reader is able to get additional information if their knowledge is lacking. However, a couple of times Matt seemed to switch quite abruptly from assuming a good level of knowledge on a particular topic to explaining what seemed to me to be basic or simple concepts in great detail.

In the first chapter, entitled Drupal and JavaScript, Matt covers the basics of Drupal, its relationship with JavaScript and recommends some essential items for any serious Drupal developer’s toolbox. This chapter provides a nice introduction of what is to come in the rest of the book and allows the reader to become acquainted with Matt’s style.

Working with JavaScript in Drupal covers the basics of the Drupal coding standards and why sticking to the standard is important. It then moves onto a quick overview of Drupal’s theme engine, PHPTemplate, and integrating JS with Drupal themes. I felt that the development practices part of this chapter could have been expanded a bit more and turned into its own chapter. Understanding the basics of theming is critical for being able to follow the rest of the book, but again I think this half of the chapter could have been developed into a separate chapter. Regardless of how the chapter was arranged, the content is well written and provides solid and practical examples.

In jQuery: Do More with Drupal, Matt gives a detailed overview of jQuery and how it is used in Drupal. Although the code sample has limited real world usefulness, it provides the reader with a very clear idea of the power of jQuery and how easy it is to use with Drupal. By the end of this chapter I was left feeling like I wanted to get my hands dirty with jQuery, unfortunately it was after 1am and I had to work the next day.

In Chapter 4, we move onto Drupal’s Behaviors, which is covered in great detail. Behaviors are a key part of Drupal’s JS implementation and essentially provide an events based hooks system in JavaScript. Once again Matt spends a lot of time explaining this feature, how it works, how to use it and where to learn more. Matt’s description of this feature had me thinking “OMG, Drupal behaviours are awesome” throughout the chapter.

Lost in Translations, is the name of a good movie starring Scarlett Johansson and Bill Murray, which I enjoyed watching a few years ago, oh and is also the fifth chapter of the book. I suspect that I am like many English speaking Drupal developers in that I use the basics of the Drupal translation engine, but pay very little attention to how it works as my target audience is English speaking like me. Not only does Matt explain how Drupal’s translation system works in both PHP and JavaScript, he makes it clear why all Drupal developers should understand and use the system - regardless of their native/target language/s.

The JavaScript Themeing chapter was a bit of a surprise for me. I was expecting Drupal to have a JS equivalent to PHPTemplate and for this chapter to outline it and provide some code samples. Instead I learn that Drupal has a very simple, and easy to use, JS themeing system. Matt spends some time discussing best practice for themeing content in JS and goes on to provide the code for his own simple yet powerful jQuery based themeing engine for Drupal.

In AJAX and Drupal Web Services, we learn about JSON, XML and XHR in the context of Drupal. Once again Matt demonstrates the ease of using Drupal and jQuery for quickly building powerful functionality.

Chapter 8 is entitled, Building a Module, and covers the basics of building a JS enabled module for Drupal. Matt also discusses when JS belongs in a theme and when it should be part of a module. The cross promotion of his other book Learning Drupal 6 Module Development ramps up a couple of notches in this chapter. I found the plugs a bit irritating (especially as I own a copy of the book), but overall the chapter is loaded with useful information.

The final chapter, Integrating and Extending, leaves the reader with a solid understanding of what can be done to make jQuery even more useful. This chapter provides a nice motivational finish to the book.

At the start of each chapter Matt recaps what has been covered and outlines where the chapter is heading which makes it easy to get back into the book after putting it down for a few days.

This book is definitely not for the copy and paste coder, nor the developer who just wants ready made solutions they can quickly hack into an existing project. Some may disagree, but I think this is a real positive of this book. Matt uses the examples to illustrate certain concepts or features which he wants the reader to understand. I found the examples got me thinking about what I wanted to use JS and jQuery for in my Drupal sites. Although some of the code samples run to several pages, Matt then spends a lot of time explaining what is happening in bit sized chunks, which makes it easy to understand. I also appreciated the links to documentation so I could get the information I’d need to write my own code for my projects.

One thing which always annoys me about Packt books is the glossy ink they use. In some lighting conditions it is too shiny, which makes it annoying to read, especially with a bed side lamp. On the positive side, the paper is solid and easy to turn.

Sprinkled through the book is some cross promotion of other Packt titles, which I have no issue with, it is a good opportunity to try to grab some additional sales. In a couple of the later chapters it becomes a bit too much. I think once or twice per chapter is reasonable.

I really enjoyed reading Drupal 6 JavaScript and jQuery, it is easy to read and the chapters are a size which lend themselves to being read in a session. I think any Drupal developer who wants to get into using JS in their sites/projects would benefit from reading this book. I finished it feeling like I wanted to start doing some hacking. I plan to update this site in the next few months, and now jQuery enabled effects is on the requirements list. I hope I can bump into Matt Butcher at a DrupalCon or somewhere else in my travels so I can buy him a beer to thank him for putting together a quality book.

Disclaimer Packt Publishing gave me a dead tree copy of this book to review it and keep. I’m glad they gave me a good title to review.

Nov 27 2009
Nov 27

After two years of working from home, I've decided it's time to make a move back into an office and look for some contract-based or perhaps permanent work in London.

I have four years of experience using, developing and helping guide the development of Drupal projects as well as a background and interest in all things geographic, from maps to open data (as you've probably seen from the topics I cover in my blog). With these skills I am looking to find some work as a Drupal developer for an organisation based in London, ideally integrating my geographic interest. Alternatively, I'm open to other opportunities that I may be suitable for.

If you have, or know of, any positions coming up from January onwards, I'd love to hear from you to discuss the details. You can find out more information about me in my CV (pdf) or on my LinkedIn profile.

Oct 28 2009
Oct 28

This article is outdated. You should probably have a look at this handbook page for up to date information.

Imagine you would want to create a Google Map showing all shops in Belgium, and let your users enter their postal code and a distance range (in kilometers or miles) to filter the results to. This would allow a user to search for shops nearby his location. As you can see in the screenshot below our map shows only shops that are within a 20 kilometer range of Brussels.

To build this functionality you will need to download and install the following modules:

Some of these packages come with multiple modules, so these are the ones you'll need to enable: Location, GMap, GMap Location, Views and Views UI.

1Step 1: First, we will configure the GMap module. Go to admin/settings/gmap to do so.
Here you can configure how you want your Google Map to look and behave. Make sure to enter your Google API key. You might also want to check the autozoom option in the "Map behaviour flags" fieldset. Most of the other settings can be left at their default value for now.

2Step 2: Next we will configure the Location module settings at admin/settings/location.
You can adjust the settings so they match the following:

You can leave the other tabs of the Location settings pages untouched.

3Step 3: Now we have to assign location based data to a content type. You can either use an already existing content type, or create a new one specifically for this purpose. I created a content type "Shop", so we can create a node for each shop we want to show on the map.
To configure our content type to collection location data, go to the content type edit page (at admin/content/node-type/shop) and find the fieldset "Locative information".
Configure the first two included fieldsets as shown in the screenshots below:

We configure our content type to have one location assigned with it.
Next, we will define which location fields a user adding a shop node will be allowed or forced to enter. In my case I made the basic address fields required to make sure to have a complete address.

You can configure the other location settings as you wish, or leave them as they are. Now save your content type and go ahead and create some shop nodes. You will have the ability to enter location data on the node submission form.

4Step 4: Now we're going to make the Location module aware of the Belgian zipcodes and their latitude - longitude coordinates. The Location module already comes with this data, so all we need to do is import the file sites/all/modules/location/database/zipcodes.be.mysql into our database (e.g. use PHPMyAdmin to import this file).

Important: at the moment of writing there is still an issue with this file: latitude and longitude values are switched. You will need to run the following SQL query to fix this:

UPDATE zipcodes SET latitude =(@temp:=latitude), latitude = longitude, longitude = @temp

More information on this issue: http://drupal.org/node/566792

5Step 5: Now we will create a View which will show our shops on the map. I assume you have at least some basic knowledge on how to create and configure a View.

First of all, create a new View, and make sure to create a Page display. Also assign a path to this page, so your users can browse to the map.

A. Add fields

Add at least the following fields to your View, and make sure to exclude them from display (unless you have a reason to show this information in the Gmap marker):

  • Location: Distance / Proximity
    configuration options:
    Exclude from display: checked
    Units: Kilometers
    Origin: Use Distance / Proximity Filter
  • Location: Latitude
    configuration options:
    Label: Latitude
    Exclude from display: checked
    Display style: Decimal degrees
  • Location: Longitude
    configuration options:
    Label: Longitude
    Exclude from display: checked
    Display style: Decimal degrees
  • Any optional fields
    Any optional fields you add and don't exclude from display will show in the marker.
    I suggest adding at least "Node: Title".

B. Save the view

It is probably necessary to save the View now, so the Latitude and Longitude labels are stored, as we need them in the following step.

C. Basic settings

Configure the basic settings in your View as follows:

  • Style: GMap
    • configuration: Data source = Choose latitude and longitude fields
    • Latitude field: select "Latitude" from list
    • Longitude field: select "Longitude" from list
      Note: if the select list shows only blank items, you didn't set a label for the latitude / longitude fields! See A. Add fields.
  • Row style: fields
  • Use pager: No
  • Items to display: 0

D. Add filters

Add at least the following filters:

  • Node: Published = yes
  • Node: Type = [your content type] (e.g. Shop)
  • Location: Distance / Proximity
    • Expose this filter
    • Operator = Proximity (Circular)
    • Form mode = Postal code (assume default country)
    • Optional = checked

E. Save the view (again)

This should give you a basic working map, with proximity search exposted filters. You can come back and tweak your view some more to fit your specific needs.

For an alternate resource on creating this View, check out http://drupal.org/node/359463

6Step 6: This is the point where things are getting a bit dirty, as we're going to have to make changes to a file in the Location module. This is not considered good practice, so you should make sure you document this step properly or create a patch file, so you can reproduce this step if you should update your copy of the Location module.

Browse to sites/[all]/modules/location/supported/location.be.inc and add the function location_latlon_rough_be to this file, as described in this comment (step 2/): http://drupal.org/node/326749#comment-1235822

Finished!

After following the steps above, you should now have a map showing your shops at the URL path you specified in the View we created, with the ability to search for pinpoints in the area near a given postalcode.

If you have suggestions or questions about this tutorial, feel free to add a comment below.

Oct 13 2009
Oct 13

UPDATE: Screencast now lives here:

[embedded content]

I recorded a quick screencast of a simple integration we did to show Open Atrium leveraging Alfresco as a formal document repository via CMIS. This leverages the CMIS Alfresco module we developed and released on Drupal.org.

As I point out in the screencast, there’s not much to the integration from a technical standpoint. Open Atrium is Drupal and the CMIS module already has a CMIS repository browser. So, all we had to do was expose the module as a “feature”, which is something Open Atrium uses to bundle modules together that create a given chunk of functionality.

Readers familiar with Alfresco Share will instantly recognize the Open Atrium concepts. Instead of “sites” Atrium uses “groups”. Instead of “pages” or “tools”, Atrium uses “features”. The overall purpose, self-provisioned team-based collaboration, is the same and many of the tools/features are the same (blog, calendar, member directory). I’m not advocating using one over the other–as usual, what works best for you depends on a lot of factors. I just thought Atrium provided a nice way to show yet another example of Drupal and Alfresco together (post).

Oct 12 2009
Oct 12

I am not reviewing a book today, but I soon will be. Packt Publishing have asked me to review Matt Butcher’s new book Drupal 6 JavaScript and jQuery. The book looks pretty interesting. Although it isn’t on the same scale, being asked to review a serious Drupal developer book, is a bit like Obama winning the noble peace prize - ok maybe I am exaggerating a little there.

I really like YUI, but Drupal has made me interested in jQuery. One of the things awesome things about Drupal is that you can use jQuery without ever having to touch jQuery. This has made me lazy about learning jQuery - especially in the context of Drupal. It look like I have run out of excuses.

The book should arrive in France by the end of the week, but I won’t be back in France for a couple of weeks, I have heading off to China for 10 days or so to catch up with friends and discuss some ideas about doing cool things with Drupal. Watch this space.

Oct 07 2009
Oct 07

One of the scripts in my bag of tools to check whether a site is still up and running is content-check.pl. This little script compares the hash of a web page against a previously created reference hash. If the content of the web page has modified, this script will detect it and send an email.

You have to be very careful which page you check because even the slightest difference in the HTML source will be detected.

In Drupal you should not use any page that contains a form because each time the form is rendered a unique form id is generated and inserted into the HTML. This results in a different page hash and will cause the script to send an warning email.

Keep in mind that, if you have enabled the User login block, each page contains a small login form and is not a good candidate for monitoring through content-check.pl. You might have to resort to monitoring your update.php file instead.

Sep 28 2009
Sep 28

Unbeknownst to people who visit my website directly, Just a Gwai Lo has served Google AdSense ads for some time now. Very early on I came to the conclusion that showing automatically-generated content-aware ads on personal sites was tacky, since those visiting directly were likely those who wanted to develop a personal relationship, no matter how loosely defined, with the author. That applies to the relationship I want to build with my readership as well: if they visit the site directly in a browser of subscribe to a feed, they shouldn't have something hawked at them.

People visiting through search engines, however, get no such treatment. They're likely people I don't know and, based on what people search for—and don't find on the site—don't want to have a relationship with. So anybody visiting in from one of the major search engines would see a Google AdSense ad, and over the years I've made enough to help fund some small vacations to the United States. (Since Google pays me in American dollars, it goes in my USD account, which I then withdraw for trips south of the border.) Nothing spectacular, and definitely not worth the amount of investment I put into the custom PHP script which looped through a list of domains returning TRUE if one of those domains were Google, Windows Live Search (as it was known then), Yahoo! and some others. Now I use Drupal, the following contributed modules and a one-line conditional in the block display filter to show ads to people finding my site that way.

  • AdSense, with the component modules "AdSense core" and "Managed ads" enabled.
  • Search Engine Referer API, which returns FALSE if the referrer is not a search engine and an object (essentially TRUE) if it is.
  • The following line of PHP in the "Page specific visibility settings" for the ad's block: I could have written a ternary operator, but I sought clarity in this case.

After my next payment, I'll disable this feature as part of an effort to simplify my Drupal install. This will cut down on the number of updates needed and ever so slightly decrease the load on the server. It's never been worth displaying them based on the time invested in this, though I can say the money plus experience with the advertising system in general did offset it a bit.

Sep 15 2009
Sep 15

People want intranets that are fun and easy to use, full of compelling content relevant to their job, and enabled with social and community features to help them discover connections with other teams, projects, and colleagues. IT wants something that’s lightweight and flexible enough to respond to the needs of the business that won’t cost a fortune.

That’s why Drupal + Alfresco is a great combination for things like intranets like the one Optaros built for Activision and why we had a record-breaking turnout for the Drupal + Alfresco webinar Chris Fuller and I did today. Thanks to everyone who came and asked good questions. I’ve posted the slides. Alfresco recorded the webinar so they’ll make it available soon, I’m sure. When that happens, I’ll update the post with a link. Until then, enjoy the slides.

[UPDATE: Fixed the slideshare link (thanks, David!) and added the links to the webinar recording below]

1. Streaming recording link:
https://alfresco.webex.com/alfresco/lsr.php?AT=pb&SP=TC&rID=42774837&act=pb&rKey=b44130d69cc9ec5f

2. Download recording link:
https://alfresco.webex.com/alfresco/ldr.php?AT=dw&SP=TC&rID=42774837&act=pf&rKey=c50049ac82e1220a

Sep 11 2009
Sep 11

I've been back in London for almost two years now and haven't met that many people working with Drupal, partly due to working from home I think, but also because there don't seem to be too many events (outside of paid training events and the like) that are aimed at Drupalers in the London area.

As I was looking today to find out if there were meetups happening already that I wasn't aware of, I came across a thread on groups.drupal.org asking about regular meetups, and left a comment to say I'd be interested if there was anything happening. In a city the size of London, there are surely enough people working with Drupal to get a group of people together every now and then for a social event. I for one would love to meet more people in the area who are working with Drupal, and maybe have a pint or two in the process.

By the end of the day, there was a meetup organised: London Drupal Pub Meet- September Meetup. Brilliant!

If you're interested in coming along, sign up to the meetup.com event, and I'll see you there!

edit: the event will be held from 7pm on Monday 28 September, at the Square Pig, Holborn.

Aug 29 2009
Aug 29

Three weeks until Talk Like a Pirate Day, and next week the Drupal project freezes its code in anticipation of the version 7 release of the CMS. In anticipation of that, I've updated the Pirate module, which turns all contributed content on a Drupal-powered website into pirate-speak for just the day of September 19th, to include a developmental release for the Drupal 7 platform. I'd like for people to test it out, and in order to do so on a day other than September 19th is to either modify the date in the following code of the module or remove it altogether.

  if (date('md') != '0919') {
    return $text;
  }

I see that similar code to my module has been included in the Dialectic module, which supports other novelty input formats. I may be convinced to officially merge my work into that project in the future, though my module differs in that it activates on only one day a year.

Any issues specific to the Drupal 7 conversion can added be to the issue for that. One sticky, longstanding issue that I could use some help with and affects all versions is a bug involving URLs that contain 'ing'.

I pledge that Pirate will have a full Drupal 7 release on the day that Drupal 7 is released. #D7CX:

Aug 29 2009
Aug 29

Today I should be in Prishtina Kosovo running Drupal workshops at the first Software Freedom Conference Kosova. Unfortunately due to work and family commitments I had to decline the invitation. I hope to make it there next year.

I will also be missing out on DrupalCon Paris next week too.

Sometimes it sucks to be in Australia, especially when Europe is so far away and so many cool things happen there.

Aug 28 2009
Aug 28

Bless me internet for I haven’t blogged, it has been 274 days since my last post.

I have wanted to blog, but I kept on finding excuses to avoid it - need to upgrade the software, need to finish x, y and z, need to focus on projects a, b and c etc. One of the main reasons is that I have been too lazy to put the effort in. I find it takes time to think of what to blog and then to bash it out, refine it and post it. When I have had the time to put that effort into my blog, I haven’t had the inclination.

While things have been quiet here, I have microblogging using the open source laconica platform through identi.ca.

More recently I have been working with a client in France who has some serious collaboration requirements. At last count they have almost 2100 instances of drupal running. This has involved a lot of work, and some travel. I will blog about this project soon - it is pretty awesome (even if I say so myself).

We have built a small drupal powered site for a local assest management consulting business. They are very happy with the results. Now their customers just log in to download the software. Everything was off the shelf drupal - except for the theme and a 60 line custom permissions module.

We have built the Newstead community website using drupal. It still needs some polish before final launch. The community has been heavily involved in the development of the site. So far over 30 locals have been trained in maintaining their page/s on the site. There is no “webmaster”, each local business and community group will maintain their own content

A couple of months ago I/we joined the drupal association. One day the buttons will be added to the site.

Where to next?

I plan to blog more about projects I am involved in. I also plan to switch this site to drupal 7 as close as possible to the release date - it looks like others will be switching too. Next week I will be commencing the build of the Newstead community wireless network.

Watch this space, lots happening - including more frequent updates from here on in.

Jun 14 2009
Jun 14

Addison Berry, aka @add1sun, presented about her experience as documentation lead for the Drupal content management system project the other day at the Writing Open Source conference in Owen Sound. In her role as chief cat-herder, she found that the most difficult people aren't poisonous. Instead they just don't know how to communicate with the community, and they need to translate where they're coming from to the way the community operates. It's hard work, she reports, to turn them into a contributor. She referred the audience to the "Poisonous People" presentation by the Subversion people, as yet unwatched by yours truly.

Addison talked about religious wars that occasionally break out. That is, the crux of the issue is more important than the resolution, and often leads to inaction. She also discussed the differences between recruiting in the corporate world and recruiting in the open source world. For private companies, they hire a skillset that they can filter for by listing the job requirements, either explicitly or implied. In open source, she says, you have the skillset first and you work with it. Many cats scratching their own itch, hence the herding to get them to scratch the community's itches too. The people you get working on a project have a rich background, both in terms of skills and life history. Skillsets include a lot of non-technical backgrounds in open source (Addison has an anthropology degree, for example, and my education is in political science).

Drupal has a large mass of documentation, and Addison is trying to whoop up energy in managing the base of existing documentation for Drupal 5 and 6 while gearing up for writing the documentation for the upcoming Drupal 7.

Open source has a natural passion that brings people together. Showing the example of a rowing team on her slide illustrated the need to hire a coach to tell them when to row. Herding involves keeping lines of communication open and opening up new ones as well as banging on pots about documentation. Instead of telling people what they can do, empower them by including them in the conversation. Addison, as leader, knows what she won't do and has so far been able to find people who will. Tracking metrics around the documentation—answering a question I had before I had the chance to ask it—Addison is not interested in, but she found someone who is. Many "soft-skills", such as facilitation, have come in handy even if the person with the skill does not claim membership in the software community. Also universities and their students have found time and energy to contribute usability testing as part of course credit or as part of their graduate studies.

Letting go and getting out of the way: Addison wanted the vision to be perfect, but quickly understood that she can't lead the charge or drag it out all the time: instead she recognized the need to let people run with things and support them. Getting people to trust you that that's the right direction.

Jun 09 2009
Jun 09

Last week I received a copy of Matt Butcher's "Drupal 6 JavaScript and jQuery" from Packt Publishing.

The book starts off with a gentle introduction into Drupal modules, Drupal themes and jQuery. These first chapters are pretty boring if you are already familiar with Drupal development but they aid newcomers in getting up to speed.

The next couple of chapters dive into the dark corners of Drupal's JavaScript capabilities. They explain things like Drupal behaviors, the JavaScript translation functions, the JavaScript theming functions and how to use AJAX. Each of these chapters is built around a very practical example that is analyzed thoroughly and each chapter also contains numerous tips about Drupal, JavaScript and jQuery. Good stuff!

I've learned quite a few things while reading this book so if you need to do any JavaScript or jQuery development in Drupal and if the cryptic JavaScript Startup Guide on drupal.org doesn't make a lot of sense to you than this book is definitely for you.

Jun 04 2009
Jun 04

In a week, I will attend the Writing Open Source conference in Owen Sound, Ontario. I'm excited to meet some of my colleagues in the field of open source documentation, having written the bulk of the support materials for Bryght, the Drupal-powered hosted service. I'm particularly interested in meeting those working to document open source tools other than Drupal, to gain some perspective on what's out there and what's needed.

Writing documentation was my first task at Bryght back in 2004. I recall spending part of that Christmas break furiously jotting down the important steps to creating dynamic and community websites. This included checklists, instructions and descriptions of module settings and how people could take advantage of them. The initial push of documentation made the subsequent job of supporting customers easy: instead of each time having to explain how to do something, I quickly pointed to the documentation, either through a link or a copy & paste. Along the way I even heard from non-customers thanking me for the handy references. After the second time someone asked we documented the answer. (We even wrote documentation after the first time someone asked a question.) Sometimes it didn't work, and sometimes the documentation wasn't all that great or hard to find. We allowed comments and opened the forums and listened to feedback when what we wrote didn't make a whole lot of sense. That's the experience I'd like to share with the conference, and I'd like to hear of others' experiences in making complex software more understandable.

After the weekend conference, I'll spend a couple of full days in Toronto proper, getting some much needed distance from Vancouver. I'd like to meet with some of the Toronto Drupal heads, and others I know (but haven't met) from other online communities I'm part of. Sadly, my favourite baseball squadron, the Toronto Blue Jays, play on the road in late June. Surely a local pub will have the games in HD?

The themes at Writing Open Source have a lot in common with two sessions I attended at FSOSS (the Free and Open Source Symposium) in 2006. I wrote two well-received pieces about the symposium, both notes on sessions at the conference:

(Audio and video for both presentations are available at http://fsoss.senecac.on.ca/2006/recordings/)

I'm looking forward to the sessions in Owen Sound next week, and to sharing what I learn there!

Jun 01 2009
Jun 01
Screenshot of the Cherry Blossom theme for Drupal

More than 5 years ago now, I sat down with Raincity Studios' Mark Yuasa to discuss the redesign of this blog, Just a Gwai Lo. It was springtime in Vancouver, and the cherry blossom trees around the lower mainland were blooming, so I suggested that as the visual theme for the blog. Then powered by WordPress, Mark delivered two designs in a few weeks and a few months after choosing the overwhelmingly pink comp, I switched to Drupal, bringing the theme along with me.

It's now time to release the theme to the general public. If you visit the Cherry Blossom theme project page on Drupal.org, you can install a Drupal 6-compatible version of the theme. I put up a demonstration site so that you can see the theme in action, as I've long moved onto another theme (currently the Deco theme). One known issue that I'd like help with is an alignment problem with Internet Explorer 6, after fixing I'll release a 1.1 version.

May 27 2009
May 27

It is looking like June is going to be quite an interesting month for participation, with a couple of projects being set up to focus on certain parts of Drupal for the month.

Last week, Advantage Labs announced Geo June, a month of focused development on the geo module. The geo module has a lot of potential to become the basis of the GeoCMS that Drupal should be (as long as the module stays generic enough), and Advantage Labs are keen to get more people interested and involved to help make that happen. During the month there are a number of physical events, but you're also encouraged to share your use cases and join in day to day with the IRC chat in #drupal-geo.

The Drupal User Experience Project also yesterday announced the launch of Microprojects to encourage user experience (UX) professionals to get involved in small bounded problems, working with a Drupal developer to implement their designs and suggestions for improvement. This seems like a great idea, not only because it's breaking down some quite large problems into bite-size manageable chunks, but also to get some outside experts - who may not have previously used Drupal - involved in the community.

If you're interested in either of these areas (or any of the other sprints which are happening), why not jump in and get involved. Having not spent much time on the Geo project yet, I'm looking to spend some time getting to know it in June and hopefully help to push it forwards, as well as starting the rewrite of the KML module to simplify it as a views display type instead of a bundle of custom code.

May 11 2009
May 11

If we are to believe in timestamps, on October 7th, 2006, I took over maintainership of the Mobile theme for the Drupal CMS. At the time there was no iPhone, and stripped down graphics-free versions of websites made it easier for people with small screens on their phones to get to the information quickly. Now, relatively larger screens coupled with effectively unlimited data plans make websites more consumable by tiny devices. My maintainership of the theme continues unabated, as today I (finally) released official 1.0 versions for Drupal 5 and Drupal 6, and dropped official support for the 4.7 version. m.justagwailo.com serves as the Mobile theme's demonstration site. Thanks to Bèr Kessels for originally writing the theme and webschuur.com for its original sponsorship.

A note about version numbers: when the new system for CVS tagging came out way back when, somehow it occurred to me that "DRUPAL-6-4" was the correct version number to assign to a developmental release. What that meant in practices was the version number for the theme ended up as "4.x". Looking at the usage statistics, at this writing, the overwhelming majority of sites that have deployed the Mobile theme use that developmental release. It is my hope that they all move to the 1.x branch, either developmental or official, as that's where all development will happen from now on.

As with any software, there are

Apr 28 2009
Apr 28

I’ll be in Chicago tomorrow for the Alfresco Meetup. I’ll be speaking during the Barcamp on Alfresco and Drupal integration with CMIS (module, screencast). I’ll also have the Alfresco-Django integration running on my laptop. I may not have time to show Alfresco-Django during my slot, but I’ll be happy to stick around and do informal demos and talk about either integration if you’re interested because I’d like your feedback on it.

Apr 20 2009
Apr 20

People often need to build a custom user interface on top of the Alfresco repository and I see a lot of people asking general questions about how to do it. There are lots of options to consider. Here are four options for creating a user interface on top of Alfresco, at a high level:

Option 1: Use your favorite programming language and/or framework to talk to Alfresco via REST or Web Services. PHP? Python? Java? Flex? Whatever, it’s up to you. The REST API is nice because if you can’t find a URL that does what you need it to out-of-the-box, you can always roll-your-own with the web script framework. This option offers the most flexibility and creative freedom, but of course you might end up building constructs or components that you may have gotten “for free” from a higher-level framework. Optaros‘ streamlined web client, DoCASU, built on Ext-JS, is one freely-available example of a custom UI on top of Alfresco but there are others.

Option 2: Use Alfresco’s Surf framework. Alfresco’s Surf framework is just that–it’s a framework. Don’t confuse it with Alfresco Share which is a team-centric collaboration client built on top of Surf. And, don’t assume that just because a piece of functionality is in Share it is available to you in the lower-level Surf framework. You may have to do some extra work to get some of the cool stuff in Share to work in your pure Surf app. Also realize that Surf is brand new and still maturing. You’ll be quickly disappointed if you hold it to the same standard as a more widely-used, well-established framework like Seam or Django. Surf is a good option for quick, Alfresco-centric solutions, especially if you think you might want to leverage Alfresco’s browser-based site assembly tool, Web Studio, at some point in the future. (See Do-it-yourself Alfresco Surf Code Camp).

Option 3: Customize the Alfresco “Explorer” web client. There are varying degrees to which you can customize the web client. On one end of the spectrum you’ve got Freemarker “presentation templates” followed closely by XML configuration. On the other end of the spectrum you’ve got more elaborate enhancements you can make using JavaServer Faces (JSF). Customizing the Alfresco Explorer web client should only be considered if you can keep your enhancements to an absolute minimum because:

  1. Alfresco is moving away from JSF in favor of Surf-based clients. The Explorer client will continue to be around, but I wouldn’t expect major efforts to be focused on that client going forward.
  2. JSF-based customizations of the web client can be time-consuming and potentially complex, particularly if you are new to JSF.
  3. For most solutions, you’ll get more customer satisfaction bang out of your coding buck by building a purpose-built, eye-catching, UI designed with your specific use cases in mind than you will by starting with the general-purpose web client and extending from there.

Option 4: Use a portal, community, or WCM platform. This includes PHP-based projects like Drupal (Drupal CMIS Screencast) or Joomla as well as Java-based projects like Liferay and JBoss Portal. This is a good option if you have requirements that match up well with the built-in (or easily added-on) capabilities of those platforms.

It’s worth talking about Java portal servers specifically. I think people are struggling a bit to find The Best Way to integrate Alfresco with a portal. Of course there probably is no single approach that will fit every situation but I think Alfresco (with help from the community) could do more to provide best practices.

Here are the options you have when integrating with a portal:

Portal Option 1: Configure Alfresco to be the replacement JSR-170 repository for the portal. This option seems like more trouble than it is worth. If all you need is what you can get out of JSR-170, you might as well use the already-integrated Jackrabbit repository that most open source portals ship with these days unless you have good reasons not to. I’m open to having my mind changed on this one, but it seems like if you want to use Alfresco and a portal, you’ve got bigger plans that are probably going to require custom portlets anyway.

Portal Option 2: Run Alfresco and the portal in the same JVM (post). This is NOT recommended if you need to scale beyond a small departmental solution and, really, I think with the de-coupling of the web script engine we should consider this one deprecated at this point.

Portal Option 3: Run the Alfresco web script engine and the portal in the same JVM. Like the previous option, this gives you the ability to write web scripts that are wrapped in a portlet but it cuts down on the size of the web app significantly and it frees up your portal to scale independently of the Alfresco repository tier. It’s a fast development cycle once you get it set up. But I haven’t seen great instructions for setting it up yet. Alfresco should document this on their wiki if they are going to support this pattern.

Portal Option 4: Write your own portlets that make services calls. This is the “cleanest” approach because it treats Alfresco like any other back-end you might want to integrate with from the portal. You write custom portlets and have them talk to Alfresco via REST or SOAP. You’ll have to decide how you want to handle authentication with Alfresco.

What about CMIS?

CMIS fits under the “Option 1: Use your favorite programming language” and “Portal Option 4: Write your own portlets” categories. You can make CMIS calls to Alfresco using both REST and SOAP from your own custom code, portlet or otherwise. The nice thing about CMIS is that you can use it to abstract the underlying repository so that (in theory) your front-end code will work with different CMIS-compliant back-ends. Just realize that CMIS isn’t a fully-ratified standard yet and although a CMIS implementation is in the Enterprise version of Alfresco, it isn’t clear to me whether or not you’d be supported if you had a problem. (The last response I saw on this specific question was a Peter Monks tweet saying, “I don’t think so”).

The CMIS standard should be approved by the end-of-the-year and if Alfresco’s past performance is an indicator of the future, they’ll be the first to market with a production-ready, fully-supported CMIS implementation based on the final spec.

Pick your poison

Those are the options as I see them. Each one has trade-offs. Some may become more or less attractive over time as languages, frameworks, and the state of the art evolve. Ultimately, you’re going to have to evaluate which one fits your situation the best. You may have a hard time making a decision, but you have to admit that having to choose from several options is a nice problem to have.

Apr 10 2009
Apr 10

Is writing Drupal tests boring? Maybe. Is running tests boring? Not anymore!

There is a patch for Drupal 7 pending that introduces a couple of new hooks into the simpletest module. These hooks will allow you to listen to the progress of a test run and react to tests that pass or fail. I have already created a new testlistener module that uses these new hooks to execute shell commands.

Some examples:

I have a Dell XPS laptop with built-in LEDs that can be controlled using the dellLEDCtl shell command (Linux). Using these new hooks, I can turn the LEDs green at the start of a test run, make them red when a test fails and turn them off when the test run has completed. This is very similar to the functionality of my Eclipse XPS plugin that shows JUnit results in Eclipse.

Another (typical) example is to have two lava lamps, a red one and a green one. You connect both of them to an X10 device (I have an X10 starterkit from IntelliHome) so you can turn them on/off with the heyu shell command (Linux). The testlistener module allows you to let the lamps reflect the status of the Drupal code. Other people are using this setup in their (non-Drupal) continuous integration environment.

If lamps are too soft for you, you might consider shooting missiles instead.

Drupal testing will never be boring again ;)

Apr 01 2009
Apr 01

Developers are all familiar with the default behavior of the drupal menu systems "local tasks" (aka tabs). These appear throughout most Drupal sites, primarily in the administration area, but also on other pages like the user profile.

Generally, developers are pretty good about creating logical local tasks, meaning only those menu items which logically live under another menu item (like view, edit, revisions, workflow, etc... live under the node/% menu item).

But sometimes, these tabs either don't really make sense as tabs or you simply want to have the flexibility of working with the items as "normal menu items", or those menu items which appear under admin/build/menu.

I recently wanted to move some of the tabs on the user profile page (user/UID) into the main menu so that I could include them as blocks.

For some reason, developers think the user profile page is a great place to put tabs for user related pages such as friendslist, tracker, bookmarks, notifications and so on. But these types of items are less a part of the user's account information than they are resources for specific users. Personally, I would not think to look at my account information on a site to find stuff like favorites or buddies. I'd expect those items to be presented somewhere much more obvious like a navigation block.

Initially, this may seem like a trivial task. My first thought was to simply use hook_menu_alter() and change the 'type' value of the menu item from MENU_LOCAL_TASK to MENU_NORMAL_ITEM. However, for reasons I don't understand well enough to explain in detail, this does not work.

In order to achieve the desired result, you must change the path of the menu item and incorporate the '%user_uid_optional' argument, replacing the default '%user' argument.

All very confusing, I know. Let's look at an example.

The notifications module (which provides notification on changes to subscribed to content) uses the user profile page rather heavily. I don't want its links there, I want them in the sidebar where users can always see them.

/**
* Implementation of hook_menu_alter().
*/
function MODULENAME_menu_alter(&$callbacks) {
 
// NOTIFICATIONS MODULE
 
$callbacks['notifications/%user_uid_optional'] = $callbacks['user/%user/notifications'];
 
$callbacks['notifications/%user_uid_optional']['type'] = MENU_NORMAL_ITEM;
  unset(
$callbacks['user/%user/notifications']);
  <
SNIP>
}
?>

So I have moved the notifications menu into my own menu, changed the type, used %user_uid_optional instead of %user, and unset the original menu item.

This works fine except for the fact that you'll lose all of the other menu items under user/%user/notifications! You need to account for all menu items in the hierarchy to properly reproduce the tabs in the main menu system, so we add the following:

    $callbacks['notifications/%user_uid_optional/thread'] = $callbacks['user/%user/notifications/thread'];
    unset(
$callbacks['user/%user/notifications/thread']);

    </span>$callbacks['notifications/%user_uid_optional/nodetype'] = $callbacks['user/%user/notifications/nodetype'];
    unset(
$callbacks['user/%user/notifications/nodetype']);

    </span>$callbacks['notifications/%user_uid_optional/author'] = $callbacks['user/%user/notifications/author'];
    unset(
$callbacks['user/%user/notifications/author']);
?>

And of course, we don't want this code executing at all if our module is not enabled, so you'd want to wrap the whole thing in:

  if (module_exists('notifications')) {
 
  <
SNIP>

  }
?>

Keep in mind that not all modules implement menu items using hook_menu(). It's becoming more and more common for developers to rely on the views module to generate menu items, and this is a wise choice. Menus generated using views (ala bookmark module) can be modified to get the desired result without any custom code.

Mar 05 2009
Mar 05

OpenBand has today unveiled its new OpenBand Labs website to help improve the information around the work we've been doing over recent years as well as hopefully invite some discussion.

OpenBand Labs

We hope to make some improvements to the new site over the coming days as we continue to add some more information and blog about some of our experiences here at DrupalCon. We'll also be adding the slides and some writeups about the XMPP talk Darren gave yesterday and the distributed enterprise talk Ben gave today.

We had a great presentation yesterday with Darren Ferguson talking about the XMPP Framework, and today with Ben Lavender talking getting a chance to demonstrate our collaboration platform.

Mar 03 2009
Mar 03

I'm in Washington, DC this week with many of the rest of the OpenBand / M.C. Dean team. We're all here to visit DrupalCon DC and meet our friends and associates in the Drupal community as well as present some of the things we've been working on.

A couple of the sessions that we proposed have been accepted, so we're going to be presenting:

If you're interested in the work we've been doing to help distributed enterprises communicate better, feel free to come up and say hi. We look forward to meeting you at DrupalCon!

Feb 27 2009
Feb 27

I was working the other night to create an integration module that would tie the existing Activity Stream module for Drupal into the Brightkite location-based social network.

The idea is that users can check in at their current location using Brightkite and have their Drupal site update their location within the site based on their last known location - handy if you want a little map that shows where you are, for example, but you could do whatever you wanted with those locations, and even use them to extend a social network you might be building up in Drupal.

While I was fighting with the SimplePie feed parsing library to work out why it didn't like the feeds from Brightkite, John McKerrell suggested that some integration for his new Mapme.at service would be nice too.

So, the first two services to be supported by the new Activity Stream location services project are Brightkite and Mapme.at. I'd also like to extend this to other services like Yahoo's Fire Eagle and Google Latitude at some point, but neither of them are quite so simple to integrate with, the former because it has no public location feeds for users and requires authentication, and the latter because it doesn't share any of its data at all (boo!).

In their most basic form, the modules pull in the updates from these services and they get included in your activity stream along with your Twitter updates and the like, but also if you have Location module installed and the user locations module enabled (plus a patch for Activity Stream for now), your user will be updated with the latest coordinates from the location service you use.

Jan 22 2009
Jan 22

Ballavayre CottagesThe new website for Ballavayre Cottages went live recently to give a new online presence to this 5 star self-catering accommodation in their 200 year old cottage in Colby, Isle of Man.

The site, built on Drupal, allows the cottage owner to change content as and when they wish, and also to update their availability calendar to let visitors know when the cottages are available.

To help visitors see at a glance where the cottages are located, and to give directions from the sea terminal and airport, we included a series of custom built maps, designed (with some very helpful tips from Steve Chilton) using OpenStreetMap data.

Directions to Ballavayre Cottages

It is always great to help promote the Isle of Man as a tourist destination, even in a small way, by giving accommodation providers a chance to promote their services to a wider market.

Jan 15 2009
Jan 15

Man, we’re getting old!

Today (January 15th) is the 8th anniversary of the day Drupal 1.0 was released. Although Dries had no idea at the time - it was a move that would not only change his life, but mine too…

January 2009 also marks the 5th anniversary of my starting to work on Drupal full time (after a few years of “hobby” involvement). My first project (at the time, actually a re-launch) still stands as one of my favourites: http://www.terminus1525.ca/ . Since then, Drupal has defined my career: from co-founding Bryght to my current life as a Lullabot. The community is home to some of my best friends and people I love.

Five years - full-time. No wonder I feel old.

Jan 10 2009
Jan 10

I'll be attending 2 major conferences the next couple of weeks.

First of all there's FOSDEM in Brussels on February 7th and 8th. I'm particularly interested in the Drupal devroom on Sunday, but there are plenty of other open source sessions throughout the weekend.

The second conference I'll be attending is DrupalCon in Washington from March 4th until 7th. I expect to pick up a couple of tips and tricks about Drupal 6 and I hope to learn a lot about the upcoming changes in Drupal 7. I'm pretty sure that being among that many Drupalistas will help me to find more ways to give something back to the community.

I never attended FOSDEM nor DupalCon (and I haven't been in the US recently) so I'm looking forward to both events!

Jan 07 2009
Jan 07

It's amazing how much spam is generated when commenting is enabled on a blog. When I first launched the Drupal version of dankarran.com, I had commenting enabled for new posts but had it set so that administrator approval was needed before any new comments went live. Coming from a MovableType blog previously, I was used to doing this as I received tons of spam on that blog and had to moderate there as well. It's not a good user experience to expect people to wait until I approve a comment though, so I was keen to let users post directly.

When the site first launched, there were very few spam comments, so leaving comments open seamed feasible, but very soon - as spammers started to pick up on the changes - they started arriving in droves, and that was just on the few comment-enabled posts that I had created since launching.

Not wanting to impede people's commenting with a captcha for every comment, I avoided Drupal's captcha module and instead opted to try out the Mollom module as an interface to the Mollom spam filtering service created by Dries Buytaert, the founder of the Drupal project.

Mollom is a free service that checks all the comments (and/or other forms) posted for known patterns of spam, blocking it where appropriate, and letting real user-created comments through unhindered. If it's not sure whether it's spam or ham (the term for real content), it then presents the user with a captcha that the user can fill in if they have been mistakenly flagged as possible spam.

So far, the service has been great, with 588 spam messages blocked in the past 16 days, the busiest days being Christmas Eve and Christmas Day with 260 spam messages between them. I'm very happy to be a Mollom user! Sorry spammers, it's nothing personal, honest.

Dec 31 2008
Dec 31

From the start of 2009 I am going to be self-employed, and while I will still be working much of my time on the projects I've been working on for the past three years at OpenBand/M.C. Dean, I would also like to start taking on some small Drupal-based projects to go side-by-side with that.

Some of the services I am planning to offer in the New Year include:

  • Drupal site creation
    If you are looking to get a site set up for your small business or organisation, and you like the power that Drupal can give to your site, then I can set a site up for you to meet your needs. I'm particularly interested in creating sites based around geographic information or related to the tourism industry, but I will happily consider any project.
  • Drupal site setup support
    If you need advice on how best to achieve your requirements with existing Drupal modules, I can help point you in the right direction and get you started with your Drupal site setup. This can either be on a remote basis, or on a face-to-face basis in the London area if needed.
  • Drupal module development
    If there isn't already a module in the Drupal community to do what you need, I can help you by building a module to meet your requirements.

If you're interested in taking advantage of these freelance Drupal services for your project, please contact me to discuss your needs.

Dec 15 2008
Dec 15

We needed to disable all of Drupal’s CSS files from our theme. Here’s how we did it:

function THEMENAME_preprocess(&$variables) {

  // Get rid of all of Drupal's CSS files
  $css = drupal_add_css();
  foreach ($css['all']['module'] as $file => $status) {
    if (!strstr($file, 'modules/MYMODULE')) {
      unset($css['all']['module'][$file]);
    }
  }
  $variables['styles'] = drupal_get_css($css);

We also wanted (no *real* need) to use screen.css rather than style.css, so we edited THEMENAME.info to have this:

stylesheets[all][] = reset.css
stylesheets[all][] = screen.css

… and we removed the line for style.css from it.

Finally, as the superuser we went to /admin/build/modules (or /themes, can’t remember now) to refresh the theme cache. We also had to tick to enable the theme at /admin/build/themes as although we’d been using the theme for ages quite fine, it wasn’t actually ticked before.

And hey presto, it worked. Should probably add that it took waaaay too long to do though, so though we’d add this snipped for others to read.

Share this:

Like this:

Like Loading...
Dec 01 2008
Dec 01

If you build quality sites that attract a large number of visitors and interaction there eventually will come a point when you have to start looking for ways to offload your files and bring down your server overhead. I have been looking into the CDN issue off and on for the past 6 months. Recently I decided it was time to get something dialed in and move forward. I wanted something that required the least number of hacks and was easy and scalable. This post isn't meant to be an end all to Drupal and CDNs, but rather just some insight into the way I have tackled this issue for the time being.

There are a number of options to choose from and a lot of different ways to go about it. You could get a new server locally and load balance your stuff, you could get a new server locally and use it as a static file server, you could team up with a big time CDN like Akamai or Limelight and go that route, or you could go the less expensive CDN route with something like Amazon S3.

Note: This article doesn't attempt to explain every little detail of what's going on, but rather act as a guide for a developer to work off.

My Goals

Being somebody that is cost conscience and wanting to always try to get the most bang for my buck the idea of a CDN like Limelight didn't quite seem like an appropriate fit for me. I wanted something a little bit less serious that I could ease into. I don't care about having servers all across the world and getting content to people a few milliseconds faster than other solutions. I just want to take a huge load off my local network and put that load somewhere else. If my site is up and the users are happy, I'm happy.

I also don't care if 100% of my site files are not offloaded to the CDN. If I can take care of 95% of my load and leave 5% on my main box it really doesn't bother me. My main goal is to get rid of the majority of the overhead and keep everything scalable and dynamic with a relatively small code footprint.

Which files to offload

With these things in mind I have basically decided that a good route for me was to mainly take care of my two main sources of files: images and videos. If I really wanted to get hardcore I could also move my CSS and Javascript, but I don't see this as important as taking care of the major problems first.

Take my site www.gunslot.com for example. If you hit the homepage you will see that there are a lot of thumbnails and image requests. Each page probably has at least 20 image requests, some as much as 50. So I would rather take care of these requests first and then I can worry about the 5 or so CSS and JS requests later if I have to.

And then I have the video content which can be large in filesize and taxing on the server. This also had to go.

Local file system

Another thing to keep in mind is that I don't want to make my CDN act as my local file system. I don't want people to upload straight to this and only have this as my main system, I just see too many errors and bugs going this route. I would rather just have everything work normally and smoothly through the default local structure and just copy stuff over to the CDN, which brings up an important point.

Synching your content

Just how should you copy your content over to the CDN? Big time CDNs like Limelight allow HTTP synchronization (which Ted Serbinski talks about in his article) which basically copies your files over automatically. S3 does not offer this type of functionality so you will need to go another route.

You could simply copy them all at once programatically and call it a day. You could also maybe set up a cron and copy any new files every few minutes. But how do you know which files are new? How do you know which files get updated and which don't? You could run some PHP scripts like a cURL or file_get_contents to check the file's last modified time, but this has some big overhead (I tried). So when is the best time to copy the file and how should you do it? You most likely want to get the file over to the CDN as quick as possible, but at the same time you want to make sure if the user updates the file or something is changed your CDN reflects that.

My S3 Solution

I chose to go with a hybrid type approach. I basically send a file over to the S3 every time it is requested, if it is not already there, or if it is newer than the current file.

I have one main routing function that when called will run through the flowchart below and figure out what path to return for any given file, either:

Local: http://www.domain.com/files/full/path/myfile.jpg

or

S3: http://s3domain.com/files/full/path/myfile.jpg

So basically my flowchart goes something like:

The Technical

S3 Interfacing

So if any of you have messed with S3 you know that you are going to need a PHP class first to put your stuff there. I found a decent class off their forums and went with that (it requires PEAR, unfortunately). At one point I'll probably change this to something better if I find one, but it gets the job done for now.

Filetype check

This is just a quick little function that basically checks if the filetype is listed in an array of allowed filetypes that I have chosen, like: jpg, jpeg, gif, png, flv, mp3, etc.

If file exists

So this is one of the most important steps that I took to make this possible. Rather than actually checking if the file exists on S3 with one of the S3 class functions (slow) or via something like cURL or file_get_contents (slow) I went with a database table on my own server that keeps track of what is on the S3 server (fast). This table keeps track of every filepath and when it was created and changed on S3. I am able to use the changed timestamp from my table to check it against the local file's timestamp (filemtime) to know when the file needs to be updated on S3.

Queuing the file for S3

Originally I didn't queue the file at all and just tried to put the file to S3 every time it was requested - mistake. It obviously took waaaaaaay to long to render my pages with this overhead so I opted for the queue method. I created another database table that keeps track of every file that needs to be put to S3. Basically every time a file doesn't exist on S3 I return the local path for the time being and add this file to the queue.

Putting to S3

As for putting the file to S3 there are a lot of ways to do this and it probably depends a bit on your situation. You could run a cron or some type of rsync if you really wanted to dial it in, but for the time being I am going for a bit more simpler method. I simply run 1 put operation at the end of each page request by the users. This seems to work really well right now and gets the files uploaded pretty much within seconds of when they are queued. At any time I have less than 20 or so files in the queue, and obviously once most of my files are on S3 this doesn't need to run anymore. I run my put function using hook_exit and I pick 1 file each time and get it over to S3.

Routing the files with Drupal

So once you have all this ready to go how do you actually get Drupal to replace the current local paths with the S3 paths? Well, there are a number of ways to do this. If you want to replace all your images, CSS, logos and all that stuff you can patch common.inc, file.inc and theme.inc (maybe more) to run through your routing function. Since I don't care too much about this stuff I skipped this part and decided just to replace imagecache and my videos (for now).

Imagecache

Imagecache comes with a sweet theme_imagecache function that allows you to simply over-ride it in your template.php. You basically just need to tweak the $imagecache_url to run through your routing function and decide which path to return.

Notes

If you are using Imagecache for profile pics or nodes you may need to flush the imagecache when these are updated so that your new file will run through the queue and be uploaded to S3. This can be done via some hook_form_alter or hook_nodeapi calls that run the imagecache_image_flush function.

Videos

My videos are done through a custom module so I can't really offer any help here. I just plugged my routing function into it.

So that's it!

As you can see for each file request that is going through the routing function it will always check to make sure the newest file is on S3. If it's not it just pulls the local file for the time being. As soon as the S3 file is ready it will start pulling that one. If you do an update it will pull the new local file until the new S3 file is ready again.

That's basically how I'm doing it right now and it's working really well. My server is thanking me and even with fairly high traffic the S3 costs are very reasonable. If anyone has any feedback or tips on how I could make this better lemme know!

 Filed under: Internet / Tech, Drupal, S3, File Server, CDN
Nov 27 2008
Nov 27

My IMCE plugin for YUI Editor has been included in drupal CVS git and the 6.x-2.33 release. Now I can claim to have code included in an official drupal release, ok it is a small plugin for a contrib module, we all have to start somewhere.

The version included in Drupal only supports YUI 2.5.x as the API has changed in 2.6. I have a new version which supports 2.6.x, but it has a layout bug, so I won’t be submitting it until this bug is fixed. If you can tolerate the visual bug or want to help fix it, grab the lastest version of the IMCE plugin for Drupal’s YUI Editor. Use the same installation instructions as last time.

Feedback welcome.

Nov 23 2008
Nov 23

In this episode: Drupal Docs and Drupal Dojo steal all the headlines.

Nov 20 2008
Nov 20

The front-end database is powered by Drupal. Includes a video demonstration.

Nov 16 2008
Nov 16

In this episode: KDI applications on hold, new theme coming for core, installation insurrection, events, site launches.

Links from the episode

KDI Apps on hold until DrupalCon DC

CHX Installation Insurrection

Albin: New theme headed for the core

Sites

  • Drupal Showcase:
  • Pregnancy.org
  • SpreadThunderbird.com

Events

Share | Download
Nov 14 2008
Nov 14

This is the podcast for November 14th, 2008. Welcome to the DrupalNews podcast. My goal with this podcast is to provide a quick overview of the most important changes in the drupal community. Story ideas are welcome and encouraged.

Share | Download

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web