Oct 04 2019
Oct 04

Over the years I have benefited greatly from the generosity of the Drupal Community. In 2011 people sponsored me to write lines of code to get me to DrupalCon Chicago.

Today Dave Hall Consulting is a very successful small business. We have contributed code, time and content to Drupal. It is time for us to give back in more concrete terms.

We want to help someone from an under represented group take their career to the next level. This year we will provide a Diversity Scholarship for one person to attend DrupalSouth, our 2 day Gettin’ Git training course and 5 nights at the conference hotel. This will allow this person to attend the premier Drupal event in the region while also learning everything there is to know about git.

To apply for the scholarship, fill out the form by 23:59 AEST 19 October 2019 to be considered. (Extended from 12 October)

Share this post

Sep 24 2017
Sep 24

Over the years Drupal distributions, or distros as they're more affectionately known, have evolved a lot. We started off passing around database dumps. Eventually we moved onto using installations profiles and features to share par-baked sites.

There are some signs that distros aren't working for people using them. Agencies often hack a distro to meet client requirements. This happens because it is often difficult to cleanly extend a distro. A content type might need extra fields or the logic in an alter hook may not be desired. This makes it difficult to maintain sites built on distros. Other times maintainers abandon their distributions. This leaves site owners with an unexpected maintenance burden.

We should recognise how people are using distros and try to cater to them better. My observations suggest there are 2 types of Drupal distributions; starter kits and targeted products.

Targeted products are easier to deal with. Increasingly monetising targeted distro products is done through a SaaS offering. The revenue can funds the ongoing development of the product. This can help ensure the project remains sustainable. There are signs that this is a viable way of building Drupal 8 based products. We should be encouraging companies to embrace a strategy built around open SaaS. Open Social is a great example of this approach. Releasing the distros demonstrates a commitment to the business model. Often the secret sauce isn't in the code, it is the team and services built around the product.

Many Drupal 7 based distros struggled to articulate their use case. It was difficult to know if they were a product, a demo or a community project that you extend. Open Atrium and Commerce Kickstart are examples of distros with an identity crisis. We need to reconceptualise most distros as "starter kits" or as I like to call them "puppies".

Why puppies? Once you take a puppy home it becomes your responsibility. Starter kits should be the same. You should never assume that a starter kit will offer an upgrade path from one release to the next. When you install a starter kit you are responsible for updating the modules yourself. You need to keep track of security releases. If your puppy leaves a mess on the carpet, no one else will clean it up.

Sites build on top of a starter kit should diverge from the original version. This shouldn't only be an expectation, it should be encouraged. Installing a starter kit is the starting point of building a unique fork.

Project pages should clearly state that users are buying a puppy. Prospective puppy owners should know if they're about to take home a little lap dog or one that will grow to the size of a pony that needs daily exercise. Puppy breeders (developers) should not feel compelled to do anything once releasing the puppy. That said, most users would like some documentation.

I know of several agencies and large organisations that are making use of starter kits. Let's support people who are adopting this approach. As a community we should acknowledge that distros aren't working. We should start working out how best to manage the transition to puppies.

Share this post

Sep 16 2017
Sep 16

While preparing for my DrupalCamp Belgium keynote presentation I looked at how easy it is to get started with various CMS platforms. For my talk I used Contentful, a hosted content as a service CMS platform and contrasted that to the "Try Drupal" experience. Below is the walk through of both.

Let's start with Contentful. I start off by visiting their website.

Contentful homepage

In the top right corner is a blue button encouraging me to "try for free". I hit the link and I'm presented with a sign up form. I can even use Google or GitHub for authentication if I want.

Contentful signup form

While my example site is being installed I am presented with an overview of what I can do once it is finished. It takes around 30 seconds for the site to be installed.

Contentful installer wait

My site is installed and I'm given some guidance about what to do next. There is even an onboarding tour in the bottom right corner that is waving at me.

Contentful dashboard

Overall this took around a minute and required very little thought. I never once found myself thinking come on hurry up.

Now let's see what it is like to try Drupal. I land on d.o. I see a big prominent "Try Drupal" button, so I click that.

Drupal homepage

I am presented with 3 options. I am not sure why I'm being presented options to "Build on Drupal 8 for Free" or to "Get Started Risk-Free", I just want to try Drupal, so I go with Pantheon.

Try Drupal providers

Like with Contentful I'm asked to create an account. Again I have the option of using Google for the sign up or completing a form. This form has more fields than contentful.

Pantheon signup page

I've created my account and I am expecting to be dropped into a demo Drupal site. Instead I am presented with a dashboard. The most prominent call to action is importing a site. I decide to create a new site.

Pantheon dashboard

I have to now think of a name for my site. This is already feeling like a lot of work just to try Drupal. If I was a busy manager I would have probably given up by this point.

Pantheon create site form

When I submit the form I must surely be going to see a Drupal site. No, sorry. I am given the choice of installing WordPress, yes WordPress, Drupal 8 or Drupal 7. Despite being very confused I go with Drupal 8.

Pantheon choose application page

Now my site is deploying. While this happens there is a bunch of items that update above the progress bar. They're all a bit nerdy, but at least I know something is happening. Why is my only option to visit my dashboard again? I want to try Drupal.

Pantheon site installer page

I land on the dashboard. Now I'm really confused. This all looks pretty geeky. I want to try Drupal not deal with code, connection modes and the like. If I stick around I might eventually click "Visit Development site", which doesn't really feel like trying Drupal.

Pantheon site dashboard

Now I'm asked to select a language. OK so Drupal supports multiple languages, that nice. Let's select English so I can finally get to try Drupal.

Drupal installer, language selection

Next I need to chose an installation profile. What is an installation profile? Which one is best for me?

Drupal installer, choose installation profile

Now I need to create an account. About 10 minutes I already created an account. Why do I need to create another one? I also named my site earlier in the process.

Drupal installer, configuration form part 1 Drupal installer, configuration form part 2

Finally I am dropped into a Drupal 8 site. There is nothing to guide me on what to do next.

Drupal site homepage

I am left with a sense that setting up Contentful is super easy and Drupal is a lot of work. For most people wanting to try Drupal they would have abandoned someway through the process. I would love to see the conversion stats for the try Drupal service. It must miniscule.

It is worth noting that Pantheon has the best user experience of the 3 companies. The process with 1&1 just dumps me at a hosting sign up page. How does that let me try Drupal?

Acquia drops onto a page where you select your role, then you're presented with some marketing stuff and a form to request a demo. That is unless you're running an ad blocker, then when you select your role you get an Ajax error.

The Try Drupal program generates revenue for the Drupal Association. This money helps fund development of the project. I'm well aware that the DA needs money. At the same time I wonder if it is worth it. For many people this is the first experience they have using Drupal.

The previous attempt to have simplytest.me added to the try Drupal page ultimately failed due to the financial implications. While this is disappointing I don't think simplytest.me is necessarily the answer either.

There needs to be some minimum standards for the Try Drupal page. One of the key item is the number of clicks to get from d.o to a working demo site. Without this the "Try Drupal" page will drive people away from the project, which isn't the intention.

If you're at DrupalCon Vienna and want to discuss this and other ways to improve the marketing of Drupal, please attend the marketing sprints.

Share this post

Apr 27 2017
Apr 27

My blog post from last week was very well received and sparked a conversation in the Drupal community about the future of Drupal. That conversation has continued this week at DrupalCon Baltimore.

Yesterday during the opening keynote, Dries touched on some of the issues raised in my blog post. Later in the day we held an unofficial BoF. The turn out was smaller than I expected, but we had a great discussion.

Drupal moving from a hobbyist and business tool to being an enterprise CMS for creating "ambitious digital experiences" was raised in the Driesnote and in other conversations including the BoF. We need to acknowledge that this has happened and consider it an achievement. Some people have been left behind as Drupal has grown up. There is probably more we can do to help these people. Do we need more resources to help them skill up? Should we direct them towards WordPress, backdrop, squarespace, wix etc? Is it is possible to build smaller sites that eventually grow into larger sites?

In my original blog post I talked about "peak Drupal" and used metrics that supported this assertion. One metric missing from that post is dollars spent on Drupal. It is clear that the picture is very different when measuring success using budgets. There is a general sense that a lot of money is being spent on high end Drupal sites. This has resulted in less sites doing more with Drupal 8.

As often happens when trying to solve problems with Drupal during the BoF descended into talking technical solutions. Technical solutions and implementation detail have a place. I think it is important for the community to move beyond this and start talking about Drupal as a product.

In my mind Drupal core should be a content management framework and content hub service for building compelling digital experiences. For the record, I am not arguing Drupal should become API only. Larger users will take this and build their digital stack on top of this platform. This same platform should support an ecosystem of Drupal "distros". These product focused projects target specific use cases. Great examples of such distros include Lightning, Thunder, Open Social, aGov and Drupal Commerce. For smaller agencies and sites a distro can provide a great starting point for building new Drupal 8 sites.

The biggest challenge I see is continuing this conversation as a community. The majority of the community toolkit is focused on facilitating technical discussions and implementations. These tools will be valuable as we move from talking to doing, but right now we need tools and processes for engaging in silver discussions so we can build platinum level products.

Share this post

Apr 21 2017
Apr 21

WOW! The response to my blog post on the future of Drupal earlier this week has been phenomenal. My blog saw more traffic in 24 hours than it normally sees in a 2 to 3 week period. Around 30 comments have been left by readers. My tweet announcing the post was the top Drupal tweet for a day. Some 50 hours later it is still number 4.

It seems to really connected with many people in the community. I am still reflecting on everyone's contributions. There is a lot to take in. Rather than rush a follow up that responds to the issues raised, I will take some time to gather my thoughts.

One thing that is clear is that many people want to use DrupalCon Baltimore next week to discuss this issue. I encourage people to turn up with an open mind and engage in the conversation there.

A few people have suggested a BoF. Unfortunately all of the official BoF slots are full. Rather than that be a blocker, I've decided to run an unofficial BoF on the first day. I hope this helps facilitate the conversation.

Unofficial BoF: The Future of Drupal

When: Tuesday 25 April 2017 @ 12:30-1:30pm
Where: Exhibit Hall - meet at the Digital Echidna booth (#402) to be directed to the group
What: High level discussion about the direction people think Drupal should take.
UPDATE: An earlier version of this post had this scheduled for Monday. It is definitely happening on Tuesday.

I hope to see you in Baltimore.

Share this post

Apr 19 2017
Apr 19

Update 21 April: I've published a followup post with details of the BoF to be held at DrupalCon Baltimore on Tuesday 25 April. I hope to see you there so we can continue the conversation.

Drupal has a problem. No, not that problem.

We live in a post peak Drupal world. Drupal peaked some time during the Drupal 8 development cycle. I’ve had conversations with quite a few people who feel that we’ve lost momentum. DrupalCon attendances peaked in 2014, Google search impressions haven’t returned to their 2009 level, core downloads have trended down since 2015. We need to accept this and talk about what it means for the future of Drupal.

Technically Drupal 8 is impressive. Unfortunately the uptake has been very slow. A factor in this slow uptake is that from a developer's perspective, Drupal 8 is a new application. The upgrade path from Drupal 7 to 8 is another factor.

In the five years Drupal 8 was being developed there was a fundamental shift in software architecture. During this time we witnessed the rise of microservices. Drupal is a monolithic application that tries to do everything. Don't worry this isn't trying to rekindle the smallcore debate from last decade.

Today it is more common to see an application that is built using a handful of Laravel micro services, a couple of golang services and one built with nodejs. These applications often have multiple frontends; web (react, vuejs etc), mobile apps and an API. This is more effort to build out, but it likely to be less effort maintaining it long term.

I have heard so many excuses for why Drupal 8 adoption is so slow. After a year I think it is safe to say the community is in denial. Drupal 8 won't be as popular as D7.

Why isn't this being talked about publicly? Is it because there is a commercial interest in perpetuating the myth? Are the businesses built on offering Drupal services worried about scaring away customers? Adobe, Sitecore and others would point to such blog posts to attack Drupal. Sure, admitting we have a problem could cause some short term pain. But if we don't have the conversation we will go the way of Joomla; an irrelevant product that continues its slow decline.

Drupal needs to decide what is its future. The community is full of smart people, we should be talking about the future. This needs to be a public conversation, not something that is discussed in small groups in dark corners.

I don't think we will ever see Drupal become a collection of microservices, but I do think we need to become more modular. It is time for Drupal to pivot. I think we need to cut features and decouple the components. I think it is time for us to get back to our roots, but modernise at the same time.

Drupal has always been a content management system. It does not need to be a content delivery system. This goes beyond "Decoupled (Headless) Drupal". Drupal should become a "content hub" with pluggable workflows for creating and managing that content.

We should adopt the unix approach, do one thing and do it well. This approach would allow Drupal to be "just another service" that compliments the application.

What do you think is needed to arrest the decline of Drupal? What should Drupal 9 look like? Let's have the conversation.

Share this post

Apr 06 2017
Apr 06

Living in the middle of nowhere and working most of my hours in the evenings I have few opportunities to attend events in person, let alone deliver presentations. As someone who likes to share knowledge and present at events this is a problem. My work around has been presenting remotely. Many of my talks are available on playlist on my youtube channel.

I've been doing remote presentations for many years. During this time I have learned a lot about what it takes to make a remote presentation sucessful.

Preparation

When scheduling a remote session you should make sure there is enough time for a test before your scheduled slot. Personally I prefer presenting after lunch as it allows an hour or so for dealing with any gremlins. The test presentation should use the same machines and connections you'll be using for your presentation.

I prefer using Hangouts On Air for my presentations. This allows me to stream my session to the world and have it recorded for future reference. I review every one of my recorded talks to see what I can do better next time.

Both sides of the connection should use wired connections. WiFi, especially at conferences can be flakely. Organisers should ensure that all presentation machines are using Ethernet, and if possible it should be on a separate VLAN.

Tips for Presenters

Presenting to a remote audience is very different to presenting in front of a live audience. When presenting in person you're able to focus on people in the audience who seem to be really engaged with your presentation or scan the crowd to see if you're putting people to sleep. Even if there is a webcam on the audience it is likely to be grainy and in a fixed position. It is also difficult to pace when presenting remotely.

When presenting in person your slides will be diplayed in full screen mode, often with a presenter view in your application of choice. Most tools don't allow you to run your slides in full screen mode. This makes it more difficult as a presenter. Transitions won't work, videos won't autoplay and any links Keynote (and PowerPoint) open will open in a new window that isn't being shared which makes demos trickier. If you don't hide the slide thumbnails to remind you of what is coming next, the audience will see them too. Recently I worked out printing thumbnails avoids revealing the punchlines prematurely.

Find out as much information as possible about the room your presentation will be held in. How big is it? What is the seating configuration? Where is the screen relative to where the podium is?

Tips for Organisers

Event organisers are usually flat out on the day of the event. Having to deal with a remote presenter adds to the workload. Some preparation can make life easier for the organisers. Well before the event day make sure someone is nominated to be the point of contact for the presenter. If possible share the details (name, email and mobile number) for the primary contact and a fallback. This avoids the presenter chasing random people from the organising team.

On the day of the event communicate delays/schedule changes to the presenter. This allows them to be ready to go at the right time.

It is always nice for the speaker to receive a swag bag and name tag in the mail. If you can afford to send this, your speaker will always appreciate it.

Need a Speaker?

Are you looking for a speaker to talk about Drupal, automation, devops, workflows or open source? I'd be happy to consider speaking at your event. If your event doesn't have a travel budget to fly me in, then I can present remotely. To discuss this futher please get in touch using my contact form.

Share this post

Sep 17 2016
Sep 17

DrupalCon Dublin is just around the corner. Earlier today I started my journey to Dublin. This week I'll be in Mumbai for some work meetings before heading to Dublin.

On Tuesday 27 September at 1pm I will be presenting my session Let the Machines do the Work. This lighthearted presentation provides some practical examples of how teams can start to introduce automation into their Drupal workflows. All of the code used in the examples will be available after my session. You'll need to attend my talk to get the link.

As part of my preparation for Dublin I've been road testing my session. Over the last few weeks I delivered early versions of the talk to the Drupal Sydney and Drupal Melbourne meetups. Last weekend I presented the talk at Global Training Days Chennai, DrupalCamp Ghent and DrupalCamp St Louis. It was exhausting presenting three times in less than 8 hours, but it was definitely worth the effort. The 3 sessions were presented using hangouts, so they were recorded. I gained valuable feedback from attendees and became aware of some bits of my talk needed some attention.

Just as I encourage teams to iterate on their automation, I've been iterating on my presentation. Over the next week or so I will be recutting my demos and polishing the presentation. If you have a spare 40 minutes I would really appreciate it if you watch one of the session recording below and leave a comment here with any feedback.

Global Training Days Chennai

Thumbnail frame from DrupalCamp Ghent presentation video

DrupalCamp Ghent

Thumbnail frame from DrupalCamp Ghent presentation video

Note: I recorded the audience not my slides.

DrupalCamp St Louis

Thumbnail frame from DrupalCamp St Louis presentation video

Note: There was an issue with the mic in St Louis, so there is no audio from their side.

Share this post

Jan 25 2016
Jan 25

One of the biggest improvements in Drupal 8 is the new configuration management system. Config is now decoupled from code and the database. Unlike Drupal 6 and 7, developers no longer have to rely on the features module for moving configuration around.

Most large Drupal sites, and some smaller ones, require per environment configuration. Prior to Drupal 8 this was usually achieved using a combination of hard coding config variables and features. Drupal 8 still allows users to put config variables in the settings.php file, but putting config in code feels like a backward step given D8 emphasis on separating concerns.

For example we may have a custom module which calls a RESTful API of a backend service. There are dev, stage and production endpoints that we need to configure. We also keep our config out of docroot and use drush to import the config at deployment time. We have the following structure in our git repo:

/
+- .git/
|
+- .gitignore
|
+- README.md
|
+- config/
|  |
|  +- README.md
|  |
|  +- base/
|  |
|  +- dev/
|  |
|  +- prod/
|  |
|  +- stage/
|
+- docroot/
|
+- scripts/
|
+- and-so-on/

When a developer needs to export the config for the site they run drush config-export --destination=/path/to/project/config/base. This exports all of the configuration to the specified path. To override the API endpoint for the dev environment, the developer would make the config change and then export just that piece of configuration. That can be done by runing drush config-get mymodule.endpoint > /path/to/project/config/dev/mymodule.endpoint.yml.

Drupal 8 and drush don't allow you to import the 2 config sets at the same time, so we need to run 2 drush commands to import our config. drush config-import --source=/path/to/project/config/base && drush config-import --partial --source=/path/to/project/config/dev. The first command imports the base config and the second applies any per environment overrides. The --partial flag prevents drush deleting any missing config. In most cases this is ok, but watch out if you delete a view or block placement.

Best practices are still emerging for managing configuration in Drupal 8. While I have this method working, I'm sure others have different approaches. Please leave a comment if you have an alternative method.

Update 11-Mar-2016:Removed --partial from base import. This prevented old configuration being removed during updates.

Share this post

May 14 2015
May 14

Update: It turns out the DA was trolling. We all now know that DrupalCon North America 2016 will be in New Orleans. I've kept this post up as I believe the information about handling unpublished nodes is relevant. I have also learned that m4032404 is enabled by default in govCMS.

When a user doesn't have access to content in Drupal a 403 forbidden response is returned. This is the case out of the box for unpublished content. The problem with this is that sensitive information may be contained in the URL. A great example of this the DrupalCon site.

The way to avoid this is to use the m4032404 module which changes a 403 response to a 404. This simple module prevents your site leaking information via URLs.

Share this post

Attachment Size DrupalCon-Philadephia.png 139.21 KB
Apr 12 2015
Apr 12

A couple of times recently the issue of managing variables in Drupal 7 has come up in conversation with other developers. This post outlines the various ways of managing variables in Drupal sites. The three things this guide ensures:

  • Sensitive data is kept secure
  • Variables are correct in each environment
  • You are able to track your variables (and when they changed)

The Variables Table

The most common place you'll find configuration variables is in Drupal's variable table (aka {variable}). The values in this table are often managed via admin forms that use system_settings_form(). Users enter the values click "Save configuration" and the data is stored in the database.

If you prefer to manage your configuration via the command line and know the variable you wish to set you can use drush vset. This does exactly the same thing as admin form, without needing to click on a mouse.

$conf Array

While the variables table is great at storing our variables, there are times when you want to enforce a setting. This might be because you want to prevent users from changing it (accidentally or otherwise) or because you need it to be different in each environment. The $conf array in settings.php always overrides any values in the variable table.

Acquia, Pantheon and platform.sh all provide environment variables so you can use different values in your $conf array depending on the environment.

Exporting Variables

In Drupal 7, the common way to export your variables is by using Strongarm with Features. I'm not going to cover how to do this as there is loads of documentation already available on this topic.

If your variable changes on a per environment basis or if it calculated on the fly, then you won't want to use strongarm+features as the exported values are static. You will need to put them in settings.php.

Note to self: I should debug and reroll my patch for adding support in alter hooks strongarm.

My settings.php is Out of Control!

This is a common problem, especially on more complex sites. To avoid this I recommend creating sites/default/settings/settings.[env].php files. Your settings.php file should check for the environment in an environment variable and then include the appropriate settings.[env].php file.

What About Sensitive Data?

You can encrypt variables on a case by case basis using the encrypt module and some custom code similar to what I recently implemented in the Acquia SDK module (see on store and on read examples). Warning: This doesn't encrypt the data if you're using drush vset.

If you are storing sensitive data in your variables table I would recommend you implement hook_sql_sync_sanitize() which will delete the sensitive data from your db when drush sql-sanitize or drush sql-sync --sanitize are run.

How to Decide?

This little code snippet should help you decide.

<?php

// Don't try using this code in your Drupal site.

if (!using_version_control()) {
  // Seriously there is no point in doing this without version control.
  abandon_all_hope();
  drupal_exit();
}

if (is_data_sensitive($var)) {
  $var = encrypt_var($var);
  if (!we_use_drush_based_workflows()) {
    learn_and_implement_drush_based_workflows();
    // I'm serious!
    }
  }
  implement_hook_sql_sync_sanitize($var);
}

if (is_unique_per_environment($var)) {
  store_conf_array($var);
}
else {
  store_in_db($var);
  if (!we_use_features_based_workflow()) {
    learn_and_implement_features_based_worflows();
    // I'm serious!
  }
  export_using_strongarm($var);
}

Share this post

Jul 21 2014
Jul 21

The Acquia Cloud API makes it easy to manage sites on the platform. The API allows you to perform many administrative tasks including creating, destroying and copying databases, deploying code, managing domains and copying files.

Acquia offers 2 official clients. The primary client is a drush plugin which can only be downloaded from Acquia Insight. The other is a PHP library which states in the README that it is "[n]ot ready for production usage".

On a recent project using WF Tools we needed some pretty advanced deployment scripts for sites hosted on Acquia Cloud. We had tried using a mix of bash and PHP, but that created a maintenance nightmare, so we switched to Python.

I was unable to find a high quality Python library, so I wrote a python client for the Acquia Cloud API. The library implements all of the features that we needed, so there are a few things missing.

Chaining complex commands together is easy because the library implements a fluent interface. An extreme example of what is possible is below:


import acapi

# Instantiate the client
c = acapi.Client('[email protected]', 'acquia-token')

# Copy the prod db to dev, make a backup of the dev db and download it to /tmp
c.site('mysite').environment('prod').db('mysite').copy('dev').backups().create().download('/tmp/backup.sql.gz')

Some of the code is "borrowed" from the Python client for Twilio. The library is licensed under the terms of the MIT license.

I am continuing to develop the library. Consider this a working alpha. Improving error handling, creating a comprehensive test suite and implementing the missing API calls are all on the roadmap. Pull requests are welcome.

The code is PEP 8 (coding standards and PEP 257 (documentation standards) compliant and uses the numpydoc for code documentation.

Check out the Python client for Acquia's Cloud API on github.

Bookmark/Search this post with

Jul 15 2013
Jul 15

Most experienced Drupal developers have worked on sites with security issues. Keeping modules up to date is a pretty straight forward process, especially if you pay attention to the security advisories. Coder can find security issues in your code.

Recently I needed to perform a mass security audit to ensure a collection of sites were properly configured. After searching and failing to find a module that would do what I needed, I decided to write my own. The Security Check module for Drupal checks basic configuration options to ensure a site configuration doesn't have any obvious security flaws. This module isn't designed to find all flaws in your site.

Security Check works by checking a list of installed modules, settings for variables and permission assignments. I hope that others in the community will have suggestions for other generic tests that can be implemented in the module. If you have any ideas (or patches), please submit them as issues in the queue.

This module isn't a substitute for a full security audit, which can be conducted in house or by a third party such as Acquia's Professional Service team. Security Check is designed to be run as part of an automated site audit to catch low hanging fruit.

To use Security Checker install it in ~/.drush and then cd to any docroot and run "drush security-check" or if you prefer to use an alias run "drush @example secchk" from anywhere.

Bookmark/Search this post with

Jan 02 2013
Jan 02

Over the Christmas break I came across gource, a software version control visualization tool. Gource produces really nice visual representations of software projects growing. About 2 years ago David Norman produced a gource video of development of Drupal up to the 7 release. This is pretty cool, but it only shows who committed the patch, not who contributed to it.

After some searching I found the Drupal contribution analyzer sandbox project. This module allows you to produce contributor tag clouds and code swarm videos. This was closer to what I was after, but I had to patch patch drupal_log_generator.py to support the gource custom log format.

The result is a 5:23 minute video showing the growth of Drupal.

The first few years things are pretty consistent and easy to follow. The Drupal 8 development cycle shows how much the community of contributors has grown. Towards the end of last year things look really chaotic.

To produce the video I used a clone of the Drupal 8 branch as at some time on January 1, 2013. The gource command I used was:
gource --log-format custom -i 500 -s 0.0001 -a 0.01 -r 30 --title "Drupal" --highlight-users --disable-progress --hide filenames -1280x720 drupal --bloom-intensity 0.2 --bloom-multiplier 0.2 --stop-at-end /tmp/commit.log -o -| ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libvpx -b 10000K ~/Videos/drupal.webm

I considered writing a script to find and download user avatars from groups.drupal.org but after reviewing the video without them I decided it would be too cluttered.

Can you find your name in the video?

Note: I gave up on trying to embed the video

Bookmark/Search this post with

Oct 10 2012
Oct 10

This is a Drupalised version of Jive's Deirdre Walsh's blog post entitled "What LEGO taught me about Community Building".

Like most kids, I loved LEGO. I would spend hours building everything from spaceships to crazy robots (true story).

As an adult, building a community has that same sense of awesomeness.

Here is a list of the top 7 things LEGO taught me about building a quality community.

Accessibility. You can find LEGO building blocks anywhere (especially in the cup holders in my car). Social business needs to be the same. A strong community should span internally and externally, across departments, geographies, and devices.

Usability. Unlike Ikea furniture, anybody can pick up a few LEGO blocks, stick them together, and build something amazing. A good community should make it easy for members to go from a newbie to expert in record time, with engaging tutorials and introductory tours.

Fun. LEGO allows people spend hours being creative. Communities should engage users. Every week Drupal events are held which help make this a reality.

Beneficial. LEGOs are more than just an entertaining toy. By playing with LEGOs, kids learn things like simple mechanics. The same should ring true for your community - members should learn through building and sharing. Community members should be free to run, study, redistribute, modify and copy the code.

Next Generational. LEGO has evolved its product offerings. When I had more free time I used to play around with LEGO Mindstorms NXT. This flavor of LEGO allows you to build and program robots - a far advancement from the standard building blocks. A good community will also adopt next-generation technologies, such as enterprise applications, deep webservices integration, html5 and responsive design.

Versatile. By buying a single set of LEGOs you can make several different creations. One day, you'll build a log cabin and the next day a castle. Building a community is similar. With an investment in one strong social business platform, like Drupal, you can build a variety of vibrant communities for areas like customer support, sales and marketing, social intranet, etc.

Leader. Most boxes of LEGOs comes with one of those cool little, plastic people. Like those minifigs, it's key to have a community manager, who can serve as the front-person. Altimeter Research’s Jeremiah Owyang studied community manager job descriptions from 16 different organizations and found four key elements: community advocacy, brand evangelism, savvy communication skills and editorial planning, and liaising between internal decision makers and community members. In the Drupal community we don't need to link to LinkedIn profiles of people who inspire us. All the cool people have accounts on drupal.org - including Dries.

While building a community might not feel like child's play, just remember that it can be fun and the hard work will pay off in the end. Communities are real things that involve people, they are more than a website built using a CMS. As an example look at recent DrupalCons or BADCamp.

Now, if I can only get belly to be as flat as a LEGO minifig's....

Bookmark/Search this post with

Oct 08 2012
Oct 08

One of the many hats I wear these days is Development and Coding Track Chair for DrupalCon Sydney 2013. As outlined in the track description we are planning on showcasing what is awesome today in Drupal 7 and the cool stuff that is coming in Drupal 8. Given that there are no core conversations in Sydney we are trying to put together a more intermediate-to-advanced level track. I want people to come to these sessions and go away with their heads full of ideas about what they can do better in their next project.

If you have a session that you think fits that brief then please submit something. If you want to ask me anything before submitting your session, feel free to contact me. The decision on which sessions are accepted will be made in late October / early November by the track team, the global track chairs, the content chair and myself in a collaborative decision making process. The accepted sessions will be announced on 13 November.

Although the event won't be as big as a northern hemisphere DrupalCon, it is going to be full of great people. The initial 100 early bird tickets sold out in less than 8 hours!

Please be aware that there is no financial support available for speakers and you will be required to buy a speakers ticket at a cost of 165USD.

Submissions close at 23:59 AEST (UTC+10) on 26 October so submit a session today!

Bookmark/Search this post with

Sep 12 2012
Sep 12

In my last blog post I outlined how to use per project installation profiles. If you read that post and want to use installation profiles to take advantage of site wide content changes and centralised dependency management, this post will show you how to do it quickly and easily.

The easiest way to switch installation profiles is using the command line with drush. The following command will do it for you:

$ drush vset --exact -y install_profile my_profile

An alternative way of doing this is by directly manipulating the database. You can run the following SQL on your Drupal database to switch installation profiles:


UPDATE variable SET value = 'my_profile' WHERE name = 'install_profile';
-- Clear the cache using MySQL only syntax, when DB caching is used.
TRUNCATE cache;

Before you switch installation profiles, you should check that you have all the required modules enabled in your site. If you don't have all of the modules required by the new installation profile enabled in your site, your are likely to have issues. The best way to ensure you have all the dependencies enabled is to run the following one liner:

drush en $(grep dependencies /path/to/my-site/profiles/my_profile/my_profile.info | sed -n 's/dependencies\[\]=\(.*\)/\1/p')

Even though it is pretty easy to switch installation profiles I would recommend starting your project with a project specific installation profile.

Edit: Jaime Schmidt picked up a missing step in the instructions above. You need to enable the installation profile in the system table. The easiest way to do that is with this drush one liner:

echo UPDATE system SET schema_version = 0 WHERE name = 'my_profile' | drush sqlc && drush cc all

Further Edit: Marji Cermak picked up a typo in the dependencies one liner. It is one word I use a lot and always misspell.

Bookmark/Search this post with

Sep 09 2012
Sep 09

Unbeknown to many users, installation profiles are what is used to install a Drupal site. The two profiles that ship with core are standard and minimal. Standard gives new users a basic, functional Drupal site. Minimal provides a very minimal configuration so developers and site builders can start building a new site. A key piece of a Drupal distro is an installation profile.

I beleive that developers and more experienced site builders should be using installation profiles as part of their client sites builds. In Drupal 7 an installation profile is treated like a special module, so it can implement hooks - including hook_update_N(). This means that the installation profile is the best place for controlling turning modules on and off, switching themes or any other site wide configuration changes that can't be handled by features or a module specific update hook.

In an ideal world you could have 1 installation profile that is used for all of your projects and you just include it in your base build. Unfortunately installation profiles tend to evolve into being very project specific. At the same time you are likely to want a common starting point. I like to give my installation profiles unique names, rather than something generic like "my_profile", I prefer to use "[client_prefix]_profile". I'll cover project prefixes in another blog post.

After some trial and error, I've settled on a solution which I think works for having a common starting point for an installation profile that will diverge overtime using a unique namespace. My solution relies on some basic templates, a bash script with a bit of sed. I could have written all of this in PHP and even made a drush plugin for it, but I prefer to do this kind of thing on the command line with bash. I'm happy to work with someone to port it to a drush plugin if you're interested.

Here is a simple example of the templates you could use for creating your installation profile. The version on github is closer to what I actually use for clients, along with the build script.

base.info

name = PROFILE_NAME
description = PROFILE_DESCRIPTION
core = 7.x
dependencies[] = block
dependencies[] = dblog

base.install

<?php
/**
 * @file
 * Install, update and uninstall functions for the the PROFILE_NAME install profile.
 */

/**
 * Implements hook_install().
 *
 * Performs actions to set up the site for this profile.
 *
 * @see system_install()
 */
function PROFILE_NAMESPACE_install() {
  // Enable some standard blocks.
  $default_theme = variable_get('theme_default', 'bartik');
  $values = array(
    array(
      'module' => 'system',
      'delta' => 'main',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'content',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'user',
      'delta' => 'login',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'navigation',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'management',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 1,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'help',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'help',
      'pages' => '',
      'cache' => -1,
    ),
  );
  $query = db_insert('block')->fields(array('module', 'delta', 'theme', 'status', 'weight', 'region', 'pages', 'cache'));
  foreach ($values as $record) {
    $query->values($record);
  }
  $query->execute();

  // Allow visitor account creation, but with administrative approval.
  variable_set('user_register', USER_REGISTER_VISITORS_ADMINISTRATIVE_APPROVAL);

  // Enable default permissions for system roles.
  user_role_grant_permissions(DRUPAL_ANONYMOUS_RID, array('access content'));
  user_role_grant_permissions(DRUPAL_AUTHENTICATED_RID, array('access content'));
}

// Add hook_update_N() implementations below here as needed.

base.profile

<?php
/**
 * @file
 * Enables modules and site configuration for a PROFILE_NAME site installation.
 */

/**
 * Implements hook_form_FORM_ID_alter() for install_configure_form().
 *
 * Allows the profile to alter the site configuration form.
 */
function PROFILE_NAMESPACE_form_install_configure_form_alter(&$form, $form_state) {
  // Pre-populate the site name with the server name.
  $form['site_information']['site_name']['#default_value'] = $_SERVER['SERVER_NAME'];
}

Some developers might recognise the code above, it is from the minial installation profile.

The installation profile builder script is a simple bash script that relies on sed.

build-profile.sh

#!/bin/bash
#
# Installation profile builder
# Created by Dave Hall http://davehall.com.au
#

FILES="base.info base.install base.profile"
OK_NS_CHARS="a-z0-9_"
SCRIPT_NAME=$(basename $0)

namespace="my_profile"
name=""
description="My automatically generated installation profile."
target=""

usage() {
  echo "usage: $SCRIPT_NAME -t target_path -s profile_namespace [-d 'project_descrption'] [-n 'human_readable_profile_name']"
}

while getopts  "d:n:s:t:h" arg; do
  case $arg in
    d)
      description="$OPTARG"
      ;;
    n)
      name="$OPTARG"
      ;;
    s)
      namespace="$OPTARG"
      ;;
    t)
      target="$OPTARG"
      ;;
    h)
      usage
      exit
      ;;
  esac
done

if [ -z "$target" ]; then
  echo ERROR: You must specify a target path. >&2
  exit 1;
fi

if [ ! -d "$target" -o ! -w "$target" ]; then
  echo ERROR: The target path must be a writable directory that already exists. >&2
  exit 1;
fi

ns_test=${namespace/[^$OK_NS_CHARS]//}
if [ "$ns_test" != "$namespace" ]; then
  echo "ERROR: The namespace can only contain lowercase alphanumeric characters and underscores ($OK_NS_CHARS)" >&2
  exit 1
fi

if [ -z "$name" ]; then
  name="$namespace";
fi

for file in $FILES; do
  echo Processing $file
  sed -e "s/PROFILE_NAMESPACE/$namespace/g" -e "s/PROFILE_NAME/$name/g" -e "s/PROFILE_DESCRIPTION/$description/g" $file > $target/$file
done

echo Completed generating files for $name installation profile in $target.


Place all of the above files into a directory. Before you can generate your first profile you must run "chmod +x build-profile.sh" to make the script executable.

You need to create the output directory, for testing we will use ~/test-profile, so run "mkdir ~/test-profile" to create the path. To build your profile run "./build-profile.sh -s test -t ~/test-profile". Once the script has run you should have a test installation profile in ~/test-profile.

I will continue to maintain this as a project on github.

Bookmark/Search this post with

Jul 13 2012
Jul 13

Months ago I was searching for a good web front end for git for doing code reviews and browsing repos. My short list ended up being Gitweb and GitLab.

Gitweb is a Perl based web front end for git that is a sub project of the official git project. Out of the box Gitweb is pretty ugly and I have never found it to be very user friendly. Even with all of its problems, it does what it does pretty well.

GitLab is a Rails based github clone that is being actively developed and shows a lot of promise. If you want a fully blown ready to go basic github clone to setup and run for your dev team, GitLab looks great. Unfortunately trying to install it on RHEL5 wasn't an easy task. GitLab offered a lot more functionality than I needed and my lack of Ruby experience meant that it was adding more complexity than I wanted.

I ended up reluctantly recommending that we use Gitweb but I have since changed my mind.

Over the last few weeks I've been playing with GitList by Klaus Silveira. GitList is a web front end for browsing git repos, that looks a little like github. Klaus describes GitList as:

GitList is an elegant and modern web interface for interacting with multiple git repositories. It allows you to browse repositories using your favorite browser, viewing files under different revisions, commit history, diffs. It also generates RSS feeds for each repository, allowing you to stay up-to-date with the latest changes anytime, anywhere. GitList was written in PHP, on top of the Silex microframework and powered by the Twig template engine. This means that GitList is easy to install and easy to customize. Also, the GitList gorgeous interface was made possible due to Bootstrap.

It is nice having an open source side project to contribute to that gets me out of the Drupal ghetto. GitList also gives me an excuse to play with Symfony components and twig more, both things that are already in Drupal 8 core. Silex has been on my "to play with list" for some time too.

I have been searching for an alternative to Gitweb that is easy to use, has a nice UI and the basic features I want in a web front end for git - commit history, diffs, blame and branch comparison. GitList ticked all the boxes, except blame was a bit broken and branch comparison wasn't implemented.

I now run GitList on my laptop for offline repo browsing and find it very useful. I will be deploying it for a client in the next few weeks as part of a broader Drupal workflow management system.

This isn't a review of GitList and I won't be providing any feature comparison of GitList vs (Gitweb|GitLab|github). Instead this post is inspired by a discussion on twitter between Klaus (@klaussilveira), Fabien Potencier (@fabpot), Peter Droogmans (@attiks) and myself (@skwashd). I want outline how we are using GitList and the underlying git library, my vision for GitList and what I can do to make it happen.

Missing Features

Based on the project description I think that GitList is almost feature complete. There are a few TODO items in the README. In my time using GitList I haven't found myself going "if only it did X (and Y and Z)" - except branch diff support. I have been working on a branch diff patch, and I hope to clean up that code this weekend and submit a pull request. I think it is good when something does one thing and does it well.

Library and App Separation

As part of the workflow management system for Drupal we're developing we needed a solid git library for PHP. The one that ships with GitList is pretty nice. I've been working with my fellow Technocrat developer, James Harvey, on extracting the git library from GitList. We still have some work to do on this, but it is usable today. We have added some enhancements to the library for our purposes and removed the Silex dependency. Our vision is to have a generic OO git library for PHP 5.3+ and that GitList use this as a sub module.

Although I am currently hosting the code in my github repo, I would love to see Klaus create a new primary repo for the library and that James, myself and others would continue to develop and support his library. It is not my intention to fork the library from GitList and maintain it myself in isolation - that is a waste of energy and resources.

Git Module for Drupal

Following on from our creation of a generic git library for PHP, we have created a git wrapper module for Drupal. We plan to release this module in the next week or so on drupal.org - we've already reserved the project namespace. As part of this change the features git module will now depend on the new library wrapper module. We will release other modules that use this library as they pass QA.

Releases

I have a hacker mentality, I am happy to clone a repo, create a branch or tag and start playing. If I find something that it broken, then there is a good chance that I'll fix it and post a patch. At the same time I understand that not everyone can work that way and that in some environments there is a focus on only using official stable releases. It is also difficult tracking bugs against git commit hashes. GitList is moving quickly, but I think that it could benefit from having official releases.

If the library and app are separated, they can have separate and independent release cycles. I am happy to work with Klaus to work out a plan for GitList releases. One of my first roles in an open source project was as a release manager.

Discussion Space

I have posted this on my blog because I don't know what the real audience is for this conversation. I know that there are 3 people interested in participating in the conversation but are there others? I know others contribute patches to GitList. I could have emailed Klaus directly, but that would have excluded others who are interested. Even though in some ways email is on the way out, some discussions require more than 140 characters. Is it worth setting up a mailing list, google group, a forum or some other channel of communication to discuss these issues?

The Future

I am already dependent on GitList for a significant piece of work. I want to work with Klaus and the rest of the GitList community to make a kick arse web front end for git, I also need a rock solid PHP lib for git. I think GitLib provides a solid foundation for both. Fabien wants to use it as a good Silex example. What do others want to do with GitList? Let the discussion begin ...

Upadte: Klaus has asked that we move the discussion to the recently created GitList mailing list/google group, so I have created a new thread to continue this conversation. I can cross an item off my list - discussion space.

Bookmark/Search this post with

May 20 2012
May 20

From: Dave Hall
To: boxes <[email protected]>
Subject: Our Relationship

Dear boxes,

I'm sorry but things just aren't working out between us. It's not you, it's me. I need some time to myself. I need to think things through. I'm not sure what I want. We should spend some time apart. We should try new things. I will miss you, but this is for the best. Let's meet for coffee in a couple of weeks.

Love

Dave
xox

Breaking up is never easy, but recently I broke up with the boxes module. I'd been with boxes for a long time, we'd done a lot of good things together. Over time I'd become oblivious to some of boxes' flaws, such as giving black eyes to sys admins who run features reverts on sites where users can edit boxes. Life was pretty good, but my life still felt somewhat empty and incomplete.

One day I was standing in a crowded room with lots of modules on drupal.org and across the room I spotted BEAN. I feel in love instantly. BEAN was decked out in all the latest Drupal 7 gear - fields, Entity API and CTools. The way BEAN moved make me weak at the knees. After another beer I got my courage up, crossed the room and asked if BEAN would like to come back to my site. I was shocked, BEAN wanted to hang out with me and it wasn't just a one night fling. I've been going steady with BEAN for a couple of months now.

Just before BEAN moved in permanently to my dev environment I had to find a way of removing all the remnants of boxes from my life (aka existing sites). It took me less than an hour. I created a new BEAN type and called it "box", I then wrote a little "drush scr" script which converted all of my boxes to BEANs and even updated my contexts to use the newly created BEANs. I hope neither bean or boxes finds my work on github. I chose github hoping neither of them would find my work and also because I don't want to maintain this as a module on d.o.

On a serious note - BEAN is awesome! If you haven't tried it, you really should. BEAN treats blocks as content, not config which is really useful for most sites. For new sites BEAN is a drop in replacement for core blocks or boxes making it super easy to get started. To learn more check out the docs.

If you're interested in seeing the full power of the BEAN module, and other cool stuff I've been working on recently, please comment on my DrupalCon session proposal - An Enterprise Scale Drupal Workflow. It will be a more technical version of my session at Drupal Business Days a couple weeks ago.

* "Full of beans" is phrase commonly used in Australia to describe energetic children.

Bookmark/Search this post with

Nov 11 2011
Nov 11

TL; DR: [spam]Please vote for my DrupalCon Denver proposal on Drupal workflows in the enterprise.[/spam]

For the last few months I've been working for Technocrat on a new Drupal based site for the Insurance Australia Group's Direct Insurance brands. The current sites are using Autonomy Teamsite.

The basics of the build are relatively straight forward, around 1000 nodes, a bunch of views and a bit of glue to hold it all together. Where things get complicated is the workflow. The Financial services sector in Australia is subject to strict control of representations being made about products. The workflow system needs to ensure IAG complies with these requirements.

During the evaluation we found that generally Drupal workflows are based around publishing a single piece of content on the production site. In the IAG case a collection of nodes need to be published as a piece of work, along with a new block. These changes need to be reviewed by stakeholders and then deployed. This led us to build a job based workflow system.

We are using the Features module to handle all configuration, deploy for entities and some additional tools, including Symfony, Jenkins and drush to hold it all together.

I've proposed the session for Drupal Downunder in January and will refine the session based on feedback from there in preparation for Denver. If you want to learn more about Drupal Workflows in the Enterprise, please vote for my session.

Bookmark/Search this post with

May 16 2011
May 16

Drush make is a wonderful tool for constructing Drupal platforms. A lot of Drupal developers are used to adding a list of modules, a few libraries and theme or 2 then running drush make to build their platform. It all seems pretty easy. What if I told you module developers could make things even easier for site builders?

Some contrib modules depend on third party libraries, and due to various reasons they can't always be stored in git repositories on drupal.org and included in the module release. To solve this problem module developers can include a .make file for their module. Drush recursively processes make files, so the module make file would be processed once found by drush make.

A good example of where this could be useful is the SMTP module, which depends on the LGPL licensed PHPMailer library. The module also requires a patch to be applied to the library, which drush make can apply for us. The following .make file could be included in the SMTP module as smtp.make:

core = 6.x
api = 2

libraries[phpmailer][download][type] = "get"
libraries[phpmailer][download][url] = "http://downloads.sourceforge.net/project/phpmailer/phpmailer%20for%20php5_6/Previous%20Versions/2.2.1/phpMailer_v2.2.1_.tar.gz"
libraries[phpmailer][download][md5] = "0bf75c1bcef8bde6adbebcdc69f1a02d"
libraries[phpmailer][directory_name] = "phpmailer"
libraries[phpmailer][destination] = "modules/contrib/smtp"

libraries[phpmailer][patch][drupal-compatibility][url] = "http://drupalcode.org/project/smtp.git/blob_plain/2acaba97adcad7304c22624ceeb009d358b596e3:/class.phpmailer.php.2.2.1.patch"
libraries[phpmailer][patch][drupal-compatibility][md5] = "2d82de03b1a4b60f3b69cc20fae61b76"

Now when the SMTP module is included a normal drush make file it will be downloaded, the PHPMailer library will be downloaded and patched ready for use.

Unfortunately there are some limitations to this approach. Firstly it assumes that the SMTP module will be installed under the modules/contrib directory, which is accepted best practice, but may not suit everyone's needs. When I tested this with the current stable version of drush make (6.x-2.2) it failed, and drush make 6.x-3.x from git needed to be patched. Hopefully a fix for this can be backported to the 6.x-2.x branch and included in a future release.

Update: I have posted the make file for the SMTP module as patch in issue #1159080.

Feb 05 2011
Feb 05

I really want to attend DrupalCon Chicago, which kicks off in just over 4 weeks. The problem is that since DrupalCon Copenhagen business has been pretty quiet and so I find that I can't really afford to fund it myself. After deciding I had to be in Chicago I got creative about how to make it happen. The buy a line project was born.

Instead of just asking people to kick in some cash to get me to Chicago, I felt it was only right to earn my keep. People can buy a line of code, or sentence of documentation for Drupal. All code and docs created will be contributed to drupal.org. Buyers are free to specify where the lines are to be contributed, or leave me to decide. I'm looking forward to writing some of the lines on the Drupal Bus.

Recently I have been working on porting the UUID module to Drupal 7. I hope to get this module into Drupal 8 core. To make this happen I have to be in Chicago! Improvements to UUID will mean that content can be packaged up and moved around like configuration can be using the Features module.

All buy a line issues will be tagged so people can watch my progress. The first lines of code have been contributed to the getID3() module, so Drupal Commons can be installed by Aegir.

Thanks to everyone who has contributed so far. I have almost covered the DrupalCon ticket I bought from the Gizra team.

Please consider buying a line (or more) to help get this Drupal geek to Chicago. This is a great way of getting a module ported to Drupal 7, better documentation or even just a bug fixed. I have a decent track record of contributing to the project.

Jan 26 2011
Jan 26

I spent the weekend at Drupal Downunder in Brisbane. The venue was excellent. I'm a fan of not using "traditional" venues for conferences, to help make them even more memorable for attendees.

I managed to catch up with a bunch of people. The relaxed feel about the event was great. Most conferences I've attended recently have either been large or I've helped organise them, this time I could relax and enjoy.

On the Saturday I presented Building Distributions with Drupal 7, which had a small turn out as I was up against Josh Koenig and his Pantheon presentation. My presentation was hampered by lack of internet connectivity, but I think it went well. I used Lego, Duplo and Quatro blocks to demonstrate the evolution of Drupal distros.

Saturday night involved a pub crawl with various DDU and LCA folks. The highlight of the crawl was the Mana Bar, which is a gamers bar, that has a good collection of retro consoles and games on display.

I spent a fair bit of Sunday in the hallway track. I discussed the D7 port of UUID with Dries, which helped confirm the direction I was heading with it. Several people wanted to discuss my $100 Drupal site blog series. I also gave my Horizontally Scaling Drupal presentation, which was very well attended. Unfortunately due to people torrenting there was no usable internet access for my presentation. I had to skip the post event BBQ so I could fly back to Melbourne.

The lack of mobile signal and wifi made it frustrating to prepare and present. I would have liked to have seen an inclusive social event organised on the Saturday night. Overall I really enjoyed DDU and the organisers are to be congratulated. The vegetarian food options were excellent.

Thanks for Four Kitchens for funding me to get to DDU. I have just started contracting with them, so I really appreciated them covering my trip.

Jan 04 2011
Jan 04

When doing development work, from time to time it is handy to be able to look up documentation. Bookmarking manuals is handy, but often you still need to search for the function you're after. Firefox, and possibly other browsers (not Chrome or Chromium), allows you to setup a keyword bookmark linked to a search.

I've setup a few search bookmarks for development resources. This is how I've done it:

  1. Select Bookmarks > Organise Bookmarks... from the menu.
  2. Right click on the bookmarks menu (or any folder) on the left pane
  3. Select New Bookmark... from the context menu
  4. Drupal bookmark example
    Fill in the information for the bookmark, the import piece is the keyword, that will allow us to search.
  5. Click save and return to the browser

Now when we want to search the Drupal 7 API, we can just type "dapi ",>

Example Drupal API search in location bar

Now we should see the appropriate page from the Drupal API documentation.

Example Drupal API page

The same method can be used for other PHP web app developer resources, here are some which I'm using.

  • I've found Google to be the best resource for getting help with javascript

I could have implemented this using OpenSearch plugins, but that requires changing the search provider everytime I want to look for something. By using keyword bookmarks I just type the keyword and the search term into the location bar.

Feel free to share your keyword bookmarks in the comments.

Jan 02 2011
Jan 02

Thanks to everyone who read the posts in my $100 Drupal site series. Today I will be responding to some of the points people have raised in comments and via email as well as adding a few closing comments.

Undervaluing Labour

Some have suggested that the only way to make a venture like this work is to work for a few dollars an hour. I completely disagree with that! If you are building every project from scratch and only charging your clients 100USD, you will be working for peanuts, but I'm not advocating that model. I'm encouraging people to rethink how they manage their development workflow. Although there are initial costs in learning new tools, over time you will end up spending less time on site builds and so increase your profits.

$100 is too Cheap

I agree that 100USD is very cheap for a high quality Drupal site and in most western markets it is probably too cheap. Sure you can do it and make the profit on the margin, but you have to sell a lot of sites to recover your initial investment. The "$100 Website" can be a useful sales pitch. Don't charge 100USD per site if the market will happily pay 5 or 10 times that. It doesn't matter if you are charging 10USD per site or 250kUSD per site, the same basic rules apply - understand your client's requirements and know the market.

There is a growing list of freemium services out there too, including Drupal Gardens and WordPress.com. Depending on your business model you may want to have a basic freemium offering.

What Does the Competition Offer?

Investigate what the competition is offering. There is a large market for hosted CMSes these days, go see what some of them are doing. What do you think they are doing right? What do you think they lack or could do better? A quick list of services I'd recommend you check out are (in alphabetical order):

I'd recommend spending some time searching for hosted CMS solutions. Sign up for the free or trial services and see what they're like.

Do I Need a lot of Money to do this?

Although in the Resources and Infrastructure post I outlined the positions I thought your team needed to make this happen, it wasn't a list of must haves. You need to figure out what works with you. It would be possible for a couple of switched on people to do this as a basement startup, so long as you have the skills and connections to get everything you need in place. If you can only drive a server using cPanel, then you need to find a sysadmin. If you can't even draw stick figures, then you will need to hire a designer. Your staffing costs are really going to dictate how much money you will need to get this up off the ground.

Why did you do this?

I wanted to put the idea out there. I wanted to see how people would react. It wasn't that long ago that people could build a profitable business selling websites built using Dreamweaver, and some people still do, but the market is changing, so too is the Drupal market.

For some time I have been thinking about how you'd develop a low cost Drupal based hosted CMS solution. There are 2 main reasons why I haven't just gone and built it already. The primary reason is I hate being on call and running a hosting service such as this requires me to never be out of range of a 3G tower for more than 30 minutes, customers won't accept that the server crashed 15 minutes after I hoped on a flight to the US or Europe. Secondly building such a solution takes time and I have a family to feed, I don't think my landlord would cop me not paying the rent until this thing started to turn a buck.

I have very consciously put enough information into this series to allow someone to head off in the right direction. At the same time I've held back on some of my execution specifics, because if I was to go build something like this I want to have some things which make my service unique.

I was hoping to present a session at DrupalCon Chicago on this topic, but I found out yesterday that my proposal hasn't been accepted. As I understand it, there will be a small number of sessions announced sometime in the next 2 weeks, but I'm not holding out much hope of being selected then. There is always London.

Can you Build this for me? / I need Help!

If you are serious about building a service like this or using some of the tools I've discussed, I'd be happy to talk to you. I have about a decade's worth of experience running my own business and developing web apps. I have offered training and consulting around many of the topics I have discussed in this series.

So neither of us waste our time, I will be up front about how I see things working on my end. I expect to be paid for my time, I'm happy to take some equity, but I've got bills to pay. I have put a lot of thought and energy into this, so I am very unlikely to sign a contract which gives you ownership of my IP. If you can live with the above, then please contact me.

It didn't Work for me

Sorry that just the way things go in business sometimes. Not everything I've tried over the years has worked either. Learn from the failure and try again, maybe doing something completely different.

Disclaimer: Don't sue me if this doesn't work out, do your own research and seek professional advice before acting on anything on this site.

Will you Write More on these Topics?

Maybe. I will probably turn some of the content from Horizontally Scaling Drupal workshops and talks into blog posts, and possibly a book. I tend to blog when I feel passionate about something. Subscribe to my RSS feed, or follow me on twitter to find out what I'm working on.

In the mean time, I'd recommend you subscribe to the following blogs / follow these people on twitter:

I welcome comments, feel free to leave one below.

Dec 29 2010
Dec 29

During this series on creating a profitable business around the concept of building Drupal sites for $100 I have attempted to demonstrate that there is a viable business model here. I don't believe it is a business that will suit everyone and nor do I believe every developer will want to work on such a project, but for some this will be an excellent opportunity. Today I will cover some of the things that I think you should consider before investing too much in this business model. Some of this is just basic business sense which isn't specific to this project.

Dollars and Sense

Oscar Wilde once wrote "What is a cynic? A man who knows the price of everything and the value of nothing". If you want to build this type of business, be frugal, but not a Scrooge. Generally a good developer will cost you more in salary than an average one, but the good developer will work out cheaper in the longer term. A similar thing goes for hosting, avoid Virtuozzo based VPS and look for someone offering the more expensive Xen or KVM based platforms, they are less likely to oversell capacity. At the same time keep an eye on the markets you are buying services in to ensure you are always getting the best value for money.

Recognise when you have to pay for something. Spending a little now might save you a lot in the long run.

In House vs Outsourcing

It makes sense to outsource some services, for example email. You want to focus on offering a high quality Drupal driven SaaS, having to support email too is a lot of hassle, use Google Apps for Domains or similar service.

Some things are better performed in house, such as design and themeing. There are businesses around who offer cheap PSD to Drupal theme conversion services, but these services usually cut corners which will cost you more in the longer term. Having someone in house means that they can learn from others on your team and do things the right way for your business.

Pricing

Although the theme of this series is "$100 Drupal site", that doesn't mean every site you sell should be $100. You need to research your target market/s and make sure you are pricing your product/s appropriately. I would seriously consider offering 3 levels of service, each level should offer more options, bandwidth and storage. Some customers will be drawn by the initial $100 price tag, but when they see the feature list, the $250 product might be more appealing. Give customers a sense of choice.

Options and Up Selling

Find opportunities to sell extras to your clients. Not everything needs to be included in the base level package. I discussed enhanced support offering in my previous post, which could provide additional revenue.

Most of the clients who sign up for this service will have no interest in paying for a custom theme and/or development, but once you have them through the door you have more chance of selling such extras to them. Find the higher traffic clients and connect with them, see if they want more. The price jump is likely to be significant, but if they can see value in it, they may pay for it.

Domain Names

Domain names is a great opportunity to generate extra revenue. By default customers should get biz-name.example.net, and charge them a fee if they want to use another domain, such as mybiz.com. Most people know that GoDaddy sells domains for around 10USD, while Melbourne IT charge around 75USD. Sign up for a reseller account with one of various domain registrars and offer customers domains directly from your service. If a customer brings their own domain charge them a fee, if they buy the domain through you advertise that you waive the setup fee, but then include it in the pricing of the domain name.

Partners

Seek out partners. If you can find a company which offers a service which is a good fit with your business, work out a deal with them. Be it bulk licensing, reseller, commissions, find something that works for both parties. Your customers can benefit from such arrangements too.

Giving Back

Previously I mentioned that you will be benefiting from the great work of the Drupal community, you should acknowledge this benefit by contributing back to the community where it is appropriate. Not only does this help build the profile of your business in the community, it can reduce your maintenance workload. For example if you have a patch which fixes a bug or adds generically useful functionality to a module, push it back up stream, so you don't have to maintain the patch. Consider having a budget for sponsoring community events to help build good will.

Data Exports

Allow customers to export their site and go host it with someone else. This might sound counter intuitive, but if they are just a base package customer and you are hosting them, they have paid you. If they are only 2 months into 12 month service, which should be non refundable, then let them take their site, that frees up server resources. It is also the ethical thing to do, don't make your money because people are locked into your service, they will eventually resent you for it.

Licensing

Drupal itself is licensed under the terms of the GPLv2 (or later) with some contrib modules depending on GPL compatible code. If you are contributing code back to drupal.org it must be under the terms of the GPLv2. You could choose to offer the non PHP parts of your code under CC-BY with an attribution in the footer, you could release your features under the terms of the AGPLv3. These are business decisions you must make in the context of the rest of your business.

Pay for Expert Advice

There are times when you need an expert. Get a lawyer to check your Terms of Service and any significant contracts. Use an accountant to get your financial affairs right from the start. If your servers are performing poorly and your Sys Admin can't work it out, pay for someone who can. Sometimes it is better to pay for something than trying to learn how to do it yourself.

Share

Blog about your experiences, share the knowledge. I'm not advocating putting your entire business plan online, but discuss problems you've solved, code you've created so others can benefit from your experiences. Don't be afraid to share, Scott Adams sums this up perfectly "Ideas are worthless. Execution is everything". I have nothing to fear in publishing this blog series, after all it is just my ideas.

What's Next?

This is the last substantial post in this series. I have tried to break it up into digestible chunks. Thanks for taking the time to read my posts and I welcome any comments you may have.

I plan to let things sit for a few days. Over the weekend I plan to review the comments posted and emails received as part of preparing a summary response and conclusion. Anything else I think of will get tacked onto that post.

Dec 28 2010
Dec 28

Through out this series, the cost of labour has been identified as one of the biggest risks for this project. As most people who have run a tech business know, support can turn into a massive black hole of wasted time. Today we will look at how to manage support in a way that helps you avoid any direct customer contact for support.

Documentation and Online Resources

People like documentation, or even better videos, to walk them through a process. Invest the time up front to create documentation to help your customers use your platform. Only offer a single admin theme, such as Root Candy or Rubik or something you create yourself, this way the UI remains consistent and users are less likely to get confused. When developing the documentation, assume the user has no Drupal experience and talk in generic lay person terms, not "Drupal babble". You should include links in the Drupal interface back to your help system, so people can access contextually appropriate help. Consider having a FAQ for each discrete piece of functionality so people can get some idea of what it does without having to play with it for half an hour.

A lot of the documentation you generate will be specific to your business and only available to your paying customers, but some of it will have generic value to the Drupal community. For the more generally useful content, publish it in a public place, such as your blog, company website or contribute it to the appropriate section of the documentation on drupal.org. You will be benefiting greatly from the work of the Drupal community, you should be looking for opportunities to give something back.

The flexibility of Drupal means that you should be able to build a documentation system that suits your needs. You probably want to start by extending the book module.

Forum

From time to time people won't be able to find what they need using your online documentation. If you have a solid user base, a forum can be a good way of crowd sourcing your support resources. This doesn't mean you can just put up a forum and hope that your users will all help each other solve their problems, you need to be in there responding in a timely manner too. If you have some users who are contributing regularly in the forums, acknowledge that in their profile and even offer them some free time on the service - after all they're saving you money.

Support Tickets

Some users will want a more personalised approach to support. This will most likely involve email. When a user emails support directly, they will expect a timely response. This has the potential to add significant costs to your business, especially if you have a global customer base. One way to deal with this additional cost is to charge for it, offer packages of X incidents per month for Y dollars and put them on recurring billing. Most customers will forget to remove the recurring billing, but stop asking questions once they are up and running. There are various support systems available, I'll leave you to find the one which works best for you.

Chat and Telephone Support

Real time support channels such as chat or telephone can really eat into your bottom line. You need to have people sitting there waiting to take the "call", whatever time the customer contacts you and they may not call for a day. Accents and cultural issues can make it difficult to communicate effectively via the telephone. I would avoid offering any telephone support to customers.

Mailing Lists

A mailing list can be an effective way of communicating with your users. I'm not suggesting something like MailMan or Google Groups, I am talking about a one way announcement list. Check out MailChimp or Campaign Monitor to take care of this for you. Each month send out a newsletter with some useful tips, new features or announce other improvements to the service. Consider including case studies in the newsletter, which you can also feature on your sales site.

Customer Service

The cheapest way to build a business is through referrals or word of mouth. You need to look after your customers, so they become your sales team. Manage their expectations in terms of the support you offer them. Don't be afraid to cut a time sink loose, apologise, give them a refund and encourage them to find another solution. Support vampires can cost you a lot more than they're worth.

What's Next?

We are almost in the home stretch now. In the next post I will cover some of the business considerations in terms of what you offer to customer and how to upsell. The final post in the series will come later this week and will be a summary of the previous posts.

Dec 27 2010
Dec 27

So far in this series we have covered a potential target market and business plan, resources and infrastructure and the tools required to deliver Drupal sites with a sale price of $100 per site. In this post I'll be covering some of the considerations when building Drupal platforms or distributions.

The sites which customers deploy will need to be based on a custom Drupal distribution or "distro". The distro should be modular and primarily driven by Features.

Customers shouldn't have to know anything about administering Drupal when they first buy their site. A customer should be able to turn functionality on and off as they want, through a simple user interface.

Features

The platform should contain a good collection of Features. The following list is an example of what you might offer customers:

  • Contact Form
  • Image Gallery
  • Products
  • Services
  • "Static" Pages
  • Blog
  • News
  • Mailing Lists
  • Social Network Integration
  • Office / Store Locations
  • Staff Profiles

When developing your list of things to include in the site, think in terms of functionality a small business would want, not what modules you should be using. The list of modules should be derived from the functionality, not the other way around.

As the features included in the platform will be modular and generically useful, you should consider releasing them publicly, via your own features server or drupal.org as full modules.

On top of the features listed above you will probably need to include some custom glue code to enhance the user experience. In my first post in this series I discussed the target audience not having high level computer skills, so the user interface should take this into account. Some of the language might need to be changed or form options modified to use sane defaults and some might even be hidden from the user.

Security

As each server may have hundreds or even thousands of sites running on it, security will be an important consideration. Like with all servers you should ensure it is properly locked down and only running the services you need. Apache should be configured to block access to most things except index.php and relevant client side files (images, css, js) and the files directory. At the Drupal level you should make sure that things like the PHP module aren't enabled and secure coding practices are adhered to. The user account given to your customer shouldn't be user 1, they should be user 2, with restricted permissions that only gives them access to what they need.

I strongly recommend that you read Cracking Drupal by Greg Knaddison.

Sales and Support

In order to attract customers you will need a site to promote the service and allow customers to sign up and hand over their credit card details. Drupal now offers 3 ecommerce projects, Drupal e-Commerce, Drupal Commerce and ubercart, you should investigate which of these best suits your needs. The sales system will need some custom code to hook into Aegir, which will be managing the actual site deployments. The sales and support platform/s should be managed in a similar manner to the customer sites.

Once you have paying customers, you will also need to provide them with some resources such as detailed documentation, video walk throughs, forums and possibly a ticketing system. This site can either be part of the sales site or a separate site. In the next instalment I'll cover support in more detail.

Deploying Platforms

We need to keep the whole process very automated, CPU cycles are a lot cheaper than workers. Building and deploying platforms should involve a few clicks or tweaking a configuration file. For example platforms could be built as Debian (or Ubuntu) packages (aka debs) using an automated build process that runs a test suite against the code base before building a deb. The debs could then be deployed using puppet and a post installation script can notify Aegir that it has been installed successfully. The whole process could involve very little human interaction. Migrating client sites to upgraded platforms could also be automated using a simple script which adds a migrate task for each site.

What's Next?

Now that we have the service almost ready to go, we should look into how we are going to get customers to part with their cash and how we will support them once they have paid.

Dec 26 2010
Dec 26

In the previous instalment of my $100 Drupal site series I covered resources and infrastructure. In this post I will be covering the development tools I think you need in order to build and sell Drupal sites at the $100 price point. Given that Drupal 7 is close to release, it is assumed that the sites will be built using D7. I don't believe that it is smart to invest heavily in Drupal 6 for new long term projects, given D8 could be out in 18 months and D6 would then be unsupported.

git

You are going to need a version control system for storing all of the code. There are many different version control systems available, but I think git is the most flexible and powerful option. Git allows for distributed development, offline commits and best of all, Drupal is switching to git. Gitorious is an open source clone of github which allows you to browse your git repository and manage integration of code from all of your developers.

If you are new to git I strongly recommend you get a dead tree copy of the book Pro Git.

drush

Drush is the DRUpal SHell, a command line interface for Drupal. It is an invaluable tool for developing and maintaining Drupal sites. Simple things like running "drush cc all" to clear all caches when developing sites, through to being able to sync databases from one server to another, or one continent to another, save many hours. Even if you're only running a handful of sites, you should have drush installed on all of your servers.

Modules

There are a few modules which you should be using if you want to be able to create polished Drupal based platforms. My list includes:

  • module allows developers to package up configuration as a Drupal module, including access to all the normal hooks that a module can implement. Feature is version control friendly and includes the ability to reset to a known good state, either via the Drupal web GUI or by using drush on the command line.

  • exports values from Drupal's variables table, allowing sane defaults to be set. This means your site will always be configured the way you want it to be.

  • allows you to manage contextual conditions and reactions for different portions of your site. It gives you control over displaying blocks,hierarchy of breadcrumbs, themes and other context sensitive configuration options.

  • is a menu module which makes it quick and easy to access items. (Disclaimer: I co-maintain the module)

  • can perform static analysis of your code to ensure it is secure and complies with the Drupal coding standards. The module can also be used to upgrade code from one major version of Drupal to the next.

  • is a collection of tools to help developers create Drupal sites and modules. Devel integrates with Admin to provide easy access to the devel actions.

  • provides a pluggable framework for integrating rich text editors into a Drupal site. Given the target market for the $100 sites, we can't expect our users to hand craft HTML, they'll be expecting "something like Word".

  • provides a way for end users to apply predefined CSS to designated parts of a site, without the user having to understand CSS.

Installation Profiles

When most people install Drupal for the first time they are blissfully unaware that they are using an installation profile. An installation profile sets out a base configuration for a site that is installed by Drupal. Until recently installation profiles contained a list of modules to install and a large chunk of hand crafted PHP code which setup all the configuration for a site. These days most of the configuration can live in Features and so an installation profile can be very lightweight. It is worth reviewing some of the more popular installation profiles, such as Open Atrium, Managing News or Drupal Commons, for inspiration. Installation profiles have changed a lot in Drupal 7, so you will need to port the ideas to the new way of doing things.

Drush Make

Drush Make is a tool for building Drupal platforms. Drush Make allows developers to specify all the components of their site in a text file, then use the command line tool to "make" the site. It is possible to specify a version of Drupal core (including Pressflow), contrib modules and themes, third party components, patches and external libraries.

Aegir

Aegir is a Drupal site deployment and management tool built using a collection of Drupal modules. With Aeigr you can manage your DNS, http (web) and database servers from a common UI. It is possible to move a bunch of sites from one server to another in a matter of minutes, not hours and the downtime will be measured in seconds. When security fixes are released, testing can involve a few minutes of work then a few more clicks to deploy it to all of your sites.

Workflow

Instead of me explaining development workflows, I'll defer to Miguel Jacq (aka mig5). Miguel's blog post entitled "Drupal deployments & workflows with version control, drush_make, and Aegir" is considered by many to be the key work on modern Drupal development workflows.

What's Next

Many of the tools I've covered today should be in your Drupal developer's tool bag. My next post will cover what I think are the important considerations in building the platforms for the service.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web