Feeds

Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Feb 11 2021
Feb 11

Way back in August 2006 we launched a holding page for Dave Hall Consulting. Then in early 2007 we launched the real Dave Hall Consulting website on Drupal 4.7. The site and logo were designed by imgageffect.

Since that time little changed with the website. The site evolved a little over time. There were quick and dirty upgrades to Drupal 5 and 6. In 2019 we stripped the design back to make it more mobile friendly.

Over the last 5 years there have been multiple plans to fix the site. They all failed, most due to lack of time and/or focus. In September 2020 we decided it was time to fix the site. Instead of doing it in house, we leaned on our years of experience managing remote and out sourced teams. We searched for the best people to deliver our new site.

After an extensive search we settled on One Week Wonders Studio. We couldn’t be happier with the results. We worked closely with the OWW team throughout the process. The communication was excellent. They understood the image we wanted to project and they provided a logo and design that fitted the brief. We have no reservations about using them again.

We prepared a brief for the logo including the general concept. The new logo incorporates the business name, while being a valid domain. We retained the orange and green from the original logo but went with stronger shades. The “/>” on the end references HTML and a command prompt.

The new website design is fresh and modern. With more white space and clean illustrations we hope the new site is more engaging. It now works better on all devices. It’s a shame that most people will never see our new 404 page which we love.

OWW didn’t have experience building Hugo sites. That wasn’t a problem. We engaged Ahmad Muhardian to take care of that for us. He did a great job delivering to our high standards. If you need someone to turn HTML into a Hugo theme, Ahmad is your person. He also does some awesome manga art.

We consider this iteration our minimum lovable product. Comments will be back in a future version of the site. More content is in the pipeline too. Hugo makes it easier for us to focus on the content, rather than the CMS and the underlying stack. This makes it easier for us to iterate and push out new content.

When rebuilding the site we wanted to adopt a principle of respecting users privacy. We use Netlify Analytics so your visits here aren’t added to your aggregated Google data profile. We host the fonts for the same reason. Youtube and most other embeds are links, rather than rich content. These privacy respecting features allowed us to implement a simple Content Security Policy.

Once again the builder no longer has the worst looking house in the street.

Nov 16 2019
Nov 16

A few weeks ago we announced our diversity scholarship for DrupalSouth. Before announcing the winner I want to talk a bit about our experience doing this for the first time.

DrupalSouth is the largest Drupal event held in Oceania every year. It provides a great marketing opportunity for businesses wanting to promote their products and services to the Drupal community. Dave Hall Consulting planned to sponsor DrupalSouth to promote our new training business - Getting It Live training. By the time we got organised all of the (affordable) sponsorship opportunities had gone. After considering various opportunities around the event we felt the best way of investing a similar amount of money and giving something back to the community was through a diversity scholarship

The community provided positive feedback about the initiative. However despite the enthusiasm and working our networks to get a range of applicants, we only ended up with 7 applicants. They were all guys. One applicant was from Australia, the rest were from overseas. About half the applicants dropped out when contacted to confirm that they could cover their own travel and visa expenses.

We are likely to offer other scholarships in the future. We will start earlier and explore other channels for promoting the program.

The scholarship has been awarded to Yogesh Ingale, from Mumbai, India. Over the last 3 years Yogesh has been employed by Tata Consultancy Services’ digital operations team as a DevOps Engineer. During this time he has worked with Drupal, Cloud Computing, Python and Web Technologies. Yogesh is interested in automating processes. When he’s not working, Yogesh likes to travel, automate things and write blog posts. Disclaimer: I know Yogesh through my work with one of my clients. Some times the Drupal community feels pretty small.

Congratulations Yogesh! I am looking forward to seeing you in Hobart.

If you want to meet Yogesh before DrupalSouth, we still have some seats available for our Gettin' Git 2 day git training course that’s running on 25-26 November. If you won’t be in Hobart, contact us to discuss your training needs.

Photo by Ivan Grynenko.

Oct 30 2019
Oct 30

For DrupalCon Amsterdam, Srijan ran a competition with the prize being an Apple Watch 5. It was a fun idea. Try to get a screenshot of an animated GIF slot machine showing 3 matching logos and tweet it.

Try your luck at @DrupalConEur Catch 3 in a row and win an#AppleWatchSeries5. To participate, get 3 of the same logos in aseries, grab a screenshot and share it with us in the comment sectionbelow. See you in Amsterdam\! \#SrijanJackpot \#ContestAlert \#DrupalCon

I entered the competition.

I managed to score 3 of the no logo logos. That’s gotta be worthsomething, right?#srijanJackpot

The competition had a flaw. The winner was selected based on likes.

After a week I realised that I wasn’t going to win. Others were able to garner more likes than I could. Then my hacker mindset kicked in.

I thought I’d find how much 100 likes would cost. A quick search revealed likes costs pennies a piece. At this point I decided that instead of buying an easy win, I’d buy a ridiculous number of likes. 500 likes only cost 7USD. Having a blog post about gaming the system was a good enough prize for me.

Receipt: 500 likes for7USD

I was unsure how things would go. I was supposed to get my 500 likes across 10 days. For the first 12 hours I got nothing. I thought I’d lost my money on a scam. Then the trickle of likes started. Every hour I’d get a 2-3 likes, mostly from Eastern Europe. Every so often I’d get a retweet or a bonus like on a follow up comment. All up I got over 600 fake likes. Great value for money.

Today Sirjan awarded me the watch. I waited until after they’ve finished taking photos before coming clean. Pics or it didn’t happen and all that. They insisted that I still won the competition without the bought likes.

The prize being handedover

Think very carefully before launching a competition that involves social media engagement. There’s a whole fake engagement economy.

Oct 04 2019
Oct 04

Over the years I have benefited greatly from the generosity of the Drupal Community. In 2011 people sponsored me to write lines of code to get me to DrupalCon Chicago.

Today Dave Hall Consulting is a very successful small business. We have contributed code, time and content to Drupal. It is time for us to give back in more concrete terms.

We want to help someone from an under represented group take their career to the next level. This year we will provide a Diversity Scholarship for one person to attend DrupalSouth, our 2 day Gettin’ Git training course and 5 nights at the conference hotel. This will allow this person to attend the premier Drupal event in the region while also learning everything there is to know about git.

To apply for the scholarship, fill out the form by 23:59 AEST 19 October 2019 to be considered. (Extended from 12 October)

The winner has been announced.

Sep 24 2017
Sep 24

Over the years Drupal distributions, or distros as they’re more affectionately known, have evolved a lot. We started off passing around database dumps. Eventually we moved onto using installations profiles and features to share par-baked sites.

There are some signs that distros aren’t working for people using them. Agencies often hack a distro to meet client requirements. This happens because it is often difficult to cleanly extend a distro. A content type might need extra fields or the logic in an alter hook may not be desired. This makes it difficult to maintain sites built on distros. Other times maintainers abandon their distributions. This leaves site owners with an unexpected maintenance burden.

We should recognise how people are using distros and try to cater to them better. My observations suggest there are 2 types of Drupal distributions; starter kits and targeted products.

Targeted products are easier to deal with. Increasingly monetising targeted distro products is done through a SaaS offering. The revenue can funds the ongoing development of the product. This can help ensure the project remains sustainable. There are signs that this is a viable way of building Drupal 8 based products. We should be encouraging companies to embrace a strategy built around open SaaS. Open Social is a great example of this approach. Releasing the distros demonstrates a commitment to the business model. Often the secret sauce isn’t in the code, it is the team and services built around the product.

Many Drupal 7 based distros struggled to articulate their use case. It was difficult to know if they were a product, a demo or a community project that you extend. Open Atrium and Commerce Kickstart are examples of distros with an identity crisis. We need to reconceptualise most distros as “starter kits” or as I like to call them “puppies”.

Why puppies? Once you take a puppy home it becomes your responsibility. Starter kits should be the same. You should never assume that a starter kit will offer an upgrade path from one release to the next. When you install a starter kit you are responsible for updating the modules yourself. You need to keep track of security releases. If your puppy leaves a mess on the carpet, no one else will clean it up.

Sites build on top of a starter kit should diverge from the original version. This shouldn’t only be an expectation, it should be encouraged. Installing a starter kit is the starting point of building a unique fork.

Project pages should clearly state that users are buying a puppy. Prospective puppy owners should know if they’re about to take home a little lap dog or one that will grow to the size of a pony that needs daily exercise. Puppy breeders (developers) should not feel compelled to do anything once releasing the puppy. That said, most users would like some documentation.

I know of several agencies and large organisations that are making use of starter kits. Let’s support people who are adopting this approach. As a community we should acknowledge that distros aren’t working. We should start working out how best to manage the transition to puppies.

Sep 16 2017
Sep 16

While preparing for my DrupalCamp Belgium keynote presentation I looked at how easy it is to get started with various CMS platforms. For my talk I used Contentful, a hosted content as a service CMS platform and contrasted that to the “Try Drupal” experience. Below is the walk through of both.

Let’s start with Contentful. I start off by visiting their website.

Screenshot: Contentful homepage

In the top right corner is a blue button encouraging me to “try for free”. I hit the link and I’m presented with a sign up form. I can even use Google or GitHub for authentication if I want.

Screenshot: Contentful sign up page

While my example site is being installed I am presented with an overview of what I can do once it is finished. It takes around 30 seconds for the site to be installed.

Screenshot: Contentful installer page

My site is installed and I’m given some guidance about what to do next. There is even an onboarding tour in the bottom right corner that is waving at me.

Screenshot: Contentful dashboard

Overall this took around a minute and required very little thought. I never once found myself thinking come on hurry up.

Now let’s see what it is like to try Drupal. I land on d.o. I see a big prominent “Try Drupal” button, so I click that.

Screenshot: Drupal.org homepage

I am presented with 3 options. I am not sure why I’m being presented options to “Build on Drupal 8 for Free” or to “Get Started Risk-Free”, I just want to try Drupal, so I go with Pantheon.

Screenshot: Try Drupal landing page

Like with Contentful I’m asked to create an account. Again I have the option of using Google for the sign up or completing a form. This form has more fields than contentful.

Screenshot: Pantheon signup page

I’ve created my account and I am expecting to be dropped into a demo Drupal site. Instead I am presented with a dashboard. The most prominent call to action is importing a site. I decide to create a new site.

Screenshot: Pantheon dashboard with no sites

I have to now think of a name for my site. This is already feeling like a lot of work just to try Drupal. If I was a busy manager I would have probably given up by this point.

Screenshot: Pantheon create site page

When I submit the form I must surely be going to see a Drupal site. No, sorry. I am given the choice of installing WordPress, yes WordPress, Drupal 8 or Drupal 7. Despite being very confused I go with Drupal 8.

Screenshot: Pantheon “Choose your upstream page”

Now my site is deploying. While this happens there is a bunch of items that update above the progress bar. They’re all a bit nerdy, but at least I know something is happening. Why is my only option to visit my dashboard again? I want to try Drupal.

Screenshot: Pantheon Drupal 8 installer

I land on the dashboard. Now I’m really confused. This all looks pretty geeky. I want to try Drupal not deal with code, connection modes and the like. If I stick around I might eventually click “Visit Development site”, which doesn’t really feel like trying Drupal.

Screenshot: Pantheon dashboard for new site

Now I’m asked to select a language. OK so Drupal supports multiple languages, that nice. Let’s select English so I can finally get to try Drupal.

Screenshot: Drupal installer language selection

Next I need to chose an installation profile. What is an installation profile? Which one is best for me?

Screenshot: Drupal installer select installation profile

Now I need to create an account. About 10 minutes ago I already created an account. Why do I need to create another one? I also named my site earlier in the process.

Screenshot: Drupal configure site form part 1

Screenshot: Drupal configure site form part 2

Finally I am dropped into a Drupal 8 site. There is nothing to guide me on what to do next.

Screenshot: New drupal site home page

I am left with a sense that setting up Contentful is super easy and Drupal is a lot of work. For most people wanting to try Drupal they would have abandoned someway through the process. I would love to see the conversion stats for the try Drupal service. It must miniscule.

It is worth noting that Pantheon has the best user experience of the 3 companies. The process with 1&1 just dumps me at a hosting sign up page. How does that let me try Drupal?

Acquia drops onto a page where you select your role, then you’re presented with some marketing stuff and a form to request a demo. That is unless you’re running an ad blocker, then when you select your role you get an Ajax error.

The Try Drupal program generates revenue for the Drupal Association. This money helps fund development of the project. I’m well aware that the DA needs money. At the same time I wonder if it is worth it. For many people this is the first experience they have using Drupal.

The previous attempt to have simplytest.me added to the try Drupal page ultimately failed due to the financial implications. While this is disappointing I don’t think simplytest.me is necessarily the answer either.

There needs to be some minimum standards for the Try Drupal page. One of the key item is the number of clicks to get from d.o to a working demo site. Without this the “Try Drupal” page will drive people away from the project, which isn’t the intention.

If you’re at DrupalCon Vienna and want to discuss this and other ways to improve the marketing of Drupal, please attend the marketing sprints.

Apr 27 2017
Apr 27

My blog post from last week was very well received and sparked a conversation in the Drupal community about the future of Drupal. That conversation has continued this week at DrupalCon Baltimore.

Yesterday during the opening keynote, Dries touched on some of the issues raised in my blog post. Later in the day we held an unofficial BoF. The turn out was smaller than I expected, but we had a great discussion.

Drupal moving from a hobbyist and business tool to being an enterprise CMS for creating “ambitious digital experiences” was raised in the Driesnote and in other conversations including the BoF. We need to acknowledge that this has happened and consider it an achievement. Some people have been left behind as Drupal has grown up. There is probably more we can do to help these people. Do we need more resources to help them skill up? Should we direct them towards WordPress, backdrop, squarespace, wix etc? Is it is possible to build smaller sites that eventually grow into larger sites?

In my original blog post I talked about “peak Drupal” and used metrics that supported this assertion. One metric missing from that post is dollars spent on Drupal. It is clear that the picture is very different when measuring success using budgets. There is a general sense that a lot of money is being spent on high end Drupal sites. This has resulted in less sites doing more with Drupal 8.

As often happens when trying to solve problems with Drupal during the BoF descended into talking technical solutions. Technical solutions and implementation detail have a place. I think it is important for the community to move beyond this and start talking about Drupal as a product.

In my mind Drupal core should be a content management framework and content hub service for building compelling digital experiences. For the record, I am not arguing Drupal should become API only. Larger users will take this and build their digital stack on top of this platform. This same platform should support an ecosystem of Drupal “distros”. These product focused projects target specific use cases. Great examples of such distros include Lightning, Thunder, Open Social, aGov and Drupal Commerce. For smaller agencies and sites a distro can provide a great starting point for building new Drupal 8 sites.

The biggest challenge I see is continuing this conversation as a community. The majority of the community toolkit is focused on facilitating technical discussions and implementations. These tools will be valuable as we move from talking to doing, but right now we need tools and processes for engaging in silver discussions so we can build platinum level products.

Apr 22 2017
Apr 22

WOW! The response to my blog post on the future of Drupal earlier this week has been phenomenal. My blog saw more traffic in 24 hours than it normally sees in a 2 to 3 week period. Around 30 comments have been left by readers. My tweet announcing the post was the top Drupal tweet for a day. Some 50 hours later it is still number 4.

It seems to really connected with many people in the community. I am still reflecting on everyone’s contributions. There is a lot to take in. Rather than rush a follow up that responds to the issues raised, I will take some time to gather my thoughts.

One thing that is clear is that many people want to use DrupalCon Baltimore next week to discuss this issue. I encourage people to turn up with an open mind and engage in the conversation there.

A few people have suggested a BoF. Unfortunately all of the official BoF slots are full. Rather than that be a blocker, I’ve decided to run an unofficial BoF on the first day. I hope this helps facilitate the conversation.

Unofficial BoF: The Future of Drupal

When: Tuesday 25 April 2017 @ 12:30-1:30pm

Where: Exhibit Hall - meet at the Digital Echidna booth (#402) to be directed to the group

What: High level discussion about the direction people think Drupal should take.

UPDATE: An earlier version of this post had this scheduled for Monday. It is definitely happening on Tuesday.

I hope to see you in Baltimore.

Apr 19 2017
Apr 19

Update 21 April: I’ve published a followup post with details of the BoF to be held at DrupalCon Baltimore on Tuesday 25 April. I hope to see you there so we can continue the conversation.

Drupal has a problem. No, not that problem.

We live in a post peak Drupal world. Drupal peaked some time during the Drupal 8 development cycle. I’ve had conversations with quite a few people who feel that we’ve lost momentum. DrupalCon attendances peaked in 2014, Google search impressions haven’t returned to their 2009 level, core downloads have trended down since 2015. We need to accept this and talk about what it means for the future of Drupal.

Technically Drupal 8 is impressive. Unfortunately the uptake has been very slow. A factor in this slow uptake is that from a developer’s perspective, Drupal 8 is a new application. The upgrade path from Drupal 7 to 8 is another factor.

In the five years Drupal 8 was being developed there was a fundamental shift in software architecture. During this time we witnessed the rise of microservices. Drupal is a monolithic application that tries to do everything. Don’t worry this isn’t trying to rekindle the smallcore debate from last decade.

Today it is more common to see an application that is built using a handful of Laravel micro services, a couple of golang services and one built with nodejs. These applications often have multiple frontends; web (react, vuejs etc), mobile apps and an API. This is more effort to build out, but it likely to be less effort maintaining it long term.

I have heard so many excuses for why Drupal 8 adoption is so slow. After a year I think it is safe to say the community is in denial. Drupal 8 won’t be as popular as D7.

Why isn’t this being talked about publicly? Is it because there is a commercial interest in perpetuating the myth? Are the businesses built on offering Drupal services worried about scaring away customers? Adobe, Sitecore and others would point to such blog posts to attack Drupal. Sure, admitting we have a problem could cause some short term pain. But if we don’t have the conversation we will go the way of Joomla; an irrelevant product that continues its slow decline.

Drupal needs to decide what is its future. The community is full of smart people, we should be talking about the future. This needs to be a public conversation, not something that is discussed in small groups in dark corners.

I don’t think we will ever see Drupal become a collection of microservices, but I do think we need to become more modular. It is time for Drupal to pivot. I think we need to cut features and decouple the components. I think it is time for us to get back to our roots, but modernise at the same time.

Drupal has always been a content management system. It does not need to be a content delivery system. This goes beyond “Decoupled (Headless) Drupal”. Drupal should become a “content hub” with pluggable workflows for creating and managing that content.

We should adopt the unix approach, do one thing and do it well. This approach would allow Drupal to be “just another service” that compliments the application.

What do you think is needed to arrest the decline of Drupal? What should Drupal 9 look like? Let’s have the conversation.

Apr 06 2017
Apr 06

Living in the middle of nowhere and working most of my hours in the evenings I have few opportunities to attend events in person, let alone deliver presentations. As someone who likes to share knowledge and present at events this is a problem. My work around has been presenting remotely. Many of my talks are available on playlist on my youtube channel.

I’ve been doing remote presentations for many years. During this time I have learned a lot about what it takes to make a remote presentation successful.

Preparation

When scheduling a remote session you should make sure there is enough time for a test before your scheduled slot. Personally I prefer presenting after lunch as it allows an hour or so for dealing with any gremlins. The test presentation should use the same machines and connections you’ll be using for your presentation.

I prefer using Hangouts On Air for my presentations. This allows me to stream my session to the world and have it recorded for future reference. I review every one of my recorded talks to see what I can do better next time.

Both sides of the connection should use wired connections. WiFi, especially at conferences can be flakey. Organisers should ensure that all presentation machines are using Ethernet, and if possible it should be on a separate VLAN.

Tips for Presenters

Presenting to a remote audience is very different to presenting in front of a live audience. When presenting in person you’re able to focus on people in the audience who seem to be really engaged with your presentation or scan the crowd to see if you’re putting people to sleep. Even if there is a webcam on the audience it is likely to be grainy and in a fixed position. It is also difficult to pace when presenting remotely.

When presenting in person your slides will be displayed in full screen mode, often with a presenter view in your application of choice. Most tools don’t allow you to run your slides in full screen mode. This makes it more difficult as a presenter. Transitions won’t work, videos won’t autoplay and any links Keynote (and PowerPoint) open will open in a new window that isn’t being shared which makes demos trickier. If you don’t hide the slide thumbnails to remind you of what is coming next, the audience will see them too. Recently I worked out printing thumbnails avoids revealing the punchlines prematurely.

Find out as much information as possible about the room your presentation will be held in. How big is it? What is the seating configuration? Where is the screen relative to where the podium is?

Tips for Organisers

Event organisers are usually flat out on the day of the event. Having to deal with a remote presenter adds to the workload. Some preparation can make life easier for the organisers. Well before the event day make sure someone is nominated to be the point of contact for the presenter. If possible share the details (name, email and mobile number) for the primary contact and a fallback. This avoids the presenter chasing random people from the organising team.

On the day of the event communicate delays/schedule changes to the presenter. This allows them to be ready to go at the right time.

It is always nice for the speaker to receive a swag bag and name tag in the mail. If you can afford to send this, your speaker will always appreciate it.

Need a Speaker?

Are you looking for a speaker to talk about Drupal, automation, devops, workflows or open source? I’d be happy to consider speaking at your event. If your event doesn’t have a travel budget to fly me in, then I can present remotely. To discuss this further please get in touch using my contact form.

Sep 17 2016
Sep 17

DrupalCon Dublin is just around the corner. Earlier today I started my journey to Dublin. This week I’ll be in Mumbai for some work meetings before heading to Dublin.

On Tuesday 27 September at 1pm I will be presenting my session Let the Machines do the Work. This lighthearted presentation provides some practical examples of how teams can start to introduce automation into their Drupal workflows. All of the code used in the examples will be available after my session. You’ll need to attend my talk to get the link.

As part of my preparation for Dublin I’ve been road testing my session. Over the last few weeks I delivered early versions of the talk to the Drupal Sydney and Drupal Melbourne meetups. Last weekend I presented the talk at Global Training Days Chennai, DrupalCamp Ghent and DrupalCamp St Louis. It was exhausting presenting three times in less than 8 hours, but it was definitely worth the effort. The 3 sessions were presented using hangouts, so they were recorded. I gained valuable feedback from attendees and became aware of some bits of my talk needed some attention.

Just as I encourage teams to iterate on their automation, I’ve been iterating on my presentation. Over the next week or so I will be recutting my demos and polishing the presentation. If you have a spare 40 minutes I would really appreciate it if you watch one of the session recording below and leave a comment here with any feedback.

Global Training Days Chennai

DrupalCamp Ghent

Note: I recorded the audience not my slides.

DrupalCamp St Louis

Note: There was an issue with the mic in St Louis, so there is no audio from their side.

Jan 25 2016
Jan 25

One of the biggest improvements in Drupal 8 is the new configuration management system. Config is now decoupled from code and the database. Unlike Drupal 6 and 7, developers no longer have to rely on the features module for moving configuration around.

Most large Drupal sites, and some smaller ones, require per environment configuration. Prior to Drupal 8 this was usually achieved using a combination of hard coding config variables and features. Drupal 8 still allows users to put config variables in the settings.php file, but putting config in code feels like a backward step given D8 emphasis on separating concerns.

For example we may have a custom module which calls a RESTful API of a backend service. There are dev, stage and production endpoints that we need to configure. We also keep our config out of docroot and use drush to import the config at deployment time. We have the following structure in our git repo:

/
+- .git/
|
+- .gitignore
|
+- README.md
|
+- config/
|  |
|  +- README.md
|  |
|  +- base/
|  |
|  +- dev/
|  |
|  +- prod/
|  |
|  +- stage/
|
+- docroot/
|
+- scripts/
|
+- and-so-on/

When a developer needs to export the config for the site they run drush config-export --destination=/path/to/project/config/base. This exports all of the configuration to the specified path. To override the API endpoint for the dev environment, the developer would make the config change and then export just that piece of configuration. That can be done by runing drush config-get mymodule.endpoint > /path/to/project/config/dev/mymodule.endpoint.yml.

Drupal 8 and drush don’t allow you to import the 2 config sets at the same time, so we need to run 2 drush commands to import our config. drush config-import --source=/path/to/project/config/base && drush config-import --partial --source=/path/to/project/config/dev. The first command imports the base config and the second applies any per environment overrides. The --partial flag prevents drush deleting any missing config. In most cases this is ok, but watch out if you delete a view or block placement.

Best practices are still emerging for managing configuration in Drupal 8. While I have this method working, I’m sure others have different approaches. Please leave a comment if you have an alternative method.

Update 11-Mar-2016: Removed --partial from base import. This prevented old configuration being removed during updates.

May 15 2015
May 15

Update: It turns out the DA was trolling. We all now know that DrupalCon North America 2016 will be in New Orleans. I’ve kept this post up as I believe the information about handling unpublished nodes is relevant. I have also learned that m4032404 is enabled by default in govCMS.

When a user doesn’t have access to content in Drupal a 403 forbidden response is returned. This is the case out of the box for unpublished content. The problem with this is that sensitive information may be contained in the URL. A great example of this the DrupalCon site.

DrupalCon in Philadephia in 2016?

The way to avoid this is to use the m4032404 module which changes a 403 response to a 404. This simple module prevents your site leaking information via URLs.

Apr 13 2015
Apr 13

A couple of times recently the issue of managing variables in Drupal 7 has come up in conversation with other developers. This post outlines the various ways of managing variables in Drupal sites. The three things this guide ensures:

  • Sensitive data is kept secure
  • Variables are correct in each environment
  • You are able to track your variables (and when they changed)

The Variables Table

The most common place you’ll find configuration variables is in Drupal’s variable table (aka {variable}). The values in this table are often managed via admin forms that use system_settings_form(). Users enter the values click “Save configuration” and the data is stored in the database.

If you prefer to manage your configuration via the command line and know the variable you wish to set you can use drush vset. This does exactly the same thing as admin form, without needing to click on a mouse.

$conf Array

While the variables table is great at storing our variables, there are times when you want to enforce a setting. This might be because you want to prevent users from changing it (accidentally or otherwise) or because you need it to be different in each environment. The $conf array in settings.php always overrides any values in the variable table.

Acquia, Pantheon and platform.sh all provide environment variables so you can use different values in your $conf array depending on the environment.

Exporting Variables

In Drupal 7, the common way to export your variables is by using Strongarm with Features. I’m not going to cover how to do this as there is loads of documentation already available on this topic.

If your variable changes on a per environment basis or if it calculated on the fly, then you won’t want to use strongarm+features as the exported values are static. You will need to put them in settings.php.

Note to self: I should debug and reroll my patch for adding support in alter hooks strongarm.

My settings.php is Out of Control!

This is a common problem, especially on more complex sites. To avoid this I recommend creating sites/default/settings/settings.[env].php files. Your settings.php file should check for the environment in an environment variable and then include the appropriate settings.[env].php file.

What About Sensitive Data?

You can encrypt variables on a case by case basis using the encrypt module and some custom code similar to what I recently implemented in the Acquia SDK module (see on store and on read examples). Warning: This doesn’t encrypt the data if you’re using drush vset.

If you are storing sensitive data in your variables table I would recommend you implement hook_sql_sync_sanitize() which will delete the sensitive data from your db when drush sql-sanitize or drush sql-sync --sanitize are run.

How to Decide?

This little code snippet should help you decide.

php

// Don't try using this code in your Drupal site.

if (!using_version_control()) {
  // Seriously there is no point in doing this without version control.
  abandon_all_hope();
  drupal_exit();
}

if (is_data_sensitive($var)) {
  $var = encrypt_var($var);
  if (!we_use_drush_based_workflows()) {
    learn_and_implement_drush_based_workflows();
    // I'm serious!
    }
  }
  implement_hook_sql_sync_sanitize($var);
}

if (is_unique_per_environment($var)) {
  store_conf_array($var);
}
else {
  store_in_db($var);
  if (!we_use_features_based_workflow()) {
    learn_and_implement_features_based_worflows();
    // I'm serious!
  }
  export_using_strongarm($var);
}
Jul 21 2014
Jul 21

Update: Acquia shutdown Cloud API v1 on 1 July 2020.

The Acquia Cloud API makes it easy to manage sites on the platform. The API allows you to perform many administrative tasks including creating, destroying and copying databases, deploying code, managing domains and copying files.

Acquia offers 2 official clients. The primary client is a drush plugin which can only be downloaded from Acquia Insight. The other is a PHP library which states in the README that it is “[n]ot ready for production usage”.

On a recent project using WF Tools we needed some pretty advanced deployment scripts for sites hosted on Acquia Cloud. We had tried using a mix of bash and PHP, but that created a maintenance nightmare, so we switched to Python.

I was unable to find a high quality Python library, so I wrote a python client for the Acquia Cloud API. The library implements all of the features that we needed, so there are a few things missing.

Chaining complex commands together is easy because the library implements a fluent interface. An extreme example of what is possible is below:

import acapi
# Instantiate the client
c = acapi.Client('[email protected]', 'acquia-token')
# Copy the prod db to dev, make a backup of the dev db and download it to /tmp
c.site('mysite').environment('prod').db('mysite').copy('dev').backups().create().download('/tmp/backup.sql.gz')

Some of the code is “borrowed” from the Python client for Twilio. The library is licensed under the terms of the MIT license.

I am continuing to develop the library. Consider this a working alpha. Improving error handling, creating a comprehensive test suite and implementing the missing API calls are all on the roadmap. Pull requests are welcome.

The code is PEP 8 (coding standards and PEP 257 (documentation standards) compliant and uses the numpydoc for code documentation.

Check out the Python client for Acquia’s Cloud API on github.

Jan 02 2013
Jan 02

Over the Christmas break I came across gource, a software version control visualization tool. Gource produces really nice visual representations of software projects growing. About 2 years ago David Norman produced a gource video of development of Drupal up to the 7 release. This is pretty cool, but it only shows who committed the patch, not who contributed to it.

After some searching I found the Drupal contribution analyzer sandbox project. This module allows you to produce contributor tag clouds and code swarm videos. This was closer to what I was after, but I had to patch patch drupal_log_generator.py to support the gource custom log format.

The result is a 5:23 minute video showing the growth of Drupal.

The first few years things are pretty consistent and easy to follow. The Drupal 8 development cycle shows how much the community of contributors has grown. Towards the end of last year things look really chaotic.

To produce the video I used a clone of the Drupal 8 branch as at some time on January 1, 2013. The gource command I used was: gource --log-format custom -i 500 -s 0.0001 -a 0.01 -r 30 --title "Drupal" --highlight-users --disable-progress --hide filenames -1280x720 drupal --bloom-intensity 0.2 --bloom-multiplier 0.2 --stop-at-end /tmp/commit.log -o -| ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libvpx -b 10000K \~/Videos/drupal.webm

I considered writing a script to find and download user avatars from groups.drupal.org but after reviewing the video without them I decided it would be too cluttered.

Can you find your name in the video?

Oct 08 2012
Oct 08

One of the many hats I wear these days is Development and Coding Track Chair for DrupalCon Sydney 2013. As outlined in the track description we are planning on showcasing what is awesome today in Drupal 7 and the cool stuff that is coming in Drupal 8. Given that there are no core conversations in Sydney we are trying to put together a more intermediate-to-advanced level track. I want people to come to these sessions and go away with their heads full of ideas about what they can do better in their next project.

If you have a session that you think fits that brief then please submit something. If you want to ask me anything before submitting your session, feel free to contact me. The decision on which sessions are accepted will be made in late October / early November by the track team, the global track chairs, the content chair and myself in a collaborative decision making process. The accepted sessions will be announced on 13 November.

Although the event won’t be as big as a northern hemisphere DrupalCon, it is going to be full of great people. The initial 100 early bird tickets sold out in less than 8 hours!

Please be aware that there is no financial support available for speakers and you will be required to buy a speakers ticket at a cost of 165USD.

Submissions close at 23:59 AEST (UTC+10) on 26 October so submit a session today!

Sep 12 2012
Sep 12

In my last blog post I outlined how to use per project installation profiles. If you read that post and want to use installation profiles to take advantage of site wide content changes and centralised dependency management, this post will show you how to do it quickly and easily.

The easiest way to switch installation profiles is using the command line with drush. The following command will do it for you:

drush vset --exact -y install_profile my_profile

An alternative way of doing this is by directly manipulating the database. You can run the following SQL on your Drupal database to switch installation profiles:

UPDATE variable SET value = 'my_profile' WHERE name = 'install_profile';
-- Clear the cache using MySQL only syntax, when DB caching is used.
TRUNCATE cache;

Before you switch installation profiles, you should check that you have all the required modules enabled in your site. If you don’t have all of the modules required by the new installation profile enabled in your site, your are likely to have issues. The best way to ensure you have all the dependencies enabled is to run the following one liner:

drush en $(grep dependencies /path/to/my-site/profiles/my_profile/my_profile.info | sed -n 's/dependencies\[\]=\(.*\)/\1/p')

Even though it is pretty easy to switch installation profiles I would recommend starting your project with a project specific installation profile.

Edit: Jaime Schmidt picked up a missing step in the instructions above. You need to enable the installation profile in the system table. The easiest way to do that is with this drush one liner:

echo UPDATE system SET schema_version = 0 WHERE name = 'my_profile' | drush sqlc && drush cc all
Sep 09 2012
Sep 09

Unbeknown to many users, installation profiles are what is used to install a Drupal site. The two profiles that ship with core are standard and minimal. Standard gives new users a basic, functional Drupal site. Minimal provides a very minimal configuration so developers and site builders can start building a new site. A key piece of a Drupal distro is an installation profile.

I believe that developers and more experienced site builders should be using installation profiles as part of their client sites builds. In Drupal 7 an installation profile is treated like a special module, so it can implement hooks - including hook_update\_N(). This means that the installation profile is the best place for controlling turning modules on and off, switching themes or any other site wide configuration changes that can’t be handled by features or a module specific update hook.

In an ideal world you could have 1 installation profile that is used for all of your projects and you just include it in your base build. Unfortunately installation profiles tend to evolve into being very project specific. At the same time you are likely to want a common starting point. I like to give my installation profiles unique names, rather than something generic like my_profile, I prefer to use [client_prefix\]_profile. I’ll cover project prefixes in another blog post.

After some trial and error, I’ve settled on a solution which I think works for having a common starting point for an installation profile that will diverge overtime using a unique namespace. My solution relies on some basic templates, a bash script with a bit of sed. I could have written all of this in PHP and even made a drush plugin for it, but I prefer to do this kind of thing on the command line with bash. I’m happy to work with someone to port it to a drush plugin if you’re interested.

Here is a simple example of the templates you could use for creating your installation profile. The version on GitHub is closer to what I actually use for clients, along with the build script.

base.info

name = PROFILE_NAME
description = PROFILE_DESCRIPTION
core = 7.x
dependencies[] = block
dependencies[] = dblog

base.install

php
/**
 * @file
 * Install, update and uninstall functions for the the PROFILE_NAME install profile.
 */

/**
 * Implements hook_install().
 *
 * Performs actions to set up the site for this profile.
 *
 * @see system_install()
 */
function PROFILE_NAMESPACE_install() {
  // Enable some standard blocks.
  $default_theme = variable_get('theme_default', 'bartik');
  $values = array(
    array(
      'module' => 'system',
      'delta' => 'main',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'content',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'user',
      'delta' => 'login',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'navigation',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'management',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 1,
      'region' => 'sidebar_first',
      'pages' => '',
      'cache' => -1,
    ),
    array(
      'module' => 'system',
      'delta' => 'help',
      'theme' => $default_theme,
      'status' => 1,
      'weight' => 0,
      'region' => 'help',
      'pages' => '',
      'cache' => -1,
    ),
  );
  $query = db_insert('block')->fields(array('module', 'delta', 'theme', 'status', 'weight', 'region', 'pages', 'cache'));
  foreach ($values as $record) {
    $query->values($record);
  }
  $query->execute();

  // Allow visitor account creation, but with administrative approval.
  variable_set('user_register', USER_REGISTER_VISITORS_ADMINISTRATIVE_APPROVAL);

  // Enable default permissions for system roles.
  user_role_grant_permissions(DRUPAL_ANONYMOUS_RID, array('access content'));
  user_role_grant_permissions(DRUPAL_AUTHENTICATED_RID, array('access content'));
}

// Add hook_update_N() implementations below here as needed.

base.profile

php
/**
 * @file
 * Enables modules and site configuration for a PROFILE_NAME site installation.
 */

/**
 * Implements hook_form_FORM_ID_alter() for install_configure_form().
 *
 * Allows the profile to alter the site configuration form.
 */
function PROFILE_NAMESPACE_form_install_configure_form_alter(&$form, $form_state) {
  // Pre-populate the site name with the server name.
  $form['site_information']['site_name']['#default_value'] = $_SERVER['SERVER_NAME'];
}

Some developers might recognise the code above, it is from the minimal installation profile.

The installation profile builder script is a simple bash script that relies on sed.

build-profile.sh

#!/bin/bash
#
# Installation profile builder
# Created by Dave Hall http://davehall.com.au
#

FILES="base.info base.install base.profile"
OK_NS_CHARS="a-z0-9_"
SCRIPT_NAME=$(basename $0)

namespace="my_profile"
name=""
description="My automatically generated installation profile."
target=""

usage() {
  echo "usage: $SCRIPT_NAME -t target_path -s profile_namespace [-d 'project_descrption'] [-n 'human_readable_profile_name']"
}

while getopts  "d:n:s:t:h" arg; do
  case $arg in
    d)
      description="$OPTARG"
      ;;
    n)
      name="$OPTARG"
      ;;
    s)
      namespace="$OPTARG"
      ;;
    t)
      target="$OPTARG"
      ;;
    h)
      usage
      exit
      ;;
  esac
done

if [ -z "$target" ]; then
  echo ERROR: You must specify a target path. >&2
  exit 1;
fi

if [ ! -d "$target" -o ! -w "$target" ]; then
  echo ERROR: The target path must be a writable directory that already exists. >&2
  exit 1;
fi

ns_test=${namespace/[^$OK_NS_CHARS]//}
if [ "$ns_test" != "$namespace" ]; then
  echo "ERROR: The namespace can only contain lowercase alphanumeric characters and underscores ($OK_NS_CHARS)" >&2
  exit 1
fi

if [ -z "$name" ]; then
  name="$namespace";
fi

for file in $FILES; do
  echo Processing $file
  sed -e "s/PROFILE_NAMESPACE/$namespace/g" -e "s/PROFILE_NAME/$name/g" -e "s/PROFILE_DESCRIPTION/$description/g" $file > $target/$file
done

echo Completed generating files for $name installation profile in $target.

Place all of the above files into a directory. Before you can generate your first profile you must run chmod +x build-profile.sh to make the script executable. You need to create the output directory, for testing we will use ~/test-profile, so run mkdir ~/test-profile to create the path. To build your profile run ./build-profile.sh -s test -t ~/test-profile. Once the script has run you should have a test installation profile in ~/test-profile. I will continue to maintain this as a project on GitHub.

Jul 13 2012
Jul 13

Months ago I was searching for a good web front end for git for doing code reviews and browsing repos. My short list ended up being Gitweb and GitLab.

Gitweb is a Perl based web front end for git that is a sub project of the official git project. Out of the box Gitweb is pretty ugly and I have never found it to be very user friendly. Even with all of its problems, it does what it does pretty well.

GitLab is a Rails based github clone that is being actively developed and shows a lot of promise. If you want a fully blown ready to go basic github clone to setup and run for your dev team, GitLab looks great. Unfortunately trying to install it on RHEL5 wasn’t an easy task. GitLab offered a lot more functionality than I needed and my lack of Ruby experience meant that it was adding more complexity than I wanted.

I ended up reluctantly recommending that we use Gitweb but I have since changed my mind.

Over the last few weeks I’ve been playing with GitList by Klaus Silveira. GitList is a web front end for browsing git repos, that looks a little like github. Klaus describes GitList as:

GitList is an elegant and modern web interface for interacting with multiple git repositories. It allows you to browse repositories using your favorite browser, viewing files under different revisions, commit history, diffs. It also generates RSS feeds for each repository, allowing you to stay up-to-date with the latest changes anytime, anywhere. GitList was written in PHP, on top of the Silex microframework and powered by the Twig template engine. This means that GitList is easy to install and easy to customize. Also, the GitList gorgeous interface was made possible due to Bootstrap.

It is nice having an open source side project to contribute to that gets me out of the Drupal ghetto. GitList also gives me an excuse to play with Symfony components and twig more, both things that are already in Drupal 8 core. Silex has been on my “to play with list” for some time too.

I have been searching for an alternative to Gitweb that is easy to use, has a nice UI and the basic features I want in a web front end for git - commit history, diffs, blame and branch comparison. GitList ticked all the boxes, except blame was a bit broken and branch comparison wasn’t implemented.

I now run GitList on my laptop for offline repo browsing and find it very useful. I will be deploying it for a client in the next few weeks as part of a broader Drupal workflow management system.

This isn’t a review of GitList and I won’t be providing any feature comparison of GitList vs (Gitweb|GitLab|github). Instead this post is inspired by a discussion on twitter between Klaus (@klaussilveira), Fabien Potencier (@fabpot), Peter Droogmans (@attiks) and myself (@skwashd). I want outline how we are using GitList and the underlying git library, my vision for GitList and what I can do to make it happen.

Missing Features

Based on the project description I think that GitList is almost feature complete. There are a few TODO items in the README. In my time using GitList I haven’t found myself going “if only it did X (and Y and Z)” - except branch diff support. I have been working on a branch diff patch, and I hope to clean up that code this weekend and submit a pull request. I think it is good when something does one thing and does it well.

Library and App Separation

As part of the workflow management system for Drupal we’re developing we needed a solid git library for PHP. The one that ships with GitList is pretty nice. I’ve been working with my fellow Technocrat developer, James Harvey, on extracting the git library from GitList. We still have some work to do on this, but it is usable today. We have added some enhancements to the library for our purposes and removed the Silex dependency. Our vision is to have a generic OO git library for PHP 5.3+ and that GitList use this as a sub module.

Although I am currently hosting the code in my github repo, I would love to see Klaus create a new primary repo for the library and that James, myself and others would continue to develop and support his library. It is not my intention to fork the library from GitList and maintain it myself in isolation - that is a waste of energy and resources.

Git Module for Drupal

Following on from our creation of a generic git library for PHP, we have created a git wrapper module for Drupal. We plan to release this module in the next week or so on drupal.org - we’ve already reserved the project namespace. As part of this change the features git module will now depend on the new library wrapper module. We will release other modules that use this library as they pass QA.

Releases

I have a hacker mentality, I am happy to clone a repo, create a branch or tag and start playing. If I find something that it broken, then there is a good chance that I’ll fix it and post a patch. At the same time I understand that not everyone can work that way and that in some environments there is a focus on only using official stable releases. It is also difficult tracking bugs against git commit hashes. GitList is moving quickly, but I think that it could benefit from having official releases.

If the library and app are separated, they can have separate and independent release cycles. I am happy to work with Klaus to work out a plan for GitList releases. One of my first roles in an open source project was as a release manager.

Discussion Space

I have posted this on my blog because I don’t know what the real audience is for this conversation. I know that there are 3 people interested in participating in the conversation but are there others? I know others contribute patches to GitList. I could have emailed Klaus directly, but that would have excluded others who are interested. Even though in some ways email is on the way out, some discussions require more than 140 characters. Is it worth setting up a mailing list, google group, a forum or some other channel of communication to discuss these issues?

The Future

I am already dependent on GitList for a significant piece of work. I want to work with Klaus and the rest of the GitList community to make a kick arse web front end for git, I also need a rock solid PHP lib for git. I think GitLib provides a solid foundation for both. Fabien wants to use it as a good Silex example. What do others want to do with GitList? Let the discussion begin …

Upadte: Klaus has asked that we move the discussion to the recently created GitList mailing list/google group, so I have created a new thread to continue this conversation. I can cross an item off my list - discussion space.

May 20 2012
May 20

From: Dave Hall

To: boxes [email protected]

Subject: Our Relationship

Dear boxes,

I’m sorry but things just aren’t working out between us. It’s not you, it’s me. I need some time to myself. I need to think things through. I’m not sure what I want. We should spend some time apart. We should try new things. I will miss you, but this is for the best. Let’s meet for coffee in a couple of weeks.

Love

Dave xox

Breaking up is never easy, but recently I broke up with the boxes module. I’d been with boxes for a long time, we’d done a lot of good things together. Over time I’d become oblivious to some of boxes' flaws, such as giving black eyes to sys admins who run features reverts on sites where users can edit boxes. Life was pretty good, but my life still felt somewhat empty and incomplete.

One day I was standing in a crowded room with lots of modules on drupal.org and across the room I spotted BEAN. I feel in love instantly. BEAN was decked out in all the latest Drupal 7 gear - fields, Entity API and CTools. The way BEAN moved make me weak at the knees. After another beer I got my courage up, crossed the room and asked if BEAN would like to come back to my site. I was shocked, BEAN wanted to hang out with me and it wasn’t just a one night fling. I’ve been going steady with BEAN for a couple of months now.

Just before BEAN moved in permanently to my dev environment I had to find a way of removing all the remnants of boxes from my life (aka existing sites). It took me less than an hour. I created a new BEAN type and called it “box”, I then wrote a little “drush scr” script which converted all of my boxes to BEANs and even updated my contexts to use the newly created BEANs. I hope neither bean or boxes finds my work on github. I chose github hoping neither of them would find my work and also because I don’t want to maintain this as a module on d.o.

On a serious note - BEAN is awesome! If you haven’t tried it, you really should. BEAN treats blocks as content, not config which is really useful for most sites. For new sites BEAN is a drop in replacement for core blocks or boxes making it super easy to get started. To learn more check out the docs.

If you’re interested in seeing the full power of the BEAN module, and other cool stuff I’ve been working on recently, please comment on my DrupalCon session proposal - An Enterprise Scale Drupal Workflow. It will be a more technical version of my session at Drupal Business Days a couple weeks ago.

* “Full of beans” is phrase commonly used in Australia to describe energetic children.

Nov 11 2011
Nov 11

For the last few months I’ve been working for Technocrat on a new Drupal based site for the Insurance Australia Group’s Direct Insurance brands. The current sites are using Autonomy Teamsite.

The basics of the build are relatively straight forward, around 1000 nodes, a bunch of views and a bit of glue to hold it all together. Where things get complicated is the workflow. The Financial services sector in Australia is subject to strict control of representations being made about products. The workflow system needs to ensure IAG complies with these requirements.

During the evaluation we found that generally Drupal workflows are based around publishing a single piece of content on the production site. In the IAG case a collection of nodes need to be published as a piece of work, along with a new block. These changes need to be reviewed by stakeholders and then deployed. This led us to build a job based workflow system.

We are using the Features module to handle all configuration, deploy for entities and some additional tools, including Symfony, Jenkins and drush to hold it all together.

I’ve proposed the session for Drupal Downunder in January and will refine the session based on feedback from there in preparation for Denver. If you want to learn more about Drupal Workflows in the Enterprise, please vote for my session.

May 17 2011
May 17

Drush make is a wonderful tool for constructing Drupal platforms. A lot of Drupal developers are used to adding a list of modules, a few libraries and theme or 2 then running drush make to build their platform. It all seems pretty easy. What if I told you module developers could make things even easier for site builders?

Some contrib modules depend on third party libraries, and due to various reasons they can’t always be stored in git repositories on drupal.org and included in the module release. To solve this problem module developers can include a .make file for their module. Drush recursively processes make files, so the module make file would be processed once found by drush make.

A good example of where this could be useful is the SMTP module, which depends on the LGPL licensed PHPMailer library. The module also requires a patch to be applied to the library, which drush make can apply for us. The following .make file could be included in the SMTP module as smtp.make:

core = 6.x
api = 2

libraries[phpmailer][download][type] = "get"
libraries[phpmailer][download][url] = "http://downloads.sourceforge.net/project/phpmailer/phpmailer%20for%20php5_6/Previous%20Versions/2.2.1/phpMailer_v2.2.1_.tar.gz"
libraries[phpmailer][download][md5] = "0bf75c1bcef8bde6adbebcdc69f1a02d"
libraries[phpmailer][directory_name] = "phpmailer"
libraries[phpmailer][destination] = "modules/contrib/smtp"

libraries[phpmailer][patch][drupal-compatibility][url] = "http://drupalcode.org/project/smtp.git/blob_plain/2acaba97adcad7304c22624ceeb009d358b596e3:/class.phpmailer.php.2.2.1.patch"
libraries[phpmailer][patch][drupal-compatibility][md5] = "2d82de03b1a4b60f3b69cc20fae61b76"

Now when the SMTP module is included a normal drush make file it will be downloaded, the PHPMailer library will be downloaded and patched ready for use.

Unfortunately there are some limitations to this approach. Firstly it assumes that the SMTP module will be installed under the modules/contrib directory, which is accepted best practice, but may not suit everyone’s needs. When I tested this with the current stable version of drush make (6.x-2.2) it failed, and drush make 6.x-3.x from git needed to be patched. Hopefully a fix for this can be backported to the 6.x-2.x branch and included in a future release.

Update: I have posted the make file for the SMTP module as patch in issue #1159080.

Jan 26 2011
Jan 26

I spent the weekend at Drupal Downunder in Brisbane. The venue was excellent. I’m a fan of not using “traditional” venues for conferences, to help make them even more memorable for attendees.

I managed to catch up with a bunch of people. The relaxed feel about the event was great. Most conferences I’ve attended recently have either been large or I’ve helped organise them, this time I could relax and enjoy.

On the Saturday I presented Building Distributions with Drupal 7, which had a small turn out as I was up against Josh Koenig and his Pantheon presentation. My presentation was hampered by lack of internet connectivity, but I think it went well. I used Lego, Duplo and Quatro blocks to demonstrate the evolution of Drupal distros.

Saturday night involved a pub crawl with various DDU and LCA folks. The highlight of the crawl was the Mana Bar, which is a gamers bar, that has a good collection of retro consoles and games on display.

I spent a fair bit of Sunday in the hallway track. I discussed the D7 port of UUID with Dries, which helped confirm the direction I was heading with it. Several people wanted to discuss my $100 Drupal site blog series. I also gave my Horizontally Scaling Drupal presentation, which was very well attended. Unfortunately due to people torrenting there was no usable internet access for my presentation. I had to skip the post event BBQ so I could fly back to Melbourne.

The lack of mobile signal and wifi made it frustrating to prepare and present. I would have liked to have seen an inclusive social event organised on the Saturday night. Overall I really enjoyed DDU and the organisers are to be congratulated. The vegetarian food options were excellent.

Thanks for Four Kitchens for funding me to get to DDU. I have just started contracting with them, so I really appreciated them covering my trip.

Jan 04 2011
Jan 04

When doing development work, from time to time it is handy to be able to look up documentation. Bookmarking manuals is handy, but often you still need to search for the function you’re after. Firefox, and possibly other browsers (not Chrome or Chromium), allows you to setup a keyword bookmark linked to a search.

I’ve setup a few search bookmarks for development resources. This is how I’ve done it:

  1. Select Bookmarks > Organise Bookmarks… from the menu.

  2. Right click on the bookmarks menu (or any folder) on the left pane

  3. Select New Bookmark… from the context menu

  4. Drupal bookmarkexample

    Fill in the information for the bookmark, the import piece is the keyword, that will allow us to search.

  5. Click save and return to the browser

Now when we want to search the Drupal 7 API, we can just type “dapi

Example Drupal API search in locationbar

Now we should see the appropriate page from the Drupal API documentation.

Example Drupal APIpage

The same method can be used for other PHP web app developer resources, here are some which I’m using.

I could have implemented this using OpenSearch plugins, but that requires changing the search provider every time I want to look for something. By using keyword bookmarks I just type the keyword and the search term into the location bar.

Feel free to share your keyword bookmarks in the comments.

Jan 02 2011
Jan 02

Thanks to everyone who read the posts in my $100 Drupal site series. Today I will be responding to some of the points people have raised in comments and via email as well as adding a few closing comments.

Undervaluing Labour

Some have suggested that the only way to make a venture like this work is to work for a few dollars an hour. I completely disagree with that! If you are building every project from scratch and only charging your clients 100USD, you will be working for peanuts, but I’m not advocating that model. I’m encouraging people to rethink how they manage their development workflow. Although there are initial costs in learning new tools, over time you will end up spending less time on site builds and so increase your profits.

$100 is too Cheap

I agree that 100USD is very cheap for a high quality Drupal site and in most western markets it is probably too cheap. Sure you can do it and make the profit on the margin, but you have to sell a lot of sites to recover your initial investment. The “$100 Website” can be a useful sales pitch. Don’t charge 100USD per site if the market will happily pay 5 or 10 times that. It doesn’t matter if you are charging 10USD per site or 250kUSD per site, the same basic rules apply - understand your client’s requirements and know the market.

There is a growing list of freemium services out there too, including Drupal Gardens and WordPress.com. Depending on your business model you may want to have a basic freemium offering.

What Does the Competition Offer?

Investigate what the competition is offering. There is a large market for hosted CMSes these days, go see what some of them are doing. What do you think they are doing right? What do you think they lack or could do better? A quick list of services I’d recommend you check out are (in alphabetical order):

I’d recommend spending some time searching for hosted CMS solutions. Sign up for the free or trial services and see what they’re like.

Do I Need a Lot of Money to Do This?

Although in the Resources and Infrastructure post I outlined the positions I thought your team needed to make this happen, it wasn’t a list of must haves. You need to figure out what works with you. It would be possible for a couple of switched on people to do this as a basement startup, so long as you have the skills and connections to get everything you need in place. If you can only drive a server using cPanel, then you need to find a sysadmin. If you can’t even draw stick figures, then you will need to hire a designer. Your staffing costs are really going to dictate how much money you will need to get this up off the ground.

Why Did You Do This?

I wanted to put the idea out there. I wanted to see how people would react. It wasn’t that long ago that people could build a profitable business selling websites built using Dreamweaver, and some people still do, but the market is changing, so too is the Drupal market.

For some time I have been thinking about how you’d develop a low cost Drupal based hosted CMS solution. There are 2 main reasons why I haven’t just gone and built it already. The primary reason is I hate being on call and running a hosting service such as this requires me to never be out of range of a 3G tower for more than 30 minutes, customers won’t accept that the server crashed 15 minutes after I hoped on a flight to the US or Europe. Secondly building such a solution takes time and I have a family to feed, I don’t think my landlord would cop me not paying the rent until this thing started to turn a buck.

I have very consciously put enough information into this series to allow someone to head off in the right direction. At the same time I’ve held back on some of my execution specifics, because if I was to go build something like this I want to have some things which make my service unique.

I was hoping to present a session at DrupalCon Chicago on this topic, but I found out yesterday that my proposal hasn’t been accepted. As I understand it, there will be a small number of sessions announced sometime in the next 2 weeks, but I’m not holding out much hope of being selected then. There is always London.

Can You Build This for Me? / I Need Help!

If you are serious about building a service like this or using some of the tools I’ve discussed, I’d be happy to talk to you. I have about a decade’s worth of experience running my own business and developing web apps. I have offered training and consulting around many of the topics I have discussed in this series.

So neither of us waste our time, I will be up front about how I see things working on my end. I expect to be paid for my time, I’m happy to take some equity, but I’ve got bills to pay. I have put a lot of thought and energy into this, so I am very unlikely to sign a contract which gives you ownership of my IP. If you can live with the above, then please contact me.

It Didn’t Work for Me

Sorry that just the way things go in business sometimes. Not everything I’ve tried over the years has worked either. Learn from the failure and try again, maybe doing something completely different.

Disclaimer: Don’t sue me if this doesn’t work out, do your own research and seek professional advice before acting on anything on this site.

Will you Write More on these Topics?

Maybe. I will probably turn some of the content from Horizontally Scaling Drupal workshops and talks into blog posts, and possibly a book. I tend to blog when I feel passionate about something. Subscribe to my RSS feed, or follow me on twitter to find out what I’m working on.

In the mean time, I’d recommend you subscribe to the following blogs / follow these people on twitter:

I welcome comments, feel free to leave one below.

Dec 29 2010
Dec 29

During this series on creating a profitable business around the concept of building Drupal sites for $100 I have attempted to demonstrate that there is a viable business model here. I don’t believe it is a business that will suit everyone and nor do I believe every developer will want to work on such a project, but for some this will be an excellent opportunity. Today I will cover some of the things that I think you should consider before investing too much in this business model. Some of this is just basic business sense which isn’t specific to this project.

Dollars and Sense

Oscar Wilde once wrote “What is a cynic? A man who knows the price of everything and the value of nothing”. If you want to build this type of business, be frugal, but not a Scrooge. Generally a good developer will cost you more in salary than an average one, but the good developer will work out cheaper in the longer term. A similar thing goes for hosting, avoid Virtuozzo based VPS and look for someone offering the more expensive Xen or KVM based platforms, they are less likely to oversell capacity. At the same time keep an eye on the markets you are buying services in to ensure you are always getting the best value for money.

Recognise when you have to pay for something. Spending a little now might save you a lot in the long run.

In House vs Outsourcing

It makes sense to outsource some services, for example email. You want to focus on offering a high quality Drupal driven SaaS, having to support email too is a lot of hassle, use Google Apps for Domains or similar service.

Some things are better performed in house, such as design and themeing. There are businesses around who offer cheap PSD to Drupal theme conversion services, but these services usually cut corners which will cost you more in the longer term. Having someone in house means that they can learn from others on your team and do things the right way for your business.

Pricing

Although the theme of this series is “$100 Drupal site”, that doesn’t mean every site you sell should be $100. You need to research your target market/s and make sure you are pricing your product/s appropriately. I would seriously consider offering 3 levels of service, each level should offer more options, bandwidth and storage. Some customers will be drawn by the initial $100 price tag, but when they see the feature list, the $250 product might be more appealing. Give customers a sense of choice.

Options and Up Selling

Find opportunities to sell extras to your clients. Not everything needs to be included in the base level package. I discussed enhanced support offering in my previous post, which could provide additional revenue.

Most of the clients who sign up for this service will have no interest in paying for a custom theme and/or development, but once you have them through the door you have more chance of selling such extras to them. Find the higher traffic clients and connect with them, see if they want more. The price jump is likely to be significant, but if they can see value in it, they may pay for it.

Domain Names

Domain names is a great opportunity to generate extra revenue. By default customers should get biz-name.example.net, and charge them a fee if they want to use another domain, such as mybiz.com. Most people know that GoDaddy sells domains for around 10USD, while Melbourne IT charge around 75USD. Sign up for a reseller account with one of various domain registrars and offer customers domains directly from your service. If a customer brings their own domain charge them a fee, if they buy the domain through you advertise that you waive the setup fee, but then include it in the pricing of the domain name.

Partners

Seek out partners. If you can find a company which offers a service which is a good fit with your business, work out a deal with them. Be it bulk licensing, reseller, commissions, find something that works for both parties. Your customers can benefit from such arrangements too.

Giving Back

Previously I mentioned that you will be benefiting from the great work of the Drupal community, you should acknowledge this benefit by contributing back to the community where it is appropriate. Not only does this help build the profile of your business in the community, it can reduce your maintenance workload. For example if you have a patch which fixes a bug or adds generically useful functionality to a module, push it back up stream, so you don’t have to maintain the patch. Consider having a budget for sponsoring community events to help build good will.

Data Exports

Allow customers to export their site and go host it with someone else. This might sound counter intuitive, but if they are just a base package customer and you are hosting them, they have paid you. If they are only 2 months into 12 month service, which should be non refundable, then let them take their site, that frees up server resources. It is also the ethical thing to do, don’t make your money because people are locked into your service, they will eventually resent you for it.

Licensing

Drupal itself is licensed under the terms of the GPLv2 (or later) with some contrib modules depending on GPL compatible code. If you are contributing code back to drupal.org it must be under the terms of the GPLv2. You could choose to offer the non PHP parts of your code under CC-BY with an attribution in the footer, you could release your features under the terms of the AGPLv3. These are business decisions you must make in the context of the rest of your business.

Pay for Expert Advice

There are times when you need an expert. Get a lawyer to check your Terms of Service and any significant contracts. Use an accountant to get your financial affairs right from the start. If your servers are performing poorly and your Sys Admin can’t work it out, pay for someone who can. Sometimes it is better to pay for something than trying to learn how to do it yourself.

Share

Blog about your experiences, share the knowledge. I’m not advocating putting your entire business plan online, but discuss problems you’ve solved, code you’ve created so others can benefit from your experiences. Don’t be afraid to share, Scott Adams sums this up perfectly “Ideas are worthless. Execution is everything”. I have nothing to fear in publishing this blog series, after all it is just my ideas.

What’s Next?

This is the last substantial post in this series. I have tried to break it up into digestible chunks. Thanks for taking the time to read my posts and I welcome any comments you may have.

I plan to let things sit for a few days. Over the weekend I plan to review the comments posted and emails received as part of preparing a summary response and conclusion. Anything else I think of will get tacked onto that post.

Dec 28 2010
Dec 28

Through out this series, the cost of labour has been identified as one of the biggest risks for this project. As most people who have run a tech business know, support can turn into a massive black hole of wasted time. Today we will look at how to manage support in a way that helps you avoid any direct customer contact for support.

Documentation and Online Resources

People like documentation, or even better videos, to walk them through a process. Invest the time up front to create documentation to help your customers use your platform. Only offer a single admin theme, such as Root Candy or Rubik or something you create yourself, this way the UI remains consistent and users are less likely to get confused. When developing the documentation, assume the user has no Drupal experience and talk in generic lay person terms, not “Drupal babble”. You should include links in the Drupal interface back to your help system, so people can access contextually appropriate help. Consider having a FAQ for each discrete piece of functionality so people can get some idea of what it does without having to play with it for half an hour.

A lot of the documentation you generate will be specific to your business and only available to your paying customers, but some of it will have generic value to the Drupal community. For the more generally useful content, publish it in a public place, such as your blog, company website or contribute it to the appropriate section of the documentation on drupal.org. You will be benefiting greatly from the work of the Drupal community, you should be looking for opportunities to give something back.

The flexibility of Drupal means that you should be able to build a documentation system that suits your needs. You probably want to start by extending the book module.

Forum

From time to time people won’t be able to find what they need using your online documentation. If you have a solid user base, a forum can be a good way of crowd sourcing your support resources. This doesn’t mean you can just put up a forum and hope that your users will all help each other solve their problems, you need to be in there responding in a timely manner too. If you have some users who are contributing regularly in the forums, acknowledge that in their profile and even offer them some free time on the service - after all they’re saving you money.

Support Tickets

Some users will want a more personalised approach to support. This will most likely involve email. When a user emails support directly, they will expect a timely response. This has the potential to add significant costs to your business, especially if you have a global customer base. One way to deal with this additional cost is to charge for it, offer packages of X incidents per month for Y dollars and put them on recurring billing. Most customers will forget to remove the recurring billing, but stop asking questions once they are up and running. There are various support systems available, I’ll leave you to find the one which works best for you.

Chat and Telephone Support

Real time support channels such as chat or telephone can really eat into your bottom line. You need to have people sitting there waiting to take the “call”, whatever time the customer contacts you and they may not call for a day. Accents and cultural issues can make it difficult to communicate effectively via the telephone. I would avoid offering any telephone support to customers.

Mailing Lists

A mailing list can be an effective way of communicating with your users. I’m not suggesting something like MailMan or Google Groups, I am talking about a one way announcement list. Check out MailChimp or Campaign Monitor to take care of this for you. Each month send out a newsletter with some useful tips, new features or announce other improvements to the service. Consider including case studies in the newsletter, which you can also feature on your sales site.

Customer Service

The cheapest way to build a business is through referrals or word of mouth. You need to look after your customers, so they become your sales team. Manage their expectations in terms of the support you offer them. Don’t be afraid to cut a time sink loose, apologise, give them a refund and encourage them to find another solution. Support vampires can cost you a lot more than they’re worth.

What’s Next?

We are almost in the home stretch now. In the next post I will cover some of the business considerations in terms of what you offer to customer and how to upsell. The final post in the series will come later this week and will be a summary of the previous posts.

Dec 28 2010
Dec 28

So far in this series we have covered a potential target market and business plan, resources and infrastructure and the tools required to deliver Drupal sites with a sale price of $100 per site. In this post I’ll be covering some of the considerations when building Drupal platforms or distributions.

The sites which customers deploy will need to be based on a custom Drupal distribution or “distro”. The distro should be modular and primarily driven by Features.

Customers shouldn’t have to know anything about administering Drupal when they first buy their site. A customer should be able to turn functionality on and off as they want, through a simple user interface.

Features

The platform should contain a good collection of Features. The following list is an example of what you might offer customers:

  • Contact Form
  • Image Gallery
  • Products
  • Services
  • “Static” Pages
  • Blog
  • News
  • Mailing Lists
  • Social Network Integration
  • Office / Store Locations
  • Staff Profiles

When developing your list of things to include in the site, think in terms of functionality a small business would want, not what modules you should be using. The list of modules should be derived from the functionality, not the other way around.

As the features included in the platform will be modular and generically useful, you should consider releasing them publicly, via your own features server or drupal.org as full modules.

On top of the features listed above you will probably need to include some custom glue code to enhance the user experience. In my first post in this series I discussed the target audience not having high level computer skills, so the user interface should take this into account. Some of the language might need to be changed or form options modified to use sane defaults and some might even be hidden from the user.

Security

As each server may have hundreds or even thousands of sites running on it, security will be an important consideration. Like with all servers you should ensure it is properly locked down and only running the services you need. Apache should be configured to block access to most things except index.php and relevant client side files (images, css, js) and the files directory. At the Drupal level you should make sure that things like the PHP module aren’t enabled and secure coding practices are adhered to. The user account given to your customer shouldn’t be user 1, they should be user 2, with restricted permissions that only gives them access to what they need.

I strongly recommend that you read Cracking Drupal by Greg Knaddison.

Sales and Support

In order to attract customers you will need a site to promote the service and allow customers to sign up and hand over their credit card details. Drupal now offers 3 ecommerce projects, Drupal e-Commerce, Drupal Commerce and ubercart. You should investigate which of these best suits your needs. The sales system will need some custom code to hook into Aegir, which will be managing the actual site deployments. The sales and support platform/s should be managed in a similar manner to the customer sites.

Once you have paying customers, you will also need to provide them with some resources such as detailed documentation, video walk throughs, forums and possibly a ticketing system. This site can either be part of the sales site or a separate site. In the next instalment I’ll cover support in more detail.

Deploying Platforms

We need to keep the whole process very automated, CPU cycles are a lot cheaper than workers. Building and deploying platforms should involve a few clicks or tweaking a configuration file. For example platforms could be built as Debian (or Ubuntu) packages (aka debs) using an automated build process that runs a test suite against the code base before building a deb. The debs could then be deployed using puppet and a post installation script can notify Aegir that it has been installed successfully. The whole process could involve very little human interaction. Migrating client sites to upgraded platforms could also be automated using a simple script which adds a migrate task for each site.

What’s Next?

Now that we have the service almost ready to go, we should look into how we are going to get customers to part with their cash and how we will support them once they have paid.

Dec 26 2010
Dec 26

In the previous instalment of my $100 Drupal site series I covered resources and infrastructure. In this post I will be covering the development tools I think you need in order to build and sell Drupal sites at the $100 price point. Given that Drupal 7 is close to release, it is assumed that the sites will be built using D7. I don’t believe that it is smart to invest heavily in Drupal 6 for new long term projects, given D8 could be out in 18 months and D6 would then be unsupported.

git

You are going to need a version control system for storing all of the code. There are many different version control systems available, but I think git is the most flexible and powerful option. Git allows for distributed development, offline commits and best of all, Drupal is switching to git. Gitorious is an open source clone of github which allows you to browse your git repository and manage integration of code from all of your developers.

If you are new to git I strongly recommend you get a dead tree copy of the book Pro Git.

drush

Drush is the DRUpal SHell, a command line interface for Drupal. It is an invaluable tool for developing and maintaining Drupal sites. Simple things like running “drush cc all” to clear all caches when developing sites, through to being able to sync databases from one server to another, or one continent to another, save many hours. Even if you’re only running a handful of sites, you should have drush installed on all of your servers.

Modules

There are a few modules which you should be using if you want to be able to create polished Drupal based platforms. My list includes:

  • Features module allows developers to package up configuration as a Drupal module, including access to all the normal hooks that a module can implement. Feature is version control friendly and includes the ability to reset to a known good state, either via the Drupal web GUI or by using drush on the command line.

  • Strongarm exports values from Drupal’s variables table, allowing sane defaults to be set. This means your site will always be configured the way you want it to be.

  • Context allows you to manage contextual conditions and reactions for different portions of your site. It gives you control over displaying blocks,hierarchy of breadcrumbs, themes and other context sensitive configuration options.

  • Admin is a menu module which makes it quick and easy to access items. (Disclaimer: I co-maintain the module)

  • Coder can perform static analysis of your code to ensure it is secure and complies with the Drupal coding standards. The module can also be used to upgrade code from one major version of Drupal to the next.

  • Devel is a collection of tools to help developers create Drupal sites and modules. Devel integrates with Admin to provide easy access to the devel actions.

  • WYSIWYG provides a pluggable framework for integrating rich text editors into a Drupal site. Given the target market for the $100 sites, we can’t expect our users to hand craft HTML, they’ll be expecting “something like Word”.

  • Skinr provides a way for end users to apply predefined CSS to designated parts of a site, without the user having to understand CSS.

Installation Profiles

When most people install Drupal for the first time they are blissfully unaware that they are using an installation profile. An installation profile sets out a base configuration for a site that is installed by Drupal. Until recently installation profiles contained a list of modules to install and a large chunk of hand crafted PHP code which setup all the configuration for a site. These days most of the configuration can live in Features and so an installation profile can be very lightweight. It is worth reviewing some of the more popular installation profiles, such as Open Atrium, Managing News or Drupal Commons, for inspiration. Installation profiles have changed a lot in Drupal 7, so you will need to port the ideas to the new way of doing things.

Drush Make

Drush Make is a tool for building Drupal platforms. Drush Make allows developers to specify all the components of their site in a text file, then use the command line tool to “make” the site. It is possible to specify a version of Drupal core (including Pressflow), contrib modules and themes, third party components, patches and external libraries.

Aegir

Aegir is a Drupal site deployment and management tool built using a collection of Drupal modules. With Aeigr you can manage your DNS, http (web) and database servers from a common UI. It is possible to move a bunch of sites from one server to another in a matter of minutes, not hours and the downtime will be measured in seconds. When security fixes are released, testing can involve a few minutes of work then a few more clicks to deploy it to all of your sites.

Workflow

Instead of me explaining development workflows, I’ll defer to Miguel Jacq (aka mig5). Miguel’s blog post entitled “Drupal deployments & workflows with version control, drush_make, and Aegir” is considered by many to be the key work on modern Drupal development workflows.

What’s Next

Many of the tools I’ve covered today should be in your Drupal developer’s tool bag. My next post will cover what I think are the important considerations in building the platforms for the service.

Dec 24 2010
Dec 24

In my previous post in this series on the $100 Drupal site I outlined a possible target market and set out why I thought very low cost sites could be a viable business model. Today I will cover the resources and infrastructure you’d need to consider to build such a service.

I am not proposing that the business is built on the premise of working for $5 per hour to build new sites for each client. The whole business should be automated. There should be no need for human interaction to close the sale and deploy the site. As soon as a worker is added to the process costs start to rise. At the same time we need some skilled professionals to build the platform and the cookie cutters.

Developers

Building a very basic Drupal site doesn’t involve that much skill, it is possible for someone to signup with a web host that uses an automated installer script, such as Fantastico, and then start building a site using stock Drupal core. Building great Drupal sites takes skills and knowledge. In order to build modular Drupal sites that can be used as part of installation profiles and deployed using automated processes, you will need someone who knows Drupal pretty well. The person driving development of the project should be passionate about Drupal as a platform. They are likely to have an opinion on the Drupal specific religious wars - Panels vs Context, CKEditor vs tinyMCE, which Slideshow / Carousel module to use etc.

Most competent PHP developers can be taught how to develop for Drupal. A second, junior, developer can be used to do a lot of the build work. They can start being a “click happy” developer and as their skills grow they can get their hands dirty writing glue modules and patches. This person should be good at driving the Drupal GUI and working with contrib.

Designer / Themer

Although customers will only be paying $100 for their site, they won’t be very impressed if they can only use Garland as their theme. They will expect a choice of themes. The themes should be designed from the ground up to be flexible Drupal themes. The designer should have experience creating designs that will be come Drupal themes. They should understand flexible regions, module themeing and how the Color module can be used to provide control to the user over their theme. Ideally the designer should be able to craft HTML and CSS, and even perform basic Drupal themeing.

System Administrator

Even though the plan is to keep costs low, it doesn’t make sense to run the $100 sites on cheap shared hosting. You need to spend money on your own solid hosting infrastructure, and if you have your own hosting infrastructure you’ll need someone to manage it. The sys admin will be responsible for keeping everything running, adding new servers, testing backups etc.

You will need a system administrator who has some experience with administering LAMP servers, previous Drupal experience is a big plus. You want someone in this role who hates downtime.

Infrastructure

One of the assumptions for this project is that 95% of the sites will have less than 50 page views per day. At first there will be very little traffic for the sites, so you can start off pretty small, but as the business grows you need to have a plan in place to handle that growth.

I wouldn’t recommend spending a lot of money on buying servers outright and putting them in a data centre. Start off with some virtual machines from a cloud hosting provider, such as slicehost, linode, Rackspace Cloud or Amazon AWS. By using cloud hosting providers you can scale your infrastructure up as you need.

To get started you will probably need 3 production virtual machines, one for serving up content, one for the DB and one for search using Apache Solr. Each of the VMs should have 1G of RAM and be located in the same data centre with the same hosting company. When things start to grow you can scale up the VMs as you need and even add more servers. Each time you scale things you will also have to retune your configuration. An opcode cache such as APC or xcache will allow you to serve more requests on the same hardware. You should include some kind of monitoring service such as nagios or zabbix to make sure things are running as you want.

Once you have more than a handful of machines to manage, you will want a centralised configuration management tool. This will allow you to add new servers and have them configured as you want them with no (or very little) human intervention. I would recommend puppet for this job. Given the effort involved in retro fitting puppet in an existing environment, it will be easier to add it from the start. You may want to consider using libcloud to even handle the deployment of the VMs too.

To ensure everything can be tested properly before deployment, you should seriously consider having a replicated testing environment. Instead of having 3 more VMs, you can just run up a Linux box with 4G RAM and a relatively modern CPU and run 3 VMs on it using KVM or Xen.

What’s Next?

Now we have the business plan, the market, the team and servers in place, it is time to start looking at some of the tools we’ll be using to make all of this possible. The next post in this $100 Drupal site series will cover the tools needed to build a sustainable platform for the business.

Dec 24 2010
Dec 24

In late October Gdzine posed the question “$100 CMS web site feasible? What do you think?” on LinkedIn and the question was also posted on groups.drupal.org. These posts lead to lengthy discussion threads. Some people accused Gdzine of trolling and others claimed that it wasn’t possible, but a few of us argued it was possible to build a Drupal site for $100.

Over the next week or so I’ll be blogging how I would go about delivering $100 Drupal sites. $100 is in United States Dollars. I won’t be providing a complete blueprint, but there should be enough information to help get you started.

I have some experience building large numbers of production sites using Drupal for a small price per site. In 2009 I built, deployed and managed 2086 sites for a European client. For most of this year I have been offering training and consulting around some of the tools and techniques this blog series will cover.

Is it Feasible?

Several companies already offer low cost Drupal site solutions. Acquia’s Drupal Gardens is probably the most well known cheap Drupal site building service with its freemium model. Wedful offers a Drupal based site for couples getting married for only 95USD for the first year, then 25USD every following year including hosting. Spoon Media’s Pagebuild service offers a customised Drupal platform for 30USD per month. I am sure there are others operating in the same space. wordpress.com offers a similar service using WordPress. I have no idea how financially viable these businesses are, but I think it is safe to assume that they’ve done some research and planning to get to this point.

These services all rely on making their money on the turnover rather than the margin. Most consultants, myself included, make our money by charging a good hourly rate, but we only get paid for the time we work. These services rely on investing up front then waiting for the long tail revenue. For example if you invest 50,000USD up front into building the service, then you have to sell 500 sites just to break even.

Target Market

In order to make these services viable, you have to target a particular market segment. Customers for a $100 site are likely to be “mum and dad” businesses, they don’t have a lot of money to spend and are also unlikely to have a lot knowledge about the web. A lot of the customers are likely to think “the internet” is that blue e on their desktop or Facebook. I know of several small businesses who think that Yellow Pages advertising is not giving the return on investment they want, but can’t afford 1500-2000USD for a decent quality brochure site. These are the people this service should be seeking to attract.

Why Bother?

Most Drupal developers aspire to work for switched on clients who want high quality sites. Working with well known brands is always a bonus. No one is really going to be interested in hearing about how you built the site for “Joe and Jo’s Diner”. There is still a lot of problem solving involved in building a service like this, but these problems are very different to those found in large scale site builds. Also many of the people seeking a $100 site are likely to be high needs clients who undervalue the skills involved in building a site.

What’s Next?

All posts in this series will be tagged with “$100 drupal site”. In my next post I will cover what I think you need in terms of infrastructure and resources to make something like this work.

I have proposed a session for DrupalCon Chicago on this topic, please consider voting for it.

Aug 08 2010
Aug 08

As a platform, Drupal has excellent javascript support. Drupal 7 will ship with jQuery 1.4.2 and jQuery UI 1.8, which will make it even easier to build rich user interactions with Drupal.

Drupal supports aggregating javascript files to reduce the number of network connections a browser must open to load a page. It is common practice for Drupal themes to put the

Jul 22 2010
Jul 22

The next couple of months are going to be a crazy ride. I will be visiting at least 7 countries, speaking on 8 or more days in a 5 week period. The talks will be focused on Drupal and Aegir. My schedule is below.

Horizontally Scaling Drupal - Melbourne

On 7 August I’ll be running a 1 day workshop around the theme of horizontally scaling Drupal. The content is built on the knowledge I developed building, deploying and managing around 2100 sites for a client. This event has very limited capacity and has almost sold out.

DrupalCon - Denmark

Denmark is hosting the European leg of DrupalCon this year. I will be attending the full conference. I won’t be presenting, but I will be getting involved with some of the BoFs. I had a ball at DrupalCon San Francisco earlier in the year.

Efficiently Managing Many Drupal Sites - Slovakia

After spending a couple of days recovering from DrupalCon, I’ll be teaming up with the crew at Sven Creative in Bratslavia, to run a 2 day intensive workshop on horizontally scaling Drupal and development workflows. For more information check out the workshop website.

Free Software Balkans - Albania

On the weekend of 11-12 September, the inaugural Free Software Balkans Conference will be held at the University of Vlore, Albania. I’ll be there speaking about Drupal and Aegir. In addition to this I will be running half day build your first Drupal site workshops around the country. The dates and locations for the workshops are still being finalised.

OSI Days - India

On my way back to Australia I will be taking a side trip to Chennai, via Delhi, for OSI Days 2010, Asia’s largest open source conference. I will be presenting sessions on Aegir and Drupal. This looks like it will be a huge event.

Other Events

I’ve launched a new site workshops.davehall.com.au to list my training and speaking engagements. As dates are locked in I’ll be adding them to the site.

If you would like to meet with me while I’m on the road, add me to your tripit network, follow me on identi.ca or twitter or add me to your network on LinkedIn.

Jun 28 2010
Jun 28

Apache Solr is an excellent full text index search engine based on Lucene. Solr is increasingly being used in the Drupal community for search. I use it for search for a lot of my projects. Recently Steve Edwards at Drupal Connect blogged about setting up a mutli core Solr server on Ubuntu 9.10 (aka Karmic). Ubuntu 10.04LTS was released a couple of months ago and it makes the process a bit easier, as Apache Solr 1.4 has been packaged. An additional advantage of using 10.04LTS is that it is supported until April 2015, whereas support for 9.10 ends in 10 months - April 2011.

As an added bonus in this howto you will be able to auto provision solr cores just by calling the right URL.

In this tutorial I will be using Jetty rather than tomcat which some tutorials recommend, as Jetty performs well and generally uses less resources.

Install Solr and Jetty

Installing jetty and Solr just requires a single command: sudo apt-get install solr-jetty openjdk-6-jdk.

This will pull down Solr and all of the dependencies, which can be alot if you have a very stripped down basic server.

Configuring Jetty

Configuring Jetty is very straight forward. First we backup the existing /etc/default/jetty file by running sudo cp -a /etc/default/jetty /etc/default/jetty.bak

Then change your /etc/default/jetty to be like this (the changes are highlighted):

# Defaults for jetty see /etc/init.d/jetty for more

# change to 0 to allow Jetty to start
NO_START=0
#NO_START=1

# change to 'no' or uncomment to use the default setting in /etc/default/rcS
VERBOSE=yes

# Run Jetty as this user ID (default: jetty)
# Set this to an empty string to prevent Jetty from starting automatically
#JETTY_USER=jetty

# Listen to connections from this network host (leave empty to accept all connections)
#Uncomment to restrict access to localhost
#JETTY_HOST=$(uname -n)
JETTY_HOST=solr.example.com

# The network port used by Jetty
#JETTY_PORT=8080

# Timeout in seconds for the shutdown of all webapps
#JETTY_SHUTDOWN=30

# Additional arguments to pass to Jetty
#JETTY_ARGS=

# Extra options to pass to the JVM
#JAVA_OPTIONS="-Xmx256m -Djava.awt.headless=true"

# Home of Java installation.
#JAVA_HOME=

# The first existing directory is used for JAVA_HOME (if JAVA_HOME is not
# defined in /etc/default/jetty). Should contain a list of space separated directories.
#JDK_DIRS="/usr/lib/jvm/default-java /usr/lib/jvm/java-6-sun"

# Java compiler to use for translating JavaServer Pages (JSPs). You can use all
# compilers that are accepted by Ant's build.compiler property.
#JSP_COMPILER=jikes

# Jetty uses a directory to store temporary files like unpacked webapps
#JETTY_TMP=/var/cache/jetty

# Jetty uses a config file to setup its boot classpath
#JETTY_START_CONFIG=/etc/jetty/start.config

# Default for number of days to keep old log files in /var/log/jetty/
#LOGFILE_DAYS=14

If you don’t include the JETTY_HOST entry Jetty will only bind to the local loopback interface, which is all you need if your drupal webserver is running on the same machine. If you set the JETTY_HOST make sure you configure your firewall to restrict access to the Solr server.

Configuring Solr

I am assuming you have already installed the Apache Solr module for Drupal somewhere. If you haven’t, do that now, as you will need some config files which ship with it.

First we enable the multicore support in Solr by creating a file called /usr/share/solr/solr.xml with the following contents:

persistent="true" sharedLib="lib">
  adminPath="/admin/cores" shareSchema="true" adminHandler="au.com.davehall.solr.plugins.SolrCoreAdminHandler">
  

You need to make sure the file is owned by the jetty user if you want it to be dynamically updated, otherwise change persistent="true" to persistent="false", don’t include the adminHandler attribute and don’t run the commands below. Also if you want to auto provision cores you will need to download the jar file attached to this post and drop it into the /usr/share/solr/lib directory (which you’ll need to create).

sudo chown jetty:jetty /usr/share/solr
sudo chown jetty:jetty /usr/share/solr/solr.xml
sudo chmod 640 /usr/share/solr/solr.xml
sudo mkdir /usr/share/solr/cores
sudo chown jetty:jetty /usr/share/solr/cores

To keep your configuration centralised, symlink the file from /usr/share/solr to /etc/solr. Do this by calling sudo ln -s /usr/share/solr/solr.xml /etc/solr/. Don’t do it the other way, Solr will ignore the symlink.

Solr needs to be configured for Drupal. First we backup the existing config file, just in case, like so:

sudo mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.orig.xml
sudo mv /etc/solr/conf/solrconfig.xml /etc/solr/conf/solrconfig.orig.xml

Now we copy the Drupal Solr config files from where you installed the module by running

sudo cp /path/to/drupal-install/sites/all/modules/contrib/apachesolr/{schema,solrconfig}.xml /etc/solr/conf/

Solr needs the path to exist for each core’s data files, so we create them with the following commands:

sudo mkdir -p /var/lib/solr/cores/{,subdomain_}example_com/{data,conf}
sudo chown -R jetty:jetty /var/lib/solr/cores/{,subdomain_}example_com

Each of the cores need their own configuration files. We could implement some hacks to use a common set of configuration files, but that will make life more difficult if we ever have to migrate some of cores. Just copy the common configuration for all the cores:

sudo bash -c 'for core in /var/lib/solr/cores/*; do cp -a /etc/solr/conf/ $core/; done'

If everything is configured correctly, we should just be able to start Jetty like so:

sudo /etc/init.d/jetty start

If you visit http://solr.example.com:8080/solr/admin/cores?action=STATUS you should get some xml that looks something like this:



  name="responseHeader">
      name="status">0
      name="QTime">0
  
  name="status"/>

If you get the above output everything is working properly

If you enabled auto provisioning of Solr cores, you should now be able to create your first core. Point your browser at http://solr.example.com:8080/solr/admin/cores?action=CREATE&name=test1&instanceDir=test1 If it works you should get output similar to the following:



  name="responseHeader">
      name="status">0
      name="QTime">1561
  
  name="core">test1
  name="saved">/usr/share/solr/solr.xml

I would recommend using identifiable names for your cores, so for davehall.com.au I would call the core, “davehall_com_au” so I can easily find it later on.

Security Note: As anyone who can access your server can now provision solr cores, make sure you restrict access to port 8080 to only allow access from trusted IP addresses.

For more information on the commands available, refer to the Solr Core Admin API documenation on the Solr wik.

Next in this series will be how to use this auto provisioning setup to allow aegir to provision Solr cores as sites are created.

Apr 17 2010
Apr 17

My experience themeing Drupal, like most of my coding skills, have been developed by digging up useful resources online and some trail and error. I have an interest in graphic design, but never really studied it. I can turn out sites which look good, but my “designs” don’t have the polish of a professionally designed site. I own quite a few (dead tree) books on development and project management. Generally I like to read when I am sick of sitting in front of a screen. The only ebooks I consider reading are short ones.

Emma Jane Hogbin offered her Drupal theming ebook Theming Drupal: A First Timer’s Guide to her mailing list subscribers for free. I am not a big fan of vendor mailing lists, most of the time I scan the messages and hit delete before the bottom. In the case of Emma, rumour has it that it is really worthwhile to subscribe to her list - especially if you are a designer interested in themeing Drupal. Emma also offered free copies of her ebook to those who begged, so I subscribed and I begged.

The first thing I noticed about the book was the ducks on the front cover, I’m a sucker for cute animal pics. The ebook is derived from Emma’s training courses and the book she coauthored with Konstantin Kaefer, Front End Drupal. Readers are assumed to have some experience with HTML, CSS and PHP. The book is pitched at designers and programmers who want to get into building themes for Drupal.

The reader is walked through building a complete Drupal theme. The writing is detailed and includes loads of references for obtaining additional information. It covers building a page theme, content type specific themeing and the various base themes available for Druapl. The book is a very useful resource for anyone working on a Drupal theme.

Although I have themed quite a few Drupal sites, Emma’s guide taught me a few things. The book is a good read for anyone who wants to improve their knowledge of Drupal themeing. Now to finish reading Front End Drupal …

Apr 09 2010
Apr 09

Today I purchased a Motorola DEXT (aka Cliq) from Optus. Overall I like it. It feels more polished than the Nokia N97 which I bought last year. The range of apps is good. Even though the phone only ships with Android 1.6, 2.1 for the DEXT is due in Q3 2010.

The apps seem to run nice and fast. The responsive touch screen is bright and clear. I am yet to try to make a call on it from home, but the 3G data seems as fast as my Telstra 3G service, so the signal should be OK.

The keyboard is very functional, albeit cramped with my fat thumbs. The home screen is a little cluttered for my liking too, but it won’t take much to clean that up. I will miss my funambol sync, which is only available for Android 2.x.

I started writing this post using the Drupal Editor for Android app, which is pretty nice. The GPL app uses the XML-RPC and Drupal core’s Blog API module. Overall it feels like a stripped down version of Bilbo/Blogilo. Drupal Editor is an example of an app which does one thing and does it simply, but well. The only thing I haven’t liked about it was when originally writing this post, I bumped the save button and published an incomplete and poorly written post. Next time I will untick the publish checkbox until I am ready to really publish it.

I would still like a HTC Desire, but Telstra is only offering them on a $65 plan with no value. The Nokia N900 was off my list, due to the USB port of death and Nokia’s spam policies. The Nexus One was on the list too, but the lack of a local warranty was a concern.

Mar 13 2010
Mar 13

I use Apache Solr for search on several projects, including a few with Drupal. Solr has built in support for replication and load balancing, unfortunately the load balancing is done on the client side and works best when using a persistent connection, which doesn’t make a lot of sense for PHP based webapps. In the case of Drupal, there has been a long discussion on a patch in the issue queue to enable Solr’s native load balancing, but things seem to have stalled.

In one instance I have Solr replicating from the primary to a secondary, with the plan to add additional secondary backends if the load justifies it. In order to get Drupal to write to the primary and read from either node I needed a proxy or load balancer. In my case the best lightweight http load balancer that would easily run on the web heads was haproxy. I could have run varnish in front of Solr and had it do the load balancing but that seemed like overkill at this stage.

Now when an update request hits HAProxy it directs it to the primary, but for reads it balances the requests between the 2 nodes. To get this setup running on ubuntu 9.10 with HAProxy 1.3.18, I used the following /etc/haproxy/haproxy.cfg on each of the web heads:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    maxconn 4096
    nbproc 4
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    maxconn 2000
    balance roundrobin
    stats enable
    stats uri /haproxy?stats

frontend solr_lb
    bind localhost:8080
    acl primary_methods method POST DELETE PUT
    use_backend primary_backend if primary_methods
    default_backend read_backends

backend primary_backend
    server solr-a 192.168.201.161:8080 weight 1 maxconn 512 check

backend secondary_backend
    server solr-b 192.168.201.162:8080 weight 1 maxconn 512 check

backend read_backends
    server solr-a 192.168.201.161:8080 weight 1 maxconn 512 check
    server solr-b 192.168.201.162:8080 weight 1 maxconn 512 check

To ensure the configuration is working properly run wget http://localhost:8080/solr -O - on each of the web heads. If you get a connection refused message HAProxy may not be running. If you get a 503 error make sure solr/jetty/tomcat is running on the Solr nodes. If you get some html output which mentions Solr, then it should be working properly.

For Drupal’s apachesolr module to use this configuration, set the hostname to localhost and the port to 8080 in the module configuration page. Rebuild your search index and you should be right to go.

If you had a lot of index updates then you could consider making the primary write only and having 2 read only secondary back ends, just change the IP addresses to point to the right hosts.

For more information on Solr replication refer to the Solr wiki, for more information on configuring HAProxy refer to the manual. Thanks to Joe William and his blog post on load balancing CouchDB using haproxy which helped me get the configuration I needed after I decided what I wanted.

Feb 22 2010
Feb 22

When you run a lot of Drupal sites it can be annoying to keep track of all of the modules contained in a platform and ensure all of them are up to date. One option is to setup a dummy site setup with all the modules installed and email notifications enabled, this is OK, but then you need to make sure you enable the additional modules every time you add something to your platform.

I wanted to be able to check the status of all of the modules in a given platform using the command line. I started scratching the itch by writing a simple shell script to use the Drupal updates server to check for the status of all the modules. I kept on polishing it until I was happy with it, there are some bits of which are a little bit ugly, but that is mostly due to the limitations of bash. If I had to rewrite the it I would do it in PHP or some other language which understands arrays/lists and has http client and XML libraries.

The script supports excluding modules by using an extended grep regular expression pattern and nominating a major version of Drupal. When there is a version mismatch it will be shown in bolded red, while modules where the versions match will be shown in green. The script filters out all dev and alpha releases, after all the script is designed for checking production sites. Adding support for per module update servers should be pretty easy to do, but I don’t have modules to test this with.

To use the script, download it, save it somewhere handy, such as ~/bin/check-module-status.sh, make it executable (run chmod +x ~/bin/check-module-status.sh). Now it is ready for you to run it - ~/bin/check-module-status.sh /path/to/drupal and wait for the output.

#
# Check the latest version of a drupal module / template
#
# Written by Dave Hall http://davehall.com.au
# Copyright (c) 2010 Dave Hall Consulting http://davehall.com.au
# Licensed under the terms of the GPLv3 http://www.gnu.org/licenses/gpl.html
#

BASE_URL='http://updates.drupal.org/release-history/'

if [[ 0 -eq $# || 3 -lt $# ]]; then
  echo Usage: $(basename $0) /path/to/drupal-instance [[exclude-pattern] drupal-version]
  exit 1
fi

DRUPAL_PATH=$1

if [ ! -d "$DRUPAL_PATH" ]; then
  echo ERROR: Invalid path - $DRUPAL_PATH
  exit 2
fi

EXCLUDE_PATTERN='(drupal)'
if [ 2 -lt $# ]; then
  EXCLUDE_PATTERN=$2
fi

DRUPAL_VERSION=6.x
if [ 3 -eq $# ]; then
  DRUPAL_VERSION=$3
fi

INFO_FILES=$(find $DRUPAL_PATH -name *.info)

modules=( )
for info in $INFO_FILES; do
  project=$(sed -e '/project =/!d' -e 's/project = "\(.*\)"/\1/g' < $info | head -n1)
  if [[ "" == "$project" || 0 -eq $(echo $project | egrep -cv $EXCLUDE_PATTERN) ]]; then
    continue
  fi
  cur_ver=$(sed -e '/version =/!d' -e 's/version = "\(.*\)"/\1/g' < $info | tail -n1)
  modules=( "${modules[@]}" "$project|$cur_ver##" )
done

modules=$(echo ${modules[@]} | tr '##' '\n' | sort -u)

echo -e "module\t\tlocal\tcurrent"
for mod in $modules; do
  color='\e[0;32m'
  new_ver=$(wget -O - http://updates.drupal.org/release-history/`echo $mod | cut -d\| -f1`/$DRUPAL_VERSION 2> /dev/null | sed -e '/<version>.*<\/version>/!d' -e '/\(dev\|alpha\)/d' -e 's/.*<version>\(.*\)<\/version>/\1/g' | head -n1 );
  if [ "$(echo $mod | cut -d\| -f2)" != "$new_ver" ]; then
    color='\e[1;31m'
  fi
  echo -e $color$mod\|$new_ver | tr '|' '\t'
done
Feb 04 2010
Feb 04

Lately I have been trying to avoid non packaged software being installed on production servers. The main reason for this is to make it easier to apply updates. It also makes it easier to deploy new servers with meta packages when everything is pre packaged.

One tool which I am using a lot on production servers is Drupal’s command line tool - drush. Drush is awesome it makes managing Drupal sites so much easier, especially when it comes to applying updates. Drush is packaged for Debian testing, unstable and lenny backports by Antoine Beaupré (aka anarcat) and will be available in universe for ubuntu lucid. Drush depends on PEAR’s Console_Table module and includes some code which automagically installs the dependency from PEAR CVS. The Debianised package includes the PEAR class in the package, which is handy, but if you are building your own debs from CVS or the nightly tarballs, the dependency isn’t included. The auto installer only works if it can write to /path/to/drush/includes, which in these cases means calling drush as root, otherwise it spews a few errors about not being able to write the file then dies.

A more packaging friendly approach would be to build a Debian package for PEAR Console_Table and have that as a dependency of the drush package in Debian. The problem with this approach is that drush currently only looks in /path/to/drush/includes for the PEAR class. I have submitted a patch which first checks if Table_Console has been installed via the PEAR installer (or other package management tool). Combine this with the Debian source package I have created for Table_Console (see the file attached at the bottom of the post), you can have a modular and apt managed instance of drush, without having to duplicate code.

I have discussed this approach with anarcat, he is supportive and hopefully it will be the approach adopted for drush 3.0.

Update The drush patch has been committed and should be included in 3.0alpha2.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web