Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jun 23 2016
Jun 23

These days, it’s pretty rare that we build websites that aren’t some kind of redesign. Unless it’s a brand new company or project, the client usually has some sort of web presence already, and for one reason or another, they’ve decided to replace it with something shiny and new.

In an ideal world, the existing system has been built in a sensible way, with a sound content strategy and good separation of concerns, so all you need to do is re-skin it. In the Drupal world, this would normally mean a new theme, or if we’re still in our dream Zen Garden scenario, just some new CSS.

However, the reality is usually different. In my experience, redesigns are hardly ever just redesigns. When a business is considering significant changes to the website like some form of re-branding or refresh, it’s also an opportunity to think about changing the content, or the information architecture, or some aspects of the website functionality. After all, if you’re spending time and money changing how your website looks, you might as well try to improve the way it works while you’re at it.

So the chances are that your redesign project will need to change more than just the theme, but if you’re unlucky, someone somewhere further back along the chain has decided that it’s ‘just a re-skinning’, and therefore it should be a trivial job, which shouldn’t take long. In the worst case scenario, someone has given the client the impression that the site just needs a new coat of paint, but you’re actually inheriting some kind of nasty mess with unstable foundations that should really be fixed before you even think about changing how it looks. Incidentally, this is one reason why sales people should always consult with technical people who’ve seen under the bonnet of the system in question before agreeing prices on anything.

Even if the redesign is relatively straightforward from a technical point of view, perhaps it’s part of a wider rebranding, and there are associated campaigns whose dates are already expensively fixed, but thinking about the size of the website redesign project happened too late.

In other words, for whatever reason, it’s not unlikely that redesign projects will find themselves behind schedule, or over budget - what should you do in this situation? The received agile wisdom is that time and resources are fixed, so you need to flex on scope. But what’s the minimum viable product for a redesign? When you’ve got an existing product, how much of it do you need to rework before you put the new design live?

This is a question that I’m currently considering from a couple of angles. In the case of one of my personal projects, I’m upgrading an art gallery listings site from Drupal 6 to Drupal 8. The old site is the first big Drupal site I built, and is looking a little creaky in places. The design isn’t responsive, and the content editing experience leaves something to be desired. However, some of the contributed modules don’t have Drupal 8 versions yet, and I won’t have time to do the work involved to help get those modules ready, on top of the content migration, the new theme, having a full-time job and a family life, and all the rest of it.

In my day job, I’m working with a large multinational client on a set of sites where there’s no Drupal upgrade involved, but the suggested design does include some functional changes, so it isn’t just a re-theming. The difficulty here is that the client wants a broader scope of change than the timescales and budget allow.

When you’re in this situation, what can you do? As usual with interesting questions, the answer is ‘it depends’. Processes like impact mapping can help you to figure out the benefits that you get from your redesign. If you’ve looked at your burndown rates, and know that you’re not going to hit the deadline, what can you drop? Is the value that you gain from your redesign worth ditching any of the features that won’t be ready? To put it another way, how many of your existing features are worth keeping? A redesign can (and should) be an opportunity for a business to look at their content strategy and consider rationalising the site. If you’ve got a section on your site that isn’t adding any value, or isn’t getting any traffic, and the development team will need to spend time making it work in the new design, perhaps that’s a candidate for the chop?

We should also consider the Pareto principle when we’re structuring our development work, and start by working on the things that will get us most of the way there. This fits in with an important point made by scrum, which can sometimes get forgotten about: that each sprint should deliver “a potentially shippable increment”. In this context, I would interpret this to mean that we should make sure that the site as a whole doesn’t look broken, and then we can layer on the fancy bits afterwards, similar to a progressive enhancement approach to dealing with older browsers. If you aren’t sure whether you’ll have time to get everything done, don’t spend an excessive amount of time polishing one section of the site to the detriment of basic layout and styling that will make the whole site look reasonably good.

Starting with a style guide can help give you a solid foundation to build upon, by enabling you to make sure that all the components on the site look presentable. You can then test those components in their real contexts. If you’ve done any kind of content audit (and somebody really should have done), you should have a good idea of the variety of pages you’ve got. At the very least, your CMS should help you to know what types of content you have, so that you can take a sample set of pages of each content type or layout type, and you’ll be able to validate that they look good enough, whatever that means in your context.

There is another option, though. You don’t have to deliver all the change at once. Can you (and should you) do a partial go-live with a redesign? Depending on how radical the redesign is, the attitudes to change and continuous delivery within your organisation and client, and the technology stack involved, it may make sense to deliver changes incrementally. In other words, put the new sections of the site live as they’re ready, and keep serving the old bits from the existing system. There may be brand consistency, user experience, and content management process reasons why you might not want to do this, but it is an option to consider, and it can work.

On one previous project, we were carrying out a simultaneous redesign and Drupal 6 to 7 upgrade, and we were able to split traffic between the old site and the new one. It made things a little bit more complicated in terms of handling user sessions, but it did give the client the freedom to decide when they thought we had enough of the new site for them to put it live. In the end, they decided that the answer was ‘almost all of it’.

So what’s the way forward?

In the case of my art gallery listings site, the redesign itself has a clear value, and with Drupal 6 being unsupported, I need to get the site onto Drupal 8 sooner rather than later. There’s definitely a point that will come fairly soon, even if I don’t get to spend as long as I’d like working on it, where the user experience will be improved by the new site, even though some of the functionality from the old site isn’t there, and isn’t likely to be ready for a while. I’m my own client on that project, so I’m tempted to just put the redesign live anyway.

In the case of my client, there are decisions to be made about which of the new features need to be included in the redesign. De-scoping some of the more complex changes will bring the project back into the realm of being a re-theming, the functional changes can go into subsequent releases, and hopefully we’ll hit the deadline.

A final point that I’d like to make is that we shouldn’t fall into the trap of thinking of redesigns as big-bang events that sit outside the day-to-day running of a site. Similarly, if you’re thinking about painting your house, you should also think about whether you also need to fix the roof, and when you’re going to schedule the cleaning. Once the painting is done, you’ll still be living there, and you’ll have the opportunity to do other jobs if and when you have the time, energy, and money to do so.

Along with software upgrades, redesigns should be considered as part of a business’s long-term strategy, and they should be just one part of a plan to keep making improvements through continuous delivery.

May 14 2015
May 14

As Drupal has evolved, it has become more than just a CMS. It is now a fully fledged Web Development Platform, enabling not just sophisticated content management and digital marketing capabilities but also any number of use cases involving data modelling and integration with an endless variety of applications and services. In fact, if you need to build something which responds to an HTTP request, then you can pretty much find a way to do it in Drupal.

“Just because you can, doesn’t mean you should.”

However, the old adage is true. Just because you can use use a sledgehammer to crack a nut, that doesn’t mean you’re going to get the optimal nut-consumption-experience at the end of it.

Drupal’s flexibility can lead to a number of different integration approaches, all of which will “work”, but some will give better experiences than others.

On the well trodden development path of Drupal 8, giant steps have been taken in making the best of what is outside of the Drupal community and “getting off the island”, and exciting things are happening in making Drupal less of a sledgehammer, and more of a finely tuned nutcracker capable of cracking a variety of different nuts with ease.

In this post, I want to explore ways in which Drupal can create complex systems, and some general patterns for doing so. You’ll see a general progression in line with that of the Drupal community in general. We’ll go from doing everything in Drupal, to making the most of external services. No option is more “right” than others, but considering all the options can help make sure you pick the approach that is right for you and your use case.

Build it in Drupal

One option, and probably the first that occurs to many developers, is to implement business logic, data structures and administration of a new applications or services using Drupal and its APIs. After all, Entity API and the schema system give us the ability to model custom objects and store them in the Drupal database; Views gives us the means to retrieve that data and display it in a myriad of ways. Modules like Rules; Features and CTools provide extensive options for implementing specific business rules to model your domain specific data and application needs.

This is all well and good, and uses the strengths of Drupal core and the wide range of community contributed modules to enable the construction of complex sites with limited amounts of coding required, and little need to look outside Drupal. The downside can come when you need to scale the solution. Depending on how the functionality has been implemented you could run into performance problems caused by large numbers of modules, sub-optimal queries, or simply the amount of traffic heading to your database - which despite caching strategies, tuning and clustering is always likely to end up being the performance bottleneck of your Drupal site.

It also means your implementation is tightly coupled to Drupal - and worse, most probably the specific version of Drupal you’ve implemented. With Drupal 8 imminent this means you’re most likely increasing the amount of re-work required when you come to upgrade or migrate between versions.

It’s all PHP

Drupal sites can benefit hugely from being part of the larger PHP ecosystem. With Drush make, the Libraries API, Composer Manager, and others providing the means of pulling external, non-Drupal PHP libraries into a Drupal site, there are huge opportunities for building complexity in your Drupal solution without tying yourself to specific Drupal versions, or even to Drupal at all. This could become particularly valuable as we enter the transition period between Drupal 7 and 8.

In this scenario, custom business logic can be provided in a framework agnostic PHP library and a Naked Module approach can be used to provide the glue between that library and Drupal - utilising Composer to download and install dependencies.

This approach is becoming more and more widespread in the Drupal community with Commerce Guys (among others) taking a libraries first approach to many components of Commerce 2.x which will have generic application outside of Drupal Commerce.

The major advantage of building framework agnostic libraries is that if you ever come to re-implement something in another framework, or a new version of Drupal, the effort of migrating should be much lower.

Integrate

Building on the previous two patterns, one of Drupal’s great strengths is how easy it is to integrate with other platforms and technologies. This gives us great opportunity to implement functionality in the most appropriate technology and then simply connect to it via web services or other means.

This can be particularly useful when integrating with “internal” services - services that you don’t intend to expose to the general public (but may still be external in the sense of being SaaS platforms or other partners in a multi-supplier ecosystem). It is also a useful way to start using Drupal as a new part of your ecosystem, consuming existing services and presenting them through Drupal to minimise the amount of architectural change taking place at one time.

Building a solution in this componentised and integrated manner gives several advantages:

  • Separation of concerns - the development, deployment and management of the service can be run by a completely separate team working in a different bounded context. It also ensures logic is nicely encapsulated and can be changed without requiring multiple front-end changes.
  • Horizontal scalability - implementing services in alternate technologies lets us pick the most appropriate for scalability and resilience.
  • Reduce complex computation taking place in the web tier and let Drupal focus on delivering top quality web experience to users. For example, rather than having Drupal publish and transform data to an external platform, push the raw data into a queue which can be consumed by “non-Drupal” processes to do the transform and send.
  • Enable re-use of business logic outside of the web tier, on other platforms or with alternative front ends.

Nearly-Headless Drupal

Headless Drupal is a phrase that has gained a lot of momentum in the Drupal community - the basic concept being that Drupal purely responds with RESTful endpoints, and completely independant front-end code using frameworks such as Angular.js is used to render the data and completely separate content from presentation.

Personally, I prefer to think of a “nearly headless” approach - where Drupal is still responsible for the initial instantiation of the page, and a framework like Angular is used to control the dynamic portion of the page. This lets Drupal manage the things it’s good at, like menus, page layout and content management, whilst the “app” part is dropped into the page as another re-usable component and only takes over a part of the page.

For an example use case, you may have business requirements to provide data from a service which is also provided as an API for consumption by external parties or mobile apps. Rather than building this service in Drupal, which while possible may not provide optimal performance and management opportunities, this could be implemented as a standalone service which is called by Drupal as just another consumer of the API.

From an Angular.js (or insert frontend framework of choice) app, you would then talk directly to the API, rendering the responses dynamically on the front end, but still use Drupal to build everything and render the remaining elements of the page.

Summing up

As we’ve seen, Drupal is an incredibly powerful solution, providing the capability for highly-consolidated architectures encapsulated in a single tool, a perfect enabler for projects with low resources and rapid development timescales. It’s also able to take its place as a mature part of an enterprise architecture, with integration capabilities and rich programming APIs able to make it the hub of a Microservices or Service Oriented Architecture.

Each pattern has pros and cons, and what is “right” will vary from project to project. What is certain though, is that Drupal’s true strength is in its ability to play well with others and do so to deliver first class digital experiences.

New features in Drupal 8 will only continue to make this the case, with more tools in core to provide the ability to build rich applications, RESTful APIs for entities out of the box allowing consumption of that data on other platforms (or in a headless front-end), improved HTTP request handling with Guzzle improving options for consuming services outside of Drupal, and much more.

Feb 27 2015
Feb 27

There are thousands of situations in which you do not want to reinvent the wheel. It is a well known principle in Software Engineering, but not always well applied/known into the Drupal world.

Let’s say for example, that you have a url that you want to convert from relative to absolute. It is a typical scenario when you are working with Web (but not just Web) crawlers. Well, you could start building your own library to achieve the functionality you are looking for, packaging all in a Drupal module format. It is an interesting challenge indeed but, unless for training or learning purposes, why wasting your time when someone else has already done it instead of just focussing on the real problem? Especially if your main app purpose is not that secondary problem (the url converter).

What’s more, if you reuse libraries and open source code, you’ll probably find yourself in the situation in which you could need an small improvement in that nice library you are using. Contributing your changes back you are closing the circle of the open source, the reason why the open source is here to stay and conquer the world (diabolical laugh here).

That’s another one of the main reasons why lot’s of projects are moving to the Composer/Symfony binomium, stop working as isolated projects and start working as global projects that can share code and knowledge between many other projects. It’s a pattern followed by Drupal, to name but one, and also by projects like like phpBB, ezPublish, Laravel, Magento,Piwik, …

Composer and friends

Coming back to our crawler and the de-relativizer library that we are going to need, at this point we get to know Composer. Composer is a great tool for using third party libraries and, of course, for contributing back those of your own. In our web crawler example, net_url2 does a the job just beautifully.

Nice, but at this point you must be wondering… What does this have to do with Drupal, if any at all? Well, in fact, as everyone knows, Drupal 8 is being (re)built following this same principle (DRY or don’t repeat yourself) with an strong presence of the great Symfony 2 components in the core. Advantages? Lots of them, as we were pointing out, but that’s the purpose of another discussion

The point here is that you don’t need to wait for Drupal 8, and what’s more, you can start applying some of this principles in your Drupal 7 libraries, making your future transition to Drupal 8 even easier.

Let’s rock and roll

So, using a php library or a Symfony component in Drupal 7 is quite simple. Just:

  1. Install composer manager
  2. Create a composer.json file in your custom module folder
  3. Place the content (which by the way, you’ll find quite familiar if you’ve already worked with Symfony / composer yaml’s):
    "require": {
      "pear/net_url2": "2.0.x-dev"
     }
    
  4. enable the custom module

And that’s it basically. At this point we simply need to tell drupal to generate the main composer.json. That’s basically a composer file generated from the composer.json found in each one of the modules that include a composer themselves.

Lets generate that file:

drush composer-rebuild

At this point we have the main composer file, normally in a vendor folder (if will depend on the composer manager settings).

Now, let’s make some composer magic :

drush composer update

At this point, inside the vendors folder we should now have a classmap, containing amongst others our newly included library.

Hopefully all has gone well, and just like magic, the class net_url2 is there to be used in our modules. Something like :

$base = new Net_URL2($absoluteURL);

Just remember to add the library to your class. Something like:

use Net_URL2;

In the next post we’ll be doing some more exciting stuff. We will create some code that will live in a php library, completely decoupled but at the same time fully integrated with Drupal. All using Composer magic to allow the integration.

Why? Again, many reasons like:

  1. Being ready for Drupal 8 (just lift libraries from D7 or D6 to D8),
  2. Decoupling things so we code things that are ready to use not just in Drupal, and
  3. Opening the door to other worlds to colaborate with our Drupal world, …
  4. Why not use Dependency Injection in Drupal (as it already happens in D8)? What about using the Symfony Service container? Or something more light like Pimple?
  5. Choose between many other reasons…

See you in my next article about Drupal, Composer and friends, on the meantime, be good :-).

Updated: Clarified that we are talking about PHP Libraries and / or Symfony components instead of bundles. Thanks to @drrotmos and @Ross for your comments.

Oct 22 2014
Oct 22

Along with Malcolm and other colleagues from the Capgemini Drupal team, I attended the recent Drupalcon in Amsterdam. And as well as admiring the Dutch attitude to cycling and its integration in the city (btw London, blue paint on the road != a cycle superhighway), we also caught up on the state of Drupal and its future. So here a few reflections from Drupalcon Amsterdam.

Drupal In The Enterprise - A Key Component In The Wider Web

I’ve been to a few Drupalcons now, and compared to previous years, use of Drupal in the enterprise (or more generally at scale) seems much more commonplace. Dries Buytaert’s (Drupal founder) keynotes have made reference to Drupal’s ability to integrate with other systems as a key strength, and in these types of projects, Drupal is not used as the all-in-one solution that maybe was more commonplace a few years ago.

Partly this is also due to the way the web has moved far beyond the idea of ‘a thing you use on your desktop computer’, and Drupal has shown itself to be adaptable to this. For example, the idea of Headless Drupal was a well covered topic this year. Of course, previous ‘cons have had talks on uses of Drupal with other technologies (e.g. node.js talk from London 2011) but whereas it seemed more an interesting edge case then, now there are many successful real-world projects adopting these ideas.

The Sessions

Based on my not-entirely-comprehensive memory of the subset of sessions I attended from past Drupalcons, this year there seemed to be many more talks which could have easily been at a frontend or PHP specific conference. Drupal 8’s use of Symfony 2 components and shift to making use of components Proudly Found Elsewhere is part of this.

A few talks that those of us who attended would recommend (not an exhaustive list). I won’t go into too much detail (that’s all in the slides and the video) but these are worth checking out.

Automated Frontend Testing

The types of frontend testing which can be automated, covering performance (Phantomas), CSS (Wraith) and end-to-end (CasperJS) and integrating this into your build workflow. Slides Video

Models & Service Layers; Hemoglobin & Hobgoblins

I think the PHP track is a welcome addition to Drupalcon. When developing custom functionality on projects here at Capgemini, we often write the business models and logic as separate classes to Drupal which are then ‘glued’ via hooks which implement those classes. That kind of separation has advantages with portability, testability and some amount of simplification in that Drupal isn’t a dependency. Video

Cory Doctorow’s Keynote

Very interesting talk on how open-source is (in some ways) critical to our individual freedom in the modern world. In an age where “a modern house is a computer that you co-inhabit”, if a system went down - or arguably worse, were controlled by overzealous authorities - it can become uninhabitable. What do we do in this case? Is the Apple iTunes/U2 debacle merely the thin end of the wedge? Interesting viewing for anyone who contributes or uses open source. Video

Drupal 8 And The Future

As Drupal 8 entered beta during the conference, it was an opportunity to check out the changes. The plugin system for extending functionality looks interesting. In Drupal projects at Capgemini we have adopted approaches such as abstracting business logic and objects into standalone libraries and classes, called from hooks and callbacks where we need integration with Drupal. This approach allows us easier unit-testing and portability of classes. D8’s plugin system looks like a good way of achieving those advantages while implementing a Drupal API.

Having spent a lot of time on projects wrestling with the various methods of deploying and updating configuration, the CMI (Configuration Management Initiative), which imports and exports YAML files to update and dump site configuration is a very welcome addition.

In the frontend, I’m looking forward to using the Twig templating. The idea of having cleaner PHP-free templates yet still with the flexibility to have filters and basic logic is going to help improve separation between the theme and module layer. It’ll be new to me (as will other things) but as with other components, they have been successfully used in other PHP projects so there is documentation and examples already out there. There are some smaller changes too - removing drupal.js’s dependency on jQuery (thereby gently encouraging use of native JS), updating the versions of included libraries (and committing to keeping them up-to-date during D8’s lifetime) and including no JavaScript by default are good steps to optimising the frontend.

Where things may be more challenging is the APIs which have both new object-oriented components and retain the hook and callback system in some combination (for example, declaring widgets via hook_element_info). To take an example from a core module, the file module’s managed_file widget functionality is spread across a number of callbacks as well as its own FileWidget class. It’s not the most straightforward development flow to follow. Where this has some advantages is that existing modules will not need a complete OO rewrite just to be compatible with D8 - a module author could do a simple port at first before rewriting to take advantage of the new APIs. But some care is need to ensure that the advantages of encapsulation, increased unit-testability and extendability that the OO patterns introduce are not compromised by dependencies on a particular hook or callback.

Taking The Leap

Finally, as Drupal 8 progressed from alpha to beta during Amsterdam Drupalcon, it does seem now that it can be realistically considered for projects coming up on the horizon. Obviously there will a lot more work going into the project to fix bugs and improve performance and so forth. But now the major API decisions and changes have been made. But with this iteration of Drupal incorporating many more features from the contrib world (Views, WYSIWYG, etc) and PHP (Symfony2 components), it looks to be a healthy position for use when that 8.0.0 finally lands.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web