Oct 09 2018
Oct 09

This is the eighth and last installment in a series presenting work on shared configuration that comes out of the Drutopia initiative and related efforts. If you've stuck with it this far, wow! If you haven't seen previous installments, you might want to start with Part 1, Configuration Providers.

The series focus has been meeting the needs of distributions for creating and updating packages of shared configuration, a use case not addressed in Drupal 8 core.

We've seen how configuration can be packaged using the Features module (Part 6). We've covered the ins and outs of updating packages of configuration, including how to assemble configuration as provided by extensions (Part 1), make needed changes to configuration from another package (Part 4), capture a previous configuration state (Part 2), and merge in configuration updates (Part 5). Along the way we've examined the particular cases of preserving configuration customizations (Part 3), working with core configuration required by multiple features (Part 7), and (bonus!) managing blocks when subthemes are in the mix.

The fact we're on installment eight - not even including the sneak bonus episode - in itself suggests a general conclusion: addressing these needs is not simple!

Working with Drupal core

In this series we've seen challenges or regressions when it comes to managing shared configuration in Drupal 8. In Part 1 we noted an issue with core code related to optional configuration. In Part 4 we saw how previously well supported requirements for shared configuration such as adding permissions to site-wide user roles or adding fields to existing content types now require dauntingly complex configuration alters.

These regressions are closely linked to Drupal 8's focus on the configuration staging use case. Progress on these fronts was not helped by what I, at least, experienced as an often boosterist environment in the Drupal 8 cycle in which critical perspectives on use cases that were and were not being addressed were sometimes drowned out. In a parallel Drupal 8 universe in which the use case of shared configuration was fully on par with staging configuration on a single site, solutions to much or all of the problems reviewed in this series might have been part of Drupal core.

At the same time, we've seen various ways that Drupal 8 core, including the configuration management system, helps with the task of filling in missing configuration management requirements. In Part 2 we saw how the configuration snapshot used in core's configuration staging can be a model for extension configuration snapshots, and also how core's LanguageServiceProvider is a useful pointer for how to provide dynamic services for configuration snapshotting.

Then there are key developments in the contributed modules space, especially Config Filter. By modelling how to extend and customize core's configuration staging workflow, Config Filter also opened the way to a parallel workflow for merging in configuration updates from extensions. Again, core's configuration management provides an essential base here. The Config Distro module takes everything that core's configuration staging API provides and, with a few small changes, repurposes it for a shared configuration use case. True, the result is only a framework - hence the many other pieces covered in this series - but the work gets a leg up from core.

CMI 2.0

The Drupal core Configuration Management 2.0 Initiative (CMI 2.0) aims to address "common workflows are not natively supported" and fill in "missing functionality" in core's APIs. For the work covered in this series, major additions to the configuration management API could be a big plus.

But they'll also have their tradeoffs. As a small hint of what may be coming, one of the first CMI 2.0 improvements led to breakage in the Config Distro module. A current big patch in the queue, Add configuration transformer API, looks like a great improvement, but since it would supersede the Config Filter module and our whole approach is built on Config Filter . . . . Yeah, a bit of work there. On the plus side, a compatibility layer between Config Filter and the configuration transformer API could mean it would all "just work" in the interim. Or, more likely, continue to break, but for other reasons ;)

Contribution

Drutopia is a small initiative. In contributing to configuration management improvements we've been able to lean on contributions from many others including Config Filter and Config Distro maintainer Fabian Bircher, Features and Config Actions maintainer Mike Potter, Configuration Update Manager maintainer Jennifer Hodgdon, and the core configuration work by Alex Pott and many more.

Still, to get from where we are to a polished and solid solution set, we'll need a lot of help. Most of the modules we've contributed are still in alpha or beta. We hope that others will see value in the start we've made and continue to adopt and adapt the tools, making them their own. Recent help with automated testing is particularly welcome.

Beyond shared configuration

In the Drutopia initiative, configuration management is only a small part of our focus. Looking to emerging models of platform cooperativism, we're building a cooperative where members will own, shape, and use the software distributions that drive their sites. The first such distribution is Drutopia, nearing its beta release.

We welcome new participants and contributors. Interested? See our guide to contributing and the many ways to get in touch.

Next steps

This series has had a decidedly technical bent. But how do you actually use the stuff?

Good question. We're working on a follow-up on how to put it all together, including:

  • How to build a distribution using Config Distro and Configuration Synchronizer.
  • How to update sites using Configuration Synchronizer.

Watch for it.

Oct 01 2018
Oct 01

Way back in January 2005, I posted a proposal to improve the governance of the Drupal project and help make it "fully 'community-driven'". In response, one commenter wrote:

Yesterday evening on the #drupal channel there was a trial vote casting for "Leave Dries alone" and unamiously everyone voted +1 on this...

Looking back at this long ago exchange, it seems to me to say something about what were and were not considered acceptable ideas in the community, and also about socially expected behaviour when there was a perceived threat to authority.

The issue of Drupal project governance has been returned to many times in the 13+ years since, but when it comes to the parameters of debate I don't know that so much has changed.

Why? Context is important. The Drupal project is structured as a centralized hierarchy--a dictatorship. While Dries Buytaert is often referred to as the project "lead", his formal, self-appointed position as noted in the project's governance documentation is "Benevolent Dictator For Life". Those skeptical of the claim that any dictatorship is benevolent - I'm certainly one - may reasonably shorten that to "Dictator For Life".

Now yet another group is tasked with producing recommendations for improving governance in the project. I have to say, I feel for them.

Because I've been there. I served for years as an elected permanent member of the now defunct Belgum-based Drupal Association (DA), and stayed on as an advisory board member for the DA in its current, US-based incarnation. From all those hundreds (probably, thousands) of hours of work, I can point to the occasional change that seemed at the time like progress, such as the process I co-led to design a community election system. But mostly in those years I learned the hard way just how difficult it is - futile might be more accurate - to try to fix a broken system from within.

The governance group itself, along with its assignment, is a product of the very power structure it's tasked with reworking. The task force was personally approved by the dictator for life. It reports to the dictator for life. Any decision on its recommendations will be made by the dictator for life.

Ouch.

There's the challenge of scope. They're tasked with coming up with recommendations for the governance of the Drupal "community". But, according to their charter, that community excludes all technical and code-related decision making and groups, as well as the Drupal Association.

So the assignment is to talk about open source governance without talking about open source governance--did I get that right?

There's the challenge of existing networks of influence, themselves deeply shaped by the project's centralized structure.

I've been particularly struck by contradictions baked into the project's supposed "Values & Principles". That document presents as the shared commitments of a very large community--and yet it's written in the first person singular? "Leadership is not something that is appointed", the document claims; but, um, isn't personal appointment by the dictator for life precisely how leadership positions are created and filled? Community members are to be treated with "dignity and respect"; and that's somehow perfectly consistent with a structure in which "The values and principles are maintained by me, Dries Buytaert"?

And so on.

Per the task force charter, a governance proposal must include "Implementing of the Drupal Values & Principles"--which themselves both reflect and encode a very specific form of governance, a dictatorship.

Anyone else got that "let me off this merry-go-round" feeling?

As far as I can tell, pretty much every previous governance exercise has produced some variation on the fantasy world in which (to quote again from the "Values & Principles") the ideals of "diversity, equity, and inclusion" magically turn out to be ideally met by a few tweaks to the cosy male dictatorship we already have.

But there's also a lot pushing in quite a different direction. Part of that is definitely the influence of the Me Too movement. Now is a time of deep scrutiny of structures of male, cis-gender dominance. Community members may be taking a long, hard look at the list of supposedly "benevolent" dictators for life and asking: is this a model we as a community would consciously choose to adopt? And if we're skeptical, what form of governance would reflect our shared values?

And part of the push for change is the state of the Drupal project itself.

Take the bold and transformative steps the project needs--but do so within the strictures of the current power structure? Sure, no problem.

But critique is easy. What about alternatives?

Well, for one, we could do a lot worse than look at a parallel process in the Backdrop CMS project (a conversation I participated in). How did that governance decision making process play out? What options were considered? What governance model did they choose, and why? How well does the current Backdrop leadership realize the values of diversity and inclusion?

What can we learn there about what it takes to achieve meaningful change?

Sep 27 2018
Sep 27

This is the seventh and (promise!) penultimate installment in a series presenting work on shared configuration that comes out of the Drutopia initiative and related efforts, beginning with Part 1, Configuration Providers.

In this series we've covered how to create and update reusable packages of configuration in Drupal, otherwise known as features.

In Part 6, we saw how the Features module can be used to package configuration that will be used by multiple different features into a "core" feature. An example is when multiple fields use the same storage. A core feature might provide a field_tags field storage, allowing multiple features to add a field_tags field to different content types. All the features that provide a field would require the core feature.

This approach helps to manage dependencies among different features, but it has at least two major shortcomings.

  • Any site that wants to install even a single feature that's dependent on the core feature will get all the core configuration--whether or not it's needed. For example, if the core feature provides five field storages but only one is required by the dependent feature, all five will still be created on the site.
  • Features from different sets or distributions will have conflicting dependencies. Say we have two different distributions, A and B. An event feature from distribution A requires the distribution A core feature, which provides the field_tags field storage. An article feature from distribution B requires the distribution B core feature, which provides an identical field_tags field storage. The event feature should theoretically be compatible with the article feature. But in practice they can't be installed on the same site, since an attempt to install both core features will raise an exception because configuration provided by the first-installed core feature will already exist on the site when the second is queued for installation.

In this installment we'll look at options for managing shared configuration that's required across multiple features--or multiple distributions.

Namespaced configuration

One workaround that's sometimes taken to address the issues of core configuration is to ensure individual configuration items are unique by giving them a namespace prefix.

There are two ways this could go. The first keeps the concept of a core feature but namespaces everything in it by feature set or distribution. In this approach, the core feature from distribution A provides a field storage called a_field_tags, while the B distribution's core feature provides b_field_tags.

More radically, in the second approach it's every feature for itself. Even within the same feature set or distribution, there are shared configuration items shared between features. An event feature from distribution A would provide its own tags field storage, such as a_event_field_tags, while the article feature from distribution B would provide b_article_field_tags.

Either of these approaches would work to avoid conflicting dependencies. But they raise problems that are just as vexing as the ones they attempt to address.

On a given site, you don't really want eight field storages for tags, one per feature you've installed. Nor, of course, do you want eight different tags vocabuaries, each with its own set of tags. You want one that will work site-wide. To take just one example, say I want to ship my distribution with a sitewide content search feature that includes the ability to filter results by tags (a taxonomy). I need all features on the site to use the same tags field referencing the same taxonomy--not eight different versions. See this issue on the Search API module for further details.

Namespacing configuration entities is a workaround that avoids but doesn't successfully address the key challenges here.

Configuration Share

The Configuration Share module is written to address the two big shortcomings of core features.

In Part 1 of this series we looked at the Configuration Provider module, which enables alternate configuration provision patterns beyond the required and optional directories supported by Drupal core.

Configuration Share provides an additional plugin, shared. Commonly needed configuration items like user roles and field storages can be designated as shared by placing them in a module's config/shared directory. Shared configuration works similarly to optional configuration as supported by Drupal core in that it's not necessarily installed when a module providing it is installed. But while optional configuration is installed after its dependencies have been met, shared configuration is installed only on demand--when another piece of configuration that requires it is about to be installed.

The basic logic:

  • When any extension is installed, the configuration to be installed is passed to all configuration provider plugins.
  • The shared plugin has a high weight and so runs after other plugins, including the ones for required and optional config.
  • The shared plugin analyzes all shared configuration with reference to the dependencies of the configuration queued for installation. If any shared configuration is required by queued configuration but is not yet installed, it is added to the list of configuration to be installed--along with any shared configuration it in turn requires.

In this way:

  • A given site will have only the specific shared configuration that is required.
  • Multiple features can share the same required configuration items without creating conflicting dependencies, since shared configuration is only queued for installation if it's not already on the site.

Compatible

While Configuration Share is one potential approach to the challenges of base configuration, it's at best an enabler.

In theory, two different distributions could both provide, say, a field_body storage and each add body fields to their respective features. But how do we know the two body fields will work together? One distribution might use Drupal core's text_with_summary field type for a field_body, while another might use Paragraphs for its body field, meaning that field_body would be of type entity_reference_revisions. A shared name - in this case, field_body - is no guarantee of compatibility.

The real work of interoperable sets of features would require collaboration among the developers of multiple distributions to co-maintain one more more curated sets of standardized core configuration. The Compatible module is a preliminary - and thus far undeveloped - step in that direction. It's intended as a repository for a standard set of configuration that can be shared among multiple extensions or distributions.

Next up

Stay tuned for the last installment in this blog series: Summary and Future Directions.

Sep 25 2018
Sep 25

In Part 4 of a current series on managing shared configuration for Drupal distributions we looked at needs and options for altering configuration provided by extensions (modules, themes, or the site's installation profile). We covered common needs such as altering user roles to add permissions. But when it comes to altering configuration, blocks are a special case--hence this bonus installment!

When you create a site based on a distribution, there may be a requirement to customize the look and feel. The usual solution is to create a custom subtheme for the site; see the drupal.org documentation on subtheming. That way you can get everything the distribution provides but give the site a custom presentation.

Using a custom theme will work fine for most configuration. But it won't work for configuration that includes the theme itself as a dependency--like blocks.

As explained in the drupal.org handbook page Working with Blocks:

Blocks are boxes of content rendered into an area, or region, of a web page (such as "User Login" or "Who's online") that can be displayed in regions (such as footer or sidebar) on your page.

Blocks are configured per theme. Typically a Drupal distribution will ship with a designated default theme--a core theme, a contributed theme, or, most commonly, one written especially for the distribution. Blocks are placed into this theme, ensuring sites get all the expected page elements.

But this approach breaks down on sites using a custom theme. For example, if a site is installed with a distribution, sets a custom theme as the default theme, and then updates to a new version of the distribution and brings in configuration updates including new blocks, those new blocks won't show up on the site. That's because the new blocks were created for the distribution's theme--not the custom theme that's the default theme on the site.

Even if a distribution-provided block did show somehow in a custom theme, it might be in the wrong position. Each Drupal theme supports a designated set of regions--page areas that blocks can be placed in. While in most cases a custom theme will be given a similar set of regions as the distribution's theme, this may not always be the case.

Block Theme Sync

In Drutopia we drafted a module to try to address these problems.

Block Theme Sync allows a site admin to configure a "theme mapping": a relationship between two themes. As well as specifying a source and destination theme, a theme mapping includes a mapping of source to destination regions.

Block Theme Sync uses an approach of cloning and then synchronizing. When a theme mapping is in place, all blocks created for the source theme are automatically cloned and created for the destination one. Any subsequent change to block configuration in the source is applied as well to the destination theme.

Say for example that there's a theme mapping with Bartik selected as the source theme and Mytheme as the destination theme. Whenever a block is added to Bartik, a corresponding block will be added to Mytheme and assigned to the corresponding theme region as specified in the theme mapping.

This approach works, but has tradeoffs. The main one is one that comes from cloning. Since we now have two distinct versions of the block configuration, we face the complexity of divergence between the two. For example, if a destination theme's block is edited, how do we handle that customization when the source theme's block is updated? That's a question we covered in the installment on Respecting Customizations and Updating from Extensions. There's a relevant issue open on Block Theme Sync: Use three-way merging when synchronizing.

Better Sub-themes

Not long after our work on Block Theme Sync, Stuart Clark (Deciphered on drupal.org) posted the Better Sub-themes module. While similar in many ways to Block Theme Sync, it takes a different basic approach.

Like Block Theme Sync, Better Sub-themes provides a way to designate a source and destination relationship between themes and also to map regions. Rather than doing so through configuration, Better Sub-themes extends the theme info file format.

But in the implementation, rather than cloning and synchronizing, for every block in the source theme, Better Sub-themes dynamically layers on a block in the destination theme. These destination theme blocks don't exist as separate configuration entities on the site. So they don't show up in the site's block admin screen for the destination theme, or when site configuration is exported. They can't be separately edited. But they show up on the site in their designated theme regions.

If Block Theme Sync implements a cloning and synchronizing pattern, Better Sub-themes uses inheritance. This avoids the problems inherent in cloning. But it introduces challenges of its own. The fact that inherited blocks don't show up in the block administration UI is potentially confusing--how do you edit them? And it makes it difficult to position other blocks in relation to the inherited ones.

Potential enhancements

Both Block Theme Sync and Better Sub-themes are affected by a quirk of Drupal core's theme system.

By default, when you install a theme, if the theme doesn't supply any blocks of its own, Drupal core copies over the blocks that are enabled at that time for the current default theme--see block_theme_initialize(). And a mea culpa: I wrote the first version of Drupal core's theme region system, and included that quirk in the implementation ;)

Here's how this quirk affects each of the modules:

  • With Block Theme Sync, synchronization is done only when a block from the source theme is saved. if the destination theme doesn't provide any blocks, it will get an initial set of blocks created--but won't get any that were created after the theme was installed but before Block Theme Sync was in place. If it does provide blocks, it won't get any from another theme. Relevant issue: Initialize theme blocks on creation of theme mapping.
  • With Better Sub-themes, when we (a) install a distribution site, getting its standard default default theme along with a set of blocks, then (b) switch the default theme to a custom subtheme with Better Sub-themes support, all the source theme blocks will appear twice on the site: first, because they were cloned by Drupal core; second, because they're inherited from the source theme via Better Subthemes. Relevant issue: Prevent or delete duplicate blocks.

Related core issue

Takeaways

  • For distributions that include block placement, subthemes present challenges.
  • There are two different modules available that attempt to address these issues: Block Theme Sync and Better Sub-themes.
  • Both modules are more proofs of concept than polished and proven solutions.
  • Each has its relative merits, and which if either is a fit will depend on the particular distribution use case.
Sep 24 2018
Sep 24

This is the sixth installment in a series presenting work on shared configuration that comes out of the Drutopia initiative and related efforts, beginning with Part 1, Configuration Providers.

Our main focus has been updating configuration from distributions--specifically, the question:

How can I update my site so that I have all the latest configuration changes from a distribution--while still retaining any customizations I made?

Updates are well and good. But before packages of configuration can be updated, they need to be produced and managed in the first place. In Drupal 8 as in previous major versions, that task is the domain of the Features module.

The Drupal 8 version of Features is a complete rewrite with major improvements over previous versions. If you're familiar with previous versions but haven't used Features in Drupal 8 this backgrounder will bring you up to speed.

Despite being a development-focused tool, Features is in the top 40 or so most installed contributed Drupal 8 modules. Features is used in building and maintaining several of the more-used Drupal 8 distributions including Open Social, Varbase, Open Church, and Quick Start. It's a key build tool for the Gitlab-hosted Drutopia project.

In this installment we'll cover Features in Drupal 8, including how to use it to produce a distribution.

Analyzing the site

With its default configuration, Features will automatically divide a site's configuration into a logically structured set of distinct features. It will even generate an installation profile to use with the set of features.

Features is architected around three main components.

  • An assignment plugin influences the way that configuration on a site is assigned to distinct packages or features. Features ships with a large set of assignment plugins and other modules can provide their own. See documentation on Features assignment plugins and related developer documentation.
  • A generation plugin determines how features are generated. Features ships with two generation plugins, one that creates downloadable archives and the other that writes features directly to the file system. See documentation on Features generation and related developer documentation.
  • A features bundle is a configuration entity that defines a set of features such as those included in a specific distribution. Each bundle has an associated set of assignment plugins configuration--so each distribution can customize the way its configuration is packaged. See documentation on Features bundles.

The easiest way to get a feel for this functionality is to install a fresh Drupal site and then use Features to turn it into a distribution.

Here's how.

Install Drupal and enable the Features UI module

One quick way is to use simplytest.me:

  • Bring up simplytest.me with the Features module preselected and click "Launch sandbox". This will spin up a site installed with Drupal core's "Standard" installation profile.
  • When the site loads, if you have not been automatically logged in, click the "Log in" link and log in with the user name "admin" and password "admin".

Alternately, install a local development site using the "Standard" install profile. See Chapter 3 of the Drupal 8 user guide for details.

When you have a site installed and are logged in as an administrator:

  • Click "Extend" and install the "Features UI" module.
  • Navigate to admin/config/development/features and click the "Configure bundles" tab to bring up the form for creating and editing feature bundles.
  • For "Bundle", select "--New--" and enter the bundle name "Exemplary".
  • Check the checkbox "Include install profile" and enter "exemplary" for "Profile name".
  • Click the "Features" tab to bring up the form for downloading features. You'll see that the site configuration installed by the "Standard" installation profile has been neatly divided into several distinct features. As well, an installation profile, "Exemplary" has been created.
  • Click the checkbox in the table header, left of "Features", to select all available features.
  • Click the "Download archive" button.

What you get is an initial draft of a Drupal distribution. You can copy the resulting directory and all its subdirectories to the profiles folder of a Drupal codebase and install a new site using the "Exemplary" installation profile.

Of course, you won't get anything that's not already in Drupal core's "Standard" installation profile. But there are two advantages you get from the exercise:

  • You can easily install just a subset of what Drupal core provides. This contrasts with the "Standard" profile, which installs everything all at once, whether or not you need it on the particular site.
  • You have a base to build on. As you continue to build out new elements on your site, following the naming conventions in the Features building workflow, Features will recognize new features as you build them and, again, automatically package them into feature modules for generation.

Features and dependencies

The "base" and "core" assignment plugins give a feel for how assignment plugins work together to assign configuration to features.

The Base assignment plugin

To automatically divide a site's configuration into distinct features, we have to know what features to create.

One way is by analyzing the configuration itself. Certain types of configuration have a whole lot of other configuration that depends on them. Content types are a key example. A content type doesn't require any other piece of configuration, but typically has many pieces of configuration of different types that depend on it: fields, view modes, search indexes, and so on. For that reason, it's often a good design decision to create a feature module per content type on the site.

That's what the base assignment plugin does: takes a particular type of configuration and created a feature for each item of that type. Concretely, on a site that has an event content type and an article content type, the Base assignment plugin will create an event and an article feature. Then it will assign to that feature every piece of configuration that's namespaced with the feature name. For example, a field called field_event_type would be assigned to an event feature.

The Core assignment plugin

The core assignment pugin addresses the problem of configuration that's required across multiple features. How do we provide that configuration without creating a bunch of cumbersome inter-feature dependencies?

Say we have a tags field we use on both an event content type (provided by our event feature) and an article content type (provided by our article feature). Like all fields, the tags field has an accompanying field storage. That means the tags field storage will be required by both the event and the article features.

If we put it in the event feature, the article feature will require events--which doesn't make sense. The reverse case - that events require articles - isn't any better.

The Core assignment plugin addresses this issue by pulling specified types of configuration into a core feature that can be required by other features.

Further reading

See the blog post How to use Features module in Drupal 8 to bundle functionality in reusable module for an introduction to the module. There's also a detailed Features 8 handbook on drupal.org.

Potential enhancements

While Features in Drupal 8 is mature and used on thousands of sites, there are loose ends and some significant missing pieces still to cover, and additional uses the module could be extended to.

Translations

Drupal 8 supports multiple "collections" for configuration. So far, Features only works with the default collection. In practice, this means most configuration is handled fine, but language translations of configuration are not. Here's the relevant issue: Support non-default configuration collections, including language.

Alters

A significant missing piece is integration for alters (covered in Part 4 of this series).

There are two parts to integrating support for alters into Features. First, when exporting items to a feature, it should be possible to export specific changes as alters. Core's configuration override system includes the ability to load configuration free of overrides, but there is no equivalent as yet for the various approaches to altering configuration-provided configuration. Second, in the export of the original item, any alters should be reversed, so that the exported item is alter-free. Potential pieces to help address these requirements include:

Additional use cases

The plugin-based architecture of Features means it can potentially be applied beyond its original use case of managing feature modules. For example, it would be relatively simple to provide a features generation plugin to generate configuration packages as Configuration Split configuration entities.

Related core issues

Next up

Stay tuned for the next installment in this series: Base Configuration.

Sep 20 2018
Sep 20

This is the fifth installment in a series presenting work on shared configuration that comes out of the Drutopia initiative and related efforts. To catch up, see Part 1, Configuration Providers, Part 2, Configuration Snapshots, Part 3, Respecting Customizations, and Part 4, Configuration Alters.

In this installment we'll start to pull it all together.

Paraphrasing a bit from Part 1, we described a key problem this way:

How can I update my site so that I have all the latest configuration changes from a distribution--while still retaining any customizations I made?

In Part 1 we mentioned Fabian Bircher's widely used Configuration Split module and its enabling API module, Config Filter, returning in Part 3 to give a more detailed introduction. In Part 2, we summarized the API and accompanying user interface that core provides for staging configuration. Here we'll take a deep dive into how we can merge in configuration updates from installed extensions through an approach that's built on Config Filter and closely parallels core's configuration staging.

Config Filter and core's configuration staging

From Part 2 in this series:

Using Drupal core, you can stage configuration between different environments or versions of a given site--say, from a development environment to a testing one. For details of how this works, see the Configuration Management section of the Drupal 8 handbook. The Synchronizing Configuration Versions section of the Drupal 8 User Guide also reviews this functionality.

When you stage configuration, core reads the configuration to import from a configuration storage (see the overview on storages in Part 1). Core uses a storage provided by the config.storage.sync service, which by default serves up files from a specified directory.

At its most basic, configuration staging is a two-step process:

  • Export configuration from one environment - say, a development ("dev") environment - and copy them to the sync directory on another instance of the site--say, a live environment.
  • Run core's configuration synchronization. All going well, your live site now has configuration that's identical to what was exported from dev.

But what if you don't want live to be exactly the same as dev?

That's where Config Filter comes in. Any module can provide one or more filters. Configuration Split allows site admins to create configuration "splits" that are applied as filters on top of core's sync directory. For details, see the module's drupal.org documentation and the blog post Configuration Split: Managing Drupal 8 Configuration for Different Environments.

Config Filter makes Drupal core's configuration staging workflow a lot more flexible. But its API is not limited to the staging workflow. A config filter can apply to any storage; core's config.storage.sync is only the default.

So, using Config Filter, the way is open for a contributed module to mirror core's configuration staging workflow, but providing a framework for merging in updates from extensions rather than staging configuration between environments.

Config Distro and Configuration Synchronizer

And that's exactly what the Configuration Distro module does.

Configuration Distro provides a filtered storage, config_distro.storage.distro that wraps core's active configuration storage. Config Filter plugins using that storage will apply their changes to the site's active configuration--just what you need to bring in updates from extensions. It then provides a UI directly parallel to core's configuration staging UI.

And it stops there, for reasons explained in the blog post announcing Configuration Distro:

If you only install Config Distro, the import screen eternally shows that there is nothing to import. This is because the module only provides the framework for this to work, ie the UI and the drush command. The module is not opinionated on how or what should be updated. All the complexity can be addressed by a distribution maintainer with the help of a ConfigFilter plugin (the same way Config Split and Config Ignore work). One such module is Config Sync [Configuration Synchronizer]. All the complexity of finding out what configuration a module has originally shipped with, what it does now and whether a user has changed the originally installed configuration is left to Config Sync and its dependencies.

Much of the heavy lifting for "all the complexity" is accomplished through the various modules profiled in this series so far, including Configuration Provider, Config Merge, and Configuration Snapshot. But there's plenty left for Configuration Synchronizer to do ;)

A config snapshot for each extension

The Configuration Snapshot module makes it possible to take snapshots, but leaves the logic of when and why to other modules. In Configuration Synchronizer, we take config snapshots and apply alters to existing snapshots when extensions are installed. We also subscribe to an event, ConfigDistroEvents::IMPORT, that the Config Distro module dispatches on completing a configuration import. This allows us to refresh the snapshot for every extension that had updates imported.

Updates = Providers - Snapshots

Determining what updates are available for each extension involves three steps:

  • First, consult Configuration Provider to find out what configuration is currently provided.
  • Next, load our configuration snapshot for the extension via Configuration Snapshot.
  • Finally, compare the two (using core's StorageComparer) to determine what's been changed or added.

A config filter for each extension

Just as we have a snapshot for each extension that provides configuration, we provide a corresponding configuration filter.

We need a filter for every extension that has available updates--a list that may change any time we update to a new version of an extension or run the Config Distro updates. So we follow a pattern from Drupal core that allows us to dynamically declare our filter plugins: plugin derivatives. An example of a plugin deriver is, BlockContent, used in core to make a block available for each piece of block content. For a detailed backgrounder, see this Tutorial on Using Drupal 8 Plugin Derivatives Effectively.

In our plugin deriver, SyncFilterDeriver, we iterate through the list of available updates by extension and add a plugin for each.

Update modes

In Configuration Synchronizer we provide two different modes for bringing in configuration updates.

The default update mode is merge. When the merge mode is selected, the configuration filter for a given extension merges in changes via a call to the relevant method in Config Merge. This way, any customization done to the site's active configuration is retained.

But in certain cases you might not want to retain customization. So a second update mode is available: reset. When the reset mode is selected, the active configuration is reset to what's currently provided by the extension. This mode is equivalent to what was called "reverting" in the 7.x version of the Features module.

Customizing the update form

Config Distro provides a form for configuration updates that's pretty much identical to the "Configuration synchronization" form provided by Drupal core. But in Configuration Synchronizer we support a separate update per extension. So we extend the Config Distro form to provide a lot more information as well as options.

For each extension, we list details of what updates are available, showing:

  • The type of update (new items vs. changed ones).
  • The type of configuration.
  • The configuration item names.

Clicking on or selecting an individual item's name will load a diff for that item.

And we provide a checkbox to select that extension's updates.

In an "Advanced" section, we offer a select to choose the update mode, "Merge" or "Reset".

Config Distro Ignore

The merge and reset update modes are useful, but there may be cases where you want to combine the two. For example, you might want to reset almost all configuration on your site to the state that currently ships with a distribution, but still retain a select few customizations. To cover that case, Config Distro ships with a sub-module.

Using Config Distro Ignore, you can specify particular configuration items that should be ignored (left untouched) when you bring in changes. Once you have done so, if you then set the update mode to "Reset" and run an import, those specific items should be retained while all others are reset.

Potential enhancements

If there's one thing that's probably obvious by this point, it's that managing configuration updates from contributed modules is complex!

A major lack in Configuration Synchronizer and its Druptopia-sponsored dependencies is test coverage.

Support for Drush has not yet caught up with what's in the module's UI; see the feature request Add Drush command for the "reset" update mode.

Another area for improvement is handling of configuration that's been deleted from the module that originally provided it; see Optionally delete config entities when they're deleted from code.

Next up

Stay tuned for the next post in this series: Packaging Configuration.

Sep 18 2018
Sep 18

This is the fourth installment in a series presenting work on shared configuration that comes out of the Drutopia initiative. To catch up, see Part 1, Configuration Providers, Part 2, Configuration Snapshots, and Part 3, Respecting Customizations.

In the next installment we'll start to pull it all together, showing how all the pieces covered so far fit into a solution for merging in configuration updates. But first there's one more piece to add. In this installment we'll be looking at creating and updating configuration alters.

First off, what is a configuration alter?

An alter is an addition, deletion, or change to a piece of configuration.

When we're producing a several packages of shared configuration - what are often called feature modules - sooner or later we need a particular feature to modify configuration that was provided by another feature.

Say I'm producing a Drupal distribution that includes two different features: an event feature and a location feature. Any given site might install the event feature, or the location feature, or both, or neither. (A big part of the point of having distinct features is to make them optional and independent.)

In this example, say the event feature provides an 'event' content type with fields including a required 'date' field. The description of the event content type is: "An event takes place at a specified time." The location feature provides a 'location' content type.

But if I have both events and locations, there's a relationship between them. Events take place at a location. So on a site with both the event and the location features installed, the event content type should get an additional required field, "Venue", that's a reference to a location. When that happens, the description of the original event content type should change accordingly. Now it should read: "An event takes place at a specified time and place."

To make this happen, the location feature (or, possibly, a third feature) is going to have to alter the event feature's configuration.

Drupal's configuration data is exported in YML format. Here's a snippet of what the event content type would look like as provided by the event feature:

name: Event
type: event
description: 'An event takes place at a specified time.'

We only want to change one property, description. Here's the result we want after an alter:

name: Event
type: event
description: 'An event takes place at a specified time and place.'

While it looks innocuously simple, even this rudimentary alter turns out to present thorny challenges when it comes to producing and updating packages of shared configuration.

Overrides and alters in Drupal core

There are several ways that configuration changes, overrides, and alters are supported in Drupal core. We'll review them before digging into specifics.

Optional configuration

Extensions can provide optional configuration: configuration items that are installed only when certain conditions are met. In our example, the event venue field could work well as optional configuration provided by the location feature. This optional venue field will be installed only when the event feature is installed and its event content created. We'll see below that the task isn't quite so easily accomplished, but getting the field in place is at least a start.

Optional configuration helps with the use case of extending one module's configuration in another module, but doesn't do alters.

Installation profile configuration overrides

If you've ever tried to install a Drupal module that provides configuration that already exists on your site, you'll be familiar with the often perplexing error that Drupal core throws to the effect that configuration items provided by the module "already exist in the active configuration". This error occurs because regular modules and themes can't re-provide configuration that's already provided by another extension. But installation profiles can. For example, if you put a customized version of the site.info.yml file in an installation profile's config/install directory, that file will be installed on the site rather than the version provided by core's system module. For more on configuration overrides in installation profiles, see this blog post.

Configuration overrides in installation profiles are useful, but have key limitations:

  • They're only an option in the installation profile.
  • They only work for a full override of a configuration item, not for a selective alter.

Programmatic changes to configuration

It's possible to make programmatic changes to configuration in custom code--for example, in an update function. That's what core does in the Standard installation profile's install function. For example, code in that function sets the site's home page, a property of the site.info configuration item.

That works for one-off changes. But:

  • As alters multiply, so does the amount of custom code.
  • Such changes can't be worked into an automated configuration update workflow. For example, the Configuration Provider module introduced in part 1 of this series has no way of guessing what arbitrary changes may have been made in custom code.

Configuration override system

Drupal 8 introduced a system to override configuration from modules--see the relevant documentation on drupal.org.

In this system, an override is a change to configuration that's dynamically applied on top of the site's active configuration. Overrides are a good fit for cases where a change needs to be made only under certain conditions. The classic use case in core is language-specific configuration overrides--see the drupal.org documentation on translating site interfaces. If you've ever switched languages on a Drupal 8 site and seen a field label change to the new language, you've seen configuration overrides in action. User interface translation fits the model of a conditional change because the text presented to users needs to be changed only if the site is multilingual and the interface language is other than the site's default language.

Configuration overrides are also used in the contributed Domain Access module. There, the need is to customize configuration per domain.

There's a contributed Configuration Override module that allows modules to provide configuration overrides alongside the other configuration they provide.

But for our purposes it won't work for alters to be layered on top of the active configuration. Instead, we need them to behave just like regular extension-provided configuration. For example, we want the resulting configuration to be editable. Configuration overrides aren't a good fit.

Overriding configuration in settings.php

A final way core supports altering configuration is in the settings.php file used to provide and customize a site's settings--see the relevant documentation on drupal.org. For more on overriding configuration in settings.php, see this blog post.

Again, while useful for other purposes, this approach doesn't help with feature-provided configuration since (a) it has the same disadvantages as module-provided configuration overrides and (b) we can't rely on site admins to manually edit their settings.php files.

Challenges of Drupal 8

In the intro we looked at an example of configuration altering: changing a content type's description. But in Drupal 8 the need for alters runs much deeper.

Shared configuration was the original use case in Drupal for exportable configuration. Drawing on innovation by Earl Miles and others in the Views module, major work took place in and around the Features module. The original announcement of the Features module in 2009 laid out the program:

A feature is a collection of Drupal entities that, taken together, satisfy a certain use-case.

And:

individual feature modules don’t script site building or use update scripts to maintain themselves. Instead, we take advantage of the powerful exportables concept pioneered by Views to allow all of your feature’s site building to live in code (exported views, contexts, imagecache presets, node types, etc.) but also be overridden and in the database.

Because work focused squarely on the Features use case, exportables were structured from the start to facilitate stand-alone configuration packages. To take two key examples:

  • A user role can have multiple permissions assigned to it. Typically, a site will have a small number of user roles and each separate feature will need to provide permissions to one or all of those roles. In Drupal 7, exportables were structured to make this easy, by exporting user permissions separately from roles.
  • A content type has multiple fields. A fairly common need is for one feature to add a field to another--like in our example of a location feature adding a field to events. As well as the field structure itself, a field typically needs to be displayed. In Drupal 7, the export of a field (then termed a field instance) included its presentation information for multiple displays (view modes).

But the focus on configuration staging in Drupal 8 meant that exportables were rewritten seemingly without reference to the use case of shared configuration packages.

Contrary to Drupal 7, in Drupal 8:

  • Permissions are exported as part of the role entity. This means that this quintessentially basic requirement of stand-alone configuration packages - adding a permission to a site-wide user role - requires complex customization via alters. Various workarounds have emerged in contrib. Two examples are Config Role Split and Role mixins / Subroles. The Features module in Drupal 8 includes a related workaround: permissions are optionally stripped on user role export, since lumping them into a single feature would break the ability to produce stand-alone features.
  • With field configuration, we're no better off. The relevant configuration entities include display information for all fields in a single entity, meaning a feature adding a field to another feature's content type can't just export its own configuration. Instead it must perform complex alters on multiple entities provided by the other feature. How? See this Drutopoia documentation page for the messy details of one approach.

Because of regressions in Drupal 8 compared to previous versions, alters are needed even for some of the most basic and common tasks of building features.

Modules for configuration alters

Fortunately, there are at least two contributed modules that provide support for module-provided configuration alters: Config Actions and Configuration Rewrite. The Update Helper module provides similar functionality, though in that case linked specifically to updates. In Drutopia we've chosen to work with Config Actions since it provides a full set of tools for adding, deleting, and changing configuration properties along with extensive documentation.

Using Config Actions, a config action is a manually edited file that you place in a module's config/actions directory. The file contains just the additions, deletions, or changes you want to make.

Here's a simple example. To accomplish the alter from the intro - changing the description value of the event content type - we would start by creating a file with the same name as the item we're altering, node.type.event.yml, and putting it in the location feature module's config/actions directory. The file's content would be:

plugin: change
path: ['description']
value: 'An event takes place at a specified time and place.'

This says:

  • Use the change plugin (a plugin determines what type of action to apply).
  • Make the change the description property.
  • Set the new value to read 'An event takes place at a specified time and place.'

Add the file, install the event feature, then install the location feature, and voila, we have the altered event content type description just like we wanted.

But what if a given site installs the location feature first, and only later adds the event feature? Sadly, in that case, we'd be out of luck. By default config actions (and the config rewrites supported by the Configuration Rewrite module) are applied only when a module is first installed. To address that issue, we might have to pull our alter into a bridging feature: a feature that provides the integration between two other features. If we make the bridging feature require both the event and the location feature, we can ensure the event content type exists when the alter is applied.

For our use case of importing configuration updates from features, alters present a series of additional challenges that don't have such easy workarounds.

Altering provided configuration

In Part 1 of this series we looked at the Configuration Provider module, which answers the question: what configuration is available to be updated? That module includes support for the configuration models supported by Drupal core: required configuration in config/install and optional configuration in config/optional.

Support for config actions is accomplished in Config Actions Provider. As noted on its Drupal.org project page:

Config Actions Provider integrates the Config Actions module with Configuration Provider, making it possible to apply config actions during configuration updates.

Concretely, Config Actions Provider provides a config_provider plugin that accepts configuration provided by other providers (such as required and optional) and applies any available alters to it.

The Nitty Gritty

For those interested in the messy details....

Alters raise the issue of ownership. Which module "owns" the snapshot of a configuration item? Typically, we'd say a piece of configuration is owned by the module that provides it. But with alters the resulting configuration structure is the sum result of input provided by multiple sources.

Also, the order in which alters are applied matters. If two different modules have changed the same configuration property, only the second change will be in effect.

In Config Actions Provider and related modules, we assign ownership to the module that originally provided the configuration. For example, when we're listing available configuration updates by providing module, any alter-provided changes are assigned to the module that provided the original item.

To emulate the altering that happens at install time, alters are applied in module dependency order--the same order that modules are installed in. Even if we're loading only a single module's available configuration, we iterate through all installed modules, since any of them could provide an alter for this module's configuration.

If determining what alters would be applied to configuration as provided sounds tricky enough, it's really nothing compared to the challenges of applying alters to configuration snapshots.

In Part 2 of this series we looked at the Configuration Snapshot module which provides the ability to take snapshots of configuration as provided by installed extensions. A snapshot should answer the question: what was the state of this configuration as provided when it was last installed or updated?

Introducing configuration alters makes that question a lot harder to answer.

Recall that alters are applied when the module providing them is installed. This means that a snapshot of the configuration as installed or updated has to include all alters that were actually applied, but not any that may have been added since installation or update. Say as a module author I add a new alter to an existing module. New sites that install that module will get the alter, but sites on which the module is already installed won't. For our purposes, what should happen in this case is:

  • The alter should show up in the data returned by Configuration Provider on what's available to be installed or updated.
  • The alter should not be included in the snapshot.

That way, when the currently-available configuration as provided is compared with the previous snapshot, the alter will show up as an available update.

In our Drutopia work, much of the handling of snapshotting for configuration alters has gone into the Configuration Synchronizer module, which we'll introduce in the next installment in this series.

Integration for snapshotting config actions is in the Config Actions Provider module. There we do pretty much the inverse of what we do for provision. Rather than iterating through all modules and applying their alters to the provided configuration, as we do for provision, for snapshotting we iterate through all existing extension snapshots and apply to the snapshots any alters provided by the extensions we've just installed. That way, when a module is installed, any alters it provides are added to existing snapshots.

Potential enhancements

The core issue Add configuration transformer API proposes an approach that dispatches and subscribes to events (not the kind in the hypothetical event feature discussed above!) to assemble available configuration. If that API is added to core, it could open the way to use a parallel approach for config alters.

Related core issues

Next up

Stay tuned for the next post in this series: Updating From Extensions. In that installment, we'll start to pull together everything we've covered so far: configuration providers, snapshots, merges, and alters.

Sep 11 2018
Sep 11

This is the third installment in a series presenting configuration management work that comes out of the Drutopia initiative.

  • In part 1 of this series, Configuration Providers, we introduced the problem of managing shared configuration in Drupal, starting with how to determine what configuration is available.
  • In part 2, Configuration Snapshots, we covered creating snapshots of configuration to facilitate comparison of different configuration states.
  • In this installment, we'll look at another key issue when updating shared configuration: how to handle customizations.

Customizations, updates, and merges

To help clarify the problem, take this scenario:

  • You install a site from a Drupal distribution.
  • One of the pieces of configuration that distribution provides is an event content type.
  • You edit the content type to tweak the help text that appears when a user creates a new piece of content.
  • Now a new release of the distribution is available.

You want to update to the new version and get any updates it provides, including any changes that were made to the event content type. But - and here's the tricky part - when you update, you don't want to lose the customization you made to event the content type's help text.

As noted in part 2 of this series, there are three different states to consider when we update configuration that's provided by extensions (modules, themes, or the install profile):

  1. The configuration as it was provided when we first installed or last updated. This is what we store in and read from a snapshot.
  2. The configuration as it's currently provided by the extension.
  3. The configuration as it's currently saved in the site's 'active' storage.

To bring in updates while respecting customizations, we can't take any one of these states on its own. If we take just the newly-provided version, we would lose the customization we did in the active version. If we stick with the active version, we don't get the changes in the newly installed version. instead, we need to merge in updates. We want to bring in any changes from the updated version, but only if they don't overwrite or undo any change that's been made in the active configuration. In our merge, we need to consider all three configuration states. In other words, we'll do what's called a three-way merge--a merge of data from three distinct sources.

Merging data from three or more sources or states is a fairly common computing need.

Both of these merge strategies are roughly analogous to what we need to do.

The structure of configuration data in Drupal

In merging Drupal configuration data, we need to take into account how configuration data is structured. Drupal configuration data is stored in array format. Since the relevant Drupal code is written in PHP, it's PHP arrays we're dealing with.

As explained in on w3schools.com, there are three distinct types of arrays in PHP:

  • Indexed arrays, sometimes called sequence or numeric arrays - arrays with a numeric index.
  • Associative arrays, sometimes called hashes - arrays with named keys.
  • Multidimensional arrays, sometimes called nested arrays - arrays containing one or more arrays.

Drupal's configuration data structure can combine the three types. Our merging strategy needs to handle multidimensional arrays and, in doing so, treat each array item differently depending on whether it's indexed, associative, or multidimensional.

PHP has a bunch of functions that work with array data including two specific to merging: array_merge() and array_merge_recursive(). Drupal core includes some specialized helper methods for merging data arrays: NestedArray::mergeDeep() and NestedArray::mergeDeepArray().

The config-extra Drush extension provides support for doing configuration merges "in instances where different team members need to change the site's configuration settings at the same time"--see relevant documentation.

None of these does exactly what we need, but they provide pointers.

Configuration Merge

The Configuration Merge module provides a method, ConfigMerger::mergeConfigItemStates(), that implements an algorithm for three-way merging of configuration items.

The method takes three arguments as input: the previous (snapshotted), currently-provided, and active states of a configuration item.

The approach is pretty straightforward:

  • Distinguish between indexed and associative arrays. When exporting configuration, Drupal uses a Symfony method, Inline::isHash(), to determine if an array is indexed or associative. So we do the same.
  • For associative arrays, merge in additions, deletions, and changes and recurse as required.
    • Additions are items with array keys present in the current data but not in the previous snapshot.
    • Deletions are the inverse: items with array keys present in the previous snapshot but not in the data as currently provided.
    • In both cases, we consult the active configuration before bringing in an update. We remove an item only if it hasn't been changed (customized) since the previous snapshot. We add an item only if it hasn't already been added.
    • Finally, changes are items with the same key but different values in the snapshot and the currently provided data. As with deletions, we consult the active configuration storage and accept a change only if it hasn't been customized. If the changes value is itself an array, we recurse. That is, we pass the value as an argument to the same ConfigMerger::mergeConfigItemStates() method, so that this array, too, can be appropriately merged.
  • For indexed arrays, merge in additions and deletions. For indexed arrays, we do much the same as for associative ones, but with a couple of important differences.
    • Because the array keys are assigned sequentially, they don't have a fixed relationship with their values. This means that while we can say what's been added or deleted, we can't really say that an existing array item has changed.

Using this merge algorithm, Configuration Merge makes it possible to determine configuration updates that don't overwrite what's been customized on a site.

A merge plugin for Configuration Filter

While Configuration Merge applies when merging in configuration updates from installed extensions, it can also apply to the configuration staging workflow that's built into Drupal core.

When it comes to staging and handling configuration, one of the most significant developments in the Drupal contributed modules space is Config Filter. The Config Filter module makes it possible to modify configuration as it's being staged or imported from one environment to another.

The original use case of Config Filter is Configuration Split, a module that makes it possible to "define sets of configuration that will get exported to separate directories when exporting, and get merged together when importing".

There are two Drupal modules that use Config Filter to protect customizations from being lost when staging configuration.

  • The Config Ignore module provides a configuration filter that allows you to specify a list of configuration items that will be ignored (skipped) when you synchronize configuration from one environment to another.
  • The Configuration Overlay module provides a configuration filter to turn the configuration export into an overlay of the shipped configuration of all installed configuration. Installing this module and subsequently exporting configuration will leave only those configuration files in your export directory that have been added or modified relative to the shipped configuration of modules and installation profiles.

Both are useful, but they have limitations:

  • Config Ignore requires explicitly listing the items to be ignored.
  • Neither module supports selective merging of configuration updates.

To fill these gaps, Config Merge has a sub-module, Config Merge Filter, that provides a Config Filter plugin that safely merges configuration updates into the site's active configuration, using the ConfigMerger::mergeConfigItemStates() method. For this three-way merge, the three states used are:

  1. Core's configuration snapshot, representing the configuration as last staged.
  2. The configuration as it's currently provided in the sync storage, with all previous configuration filters applied.
  3. The configuration as it's currently saved in the site's 'active' storage.

If you install Config Merge Filter and then run core's configuration synchronization, updates will be merged into the active storage, retaining customizations.

Potential improvements

While the current algorithm used in Configuration Merge covers the basics, there's plenty of room for improvement.

For one thing, we're currently playing loose with ordering when merging arrays. For associative arrays additions, we're just adding new keys to the end of existing arrays. For indexed arrays, we're doing some very rough handling of ordering by keeping track of deletions and swapping in additions. In both cases, we may end up with insignificant array ordering differences that show up as later differences between array states. An improvement would be to retain sort order when merging in array additions; see this issue.

More broadly, in Drupal 8, modules can provide configuration schemas that define the configuration they provide; see the documentation on configuration schema/metadata. A merge algorithm based on the configuration schema could be way more robust. See the issue Use config schema to ensure valid merging.

Related issue on Drupal core

Next up

Stay tuned for the next post in this series: Configuration Alters.

Sep 06 2018
Sep 06

This is the second installment in a series presenting configuration management work that comes out of the Drutopia initiative.

In part 1 of this series, Configuration Providers, we introduced the problem of managing shared configuration, starting with how to determine what configuration is available.

But knowing what configuration is available is only part of the picture. To import configuration updates, we also need to answer the question: what's changed since the configuration was originally installed? That's where snapshotting comes in.

Comparing states

To break the problem down a bit, when we update configuration that's provided by extensions (modules, themes, or the install profile), we have three different states to consider:

  • The configuration as it was provided by an extension when we first installed or last updated.
  • The configuration as it's currently provided by the extension.
  • The configuration as it's currently saved in the site's 'active' storage.

The second and third states are easy--we just read what's there. But the first state is tricky. When we update to a new release of a module or theme, we lose the previous version of the configuration. Not only that, but there may have been many intermediate versions between when we last installed or updated and now. So even if we knew what the version was right before we updated, that may or may not be the same as the version we installed or updated.

What we need is a snapshot: a record of the state of the configuration at the time it was installed or updated.

And if we're searching for a model of configuration snapshots, we don't have far to look. Drupal core uses snapshotting as part of its solution for staging configuration.

Snapshots in core

Using Drupal core, you can stage configuration between different environments or versions of a given site--say, from a development environment to a testing one. For details of how this works, see the Configuration Management section of the Drupal 8 handbook. The Synchronizing Configuration Versions section of the Drupal 8 User Guide also reviews this functionality.

When you're staging configuration, you might see this message:

The following items in your active configuration have changes since the last import that may be lost on the next import.

How does Drupal know there are changes? Because it has a snapshot of configuration as last imported.

Core uses a service, config.storage.snapshot, to provide a storage for configuration snapshotting. Shapshots are stored in a database table.

One of the components Drupal 8 gets from its use of Symfony is a system for creating and responding to specific events. For more on events in Drupal 8, see:

Core uses an event to create snapshots. When configuration is staged (imported), the ConfigImporter dispatches an event, onConfigImporterImport. The config_snapshot_subscriber service subscribes to that event and saves a snapshot. So the next time configuration is staged, there's a prior baseline to compare to.

Differences from the core use case

That's all very similar to what we need for the purposes of staging configuration. But there are important nuances.

For one thing, core needs only a single storage. A site's configuration is staged as a whole, so all configuration can be snapshotted together.

But when we're bringing in configuration updates from modules and themes, we have many different sets of configuration. If we import configuration updates from a single extension, we want to compare against and then refresh the snapshot for just that extension. So, rather than a single snapshot for the whole site, we need multiple snapshots, one per extension that provides configuration.

For another thing, in core the snapshot happens after staging configuration. Because the snapshot captures the result of staging, there's no need to stage the snapshot itself.

When importing configuration updates from extensions, however, the snapshot typically happens before staging, and what we stage includes the result of the update operation. So we do need to be concerned about staging the snapshots. That means, rather than being stored just in a database table, the snapshots themselves need to be stored as configuration that can be exported and staged.

Configuration Snapshot

The Configuration Snapshot module addresses the need for creating and using multiple configuration snapshots that themselves can be staged.

In Drupal 8, configuration can be stored using Drupal's entity system. To do so, you create a custom configuration entity type.

So the first step is to create a custom configuration_snapshot entity type. This allows us to store snapshots.

But reading and writing directly to an entity isn't going to work for our purposes. As noted in part 1 of this series, the standard ways to read, write, and compare configuration use configuration storages. So we provide a custom ConfigSnapshotStorage that provides all the standard configuration storage functionality, such as the ability to read from and write to a storage. Only in this case the storage reads from and writes to a snapshot configuration entity. Modules using configuration snapshots don't have to directly work with the snapshot entities. Instead, they can use storages that take care of the details.

The Nitty Gritty

For those interested in the messy details, here they are!

To facilitate working with the configuration snapshot storages, we register a service for each storage.

The usual way to register a service is by adding it to a simple *.services.yml text file; see relevant documentation. In our case, though, we can't know in advance which services are needed on a given site since we need to register one service per installed module or theme that provides configuration. So we use a slightly more complex pattern to provide dynamic services.

An analogous requirement in core is for multilingual-related services. These services are needed only if a site is multilingual. Determining whether a site is multilingual, in turn, depends on the number of languages installed and enabled on the site. In other words, it depends on the current state of configuration.

Core handles this case by providing a service provider class, LanguageServiceProvider. Using the ::register() method, that class tests the installed languages and, if appropriate, dynamically registers the relevant services.

We do pretty much the same thing in ConfigSnapshotServiceProvider. We load all snapshot entities and register a service for each one.

Often you need to call the same snippets of code in multiple contexts. One pattern for doing this is to use a trait. Traits allow code reuse. More technically, as explained in the PHP documentation, traits

[enable] a developer to reuse sets of methods freely in several independent classes living in different class hierarchies.

Traits are used extensively in Drupal core. One of the most commonly used traits provides text (string) translation. It's called, appropriately enough, StringTranslationTrait.

There's a bit of complexity to using snapshot configuration storages. If there's an existing service we want to use it, but if not we need to spin up a new storage. So we make this happen in a trait, ConfigSnapshotStorageTrait.

A final issue addressed in Configuration Snapshot has to do with deciding what differences are actually significant.

Sometimes what shows up as a difference can turn out to be irrelevant. For example, a user role in Drupal can have multiple permissions assigned to it. The order of those permissions in a piece of configuration makes no difference to what permissions the role will have. So, if the order of permissions is different between a prior snapshot and the current configuration state, there's no actual update to import.

To avoid spurious updates, we normalize the data by sorting in a predictable way. Here we base our solution on relevant code from what is by far the most-used Drupal 8 contributed module for configuration management: Configuration Update Manager. We crib from the ConfigDiffer::normalizeArray() method in that module that normalizes configuration.

The StorageComparer class in core is used to compare one storage to another to determine differences (items to create, update, delete, or rename). To test for updates, StorageComparer::addChangeListUpdate() uses simple equality between a given item as read from two different storages, meaning that differences in ordering will indeed show up as available updates. This is the relevant line:

if ($source_data !== $target_data) {

To change the behaviour of the storage comparer, we need to alter the way configuration is read from the storage. In that alter, we'll add normalization. That way, when the storage comparer does its work, insignificant differences won't show up.

Importantly, we don't want to change our storage so it always normalizes. Doing so could have unintended consequences. Rather, we want to provide the option of normalizing on read, and trigger that option only when we're reading for the purposes of comparison. To do so, we need to change a standard configuration storage into one that will optionally normalize data on read.

We have an interface to work with, StorageComparerInterface. We extend this interface to add our own methods. Our custom interface, NormalizableStorageInterface, defines methods that relate to whether data should be normalized on read.

Then, in our ConfigSnapshotStorage, we implement NormalizableStorageInterface by providing all the public methods required by core's StorageInterface and our custom ones defined in NormalizableStorageInterface. That way, we can invoke normalization when needed while leaving the default read behaviour for all other cases.

Potential enhancements

The normalization we're doing is needed to minimize spurious differences showing up, but it cuts a lot of corners. For one, not all differences in ordering can be safely ignored. Some may in fact include important data that we're incorrectly filtering out when we update. Beyond that, a module focused on snapshotting is not a great place to put a general-use interface for normalizing config data. The whole point of normalizing is to facilitate comparing different storages, and in our case that means comparing a snapshot to another storage where we will also need to normalize.

There are a couple of issues on Configuration Update Manager to try to sort these questions out.

Next up

Stay tuned for the next post in this series: Respecting Customizations.

Aug 27 2018
Aug 27

At the Drutopia project, one of our big focuses has been improvements to configuration management in Drupal 8. In this series, I'll be covering our work to date along with related efforts and contributions.

Drutopia is a platform cooperative initiative, building out cooperatively owned and hosted Drupal distributions. In our 2016 white paper, we recognized that the Configuration Management Initiative (CMI) in Drupal 8 "produced a lot of improvements in configuration handling" while noting that these "mainly addressed the use case of 'staging' configuration from one version of a site to another, a site-building technique that lower budget sites often don’t have time or money for." We committed to focus on "the free software use case left out of Drupal core: reusable configuration that can be shared across multiple sites". For background, see Drupal 8 configuration management: what about small sites and distributions? and sections on Drupal 8, corporate influence, and the CMI in this interview.

There's a current initiative to improve configuration management in Drupal core. Dubbed "CMI 2.0", the effort comes out of a similar conclusion that limitations and missing use cases in configuration management are a major barrier to Drupal 8 adoption; see Angie Byron's post proposing the initiative.

In the past three years, we at Drutopia have contributed to a growing collection of Drupal plugins that together address some of the tricky problems involved in managing shared configuration. As well as in kind contributions by Chocolate Lily, some of our work was sponsored by Agaric and the National Institute for Children's Health Quality (NICHQ) to meet their needs for an in-house platform of community sites.

Just what do we mean by managing shared configuration?

Say I have a site built on a Drupal distribution that's for community organizing. I installed the site a month ago and got groups-related configuration such as a group type. Then I made some modifications of my own. I've just downloaded a new release of the distribution, including enhancements to the groups-related configuration. How can I update my site so that I have all the latest changes from the distribution--while still retaining any customizations I made? That's the key question we've tried to tackle.

A more abstract way of putting the problem is: how can we provide packages of shared configuration in a way that lets site administrators both customize their sites and merge in configuration updates?

This series will cover distinct aspects of the problem of managing shared configuration packages and, along the way, highlight specific solutions we at Drutopia have sketched in. Our efforts are very much works in progress. We're not sure we've even got all the problems right, let alone fully addressed them ;) But have we made progress? Yes, we have. By sharing it here, we hope to raise the profile of these problems and solutions and invite further perspectives and contributions.

Here are the aspects we'll cover, along with associated Drupal modules:

  1. Configuration Providers (Configuration Provider)
  2. Configuration Snapshots (Configuration Snapshot)
  3. Respecting Customizations (Configuration Merge)
  4. Configuration Alters (Config Actions, Config Actions Provider)
  5. Updating from Extensions (Configuration Distro, Configuration Synchronizer, Config Filter)
  6. Shared Configuration (Configuration Share)
  7. Packaging Configuration (Features)
  8. Summary and Future Directions

As we go, we'll briefly introduce some relevant concepts in Drupal 8 development as they come up, while pointing to sources of further reading.

First up, configuration providers!

Updating configuration from extensions

When you install a Drupal extension (a module, theme, or install profile), you'll often get configuration installed on your site. That's because the extension provides that configuration. Technically, the configuration is sitting in files in one of two directories within the extension.

Files that are in config/install are installed when the extension is installed. There's also optional configuration: files in config/optional that are installed when certain conditions are met. For details about how this works, see the documentation on drupal.org.

This system works great for the use case Drupal core supports: one-time installation of configuration provided by extensions. But when it comes to updating extension-provided configuration, we run into problems.

In order to run configuration updates, we need to know what updates are available. For extensions, this comes down to the question: what configuration would be provided if the currently installed modules were reinstalled?

Because core is tightly tied to the model of one-time installation of extension-provided configuration, relevant code is not available in a form we can use. A key problem is with optional configuration. In core, the logic to determine what optional configuration will be installed is embedded in ConfigInstaller::installOptionalConfig(). But this method doesn't just answer the question of what will be installed--it also goes ahead and installs it. Ouch.

Beyond optional configuration, a second issue is that core's code is limited to the two use cases covered by config/install and config/optional. In practice, though, contributed modules define configuration-providing directories of their own to address additional use cases. Examples of such are the Configuration Share module, which provides a config/share directory, and the Config Actions module, which enables programmatic alteration of configuration through actions provided at config/actions. (More on both of these modules later in this series.)

So, as one part of the puzzle of how to update configuration from modules and themes, we need a solution that can support configuration providers, both those in core and any that contributed modules might provide.

Configuration Provider

The Drutopia-supported Configuration Provider module addresses this use case by providing a ConfigProvider plugin type.

Drupal 8 makes it possible for modules to define their own types of plugins using the plugin API. For example, an image effect like cropping or rotating can be provided as an ImageEffect plugin.

A plugin of the type ConfigProvider satisfies a given use case for providing configuration from an extension. Configuration Provider ships with plugins for core's two use cases, config/install and config/optional.

Any module can meet additional use cases not covered in core by providing a new plugin. Two such examples are configuration provider plugins provided by the Config Actions Provider and Configuration Share modules.

The nitty gritty

For those interested in the technical details, here you go!

Drupal 8 modules can provide services: objects that do work and allow other code such as modules to access the functionality they provide. Services are one of the main approaches Drupal gets by incorporating elements of a framework called Symfony. For more background, see the relevant Symfony and drupal.org documentation. One type of functionality that can be exposed in a service is a storage for configuration. For example, Drupal core exposes services for the site's 'active' configuration storage and for the 'sync' storage that's used to synchronize configuration between different environments.

Drupal's configuration storage API allows configuration to be stored in multiple ways, like in the database or in files. Each configuration storage service can specify what type of storage is used. The way this is done is that Drupal core provides a storage interface. An interface is like a blueprint or contract for what a particular class (a code object) needs to do. Interfaces make it possible to have a standard answer to what a class needs to do, while leaving the question of how to do that open.

A particular class might have any number of methods, functions that are called to do something. A particular method in a class has what's called visibility, which determines where the method can be called from. Many may be private (accessible only from the class itself) or protected (accessible from the class or any of its parents or children). But the methods defined in an interface are public, meaning they can be called from anywhere. This makes sense. The whole point of an interface is to define what a class can reliably be called upon to do. So at its most basic, an interface defines a specific set of public methods a class needs to have. To implement that interface, a class just needs to have all of those methods. For more on interfaces, see the PHP documentation and a relevant section of Drupal's coding standards.

In core, there are multiple classes that implement the StorageInterface. The active configuration service uses database storage by default, while the sync service uses files.

Okay, back to the example at hand. In Configuration Provider we don't need to permanently store the configuration, so we don't need a database or file storage. Instead, through our own StorageInterface implementation, we provide a custom storage type: InMemoryStorage. This does pretty much what its name suggests: hold configuration in memory. Then, using that storage type, we provide a storage service, config_provider.storage. This is used to store configuration during the process of determining what's available on the site.

In a second service, config_provider.collector, we provide a public method ::addInstallableConfig() that can be used to populate the config_provider.storage storage with configuration that would be installed if all currently installed extensions (modules, themes, and the install profile) were reinstalled. Here's a sample usage:

$collector = \Drupal::service('config_provider.collector');
// Add installable configuration to the `config_provider.storage` storage.
$config = $collector->addInstallableConfig();
// Get the service for the storage that contains the installable configuration items.
$provider_storage = \Drupal::service('config_provider.storage');
// List available items.
$item_names = $provider_storage->listAll();

This approach can be used as part of an update workflow, to determine what updates are available from installed extensions. Spoiler alert: that's what's done in Configuration Synchronizer, which we won't cover until much later in this series.

Okay, confession time: to get around a thorny issue, we've resorted to what can only be called a messy workaround.

Because we're making it possible for contributed modules to define their own logic for providing configuration, we need to allow them to act when extension configuration is installed. When Drupal core is installing configuration, we need to also call our provider types. That's the domain of core's config.installer service. So, to add our bit, we need to alter core's service.

On the plus side, the ability to tweak and refine the way a service works is built into services--in fact, that's a big part of why Symfony and by extension Drupal provides services in the first place.

When altering the default way an existing service works, there's a "right" way to do it. That's to use a pattern called "decoration"--see the relevant Symfony documentation. By decorating a service, we add just the specific customization we need, while leaving the way open for other modules to also make their own tweaks. For more on decorators, see the blog posts Drupal 8: Service Decorators and Get a decorator for your Drupal home.

But whether or not decorating a service works depends on what we were talking about before: the interface that backs a service. As noted above, an interface defines a set of public methods a class needs to have. When you're decorating a service, it's just the interface-defined public methods you've got to work with.

And there's the rub. In our case, the method we need to alter is ConfigInstaller::getConfigToInstall()--and it's not public but protected. Again, this situation probably reflects the use case core was designed to meet. Since the assumption was that extension-provided config is only installed once, there was no need to make the ConfigInstaller::getConfigToInstall() method public.

Since decoration is out, we need to resort to a more heavy-handed approach: a service alter.

In our alter, we swap in a custom class that extends the core ConfigInstaller class to provide our own ::getConfigToInstall() method. In it, we pass the request to all available configuration provider plugins. We also pass configuration data provided by previous provider plugins, allowing a plugin to accomplish tasks like (a) fulfill configuration entity dependency requirements on demand (as in Configuration Share) or (b) modify configuration prior to it being installed (as in Config Actions).

Altering the service in this way is heavy-handed because it can only be done once on a given install. If another module used the same trick, one of the alters would fail, leading to breakage. Hence the appropriate warning in the drupal.org documentation:

you should use caution when making use of this feature.

Potential enhancements

Configuration Provider kinda fills the need. But are there ways it could be improved? For sure.

Configuration Provider introduces plugins that allow altering of configuration storages. That's more than a little reminiscent of what Config Filter does in a much more general way. So one option would be switch to using Config Filter plugins instead.

But Fabian Bircher, the intrepid developer of Config Filter and the related Configuration Split module, is working up a new and improved architecture for a "configuration transformer API" in this core issue. The basic idea: instead of plugins, pass and transform configuration using Drupal's event and event subscriber pattern. So, likely, whether in Drupal core or in contrib, a future iteration of Configuration Provider should wait until that proposed API is ready to roll.

Or the need could be met directly in core through a switch to an event-driven architecture when marshalling extension-provided configuration to be installed--or updated.

Related core issue

Add a Config Filter storage and plugins for extension-provided configuration.

Next up

Stay tuned for the next post in this series: Configuration Snapshots.

May 05 2017
May 05

In the current context of a Drupal leadership crisis and debate about project governance, it's important to reflect on ways an authoritarian structure has shaped and continues to shape the culture of the project. In this vein, response to a 2007 post by Drupal contributor Gus Geraghty makes for a fascinating, if disturbing, case study.

I recommend reading (or rereading) that thread before continuing here. I've deliberately chosen an example from earlier days to emphasize how tensions in the project, and patterns of response, have persisted and shaped the project at key junctures. I also hope that some distance may help to set those events in a reflective light, where the focus is not on who did what but on what we can learn about the overall organizational culture.

Those who raise critical questions are making a valuable contribution. Particularly in an authoritarian structure, speaking up is risky.

In his post and followup comments, Geraghty directly questioned the dictatorship power structure of Drupal, focusing on the then-new commercial interests of Drupal founder and dictator for life, Dries Buytaert, and his company, Acquia. Geraghty proposed a concrete alternative: reorganizing the project along cooperative lines. In follow-up comments, he pointed to the Linux Foundation as a possible model, structured to ensure no one company could attain dominance in the software project:

it fosters the growth of Linux by focusing on protection, standardisation and providing a neutral forum for collaboration and promotion. It also sponsors the work of Linus Torvalds, as opposed to a commercial interest paying Linus.

The response was immediate, pointed, and overwhelming.

Key elements:

  • One after another, prominent Drupal community members publicly declared their faith and trust in the project leader.
  • Commenters appeared to claim they were demonstrating a high level of patience and tolerance while publicly denouncing Geraghty in sharp and unrelenting terms.
  • In the midst of the discussion, Drupal site admins revoked Geraghty's privileges as a Drupal.org administrator. They insisted this punishment was in no way linked to his critical comments.
  • Ten days after the initial post, a site admin locked down the post, preventing further comments and leaving a note that seemed to cast blame exclusively on Geraghty.

Whatever the intentions of those who denounced him, the message that came across to Geraghty was loud and clear. He wrote:

the cheerleaders jumped down my neck and attacked me, insulted me and generally lambasted me for just raising this topic.

He also noted:

I've received more than a few emails since this thread kicked off that suggests not all of the community share the same sentiments, but, all are reluctant to speak their minds.

For obvious reasons. In an authoritarian system where key positions of prestige, influence, and power derive from the favour of the leader, loyalty may be the highest currency.

The public attacks on Geraghty's credibility and character fit with familiar patterns in authoritarian systems of delegitimatizing dissent.

Attacks on critical voices do more than drive people from the community. They also cast a long shadow on those who remain.

I'm not suggesting critics should get a free pass. Still, what's striking to me, looking back, is not only the overwhelming attack. Rather, it's the apparent near consensus that Geraghty himself was solely responsible for the excoriating treatment he was subjected to.

"The Scientology of open source"?

In a thought provoking reflection after he left his central roles in the Drupal project, Steven Wittens - for years Buytaert's second in command as core maintainer, author of much of the early core code base, co-designer of the Druplicon - called Drupal "the Scientology of open source". Some dismissed his analogy, while others cheerfully adopted the label of cult member.

Of course, Drupal is not a cult. But analyzing the project as an authoritarian system, the lens of a cult is not entirely misplaced.

The International Cultic Studies Association journal maintains a list of characteristics of cultic groups. Here is a selection:

  • The group displays excessively zealous and unquestioning commitment to its leader
  • Questioning, doubt, and dissent are discouraged or even punished
  • The group is elitist, claiming a special, exalted status for itself, its leader(s), and its members
  • The leader is not accountable to any authorities
  • The leadership induces feelings of shame and/or guilt in order to influence and/or control members
  • Members are expected to devote inordinate amounts of time to the group and group-related activities
  • The most loyal members (the “true believers”) feel there can be no life outside the context of the group

Reflecting on these attributes with reference to Drupal can be sobering.

Making space

The current discussion of governance in the Drupal project does not start from a blank slate. Rather, it occurs in an authoritarian context where many positions of power and influence in the project are closely associated with the dictator for life and his company, Acquia. In diverse ways and to different degrees, individuals who owe their positions in the project to Buytaert, who were in part selected by and serve under Buytaert at the Drupal Association, who are current or former Acquia employees, or who are owners or staff at Acquia partner companies may be materially invested in the status quo power structure.

I'm very familiar with how this works and feels. My first Drupal job was a direct result of Buytaert recommending me to CivicSpace Labs, based on my contributions to Drupal core. Like uncounted others, I owe my career in Drupal directly to the intervention of the project founder and dictator.

All community members should feel free to express their views. But placement makes a difference. In the current leadership crisis in Drupal, of the many individuals who seemed to call for a quick return to the status quo, few have been up front about their placement and investment.

How do you make space for those who have been silenced? I'm no expert, but I can offer some suggestions based on my own experience.

  • You start with acknowledgement. Drupal has a long and troubled history when it comes to dissenting views, and you don't deal with that legacy through omission or denial.
  • You're self-reflective and honest about your own placement and privilege.
  • You listen.
May 04 2017
May 04

In recent weeks Drupal community members have raised many questions about the Drupal Association (DA). Can the DA serve as an effective balance to the powers of the project's dictator for life, Dries Buytaert? Why does Buytaert have his name written into the bylaws with a reserved board seat? Is the DA structured in such a way that it could reliably address potential conflicts of interest involving Buytaert's company, Acquia? What's the history?

Community members have also raised questions about transparency in the Drupal Association.

In the absence of reliable information, speculation has been rampant.

These questions come as the Drupal Association has announced plans to facilitate a process of community consultation around governance of the Drupal project. For these discussions, it's especially important that community members have access to frank information and perspectives about the DA.

In this piece I hope to help address the knowledge gap by filling in some of that missing background. I have years of experience in the two incarnations of the Drupal Association--the Belgian-based VZW (the first Drupal Association, which I'll here call DA-VZW) and the US-based Drupalcon Inc (the current Drupal Association, which I'll here call the DA). This experience includes:

That said, comments here are my own and in no way reflect the views of either DA-VZW (which, in any case, is now defunct) or the DA.

A bland backgrounder I compiled in 2012 may be useful by way of intro, but in what follows here I promise way more of the gritty detail!

Conflict of Interest in DA-VZW

By 2008 when I was elected as a permanent member of DA-VZW community members had raised concerns of potential conflict of interest related to Acquia, the company Buytaert had co-founded in 2007.

I proposed we as DA-VZW take a proactive approach that would include publicly acknowledging perceived issues. In March, 2008 I drafted and circulated internally a reflection: "What does Acquia mean for the Drupal Association?" I hoped we could use this draft as a springboard for something we would share publicly and use as a guide going forward, providing transparency to our community and clarity in distinguishing roles and responsibilities. See the end of this article for part of what I wrote.

I also drafted bare-bones conflict of interest guidelines for the DA-VZW, circulated them for review, and then posted them to the DA-VZW website.

The immediate response to my efforts was a wave of personal attacks against me. To say that attention to potential conflicts of interest and questions related to multiple roles of the Drupal founder were not encouraged would be an understatement.

In the absence of effective (really, any) response by the DA-VZW leadership, these attacks succeeded in shutting down discussion, let alone action, related to conflicts of interest or the multiple and potentially conflicting roles of the project founder and dictator for life. The tame and minimal conflict of interest guidelines I'd posted came down and were not soon replaced.

Transition to the Current Drupal Association

In 2011 a DA-VZW board committee proposed sweeping changes in the organization, including:

  • Eliminate the voting membership, which until that point had elected the board.
  • Merge most functions of DA-VZW into the DA, effectively moving operations from Europe to the US.
  • Write Buytaert into the DA bylaws with a permanent, dedicated director class.

These discussions took place behind closed doors, with little or no information shared with the Drupal community until all details were effectively locked down.

I and several other members argued strongly against several aspects of the proposal. But in a heated atmosphere in which concerns and suggested changes were anything but welcomed, all significant proposals for change in the plan were denied. Multiple permanent members resigned.

By the time the proposed restructuring was posted publicly, it was pretty much a fait acompli. There was little or no Drupal community engagement on the key proposals. At the time in a public post I noted concerns with the restructuring:

It would see us eliminate the permanent membership, leaving the board as the sole decision making body. I'm concerned that this change could if anything increase the degree of centralization in the DA. Could we instead expand voting rights to the full membership, currently at over 2,000 individuals and organizations? I'm also concerned at the idea of relocating the DA to the United States. Would this increase the degree to which decisions in the DA come from a mainly US perspective?

Advisory board

As part of the structuring of the current DA, an advisory board was created. The current listing (from which, at my request, I was only recently removed) claims this "international network of trusted community members" was

selected for their acute experience, knowledge, expertise, and talents in many different areas including community management, policy, outreach, law, technology, project guidance, or education.

The actual story is a lot more prosaic.

As part of the restructuring of the DA, we as DA-VZW permanent members could opt to stay on as members of an advisory board. Of those permanent members who hadn't already resigned (in protest or otherwise), many did choose to stay on. Since that time, no new members have been added to the advisory board, though some (me among them) have resigned.

First Drupal Association conflict of interest committee

I served as a member of the inaugural DA conflict of interest committee.

Given the confidential nature of reporting, I can't discuss specific allegations that were brought to our attention. I can say, however, that after months of what appeared to me futile and ineffectual effort I concluded we as a committee lacked the mandate or institutional backing that would be required to minimally meet our responsibilities. Feeling my continued membership on the committee could only lend legitimacy, I tendered my resignation.

First Drupal Association Board Nominations

In 2011 I also served on the nomination committee that selected a first slate of candidates for the reorganized DA.

Our committee was composed overwhelmingly of cis-gender male developers, me being one.

I did outreach and worked my networks, focusing especially on female-led organizations, grassroots nonprofits, and the global South. After a long process involving compromise all around, we as a committee put forward a slate of candidates representing our unanimous agreement. While largely status quo, the resulting slate did include some expertise and representation that would address key gaps in the DA.

However, the three-person DA board overruled our recommendations and did not accept the full set of candidates we put forward, with the result that sectors and areas of expertise were not - and have never since been - directly represented on the DA board.

Then as now, Buytaert was a member of the nomination committee and also a board member who voted on the resulting slate.

Following the board appointments, I wanted to provide our community with detailed information on our process, information that would serve to keep our community in the loop and also could inform future nomination processes. Removing all personally identifying information on candidates or other sensitive information, I prepared an extensive writeup on the process with details including our selection criteria. My effort had support on the nomination committee.

But, after extensive delays, the DA board forbade me to release the report.

Community elected board members

In 2011 I facilitated the process that designed and carried out the first DA at-large director elections. After years of closed-door processes where minority and critical voices were repeatedly drowned out and information sharing blocked, my first aim was to design a process that would be open and inclusive. Working with a skilled and dedicated team of community volunteers, we focused on facilitation, reflective and respectful debate, and building consensus.

From the start, every detail would be worked out in public. You can't block the release of what's not secret in the first place.

The DA's record

With reference to questions community members are raising, what does the current DA structure look like on the ground?

Here are some vignettes:

  • By 2014, multinational corporate affiliations were front and centre when then executive director Holly Ross announced new DA board appointments:

    I’m excited to announce that the Drupal Association Board of Directors has filled two open seats on the Board. Please welcome Mike Lamb of Pfizer and Rob Gill of NBC Universal.

    Ross cited their "extensive experience as end-users of Drupal" representing "a perspective that currently isn’t fully realized on the board". Clearly, however, it was a very specific sort of end user that was deemed appropriate for board membership.

    Both Pfizer and NBC had been profiled as clients on the Acquia website prior to these board appointments. Mike Lamb took over the DA board executive position of secretary after Acquia employee Angela Byron left that position. Following his DA board appointment, Rob Gill was hired at Acquia, where he served as vice-president. He has since left the DA board.

  • Acquia has a partnership program in which companies active in the Drupal space can apply for and achieve partnership status. More than 1,200 companies are listed as partners on Acquia's website.

    Today, according to company information on Drupal.org and Acquia's website, a third of the DA board members are staff or owners at Acquia partner companies, while a further board member is from a company that is part of the "WPP-Acquia Alliance", a "global partnership" with Acquia

  • Drupal.org includes a "marketplace" section where companies offering Drupal services are listed. In 2015, the DA introduced a ranking system for marketplace listings. Prior to this change, companies were listed alphabetically, with a default filter based on demonstrated contribution to the Drupal project. A DA staffperson responsible for the new ranking system attributed the initiative to Buytaert: "Back at DrupalCon Austin (2014), Dries presented a sketch of an idea for highlighting organizational contributions to Drupal to myself and several business owners in the Drupal community."

    Since the algorithm was introduced, Acquia has perennially appeared first in marketplace listings.

    When the system was introduced, community members (me among them) pointed out an inherent bias towards large Drupal companies. Although many other adjustments were subsequently made to the algorithm, addressing that bias was not among them. Acquia retained its first-place ranking.

  • In 2016 the Drupal Association introduced a new advertising program in which the first prominent link on the Drupal.org home page, "Try Drupal", points to a page with sponsored ads. At time of writing, Acquia is one of three companies in the "Try Drupal" advertising program (and is the only member of the "Signature" hosting supporters program, the highest level offered by the DA).
  • The DA has a conflict of interest committee, described as an "independent body". The three members of the all-male committee, drawn from the advisory board, are executives or owners of their respective Drupal companies, two of which are Acquia partners. As former permanent members of DA-VZW, the three all could be described as longtime insiders.
  • One of these insiders also sits as the only non-board member on the DA nominating committee, which is responsible for vetting and putting forward board candidates. Buytaert is also a perennial member of this currently all-male committee.
  • Of the 12 DA board members, 7 are listed as being in the US. There is one board member from the global South, who was selected not thanks to the nominating committee but instead through community elections (which as mentioned fill only two of the board positions).

Inclusion/Conclusion

The DA today is calling for a respectful, open, facilitated, and inclusive dialogue around Drupal project governance. That's great. But these laudable ideals are far from the conditions in which the DA itself was formed.

In a recently-conducted informal "State of Drupal Poll", a majority of respondents indicated "there are conflicts that need to be resolved" regarding Buytaert being "at the top of both Drupal and Acquia" while nearly half of respondents indicated Drupal's leadership structure including the DA is either "broken and needs fixing" or "needs replacing in order to meet the needs of the community."

Returning to what I wrote more than nine years ago:

There is a potential concern that, consciously or unconsciously, the Drupal Association may select or carry out projects and activities in ways that disproportionately benefit Acquia or align with its particular corporate interests.

If, to many community members, the Drupal Association appears broken in terms of its ability to serve as a balance to the unchecked powers of the Drupal dictator for life, or to the corporate interests of his company, the outcome is not through mere oversight or omission. Rather, there is a very specific history, one in which those who have raised questions have repeatedly been shouted down, sidelined, and silenced.

I know there are many well intentioned and dedicated people working within the Drupal Association. But their presence hasn't altered the built-in structural problems of the organization.

I and others warned that, as designed, the Drupal Association would risk becoming an even more insular, US-centred body, dominated by a narrow set of interests and neither representative of nor accountable to its community.

Our warning presaged themes and issues community members are raising today.

But if in this article I've been speaking so far as a critic, I have to acknowledge I've also been part of the problem. If the DA is the product of an old boys' network, I was a longtime member. In some ways I still am.

More than that. Even after explicitly calling out inherent flaws, I stayed on in the misplaced belief that, as a supposedly critical voice, I could make a meaningful difference. Even the energy I brought to what I hoped was an inclusive, consensus-building process to design community elections arguably served only to legitimate, since - by design - the two elected community representatives on the board could never form more than a marginal segment of the otherwise self-perpetuating board.

I won't be participating in the community consultations the DA is convening. Given that on multiple occasions I've been targetted in the community in relation to critical perspectives I've raised, and that little or nothing has been done to acknowledge (let alone address) these attacks, there are many fora where I don't feel safe.

To belabour the obvious: an organization presided over by a dictator is not ideally positioned to lead an inclusive process of renewal.

The DA has yet to come to terms with the deep ways that it itself is a product of an authoritarian project structure or how it has been shaped by particular economic interests.

A good first step? Buytaert could step down from his role as president and as board member. That wouldn't do everything, but it would send a first essential signal.

[Draft prepared by Nedjo Rogers for discussion in March, 2008. Never adopted by the Drupal Association.]

The Association and Acquia

The Drupal Association was founded in 2006 with the mission of supporting and promoting the Drupal software project and the needs of the Drupal community. The Association receives donations and in-kind contributions from the Drupal community to support its work. A key requirement of carrying out its work and mandate is ensuring the Association both is and is perceived to be balanced and free from any particular private influence or motivation. Meeting this requirement includes ensuring that no particular group or interest receive special treatment or access.

Acquia, a new startup Drupal-based company, presents special challenges in terms of relations with the Association. Compared with other existing companies active in Drupal development, Acquia is special in two ways.

The first is Acquia's core strategy and aims. Acquia has announced goals that closely parallel those of the Association itself. Already Acquia has served as the largest single contributor to the Drupal Association's 2008 Boston Drupalcon.

The second is the position of Dries Buytaert as Acquia co-founder. Dries is the founder and ultimate decision maker in the Drupal software project. Dries is also a principal founder and current founding president of the Drupal Association. Dries' triple leadership role - in Drupal, in the Drupal Association, and in Acquia - creates unique possibilities and challenges. These challenges are increased as other leaders in Drupal development and in the Drupal Association converge in Acquia.

Challenges

The Drupal Association is aware of two main sets of potential concerns in relation to Acquia.

  • The first concern area is about influence. There is a potential concern that, consciously or unconsciously, the Drupal Association may select or carry out projects and activities in ways that disproportionately benefit Acquia or align with its particular corporate interests.
  • The second concern area is about role confusion or conflict. If the Association and Acquia are both active in the same areas or projects, there may be confusion in the community as to which capacity individuals are acting in.

Addressing Concerns

Dries and others in Acquia are respected as leaders in the Drupal community. Personal integrity is invaluable but is not in itself enough to provide full transparency and accountability.

The Association membership and directors are very concerned with the responsibility of impartiality. To address the possibility of conflict of interest, all directors and members are required to communicate about any potential conflicts and to recuse themselves from discussions and decisions that would put them in a real or perceived conflict.

Given the special considerations noted here, without unfairly penalizing Acquia, the Association is taking extra steps to review relationships with Acquia. The Association will [insert specific steps here].

The Association looks forward to working with Acquia as with other Drupal firms active in Drupal development and promotion. As always, open and direct communication with the Drupal community is a key part of understanding and addressing concerns. Please let us know how we're doing. You can address questions, concerns, or suggestions via the contact form at [link].

Apr 05 2017
Apr 05

As controversy swirls around the Drupal project leadership, community members are asking searching questions about the role of corporate influence in the project.

It's a good time to take a step back and look at broader questions of economic structure. What different economic models could help spur alternatives? What would an economic model look like where free software is owned and shaped by users as well as producers?

Fortunately, there are many dynamic and successful models to learn from.

User Economics

Users are the experts in their own needs. As they start to find their way around software, users have the potential to be invaluable peer teachers and mentors, able to pass on information and tips in a style free of the jargon that often seeps in when developers try to teach. Yes, exercises like bringing together focus groups and doing usability studies have their place, but they’re no substitute for users who are fully engaged and involved.

However, there are significant barriers to bringing users in as shapers of software. One is simply available time and resources. At a typical small organization, staff or volunteers are hard-pressed to keep on top of their own work. Taking on a significant role in outside projects that seem to have little to do with their job description or organizational mandate? Not likely.

Related to time constraints is the lack of familiar models. Most groups are used to contracting for services, but less familiar with models that may involve contributing back. Viable alternatives need to fit with the resources that organizations can realistically contribute, and do so in ways that are attractive and understandable.

For example, members of cooperatives may be used to the idea of working with other cooperatives. Similarly, nonprofit staff and volunteers may be used to the idea of working in networks. Strategies that fit with these existing networking links may be more compelling. And groups may be better able to allocate staff resources if doing so is structured as part of receiving services, like through a website build or support contract.

Scrum Writ Large?

Scrum is a software development approach familiar to many in the tech sector.

In a scrum project, stakeholders in the software are represented by a “product owner” who develops and prioritizes the “user stories” (use cases) that the project will meet, and also has a key role in ensuring that those requirements have been met.

On a scrum-style individual Drupal site build, the product owner might be a staff person at the client organization. By consulting with colleagues and others at their organization, the product owner gathers and channels input to ensure that what’s built intimately reflects actual needs and priorities.

In a sense, scrum can be thought of as an analogy for the challenge of creating new user-centered economic models. How can users gain an effective collective voice to shape the software for their needs and priorities?

Incentives to Share Design and Configuration

Alternative models need to have logics that mean that pooling design and configuration work – not just abstract functionality – is built in rather than being merely incidental. Likely it means something along the lines of what have so far been produced as Drupal distributions. However, rather than one-off and incompatible distributions produced by single shops, viable alternatives will need to channel efforts into collaborative and compatible solutions.

Crowd Sourcing

The idea that catering to the largest users will bring in resources that makes the software better for everyone – what might be called the “trickle down” theory of free software design – has obvious problems. In some cases, what’s good for the biggest multinational corporations will be antithetical to the needs and priorities of the vast majority.

A successful model will need to enable many users to pool their resources – both financial and in kind – to make small contributions that add up to big changes.

Also important is enabling collaborative development among many developers and shops. Individual collectives or companies may have their focuses, such as particular distributions, but they need to fit into a larger whole, including pooled components.

Cooperatives and Shared Solutions

Free software movement founder Richard Stallman says of the early days of the GNU project, “our goal wasn't just to be popular; our goal was to give people liberty, and to encourage cooperation, to permit people to cooperate.” With its focus on pooling knowledge and working for common benefit, free software from the start had strong ties to the ideals of the cooperative movement.

The seven international cooperative principles – voluntary and open membership; democratic member control; member economic participation; autonomy and independence; education, training and information; cooperation among cooperatives; and concern for community – provide solid bases for cooperative ventures.

Two common types are consumer and worker cooperatives. A consumer cooperative is owned by the consumers or customer-members and run for their mutual benefit; a worker cooperative is owned by the people who work for it – the worker-members.

In the U.S., the National Information Solutions Cooperative (NISC) is an example of a larger software-focused consumer cooperative. Member-owners of the cooperative are mainly utility cooperatives and telecommunications companies. With more than 1,000 employees, NISC produces software products that run systems used by more than five million consumers.

In the NISC case, though, access to the software is an incentive for membership. If the software is available without charge, what is the incentive for users to pay for it?

Likely the best way to learn about alternative models is to sample what’s already out there. What are some successful models for inspiration?

Alternate and Local Currencies

The Spice network enables members in the United Kingdom to earn time-credits through volunteer activity and, in turn, spend them on any of a huge range of services.

In Japan, Fureai Kippu (Caring Relationship Tickets) provide care for older people through exchanges of time credits.

These are just two of thousands of alternate and local currencies that are being used to bridge the gap between economic need and ability to pay by resolving the dichotomy between buyer and seller.

There are at least two Drupal-based alternative currency initiatives: Community Accounting forms the basis of the Community Forge distribution. And the Integral Community Exchange System is a suite of modules for local currency management.

Used creatively, time banking could play a role in facilitating users to both contribute to and benefit from software development.

Donation Ware

Some software projects incorporate voluntary financial contributions into the workflow for download. Libreoffice, for example, presents users who download the software with a donation form and the following message:

LibreOffice is Free Software and is made available free of charge. Your donation, which is purely optional, supports our worldwide community. If you like the software, please consider a donation. We have only a very few employees, supporting a community of hundreds of developers and other volunteers, serving tens of millions of users worldwide.

Free Software Cooperatives, Collectives, and Other Alternatives

In Buenos Aires, Argentina, the gcoop workers cooperative use Drupal and other free softwares as a base for their socially engaged coding. See the brilliant video, “Códigos Cooperativos” (Cooperatives Codes) for an introduction to their work.

In North America, the Tech Co-op Network highlights technology worker co-ops and helps new ones start up through resources including “A Technology Freelancer's Guide to Starting a Worker Cooperative.”

Cooperatives active in the Drupal and Backdrop spaces include Palante, CanTrust, Praxis and Agaric.

Groups like Koumbit are similarly oriented, though not formally structured as co-ops.

Tech-worker cooperatives and related groups have know-how, connections, and commitments necessary to ground alternative models.

Drutopia (which I'm active in) is an alternative initiative in the Drupal space, while Backdrop is a fresh software project based on a fork of Drupal.

Crowd Funded Journalism

In traditional mainstream media models, a publication or outlet assigns staff writers to cover particular stories, or considers submissions from freelancers. The audience or readership has only an indirect role – as purchasers or viewers or, even more tangentially, as targets of the advertising. Inevitably, many topics and perspectives – particularly those at odds with the interests of advertisers or media corporations – are underrepresented or missing altogether.

An emerging alternative is crowdfunded journalism. A diversity of sites now enable journalists to directly engage with audiences, pitching stories and funding the stories that don’t make it through mainstream news rooms. Contributoria and the Dutch-language site Yournalism enable journalists to crowdfund projects and publish the results. Through the Cracks gives crowdfunded journalism a shared showcase.

Crowdfunded journalism is demonstrating how involved consumers can spur innovation and alternatives.

CiviCRM and User-Based Development

CiviCRM, a Constituent Relationship Management (CRM) software, is an example of how software projects can draw on crowdfunding and related approaches to channel development effort solidly into reusable enhancements focused on end users.

The core CiviCRM team is available for hire for work that is “of general use to the community.” Clients can hire the team that knows the software best, but only if they reconceive their requirements in ways that others can also benefit. The CiviCRM team offers this tip: “Keep in mind that many projects that at first seem very specific to your needs can actually be turned into something that is useful to others in the community.”

And for each release of CiviCRM, “Make It Happen” campaigns allow users to plan and collectively fund new improvements. Users are encouraged to think of Make It Happen campaigns as “an opportunity for you [to] play an active role in getting some functionality you and others need into CiviCRM”. Users are involved in every stage of the process, from drafting the initial proposal and rallying others, to pitching in with development and user acceptance testing.

By bringing users directly into the development process, CiviCRM helps ensure that effort is solidly shaped by – and oriented to – the priorities of its users.

Platform Cooperativism

Platform cooperativism draws together many of these currents. The recently-published book Ours to Hack and to Own provides in depth information on this new movement.

Platform cooperatives are online platforms that are "collectively owned and democratically governed". An example is Stocksy, a stock photo platform by and for artists. Stocksy describes itself as

an artist-owned cooperative founded on the principles of equality, respect, and fair distribution of profits. Our contributing artists receive 50% of a Standard License Purchase and 75% of an Extended License Purchase – and every single co-op member receives a share of the company.

A recent article by Agaric cooperative member Micky Metts profiles a specific project blending Drupal-based initiatives with platform cooperativism.

Strategies for Renewal

Many of us have been actively engaged in issues and debates around corporate influence in Drupal and the “enterprise” orientation of the software. But the point is not to criticize – it’s to change.

Those of us interested in platform cooperativism as a model for Drupal or Backdrop have our work cut out for us. It's fairly clear how a single online platform like Stocksy lends itself to the model. A broader initiative that might embrace many related platforms is more of a challenge. But it's one with lots of real world examples to draw on. Strategies that might help ground a user-focused alternative model include:

  • Base software development in formal or informal consumer cooperatives that channel improvements into shared solutions for mutual benefit.
  • Build cooperatively owned hosting platforms.
  • Incorporate contribute back screening and language in site building contracts,
  • Organize through cooperatives, activist, and nonprofit networks,
  • Use alternative currencies, such as the ability for user-members to earn time credits for their contributions and redeem them for services,
  • Crowd fund to enable user-members to propose and jointly resource priority improvements,
  • Draw on existing tech workers cooperatives and similarly structured projects as a base.
  • Build distributions on shared bases so they don't each need to reinvent the basics.

To echo songwriter and labor organizer Joe Hill: “Don’t mourn – organize!”

Earlier drafts of this article benefitted from reviews by several friends and colleagues.

Mar 31 2017
Mar 31

In recent days there's been a bunch of insightful and thought provoking reflection within the Drupal community (as well as a share of bullshit). I've benefited from hearing perspectives that remind me of my biases and privileged placement as a cis white male. A comment by Melissa Anderson, someone I know and respect, had particular impact for me.

A lot of attention has focused on a particular action by Drupal project owner Dries Buytaert. But many are going deeper.

The trouble with Drupal is not so much any individual action.

The trouble is that, for all its collective trappings and thousands of contributors, Drupal is formally structured as a dictatorship.

Really? In 2017? Yes, really.

The Thing About Dictatorship

As detailed in documentation of the Drupal project structure, the self-anointed "benevolent dictator for life" not only exerts ultimate control over code but also "preserves [that is, controls] the philosophy, culture, and principles of the project."

Wow. Think about that for a minute.

The occasional overt dictate can indeed be worrisome. But I'm actually much more troubled by what's so normalized in the project that it passes without comment.

I won't repeat what I've gone on (and on!) about in previous comments and reflections on Drupal's power structure going back over a dozen years. Here's a selection:

But I will add I'm struck anew by what seems to me the unusual depth and reach of the authoritarian model in the Drupal project.

I've often heard it said, for example, that Linux, too, has a so-called "benevolent dictator for life".

True. But, contrary to Drupal, the Linux dictator doesn't individually set the terms of reference of, and appoint every member to, key community structures. (Come to think of it, isn't there something Orwellian about a so-called "Community Working Group" appointed by a dictator?) Unlike Buytaert with the Drupal Association, the Linux dictator doesn't have his name written into the bylaws of the Linux Foundation as a director with own reserved slot, nor has he served as the de facto permanent board president of the Linux Foundation since its inception. He doesn't have a seemingly permanent seat on the committee that vets every nomination to the Linux Foundation board.

And, crucially, unlike Buytaert in Drupal, he isn't a founder and key executive of the Linux company that exerts the deepest influence on the software.

Effective checks on the absolute power of the project founder? It's really hard to find any.

For thousands of people caught up in the Drupal project, what does all this mean in practice? As in many communities, boundaries often blur. Drupal can come to define not only one's work life, but also leisure activities like volunteer coding or meetup organizing, even key daily social links and interactions.

Put that together with a patriarchal model, intimately tied to capitalism and corporate power, permeating all these realms - work, leisure, friendship, community - and you get a deeply troubling degree of influence. One that, precisely because it's everywhere, may be almost invisible. In a community where it's all about personal ties and influence, power seldom needs to act overtly.

"Dictator for life". This, too, is something to think long and hard about. That's a lot of future years. For those who stay, what does it mean, this prospect of being part of a dictatorship culture most of one's life?

There's a tonne of beautiful energy in the Drupal project. There are brilliant and passionate people who care deeply about our community and are rightfully proud of our collective project.

Not thanks to the dictatorship model. In spite of it.

Is what we're seeing the beginnings of a "Drupal spring"? If so, where might it lead?

The Coming Fork

Conditions are ripe for a fork of the Drupal project. But what kind of fork?

For a software community mired in regressive power dynamics, a fork can be a positive source of renewal, allowing participants to resolve contradictions and carry forward the project's best attributes.

Or a fork can replicate the same regressive crap that prompted problems in the first place.

Worse--given the current context, a fork could reinforce and enshrine forms of cis white male privilege.

So the key question is not so much whether to fork. Rather, it seems to be: if so, how?

A cultural fork

Yes, there are deep problems with Drupal's code base, many resulting from the warping effects of corporate interests. But the primary challenge of a fork is not about code. It's about culture.

The dictatorship model in Drupal runs deep. So, no, a light makeover isn't going to cut it. A fork needs a radical cultural reset.

We need to look to voices of diversity and inclusion.

We need to create room for critical perspectives and insights that too often have been shouted down by louder voices in the project--ones that, over and over, have rushed to attack questioners of the founder's prerogatives. (I speak as someone who's repeatedly been targeted for my critical voice. And sometimes, yes, silenced.)

We need to deeply question a culture that promotes living for the cause as a positive or even a required leadership quality.

In the Backdrop fork of Drupal, I and others promoted a "project management committee" structure, replacing the single dictator with a group of lead contributors. And I do think the more diverse and inclusive Backdrop leadership team is a huge improvement over Drupal.

But, here, is it enough? Not nearly.

I personally want to look for inspiration and ideas to the platform cooperative movement, which is opening horizons for free software collectively owned by those who use and build it.

That vs. dictatorship? I know where my heart is.

Existing strengths

A fork should draw in existing progressive initiatives and structures in and around the Drupal space.

One that I'm involved with is Drutopia.

We've also got an expending number of engaged, radical organizations and cooperatives in the Drupal sphere. How do we draw them in? Or, maybe better put: how can we be open to them drawing us in?

A fork shouldn't require people to switch

Strategically, a new fork will probably have the most scope and impact if it doesn't force people to switch immediately to something new. Instead, it could work as a drop-in replacement for Drupal 8--and future Drupal versions.

For those familiar with the MySQL database, think MariaDB, the community-led fork of MySQL. If you already use MySQL, you can switch very painlessly to MariaDB--and get some great improvements for your effort. Your existing MySQL databases just work. The MariaDB project maintains compatibility by merging in changes from MySQL.

In the same way, a Drupal fork could "just work" if your site was originally built on Drupal.

Moving Forward

Crises in authoritarian systems play out in familiar ways. There will be - there already are - calls for a brand of "healing" that involves returning to the fold, reflecting sagely on lessons supposedly learned, and pledging renewed faith in the beneficent leader. Ah, I see one such post just appeared from the Drupal Association. Right on cue.

And there will be organizing on the ground. By those of us truly fed up with a corrosive patriarchal agenda, one that once again masks its power behind a false and exploitative language of inclusion.

Who are hungry for progressive change.

Jan 26 2015
Jan 26

This is part two of a series on configuration management challenges in Durpal 8. Part 1 looked at challenges for small sites and distriubtions.

What is the state of support for distributions in Drupal 8?

Trying to gauge the state of anything in Drupal 8 has its inherent pitfalls. The software itself is still changing rapidly, and efforts in the contributed extensions space have barely begun. That said, various initiatives are in process.

For background on configuration management in Drupal 8, see the documentation on managing configuration and the configuration API. Drupal 8 configuration is divided between a system for one-off simple configuration like module settings and configuration entities that store items you may have from zero to any number of, like content types and views. The Drupal 8 handbook pages on configuration are useful but not fully up to date.

Two recent blog post series that provide background and technical details are:

The challenges

Distributions in Drupal can be divided into two main types:

  • Starter-kit distributions like Bear are designed to get you started in building a site that you then take in your own direction.
  • Full-featured distributions like Open Atrium or Open Outreach are designed to fill a use case and support upgrades.

This distinction is important in light of the Drupal 8 assumption that sites, not modules, own configuration. Starter-kit distros will work fine with this assumption, but for full-featured distros it presents major challenges; see part 1 of this series.

Configuration management in Drupal 8 is built primarily around the single-site staging or deployment problem rather than the requirements of distributions. Back in 2012 a discussion tried to assess what was needed to make Drupal 8 distribution-friendly, but it didn't get far.

Two types of tools look to be needed to fill the gaps.

  • Developer tools. Managing configuration in distributions will require exporting it into feature-like modules. Because any extension (module, theme, installation profile) can include configuration, most of the needs of distribution authors are a subset of what any extension developer will need. For example, the built-in Drupal 8 configuration export functionality is designed only for use on a single site.
  • Site tools. Since Drupal core's single-site configuration management model conflicts with the requirements of updatable distributions, specialized modules will be needed to provide distribution-based sites with the ability to receive configuration updates.

Emerging solutions

  • Features 8.x
    Some of the first efforts to provide distribution-related functionality came in the form of sketches towards a Drupal 8 version of the Features module. The sandbox module contains a small collection of methods that can be called from the Drupal command line utility Drush for editing and reverting configuration modules.
  • Configuration Development
    Configuration Development provides automated import and export of configuration between the active configuration storage and exported modules.
  • Configuration Revert
    The sandbox Configuration Revert project provides a set of reports that allows you to see the differences between the configuration items provided by the current versions of your installed modules, themes, and install profile, and the configuration on your site. From these reports, you can also import new configuration provided by updates, and revert your site configuration to the provided values.
  • Configuration Packager
    Configuration Packager enables the packaging of site configuration into modules, like Features for Drupal 7. Rather than producing manually authored individual features, Configuration Packager analyzes the site configuration and automatically divides it up into configuration modules based on configured preferences.

Remaining work and coordinating efforts

See the drupal.org issue META: Required functionality for Drupal 8 distributions for an initial inventory of the work outstanding to prepare for Drupal 8 distributions.

As usual, the main challenges are probably not so much technical as strategic and organizational. To prepare the way for Drupal 8 distributions, we need to coordinate to understand barriers, explore solutions, and pool efforts.

Part of this work will be developing shared, generic tool sets. Already, there's a lot of work in modules like Features 8.x and Configuration Packager that isn't specific to features or packages of configuration and would better be merged into a more generic solution; see the issues #2383959, #2405015, and #2407609. Configuration Development is the most likely candidate (#2388253), though there are some outstanding issues.

Interested in helping? Please comment on and help flesh out the meta issue or the issues and projects referenced there.

Packaging configuration

My own efforts have been focused recently on taking a fresh approach to packaging configuration in Drupal 8 in the Configuration Packager module. In my next post in this series, I'll introduce that project.

Jan 23 2015
Jan 23

In a recent blog post, Drupal 8 co-maintainer Alex Pott highlighted a seismic shift in Drupal that's mostly slipped under the radar. In Drupal 8, he wrote, "sites own their configuration, not modules".

To see why this change is so far-reaching, it's useful to back up a bit and look at where exportable configuration comes from and what's changed.

In Drupal 7, a lot of site configuration (views, rules, and so on) can be exported into files. There are two main use cases for exportable configuration:

  • To share configuration among multiple sites.
  • To move configuration between multiple versions of a single site.

By and large, the two use cases serve different types of users. Sharing configuration among multiple sites is of greatest benefit to smaller, lower resourced groups, who are happy to get the benefits of expertly developed configuration improvements, whether through individual modules or through Drupal distributions. Moving configuration between different instances of the same site fits the workflow of larger and enterprise users, where configuration changes are carefully planned, managed, and staged.

In Drupal 7, both use cases are supported. An exported view, for example, can be shared between multiple sites or between instances of the same site. The Views module will treat it identically in either case.

If a site admin chooses to customize exported configuration in Drupal 7, the customized version is saved into the site database and overrides the module-provided version. Otherwise, though, the site is on a configuration upgrade path. When the site is upgraded to a new release of the module that provided the configuration, it receives any changes that the module author has made--for example, refinements to a view. At any time, a site admin can choose to toss out changes they've made and get the module-provided view--either the one they originally overrode or a new, updated version.

If anything, the multiple site use case was a driving force behind the development and management of configuration exports. The Features module and associated projects - Strongarm, Context, and so on - developed configuration exporting solutions specifically for supporting distributions, in which configuration would be shared and updated among tens or hundreds or thousands of sites. Yes, Features could be and is used for staging changes between instances of a single site; but the first and foremost use case was sharing configuration across sites.

For Drupal 8, however, the entire approach to configuration was rewritten with one use case primarily in mind: staging and deployment. The confiugration system "allows you to deploy a configuration from one environment to another, provided they are the same site."

In Drupal 8, module-provided configuration is imported once and once only--when the module is installed. The assumption is that, from that point onward, the configuration is "owned" by the site. Updated configuration in modules that have already been installed is, by design, ignored. Importing them, as Pott notes, might lead to "a completely new, never-seen-before (on that site) state." However, "Drupal 8 does not work this way."

What are the implications of this change? It has clear benefits if you're building a site that will be carefully managed through staging and, in particular, places a premium on locking down and controlling every detail of configuration.

But for most current Drupal sites and for distributions? The benefits are not so clear cut.

On the plus side, much of what previously was unexportable in Drupal core (content types, fields, variables, and so on) is now supported natively. No more heavy handed workarounds in the Features module for so called "faux exportables"--components like user roles, content types, and fields that Drupal 7 core stores only in the database.

But, with Drupal core firmly on the "single site" configuration management side, users wanting to benefit from module-provided configuration updates and developers of distributions may be left fighting core every step of the way.

It's hard not to conclude that Drupal 8 ties configuration management to a (primarily, enterprise-focused) single-site staging model, and in the process, neatly undermines the use cases that largely brought us exported configuration in the first place.

That said, there are emerging initiatives including Configuration Update Manager that may help. More on those in part two of this series.

Updated March 11, 2015. Changes:

  • Removed unintended implied criticism.
  • Updated links.

Also, see the draft Configuration Synchronize module for an initial attempt to address the use case of automated import of configuration-provided configuration updates.

Oct 04 2013
Oct 04

Browsing joomla.org, I recently noticed the following in the Joomla roadmap:

The Joomla! CMS [content management system] seeks to create a variety of distributions of the CMS to address a variety of common niche markets.

Good idea!

As a Drupal developer with a longtime focus on distributions ("distros"), I'd like to support this Joomla initiative. As a first step, as followup to posting in the Joomla development forum, I thought I'd try to write up some observations on Drupal, Joomla, and distros.

I've written this with an intended audience of Joomla developers. But if you're a Drupal contributor, please wade in as well!

What is a distro?

Technically, any time you package up and distribute a version of Joomla that's different from what's on joomla.org, you've got a new distribution. Great! But that's not really what I'm talking about here.

I'm talking about: how can you produce a bundled version of Joomla that provides free and out of the box pretty much everything that's needed for a particular type of website?

Say, for example, I'm going to build an ecommerce site. Rather than having to select, download, install, and configure and integrate a bunch of plugins, plus maybe write some custom code, I can instead just download and install a version of Joomla that has everything I need. Presto, there's my ecommerce site. Yes, from there I might hire someone to train me or extend the site, but out of the box I get all the basics covered.

And not just on some hosted online service. For example, the software as a service Nonprofit Soapbox Joomla-based product is sometimes, mistakenly, referred to as a distribution. Nope. For starters, it's not distributed. It's not even open source.

Some distros may focus on bundling software or solving a technical need rather than creating a preconfigured site for a particular sector. In the Drupal world, this type of distro includes Acquia Drupal, which bundles Drupal core with a bunch of contributed modules, and Pressflow, which provides performance enhancements. This type of distro serves a need. But, again, it's not my primary focus here.

Instead, I'm talking about the niche-focused distributions hosted on drupal.org. Like the Open Atrium groupware, Commerce Kickstart for ecommerce, and OpenChurch for, yes, churches. And if you're wondering who I am and why I'm interested, have a look at the Open Outreach distro, of which I'm a lead developer.

Why distros?

Given the millions of Joomla (or Drupal, or Wordpress) sites out there, there's a ton of duplication. Yes, it's great to have a CMS that's flexible, but endless repetition of variations on the same custom site development kinda defeats the purpose of open source development. Not to mention it gets tedious.

A niche-focused distro captures the configuration you'd otherwise repeat and presents it as a starting point. Further development can benefit not just one site but all adopters of the distro. Like other Joomla or Drupal development.

Some Drupalisms and their Joomla equivalents

For lack of an established vocabulary, each CMS develops its own jargon, often - just to make things extra confusing - taking the same terms and giving them different meanings. See Plugins, Modules, and Themes, Oh My! for a handy glossary.

Exportables and bundles of configuration

Before digging into specifics, some key concepts by way of background.

A niche-focused distribution is more than a bunch of general-use extensions. It's all the configuration that's the stuff of site building. And it's not just about a one-time initial setup. Sites on a distribution also need to get updates--to configuration as much as to code.

The obvious shortcut to a distribution is a variation on a database dump. That's basically the approach outlined a couple of years back on how to use a commercial backup extension to roll your own Joomla distribution. It's also the approach that was taken for the first Drupal distros. Just configure a site the way you like it, export it at that point, and you're set to clone it as many times as you like. Easy! Quick!

But not so flexible or updatable. What if some of your users don't need or want all the stuff you enabled? What if, next week, you extend your configuration or fix a problem--how do existing users get your fix?

In Drupal, the answer is so called "exportables"--configuration items that are exported into code. See an overview of exportable configuration in Drupal. While it's possible to export a whole site's configuration into a single bundle, for purposes of distro development what's preferable is to export into distinct bundles of configuration, each meeting a specific use case. That way, users of a distro can pick and choose, customizing their site with just the functionality they need. For example, a project management distro might include several optional bundles of configuration: project, group, issue, event, and so on.

In Drupal 7, the Features module is the standard tools used to package up bundles of exported configuration. As well as a UI for creating and managing "features" (configuration bundles), it provides bridging for types of configuration that don't natively support exportables.

For Drupal 8, the Configuration Management Iniative (CMI) has extended exportables to cover all types of coniguration. A recent blog post outlines how exportable features are shaping up in Drupal 8. The CMI is one of the elements slated for inclusion in Backdrop, a recent Drupal fork.

Distro enablers and what Drupal does--and doesn't do

So, in no particular order, here are some notes on what Joomla might be able to learn from Drupal. I've arranged them around a number of "distribution enablers"--that is, factors that - at least for Drupal - have been important in enabling or supporting distribution development and adoption. I've also added some notes about Joomla--sometimes just questions, since I'm a Joomla newbie. If you can answer the questions, please do and I'll update the list.

Support in core for distributions

  • Why it's important

    As a distro developer, I want the core software to work with me--not against me.

  • How it's done in Drupal

    Drupal supports natively multiple "install profiles", which are similar to modules (extensions). Install profiles can alter the site installation process, adding their own steps. They can also include plugins (modules, themes, libraries). Core includes two install profiles, "standard" and "minimal". To create a distribution, you simply add a new directory, e.g., profiles/myprofile, and put the profile for your distribution there, along with any contributed plugins it requires. See documentation on distribution development.

  • Room for improvement

    The install profiles that ship with Drupal 7 core aren't particularly useful as models, as they rely on direct database or API calls rather than exported configuration. This should change in Drupal 8, thanks to the Configuration Management Iniative (CMI).

  • Joomla notes

    Is there an API for modifying the Joomla install steps? Where is relevant documentation?

Infrastructure to support distributions

  • Why it's important

    Packaging up a distribution is a lot of work. As a distro developer, I want some tools that make the job easier.

  • How it's done in Drupal

    While the first Drupal distros were hosted externally, the drupal.org site now includes highly specialized infrastructure support for developing and packaging distributions. Developers focused on drupal.org infrastructure development added this in two rounds of community-supported development. First,  the core infrastructure was put in place. Then in a followup round of development, infrastructure devs funded by companies with an interest in distros put in place a major set of enhancements. The Drupal Association played and continues to play a key and direct role in funding, coordinating, and providing this and other Drupal project infrastructure.

  • Room for improvement

    The solutions require a lot of custom code, something that's added to the upgrade burden for drupal.org and is part of the reason the site is still on Drupal 6, years after Drupal 7's release.

  • Joomla notes

    Joomla legacy code has the JoomlaCode repository, while core has moved to Github, supported by the jIssues application. What would it take to extend jIssues to support distro packaging?

Source code and updates available through publicly accessible repositories and download packages

  • Why it's important

    Building a distro typically involves packaging up a lot of extensions along with core and supporting libraries. To use the code, it's not enough that it has an open source license. It also needs to be available.

  • How it's done in Drupal

In Drupal, developers and shops typically sell services--not code. The drupal.org site offers more than a listing of plugins. Rather, drupal.org provides a (Drupal-based) project management solution for Drupal code projects, including Git source code repositories, release management, a ticketing system, and security oversight.

  • Room for improvement

    A custom project management solution is costly in terms of development and can easily fall behind other available options. Some Drupal developers are drifting elsewhere, mostly to Github.

  • Joomla notes

    Much of the Joomla scene centres on selling software. Even presumably "non-commercial" projects often include blocks to code access, like mandatory registration on a host site.

    If there's one key barrier to distros in Joomla, I'd say this is it. Commercialized code sets up huge barriers to distro development, since as a distro developer what I need above all is the ability to package up and use the best my open source software has to offer. If that code is stuck in some private repository, it's a drain rather than a gain.

    The JoomlaCode site featured some relevant infrastructure and the newer jIssues application is promising.

    Probably the single most important step to enable distro development in Joomla would be to strengthen and promote publicly accessible code. This might require changes to the extensions directory to, at least, provide profile to extensions with publicly accessible source code--ideally, code managed through a Joomla community infrastructure including jIssues.

Strong GPL protections

  • Why it's important

    As a distro developer, I don't want to have to be concerned with licensing issues and the integrity of the code I'm including.

  • How it's done in Drupal

    Rather than relying on soft, voluntary compliance, Drupal includes clear developer documentation and direct protections for GPL integrity, including enforcement and removal from drupal.org code hosting. Longtime Drupal contributors including Gerhard Killesreiter (killes), Larry Garfield (Crell), and Derek Wright (dww) have provided focused and sustained leadership in this area. GPL protection is written into the distribution packaging system on drupal.org via a whitelist approach. Non-drupal.org code must be explicitly whitelisted before it can be packaged into a drupal.org hosted distro.

  • Joomla notes

    For someone coming from Drupal, Joomla looks in comparison (no offense intended!) like a GPL wild west. One of the first "non-commercial" extensions that I browsed from a link on the Joomla extensions listing offered a "personal use only" download and then wanted to charge for a "commercial use" version. Without a central repository with clearly enforced rules, I'd be at a loss as to how to ensure code integrity, short of a painstaking process of individually evaluating each package I wanted to include.

Update notifications

  • Why it's important

    As a distro developer, I include a lot of code in my distro and I need an easy way to make sure it's all up to date.

    As a distro user, I want to make sure the code I'm running is up to date, particularly for any security releases.

  • How it's done in Drupal

    The Drupal update manager allows sites to poll a central server that provides update data in an XML format. Since almost all Drupal code is hosted on drupal.org, projects get update notifications out of the box.

  • Joomla notes

    Joomla core includes update notifications. For extensions, update notifications rely on the individual extension authors implementing an update server. In practice, only a subset of extensions support updates.

Strong institutional backing for distro development

  • Why it's important

    It's a lot of work to build out and support the enabling toolset for distributions and the distros themselves.

  • How it's done in Drupal

    The Drupal distro toolset was built out alongside the first "modern" Drupal distro, OpenAtrium. Development Seed, a (then) Drupal shop, built OpenAtrium as a generic tool for use by large clients including the World Bank, and at the same time built out the Features toolset. When Development Seed changed its focus, OpenAtrium and the enabling toolsets moved to Phase2, a company already focused on building out Features-based distros including OpenPublish. Most of the well maintained distros are produced by Drupal shops that generate business from building sites with their distros. In many cases, distro development includes a range of other independent contractors or shops that also build sites with the distros and contribute plugins and other enhancements.

Standard site-building tools

  • Why it's important

    As a distro developer, I need tools to build out configuration, and the more standardized those are the better.

  • How it's done in Drupal

    In recent major releases, there's been an increasing move to bring key configuration management tools into Drupal core, including custom content types (added in Drupal 6), fields (added in Drupal 7), and Views (added in Drupal 8). Even in contrib, there is a tendency for usage to converge on a best solution. With Views, for example, there wasn't a question of which query builder would be a candidate for core inclusion--Views was already the clear choice. Factors that facilitate this convergence include (a) non-commercial code, so that the best solutions are universally accessible and (b) the usage statistics ranking of projects on drupal.org, so that site builders easily see which solutions are gaining adopters.

  • Joomla notes

    Joomla looks to be relatively challenged in both standard tools and the bringing of those tools into core. Consequently, it may be a bit harder to build integrated solutions.

    Standardizing on site building solutions could be a key step.

Extensions that facilitate distribution development

  • Why it's important

    As a distro developer, I want to have handy tools I can use both to author and to deliiver my distro.

  • How it's done in Drupal

    Drupal contributors have produced a standard toolset for producing Drupal distributions. At the core is the Features module, which includes an advanced UI for capturing and exporting bundles of configuration that meet a particular use case.

  • Joomla notes

    Are there equivalent extensions in Joomla?

Command line tool and packaging format

  • Why it's important

    Distro development involves a bunch of repeated tasks, like exporting bits of configuration. Please, save me endless clicks! It also involves gathering up code from disparate places.

  • How it's done in Drupal

    Alongside Features, the Drush command line tool includes key functionality supporting distro development. The "drush make" format makes it possible to specify, in a simple text file, a custom set of code that will be packaged up to produce a full distro.

  • Joomla Notes

Is the Joomla CLI an equivalent base here?

Flexible core

  • Why it's important

Key to being able to produce radically different distros on the same code base is ensuring that very little in the core CMS is fixed or required.

  • How it's done in Drupal

    Drupal 7 has a minimum of required core modules. Modules providing more specialized functionality have steadily been moved out of core into the contributions repository.

  • Joomla notes

    There have been efforts to build a "light" version of Joomla to facilitate distributions, including Square One and JLite. These are useful examples, but they've hit the same barriers that early Drupal distros did--it's way too much work to have to maintain a largely forked version. Instead, it will be important to focus effort on refactoring the core CMS so it has a minimum of required components.

    Tangentially related: there was public musing about whether Molajo was a Joomla distro or a fork, but according to lead developer Amy Stephens it's neither--see comment below.

Distribution-friendly hosting solutions

  • Why it's important

    As a distribution user, I want hosting options that make it easy to install and update my preferred distro.

  • How it's done in Drupal

    Many Drupal hosts offer special support for distros. The Drupal Aegir hosting platform includes built in support for distributions. The Pantheon system supports distributions.

  • Joomla notes

    Are there Joomla hosts interested in supporting distros?

Further reading on distros in Drupal

Key Joomla strengths

Looked at from outside, Joomla has some tremendous strengths that should provide a solid base for building out distro support.

  • Huge install base

    The sheer number of Joomla sites out there means there's a lot of potential market for distro-based sites. As site owners face the inevitable need to upgrade to new versions, getting off a customized one-off site build and onto a distro can be a big attraction. Upgrading an existing site to a distribution can save a ton of time and expense in upgrade costs and future maintenance.

  • Strong production and community leadership

    Having an inclusive and participatory structure should help a lot in marshalling resources to achieve the (significant, but doable) tasks of enabling and supporting distro development.

  • Strong developer and adopter communities

    There's no shortage of skilled Joomla talent to draw on.

  • Institutional backing

    It takes a lot of backing to successfully make changes and fund infrastructural development. Open Source Matters and other key Joomla institutions would be important players.

Next steps

So that's it for an initial brainstorm. To emphasize again: my aim here is to start a conversation about what Joomla devs might learn from the Drupal experience re dos and don'ts of facilitating distro development. I'm not at all suggesting that Joomla should follow Drupal's "model"--just that a close study of how Drupal has done distros is probably a valuable input into making it happen in Joomla.

What did I get wrong? What did I miss? Where can we take the conversation from here?

Sep 26 2013
Sep 26

Back in 2008, when Drupal 6 was released and version 7 was only a faint glow on the horizon, the Drupal project stood at a crossroads with multiple possible futures. One of those futures was the one that's unfolded: the path to what's emerging in Drupal 8.

This is a story of pursuing maximum growth and enterprise clients. Its key players include Acquia, the multifaceted enterprise founded by Drupal lead Dries Buytaert, and other enterprise level Drupal-focused companies. In these circles, the inherent advantages of growth, professionalization, and market maturity seem self-evident. In this future, Drupal core development is increasingly carried out in association with or directly within corporations. If some users or contributors are left behind, the accepted wisdom says, that loss is easily compensated for by growth statistics and new market offerings. Contractor fees and investor profits keep rising along with demand. What's not to like? Progress isn't for everyone, is it? "I really think we can say we've built the best CMS for enterprise systems," the project lead can conclude with satisfaction.

But what if we'd taken a different path? What if profitability and expanding market share and wooing enterprise clients weren't the driving aims? What if, instead, we'd focused on stability, usability, and accessibliity for small and medium sized organizations--the ones that were the core of the Drupal community up to Drupal 6? What would Drupal look like today?

Thanks to Jen Lampton and Nate Haug, we may just get to find out. Their Backdrop fork of Drupal aims to chart the road not taken.

Personally, I haven't felt this excited about the Drupal community in a long time.

Not surprisingly, response to Backdrop has included the familiar claims that greet any radical departure: that its proponents are stuck in the past, afraid of progress, shortsighted, disruptive, and so on.

I'm hopeful we can soon move past these reactive responses and recognize Backdrop for what it is: a positive addition to the open source world. A sister open source project built on the same base but answering a distinct need. A fresh approach with shared roots.

I'm far from saying goodbye to Drupal. For now at least I plan to keep working and contributing in the Drupal space for some time to come. But I'm also planning to dig into Backdrop. Why not? No one knows yet where it's going. But if together we can take it somewhere, I could happily work in both Backdrop and Drupal. Hey, it's much more attractive than having to learn Joomla or Wordpress!

Backdrop reminds us in the most direct way possible that, whoever may claim the trademark or exert founder's rights, no one person or company "owns" Drupal. It's all of ours. And diversity is a strength. When we look at the Drupal principles (disclosure: I wrote the original version myself), doesn't the Backdrop project ring at least as true as Drupal 8 does?

Yes, Jen and Nate and the rest of us who pitch in will take the code in a different direction from where Drupal goes in the future. But is that so different from the ways that Drupal itself has diverged radically from its own past?

They say you can't go back. But in some cases, maybe you can.

Oct 10 2012
Oct 10

Despite the great Drupal distributions available, most Drupal site builders have yet to start using them regularly for building client sites.

There seems to be a bunch of perceptions out there about the limitations and difficulties of using distros, some of which may have been true at one time but are increasingly less so. So, following up on my "There's a distribution for that" post yesterday, here's a top ten list of reasons given for not using a Drupal distribution, with some reasons you might want to think again.

  • It's all or nothing

    Site builders might be concerned that using a distro means being locked into a monolithic site design with many non-optional features. Fortunately, that's less and less the case.

    Many distros are now built on Apps, with each app being an optional and stand-alone piece of functionality. You can pick what you want and happily leave the rest. Even distros not based on Apps usually have small cores enhanced by many distinct, optional features. It's even possible to mix Apps from one distro with those of another. Increasingly, distros are flexible enough that you can take just what you need and leave the rest.

  • It's someone else's product

    There seems to be a perception that distros are proprietary "products", as opposed to modules, which are community contributions. It's true that some distros are strongly identified with particular companies. But so are many modules--and that doesn't keep us from using them. In fact, having one or more companies actively engaged in backing a module, theme, or distro probably increases the chances it's going to get the long term attention it needs.

    In my experience, distro maintainers are just as happy as any other Drupal dev to get community feedback, fixes, and improvements. Adopting and contributing to a distro isn't "helping the competition". It's using and improving our shared solutions.

    And, just like with modules or themes, some distro maintainers will be more than happy to add you as a maintainer once you start making steady contributions. So "their distro" becomes "our distro".

  • I don't have the time to try out distros

    When you look at a new module or theme it takes a bit of time to try out and evaluate. Distros are no different. But since almost all distros come fully packaged, it's usually quick and easy to install one. Or you could jump straight onto the Pantheon system or another place where you get one-click distro installation.

  • My site is unique

    True. But it probably shares at least some back end structure or features with other sites built for a similar purpose. When you use a distro, it's not a case of either/or. You can select just what you want from the distro and do any customization you need on top of it.

  • I want my own look

    And you can have it. Most distros are very flexible when it comes to design. If you've got room in your budget for a custom theme, a distro will happily accept one. Some will even give you a cleanly designed starter theme specifically built for the distro.

  • I'm skilled enough that I don't need to use a distro

    That's what a lot of PHP developers said when Drupal and similar projects first came out. Then after a few years they found that having to do everything themselves was wearing, while pooling efforts and collaborating was infinitely more rewarding. It's pretty much the same with distros.

  • Using a distro would hurt my bottom line

    Custom site development is the bread and butter of many Drupal developers and shops, and distros may seem to threaten that. After all, if you don't get paid for building out basic site functionality, where will the revenue come from?

    But here's another way of asking the same question: Now that you don't have to charge out for all the preliminaries, what more interesting features may you be able to fit in? In what ways will you be able to give your clients better value for the same investment? How many items can you convert from "future wishlist" to "phase 1 deliverable"?

  • It doesn't do everything I need

    No, probably it doesn't. But does it do a lot of what you need? Is it a good leg up?

  • I tried it but I didn't like X

    For sure, there's going to be small or even large details about any distro that don't fit with the way you'd want things done. We each have our own way of doing things and distros aren't perfect any more than modules are.

    The good news is, just like any other Drupal project, you can help improve them. Got a better idea about how to present content, design a page, or structure a content type? Go ahead and post an issue. Better yet, work up a patch. Speaking for myself, I'm thrilled whenever someone comes along with a better way to do things.

    And you can customize. It's true there's probably not a lot of sense in using a distribution if you're going to override and customize everything. But you can get value out of a distro and still do some things your own way.

    At the easiest level, since distro features tend to live in code, you can simply override selected elements like views or content type displays. If you want to play by the book, you can put your customizations into a site-specific override module, using the standard alter hooks that are available to customize code-based configuration, like hook_views_default_views_alter(). Or you can try out the Features override module, which can automate the process of capturing your customizations in code.

    So, yes, you can have your code and tweak it too.

  • A distro would take too much time to learn

    Yes, there's a learning curve in developing a site off of a distro. You have to spend the time to wrap your head around how the distro is structured. You might need to read some developer documentation or bone up on some modules in the distro that you haven't used before.

    But you might be surprised at some of the benefits. For one, the features built out in a distro often provide some great examples to work from.

    Basing a site on a distro has a way of enforcing some good habits, like systematically registering your customizations. The time you save in basic site development can help free up budget and development hours so you can do things right, rather than rushing to deadline.

    And developing expertise in a distro can be a long term boon, as it means your next project will go a lot quicker and smoother now that you've learned the ropes.

  • So if you haven't started building distro-based sites, now might be a good time to consider starting. At their best, distros are just another facet of what community open source is all about: pooling efforts and collaboratively building great solutions.

    Oct 09 2012
    Oct 09

    If you're a group having a Drupal website built for you, one of the first questions you should ask is, "What distribution will you base the site on?"

    And conversely if you're a Drupal site builder with a new site to build, you should start by asking, "What distribution will I base this one on?"

    True, not every site will lend itself to being based on a distribution ("distro" for short). A few sites will be so large, or idiosyncratic, or both, that basing them off a distro doesn't make sense. And some will have focuses that don't yet have a solid distro option. But for the rest - and that means most sites built today - a distro is likely to be a great fit.

    Why use a distro? The more pertinent question might be, why wouldn't you?

    For both users and developers, well designed distros offer a ton of advantages over a one-off site:

    • They're built by experts who know the toolsets inside out.
    • They're designed with a broad set of users in mind and have already solved many usability and design issues.
    • They have been tested on dozens or hundreds of sites so many of the issues you might face have already been caught and fixed.
    • They're flexible and allow you to start off with just what you need now.
    • They come with built-in solutions for future needs that a site admin can just turn on and use when the time comes. And as new solutions are developed and shared by other users, you'll be right in line to add them in.
    • They come with detailed and specific documentation both for users and developers.
    • You can access community support from others using the same solution as you are.
    • Rather than being stuck with a one-off product that only the individual site builder or company is familiar with, you as an organization get something that multiple developers and shops can work with and understand.
    • You're on an update path and can look forward to getting future updates that have already been tested, with update issues addressed for you.

    If these benefits sound familiar, they should. They're all the key reasons why you'd choose Drupal and open source in the first place.

    "There's a module for that," Drupal users are often heard saying. Meaning, rather than start from scratch, why not look around and pick an existing module that fits the bill? True, it might not do absolutely everything you want exactly the way you want it to. But with a bit of evaluation you're likely to find a module that's going to save you a ton of time, and probably be better than what you'd throw together yourself.

    Ditto with design. Relatively few themers start from scratch. Instead, they choose one of the many available base themes and use that as a huge leg up in building their own theme. In turn, site builders help improve the modules and themes they use. The result saves time and money all around.

    But, when it comes to building a new Drupal site, most developers still start pretty much at zero. They download Drupal core, select numerous custom modules, and start into the time consuming work of building out content types, fields, views, panels pages, and the rest, along with whatever custom coding is needed. Or, at best, they use their own private, in-house startup scripts or features.

    People, we can do better than that!

    Distributions are the next obvious step in Drupal evolution. There are now dozens of distributions meeting many or most of the major Drupal site use cases--see the distributions hosted on drupal.org and distribution documentation. In some cases there's an embarrassment of riches, with three or more distros to choose from.

    The current redesign of the flagship groups.drupal.org site is a case in point. Drupal Groups was originally developed as a custom site. For the Drupal 7 update, rather than continue custom development, the developers decided to switch to a distribution and chose the groups-focused Drupal Commons distro. The discussion offers a good example of reasons to build off a distro and also how to evaluate available distros and make a good choice.

    Got a site to build? Chances are, there's a distribution for that. Give it a try. If building off a distro is right for sites like groups.drupal.org, isn't it a good option to consider for your next project too?

    Sep 13 2012
    Sep 13

    The latest release of the Open Outreach distribution for nonprofits comes with built-in CRM (contact relationship management) support, based on the new RedHen CRM.

    CRM is fancy shorthand for contact management (or "client", or "constituent", or "customer", depending on who you're relating to).

    It's over two years now since we started work on Open Outreach, and from the start CRM has been high on our list. For most nonprofits and community organizations, contacting and engaging with members or residents or supporters is key.

    But we knew CRM integration was going to be a challenge. I'd worked for two years alongside many talented Drupal developers at CivicSpace (one of the first Drupal shops) trying to integrate Drupal with CiviCRM, a leading open source CRM software. That experience was enough to convince me that deep integration of Drupal with an external CRM was no picnic.

    The alternative would be a CRM built natively in Drupal. But two years ago, despite many rounds of discussion and several attempts, there was little sign of that. Having no shortage of other functionality to build out for the distro, we held off and hoped for the best.

    And, happily, we got it.

    In June when I rolled up my sleeves to evaluate the options for CRM integration, I was pleasantly suprprised to find not just one but several active and innovative native CRM initiatives. Joachim Noreiko and Robert Mumford were building something called Party, folks at Trellon were busily coding up CRM Core, and the ThinkShout developers were hard at work on RedHen CRM. Would one of them save me from having to wade back into CiviCRM integration land? Please, please?

    Party allows tracking of individuals and organizations and their relationships and is planned as a component of an eventual integrated CRM. CRM Core doesn't have a stable release yet but is in active development. RedHen has a beta release and has minimal field dependencies, shipping with just the core pieces needed for any CRM integration. (For more on the three projects, see July, 2012 episodes of the Modules Unravelled podcast.)

    Of the three, RedHen stood out to us as the right combination of solid architecture and extensibility. RedHen takes the same approach to CRM as Drupal Commerce takes to e-commerce: do relatively little, enable much. Like Drupal Commerce, RedHen models its basic data types as Drupal 7 entities and ships with a core set of extension modules.

    To help give an idea of how to use RedHen, the ThinkShout devs posted a demo install profile, RedHen Demo. We started out by cloning the feature module included in RedHen Demo, stripping it down a bit, and then building from there. The result is Debut RedHen, a feature module that provides RedHen-based CRM functionality. Like other Debut features, Debut Redhen can be installed on any site as an app using the Apps module--see documentation on the Debut RedHen project page.

    RedHen is still in early beta and there are definitely rough edges and missing pieces. If you're working with it as an early adopter, expect to do some troubleshooting and post patches. The good news is that RedHen devs Lev Tsypin, Tauno Hogue, Sean Larkin, and Brandon Lee have been super responsive in the issue queue, giving lots of tips and reviewing and applying patches quickly.

    RedHen is built around two core data types: contacts and organizations. Like many other Drupal 7 entity types, contacts and organizations support different types (bundles), each of which can have distinct sets of fields.

    Contacts can be linked to each other and to organizations via relationships modelled with the Relation module and can be tied to regular Drupal user accounts. A RedHen Membership plugin, packaged as a separate module, provides organizational membership handling, including optional linking of membership status with a Drupal user role. Activities like contact updating or membership renewal are automatically logged via Message module integration. An "engagement scoring" plugin allows assigning scores to specific types of contact interactions, like when a contact posts a website comment. And there's a central UI where users with appropriate permissions can browse contacts, add notes, review activity, and make new linkages.

    Much of this is what any CRM user will be familiar with. So why not just turn on CiviCRM?

    For starters, RedHen makes it easy to customize a CRM with just the pieces you need. Ever created a contact record in CiviCRM? The default forms are packed with every conceivable field and data type you could possibly need. Great if you want and need all that--but in our case we wanted to start small. Drupal site and content administration is challenging enough without adding a huge new layer of contact management. With RedHen we could add just the fields we wanted.

    A RedHen-based CRM isn't some external software tacked onto your website--it is your website. For example, the admin user interface will be comfortably familiar to anyone who's done Drupal content administration.

    The big payoffs started when we went to integrate with our existing features. For starters, rather than introducing a whole different WYSIWYG interface just for the CRM, we could simply turn on the one that already ships with Open Outreach.

    Open Outreach includes Debut Event, an event feature with calendar displays, event categories, repeating events, and some iCal integration. We'd long wanted to support event registration--and happily RedHen includes a registration plugin integrating RedHen with the Entity registrations module. With a bit of quick bridging we had a Debut RedHen event feature, providing event registration that automatically integrates with RedHen contacts. Nice!

    One glaring gap in RedHen is phone number handling. Nada! But even here I have to say I'm with the RedHen devs. I looked at available Drupal phone number field modules, but my sad conclusion was that there just isn't anything that's solid enough and a good fit for CRM usage. ThinkShout has plans to fill the gap, probably with a new custom field like the email one they built into RedHen. Meantime, in Open Outreach we're making do with a general-use text field.

    With contact and organization types, Rosemary Mann (the other main Open Outreach developer) and I decided to start simple. Rather than second-guess what's needed and introduce a bunch of different contact and organization types, we started with catch all "general" types that can be classified with taxonomy. Any site developer wanting more specific types can go ahead and add it along with any specialized fields that are needed.

    We've only scratched the surface of what we'd like to do with RedHen. Next steps could include mail integration, donation handling, or hooking up Rules for engagement scoring and collecting contact records from website users. Then there's our new Debut Location mapping feature, which includes address-based geocoding. Why not hook it up to Debut RedHen so we can map contacts and organizations based on their addresses?

    Why not indeed? Having a native Drupal CRM means that this kind of integration is as easy as cloning a few items and making some tweaks.

    I've come away from my first few weeks of working with RedHen refreshed and invigourated. With its solid, entity-based design, integrating proven Drupal core and contrib APIs and innovating where needed, RedHen is off to a very strong start. For Open Outreach, RedHen is helping us hone in on the key nonprofit and activist needs that got us going in the first place.

    We'd love to see more shops building on and contributing to Open Outreach. Whether or not you've already tried Open Outreach, please have a look at the new release with RedHen (and mapping, via Debut Location) built in. And if your response is "Yes, but...", consider that that's only another way of saying "Here's how I could contribute improvements!".

    If nonprofit networks can work together to tackle common problems, shouldn't we as Drupal shops and developers in the nonprofit space be able to do the same?

    Sep 12 2012
    Sep 12

    Chances are you're reading this because you're passionate enough about Drupal to read the Drupal Planet feed with your morning coffee or afternoon doughnut. If so, you or someone you know might be just the person we need on the Drupal Association board.

    Nominations for the 2013 "at large" Drupal Association directors close in just five days, on Sept. 16th. We need candidates!

    Last year we completely rewrote the structure of the Drupal Association, the nonprofit that supports the Drupal project. While we were working out the new structure, we discussed and argued and fretted and occasionally ranted about how we would meet the needs for experience and expertise on the board while staying true to our global community.

    How could we attract people with the ideas and skills and insights to carry the Drupal Association forward? Would hand-picking board members reduce diversity? How could we ensure the board didn't become even more US-centric than it already was?

    In the new structure we eventually approved, the board includes three types of members: Drupal project founder (Dries Buytaert); several members selected by a nominating committee and approved by the other current board members for three-year terms ("class" directors); and two community representatives ("at large" directors) elected annually.

    I served on the 2012 board nominating committee. It was a ton of work reaching out to candidates and then reaching consensus among ourselves on what we hoped was a strong and balanced board. Then one of our recommended candidates had to back out, while two others weren't ratified by the current board members.

    We ended up with a talented board that combined both longtime Drupal Association leaders and several newcomers. But I, for one, came away feeling that despite our efforts we'd fallen far short in drawing in the full diversity of the Drupal community.

    After the board nominees were approved it was time for our first open elections. I chaired the 2012 election committee, with two Drupal Association board members and many community members pitching in time, energy, and ideas.

    Just as with the nominating committee, it was our first time and we had to invent the process as we went. But a few weeks later we had ten candidates from five continents. Two lively all candidates meetings and a vote later, we had two new Drupal Association board members: Donna Benjamin and Steve Purkiss.

    What's it been like for them? An interview with Donna at Drupalcon Munich gives a good feel for her experience. Also take a look at her Drupalcon Munich workshop where she described herself, only partly tongue in cheek, as "your mole in the hole," increasing transparency by "diving into the inner workings of the Drupal Association" on your behalf.

    Having community "at large" directors has made a difference. Donna and Steve have raised questions and made contributions that might otherwise have been missing. They've increased the diversity of voices and perspectives at the table when key decisions are made.

    Donna's not running for re-election this year. Instead, she's heading up the elections process. Which brings us to the point here. Elections need candidates! Yes, that might be you!

    See the election announcement for details.

    And whether or not you run, or convince a friend to, please check the Drupal Association website after nominations close to see who's running and participate in the election discussion and debates (Sept. 17-23) and voting (Sept. 24-Oct.7).

    If not you, who?

    Jul 25 2012
    Jul 25

    "We think Open Outreach on Pantheon is an incredibly powerful tool for site builders and web-savvy nonprofit staff and a great place to begin for those new to Drupal." - Andrew Ward, East Bay Development

    At Chocolate Lily we've been working on Open Outreach for the past two years basically because we believe the best online tools should be accessible to all. And we're getting somewhere. This month the number of Open Outreach sites topped 350. Having Open Outreach hosted on Pantheon means groups can try Open Outreach for free. With the basics in place we're starting to dig into more activist focused pieces like a RedHen-based CRM and community mapping tools. For a two person shop (plus our son who we've hired on part time over the summer), it's not a bad start.

    But we need a lot of help to take it further. To turn Open Outreach from a fine idea into a tool that's really making a difference in bringing down the barriers and tooling up movements for change, we need you.

    There's a ton of ways that any Drupal site builder or shop can get involved and help turn Open Outreach into a community project and the NGO tool of choice for online advocacy and engagement. Here's a few:

    • Use Open Outreach for your pro bono work. See Andrew Ward's writeup of how his shop built a client site on Open Outreach in a single work day.
    • In your next nonprofit site build, consider using Open Outreach. Will it do everything you'd want? Nope. But it could be a great kickstart that saves both you and your client a lot of initial development time and expense.
    • Talk up Open Outreach in your local Drupal or nonprofit meetup or host an Open Outreach training in your community. We're finding an Open Outreach intro a great way to make Drupal accessible to new users. If you schedule it, we'll do our best to be available by voice or video to help out.
    • Test drive a draft Debut app. We and others are always working on new apps to add to Open Outreach. You can find them in the list of candidate Debut features. Try one out and post your review or improvements.
    • Contribute written documentation or screencasts. We've made a start at user documentation and screencasts but there's lots of room for improvement.
    • Contribute translations. We get a growing number of requests for Open Outreach translations to other languages. We're targetting Spanish as our first non-English language. But whether you build sites in Spanish or Portuguese or Esperanto, we'd love your help getting translations in place.
    • Let us know about good hosting options. We're excited to have Open Outreach hosted on Pantheon and other hosts but affordable hosting remains a key barrier to low-resourced organizations. We particularly need more options with point and click installation like Aegir. If you work with or know of a good host we should be talking to, please let us know.
    • Blog about your Open Outreach site builds. If you're building sites with Open Outreach, please take time to share your tips and tricks.
    • Bring your design expertise. Please. We've been pretty focused on functionality, pretty loose on design. That's not all bad--it means site builders using Open Outreach have a pretty clean slate. But we could use a lot more design flair, whether as enhancements to existing features or, say, Open Outreach themes.
    • Develop new apps for Open Outreach. If you're building out new functionality, consider doing it as an Open Outreach compatible app. We'll be happy to give you guidance and review along the way. See the Debut documentation for detailed pointers.
    • Get the word out about Open Outreach in your networks. So far our profile has been pretty low outside the core Drupal scene (and even within it!). If you use Twitter, you can follow @OpenOutreach and tweet about your Open Outreach work to your network.
    • Present at camps and conferences. We don't get around much. Okay, personally, I have to admit the last DrupalCon I went to was in Boston, and that's going back a ways. If you want to make a Drupal camp presentation, we'll be happy to help out with prep and presentation materials.
    • Do a site building sprint for local groups using Open Outreach. This can be a great way of bringing Drupal folks together or as a project of a local meetup.
    • Partner with us. Have a project that fits well with Open Outreach but need some help? We're always looking to partner with other Drupal shops.

    Comment here or contact us to get rolling!

    Jul 14 2012
    Jul 14

    The Apps approach to making Drupal site building easy is gaining support. But there's a long way to go before Apps can provide sites seamlessly integrating components from multiple providers.

    Apps: the glue between custom sites and Drupal distributions

    Apps promises to fill the gap between custom site building on the one hand and Drupal distributions on the other.

    To its credit, Drupal is hugely flexible. With great tools like Views, configurable content types and fields, and a host of solid extensions for everything from groups functionality to commerce, Drupal can help you build pretty much anything. As Larry Garfield has argued, Drupal is a content management framework rather than a content management system.

    But should you really need to build every site from scratch? With hundreds of thousands of Drupal sites already out there, chances are most of what you need has already been built many times over.

    The main answer to the aim of making site functionality reusable is Drupal distributions. Want a news-focused site, a nonprofit site, an online community? Browse the list of distributions on drupal.org, select and install one, and you're well on your way.

    But a distribution tends to come with some tight constraints. Design decisions. Interdependencies. Reliance on a particular theme. What if you don't need or want the whole deal? What if you want to pick and choose components that might come with various different distros--a bit of commerce functionality, some mapping, some search enhancements, a newsletter?

    That's the space that Apps aims to fill. An app is a focused set of functionality that you can easily download and install--along with all of its dependencies.

    In the 16 months since the Apps initiative was launched, dozens of apps have been produced and shared. Several distributions have been built on or converted to use Apps. The recently released Panopoly apps set provides a base designed to be used by various distributions as well as individual sites.

    There are two ways to get apps:

    • Install an apps-based distribution. You'll be prompted to select and install apps as part of the site installation process.
    • Install the Apps module and an apps connector module. Some available connector modules are LevelTen apps and Debut apps. Log into your site and select Apps from your admin menu. You're connected to one or more apps servers and given a selection of apps to install.

    If you're a Drupal developer and want to produce and share apps, your process will include:

    • Build an app. For hints, see the Phase2 documentation.
    • Either submit your app to an existing app server or create your own app server using the App server module. The App manifest module can help with generating the code to use on the app server.

    So far so good. But how well do the available apps play together?

    Interoperability: the Apps standard and Kit

    Ideally, different apps produced by different Drupal shops and developers will seamlessly integrate. Interoperability is a key aim in the Open App Standard from which the Apps module emerged:

    [t]he high level goal of the project is to achieve interoperability of apps between sites so that the Drupal platform does not become fragmented as specific distributions become widely adopted. That is, an app which conforms to the open app standard will be capable of being installed onto any site which conforms to the same standard.

    An early initiative to promote interoperability was the Kit feature specification, which laid out "a set of guidelines that facilitates building compatible and interoperable Features". Kit was a good start, but it languished when its authors mostly moved on from Drupal development.

    The close tie between apps and specific Drupal distributions has been a big factor in limiting interoperability. It's relatively easy to build an app that will play nice with other apps in the same distribution, since the apps will share a lot of common assumptions. But start mixing apps from one distribution with those of another and you'll quickly hit problems, ranging from relatively innocuous inconsistencies all the way to fatal page errors.

    Issues and barriers

    Layout and design

    There are two main approaches to layout in Drupal distributions: Context and Panels. While it's possible to combine the two in a given site, they can easily conflict.

    Can Panels-based apps mix with Context-based ones? Can we come up with a standard Apps layout approach? Discussions are ongoing both within app development circles and in the related issue queues, like this issue on Panels.

    Standard solutions?

    Ask any three Drupal site builders what modules they'd use to design a given feature and you're likely to get three different answers. Take mapping for example. One developer might choose OpenLayers, another Gmap, and a third Leaflet. There are valid reasons for each choice. But if you install three different apps that use three different mapping libraries, you quickly end up with a mess.

    There's probably no easy answer to the question of competing solutions. The Debut feature specification makes a stab, but isn't used beyond the Debut apps set. A good start would be a inventory of existing apps and their preferred modules, so we can get an idea at least of where we stand.

    Duplicate or conflicting components

    What do you name the image field that accompanies an article? If you're using a Debut app, it will be called field_content_image. Install Panopoly pages and you'll get a field_featured_image. Enterprise base? field_inline_image. Add them all after installing Drupal core with the standard install profile (which creates field_image) and you'll have four different fields doing pretty much the same thing.

    The problem gets worse with user roles. To work out of the box, an app needs to provide appropriate user permissions. But what roles should those permissions be assigned to? Even for virtually identical use cases, there's no consensus. For example, Enterprise base creates a 'copywriter' role for users who write content, while Debut provides a 'contributor' role for the same purpose. Install them both and - presto - duplicate roles, each with their own mess of permissions.

    Fixing this type of duplication could be as easy as extending Kit to cover more naming conventions. Doing so would require some good collaboration between app authors.

    Dependencies

    A bit more complex than naming conventions is dependency handling.

    Say you want to add a 'tags' taxonomy field to a content type. Easy--create a 'tags' vocabulary, add a taxonomy term field to your content type, and export the field to Features.

    However, the result will be that the tags vocabulary is exported along with the field, meaning that any other app using a 'tags' vocabulary will conflict with - or be dependent on - your app. Moving more generic components into generic apps doesn't in itself solve the dependency issue. For example, if my_tags and your_tags both provide a tags vocabulary and are dependencies for various my_ and your_ apps, the my_ and your_ apps will have conflicting dependencies.

    One way to avoid these sort of dependency problems is to keep generic components out of features. A sample implementation is the handling of the tags vocabulary in Debut Article (see debut_article.install and debut_article.module): the vocabulary is created at install time and then explicitly removed from features dependency handling.

    Another approach would be to develop and agree on a set of core apps that many other apps could require. Panopoly is a potential step in this direction. For example, apps needing responsive images could require Panopoly images. The planned Spark distribution may also produce some components or apps that could be used by various distributions and app authors.

    Enhancing the components of other apps

    Say you're using an app that provides social linking for different content types and you're also installing other apps that provide content types (events, pages, and so on). How does the social linking app know which of these content types should have social links?

    One step of complexity higher is the question of applying fields to entities provided by other apps. Say you have a mapping app that provides a mapping field. How can the mapping app enhance an app providing an event (which has a mappable address field) by adding a mapping field?

    Within the the Debut apps set, individual apps can declare components available for enhancement through a set of defined .info file keys. Two examples from debut_article:

    // Declare the article content type as suitable for social links.
    debut[social][node_types][article] = article
    // Declare the article content type as suitable for a media field.
    debut[media][node_types][article] = article

    Code in Debut Social and Debut Media interprets these keys, adding social links and a media field.

    Enterprise base takes a different approach to providing fields, with various apps requiring Enterprise base to create an image field on their content types.

    Developing some common approaches here would allow different apps with distinct approaches to recognize and enhance each other.

    Visibility

    Apps are provided by their own app servers. But how do you find an app server?

    Look on drupal.org and you'd be lucky to find out that apps even exist. Drupal distributions are highly profiled. But the apps that many of them are built on are practically invisible. Some are listed, but they're lost among the thousands of available modules. No app yet has a large install base. To fix this, we'll need at least an "apps" category for modules. Better yet, as the Apps model matures, we could have a separate space for browsing apps alongside distributions, modules, and themes.

    Security

    Alongside visibility, there are security concerns with the Apps model. Regularly with Drupal, you're downloading and installing code from drupal.org and hence can rely on some security standards. In contrast, Apps has you connecting to and downloading and installing code from various servers that you as a site administrator don't know anything about.

    Part of a more robust Apps framework might be hosting apps directly on drupal.org. A lot of the technical infrastructure and protocols that would be needed has already been put in place as part of the work to fully package distributions on drupal.org.

    Apps tools

    Apps enables an easier site admin experience by handling the download and install of dependencies, including external libraries. But it also introduces some barriers. Install an Apps-based distribution and you'll be prompted for ftp or ssh authentication--even if all the code you need is already in the install.

    Overall the Apps toolset is at a relatively early stage. The good news is the Apps and Appserver modules are starting to get some community contributions building on the great start made by Phase2.

    Apps: a promising start

    The whole point of community open source is pooling efforts and sharing solutions. We do so with code. Can we do the same with site functionality?

    Apps and the features they're built on provide a promising start. But there's a lot more work to do.

    Those of us involved in building apps need to work harder on developing shared standards to make the ideal of fully interoperable apps a possibility--and then a reality.

    Jan 31 2012
    Jan 31

    What is Drupal good for? Drupal is often used for creating complex organizational websites that might take weeks or months to develop from initial plans to launch. But what if you need a site, like, tomorrow?

    Rob Wipond, a journalist friend of mine, came to me recently with just such a request. Municipal elections were two weeks away and he needed to get a website up immediately where candidates and citizens could sign onto an alternative economic statement ahead of the elections. He was familiar with Wordpress and had tried to get it to do what he needed, but no dice. His needs weren't overly complex, but they were specific and, at least with his skill level, Wordpress wasn't doing it for him.

    Build a site in a day? Rosemary Mann and I at Chocolate Lily like a challenge, so we signed on. Was Drupal up for the task? You bet.

    Building an instant but customized site is a good test case for what Drupal can do, and also a good entry point to understanding the basics of site building with Drupal.

    So here's how our day went.

    Defining the specs

    Sometimes we'll allocate weeks for a "discovery" phase of a project, carefully working through with a client just what their needs are on a site. Here we had maybe an hour.

    Luckily, Rob had a pretty clear idea in his head of what he wanted and - just as important - was great at describing that picture. He wanted:

    • A home page that would present a "consensus statement" on community economies with the names of initial signatories, maybe in two columns, and, below that, in three columns, a list of groups, citizens, and political candidates who had signed on. If a list of signatories got too long, it should be cut off with a link to a separate page where the full list could be browsed.
    • An easy form where anyone could sign on, anonymously, adding some information about themselves and an optional comment. Some of this information was for public display, but some - e.g., email addresses - was private and should be accessible only to the site administrator.
    • A separate page where just the comments would be browsable in a pagable list.
    • A contact form.
    • Some measure against spam.

    Okay, sounds straightforward at least. Let's get to work.

    Building the site

    To get this done pronto, we had to rely on the simplest and most proven Drupal solutions that fit the spec.

    Content type and fields

    Okay, so let's start with the basics. One of the key bases of Drupal's flexibility and power is its ability to produce highly customized content types.

    Here, we have only one custom content type to build: a signatory. This content type will produce the form site visitors fill in to sign onto the statement and also the information about signatories we display on the site. A content type is built of fields, defined by the particular attributes that need to be captured and displayed. Here that includes things like the signatory's name and website address.

    According to Rob's spec signatories needed to be classified by type - organization, citizen, electoral candidate - so we used Drupal's built in taxonomy system for this, adding a vocabulary and terms.

    While Drupal core comes with basic functionality, a wealth of modules can be used to add particular pieces of functionality. Determining which modules are available and best suited for a particular purpose takes some learning. For additional fields on the content type we used some of the many Drupal modules that provide specialized fields: Email field for capturing signatories emails and Link for website addresses.

    To ensure private information wasn't publicly exposed, we configured the display for each field in the "signatory" content type. And we configured permissions so that anonymous site visitors could post new "signatory" content.

    Great, got our content ready. Now to work up some displays.

    Lists of content: Views

    Recall that a key requirement for Rob was that we display signatory information in very particular ways. On the home page we needed three different lists of signatories, each filtered by type (organization, citizen, candidate). Only certain fields should show in these lists.

    Did someone say lists? Enter the most-used Drupal module: Views. Views makes it easy and quick to produce customized lists--of comments, site users or, as in this case, content. While building views takes a bit of learning, there's great inline help available - just install the Advanced help module - as well as dozens of online tutorials and resources.

    We configured a quick view with multiple displays (variations), including one using a contextual filter to allow filtering by different types of signatories. We used a special kind of views display, a "content pane"--more on content panes and filtering below.

    What's next?

    Putting it all together: Panels

    With our content type and views in place, we needed a quick and easy way to arrange them on a page, producing the custom home page layout Rob was picturing.

    Here the Panels module was just the ticket. Panels makes it quick and easy to produce customized layouts and add to them anything you want--like views displays. You can also create custom content to add to panels, which is what we did for the "consensus statement" as well as the list of original statement signatories.

    Panels comes with several different page layouts and also allows you to create your own. We started with a simple mini-panel, where we displayed the original signatories in two columns. Then we added this mini-panel to a page layout where we also put the consensus statement. Below that, in three columns, we added the signatory lists.

    Here we could take advantage of the views "contextual filters" we set up before. What we wanted were really the same list three times, only with different filtering. Panels configuration allows just that. We could add the same view display three times, each time specifying a different signatory type. As a bonus, Panels used the correct signatory type in the title for each of our lists. Nice!

    Picking a theme

    A theme defines the look and feel of a Drupal site. Often an organizational website will feature a custom theme, specially produced and tweaked to present the image a particular groups wants to present to the world.

    But for a one-day site? Forget it. What we needed was something that worked for Drupal 7, fit the general focus of the site, and was simple enough to work with our minimal design.

    Finding a ready made Drupal 7 theme isn't as easy as it could be. When you browse the list of most used themes filtered by version, what you get mostly isn't ready to use themes at all. Instead, the list is mostly filled with what are called base themes--starter kits that make it easier to design your own Drupal theme.

    Great no doubt if you're a designer, but useless for our purposes today. We dug a bit deeper and came up with Tarski, one of several themes that have been ported to Drupal from other content management systems. Simple, clean, and it installed with no obvious errors.

    Just one detail to fix. The theme had built in advertising for the company that had ported it to Drupal. Ugly--but easy to fix. Themes are built using fairly simple template files. I did a quick text search and edited out the ad.

    So far so good, but most of our day's up. What's left?

    Finishing touches

    Enable and configure the contact module so there's a contact form people can use.

    Add a couple of blocks to display on the contact page and at the bottom of the site.

    Add a top of page menu with links to the main pages we've created.

    With our site roughed in, it's time for some quality assurance. Quick, throw the site up on Rob's web host and give him the login details.

    What, no spam prevention? Oh, right. Download and install Captcha and configure it to apply to the form for signing on to the statement. Now we'll get those annoying but, yes, effective tests that supposedly sort humans from machines. (Why do I always seem to fail?)

    It's great that emails are hidden, Rob notes, but what if I need to see them, like to generate a mailing list? Good point. Talk through Views with Rob over the phone. Lead him through the steps to create a new view listing all signatories with emails, and then restrict access to the new view. Wow, Rob says, this is cool, I'm starting to get a feel for this!

    Hey, you know, Rob says, it'd be great to see the number of signatories displayed, so that people have an immediate sense of how this is catching on.

    Uh oh, scope creep. But what the heck. I resort for the first time all day to coding up a PHP snippet, albeit only a line or two, and add the result to the front page panel.

    And, Rob adds, I'd like some emails to go out when someone signs on, one to them to thank them for signing on and one to me so I know to review their post.

    That's kind of specialized functionality, but luckily the Rules module makes it quick. Rules are actions that are taken when certain events occur and given conditions are met. I ran Rob through the process of creating an configuring a rule to send out the emails (action) when new signatories were added (action) by other than a site admin (condition).

    And, Rob adds, I had this cool idea that if.... Okay, okay, I get it Nedjo! I'll add this to the follow up wish list!

    And we're ready to launch. Rob had already registered a domain, so the site was now live at http://communityeconomy.ca. Rob and his colleagues blitzed their email contact lists and crossed their fingers. With only two weeks before the elections, was there still time to make a difference?

    Celebration time

    Absolutely.

    Sign-ons started coming in from the day the site went public. By the end of the campaign nearly 150 people had signed the ambitious and progressive community economy statement, among them many electoral candidates. And nine of the signatories were subsequently elected: seven counsellors, a mayor, and a school board trustee.

    With their one-day Drupal site, the Metro Victoria Community Economy Group had opened an important space for public engagement. They also came out of it with a ready made outreach list as well as a site ready to expand in new directions as their public engagement grew.

    For us at Chocolate Lily, building a site in a day was intense and demanding but also tremendously rewarding. It reminded us of why we chose Drupal in the first place--to be able to put powerful and fitting tools in the hands of community activists and grassroots leaders.

    As a test case, this site build affirmed Drupal's power and flexibility. Building on Drupal takes some knowledge, but it doesn't have to be expensive or painful. Fitting the right tools to the task, Drupal can get you just the particular, customized solution you need when you need it.

    In a single day if need be.

    Jan 20 2012
    Jan 20

    A current project we're working on at Chocolate Lily involves upgrading a Drupal 6 site to use the Drupal 7 Open Outreach distribution. It's the first time we've used a Drupal distribution for an existing Drupal site instead of a new one. Having worked on my share of hairy Drupal upgrade projects, I went into this project with a fair bit of apprehension.

    But I've been pleasantly surprised. The basic upgrade process was relatively quick and painless.

    As we went I wrote up some notes on how to upgrade an existing site to Open Outreach. Here I'll generecize that a bit to answer the question: how do you upgrade an existing site to use a Drupal distribution?

    Fresh install or upgrade?

    Drupal distributions are usually used as a starting point for developing a new website. The most straightforward way to use a distribution is to install it from scratch. From there, you can selectively migrate in content from other sources, including one or more existing websites. Migrating data here might be as simple as copying and pasting several pages from an old site into the new site. For larger sites, it could involve writing custom migration scripts or using an existing Drupal module like Migrate or Feeds.

    If you already have a Drupal website that has a considerable amount of content or a large number of users or has custom functionality you want to keep, though, you might want to try directly upgrading the site to use the distribution. Doing so could save you a lot of time in migrating content. But it will also raise a number of challenges. Upgrading to a distribution is probably something you should try only if you have the skills and time needed to do troubleshooting and some custom data work.

    Basic upgrade to a Drupal distribution

    Here are basic steps to take to upgrade an existing Drupal site to use a distribution. These are rough guidelines only. Because each site and each distribution is configured differently, you're likely to run into additional complexities. But these notes should at least get you started in the right direction.

    1. Select a distribution to use. You'll want to find the distribution that most closely matches your site requirements. See the listings and comparisons of Drupal distributions on drupal.org.
    2. Back up. Don't work with your current live site. Instead, create a copy of the site and its database and work with the copies.
    3. Update the site to the same Drupal core release as is used in the latest release of the distribution you're using. One way to find out what version of Drupal the distribution is using is to download it and look in the file includes/bootstrap.inc for a line like this: define('VERSION', '7.10');. If your site is in Drupal 6 and the distribution is in Drupal 7, you'll need to upgrade to Drupal 7. See the drupal.org documentation for detailed upgrade instructions. Here is a synopsis of some of the key steps.
      • Update your site to the latest version of Drupal 6 (core and contributed modules). Before upgrading to Drupal 7, disable and uninstall modules you know you won't be using in the new site. To help determine which modules to uninstall, you could review the list of modules included in the distribution, which often are found in the download in the directory profiles/[distribution_name]/modules. If a given module is not in the distribution and you don't foresee needing its functionality on your new site, you may choose to uninstall it.
      • Do an upgrade to Drupal 7 core.
      • Also upgrade any custom (non-core) data that you intend to keep on your new site. At the very least, you'll likely need to migrate Drupal 6 CCK custom fields to Drupal 7 fields.
    4. Download the latest release of the distribution. Download and extract the latest release and copy your existing site's Drupal 7 version settings.php file and files directory to the sites/default directory in the distribution install. This will point the new site to your existing site database.
    5. Change your site's 'install_profile' variable to the "machine name" of the distribution. Typically a distribution is installed via an "install profile". To determine the machine name of your distribution, look in the profiles directory for a directory that's the same as or similar to your distribution's name. To tell your site that it is using this distribution, you likely will need to reset a variable in the 'variable' table. To do so, download and enable the Devel module and then run the following code at devel/php: variable_set('install_profile', '[distribution_name]'); where [distribution_name] is the machine name of your distribution. Alternately, you can reset the variable by running a command in your database. For openoutreach, this would be: UPDATE variable SET value = 's:12:"openoutreach"' WHERE name = 'install_profile';. Until you make this change, you likely won't see any of the modules or themes that ship with the distribution, as they're in the install profile's directory.
    6. Run updates. Run update.php to update any contributed modules that were enabled on your existing site and that are also present in the distribution you're using.
    7. Enable the distribution's install profile. If your distribution is based on an install profile, you need to enable it now. Install profiles behave much like modules, and so can be enabled either using Drush or via the user interface at admin/modules. However, by default, install profiles are hidden, to prevent accidentally disabling or uninstalling them. If you're using the user interface and don't find a module with a name the same as or similar to that of your distribution, you may need to manually edit the install profile's .info file, adding the line hidden = FALSE. See the drupal.org documentation on the .info file format for details.
    8. Select and enable the features you want to use. At admin/structure/features, under the "Features" tab, enable features that you wish to use on the site. Recommended: enable the features one by one, so that you can note and respond to any errors. If you are prompted about orphaned dependencies, select "Leave enabled". For more information about the individual features, read any available documentation on the distribution you're using. See also the section below on "Assessing and preventing component naming conflicts".

    Cleaning up your site

    The steps above outline how to get a distribution minimally installed on an existing site. But you'll still have a lot of work to do to reconcile your existing site content and structure with what has been created by the distribution. Here are a few tips to get you started--but you should begin with the assumption that there will be lots more you'll discover and need to fix.

    • Roles. Many distributions will create one or more custom roles. Examples include contributor, editor, and administrator. If any of these roughly correspond to existing roles on your site, delete the roles created by the distribution and rename the existing ones to use the distribution role names. For example, if your site has a role called "site administrator" and the distribution has created a role called "administrator", delete the newly created "administrator" role and rename "site administrator" to "administrator". After adjusting the role names, visit admin/config/people/accounts, ensure the "administrator" role is selected as the "Administrator role", and save the form. Consolidating the roles in this way will get you the functionality of the distribution's roles and permissions while maintaining your existing users' role allocations.
    • Content types and fields. You may have existing content types on your site that overlap with those provided by the distribution. For example, if you have an existing content type called 'news', it might have a function very close to that of an 'article' content provided by one of the features you've enabled. To begin to use the new feature's functionality, you could consider converting the existing 'news' content into 'article' content.
      • Download and install the Node convert module.
      • At admin/structure/node_convert_templates, create one or more templates for converting your content. Then use the admin/content page to filter for the type of content you are converting from, select the appropriate conversion from the "Update options", and apply.
    • Blocks and contexts. Many distributions use the Context module to position blocks. Your existing site may use the core Block module for this purpose, may use Context or some other tool, or may use a combination of tools for block placement. With your new distribution's blocks displaying as well as those enabled by your existing site, you may get more than you need or want. To address this issue, selectively disable blocks left over from your existing site. If they were custom blocks, you may wish to delete them.

    After each step of reconciling existing site elements with those created by the distribution, you may need to "revert" one or more features from the distribution. See the Feature module documentation on reverting features.

    Assessing and preventing component naming conflicts

    Most distributions are built using the Features module, which allows exporting configuration from a Drupal site - content types, fields, views, and so on - into code, so that it can be enabled on multiple sites. For components like content types and fields to be exportable, they need to have a "machine name"--a unique name that will be the same on every site they're enabled on. For example, a date-type field used to store the date of an event might have the machine name field_date.

    If the distribution is installed from scratch, we can be sure that the components we're creating won't conflict with existing components on the site. But when we're converting an existing site into a one based on a distribution, there's the potential that a component we're creating will have the same name as one that already exists on the site. In certain cases, such a conflict can cause a site error that's difficult to resolve.

    The main issue to watch out for is: the site already has a field with the same name as a field that will be created by the distribution and the existing field is of a different type than the distribution's one. Drupal is unable to change the type of an existing field, so this situation will trigger an error.

    There's no totally easy way to determine if a feature will conflict with existing components on your site. For fields, try the following:

    • At admin/structure/features, click the "Rebuild" link for the feature you're looking at. Look for any fields listed under a heading "Fields". The names here are of field "instances"--the version of a field that's applied to a particular content type. What we're concerned with is the actual field name, which is the part at the end beginning 'field_'.
    • Ensure the Fields UI module is enabled. At admin/structure/types, click the "Manage fields" link for a content type and look for fields with the same name as one of the fields in the feature. If you find one, you have the potential for a conflict--an error will occur if the field types are different.

    Errors and what to do about them

    • I get the error message Notice: Undefined index: distribution_name in drupal_install_profile_distribution_name(). This means you haven't yet enabled your install profile.
    • I get a fatal error saying a class or function isn't available. This is a common problem when code has been moved around. Two sources are (a) a required module is not present and (b) the Drupal files registry needs to be rebuilt. Ensure the modules that were on your old site are also available on your new one. Make sure you've set your install profile so that modules that ship with your profile are available. You may need to visit admin/modules to refresh the location of your module files. If all else fails, consider whether you need to use Registry rebuild.

    Wrapping up

    Upgrading an existing site to use a distribution like Open Outreach is more work than just starting fresh. It takes some planning and the willingness to work through some potential challenges.

    If you're up for a few bumps, though, merging an existing site's data with a distribution can get you the benefit of quickly ramping up functionality and design without losing the core of your site: your content and users.

    Oct 31 2011
    Oct 31

    As the first few organizations start to try out our Open Outreach distribution, I'm realizing there's a lot to figure out as you build a distribution-based site. Like how much should you customize the site versus taking it as it is? And how do updates and upgrades work? So here are a few pointers. If you have more, please leave comments below.

    The first maybe obvious point to realize is just that building a site off a Drupal distribution, especially one that provides a lot of out of the box functionality, is different in important ways from building a site "from scratch" by downloading Drupal core and then adding modules and configuring. Customizing works differently. Updates work differently. So there are some important questions to think through at the start.

    Working through these questions will often call for some familiarity with Features, the framework used to build most Drupal distributions. If you're considering customizing a site off of a distribution and aren't yet familiar with how Features works, you'll want to spend a bit of time looking over one or more of the various tutorials out there like Drupal Features module abridged or An introduction to the Drupal Features module and/or the module's documentation.

    Key decision

    When building a site on a distribution, probably your most important decision is going to be between three options:

    1. Stay on the distribution's update (get a new release for the major version you're using) and upgrade (move to a new major version) path. For most site developers, staying on the distribution's update path will have big advantages. Doing so doesn't mean you can't customize the site somewhat to your own needs. It does mean, though, that you'll need to be careful in the customization you do--more on that below.
    2. "Fork" the site, taking it in its own direction and handle updates yourself rather than relying on future releases of the distribution. This option may be a good one if you have extensive needs that won't be met by the distribution out of the box and are, or have working for you, an experienced site developer who's capable of customizing and maintaining the site. You might also consider this option based on the release status of the distribution. Is it still in alpha or beta? If so, you probably can't count on a stable update path even if you want one. Until it gets a full release, the distribution is likely to go through changes that don't easily update, so you might be best off just taking the site in its own direction.
    3. Combine the two. With this approach, you plan to stay on the distribution's update path, but also do the occasional "manual" update as needed. You might take this approach if you find you need a different version of a module than what ships with the distribution, or want to handle security updates yourself rather than waiting for the distribution to catch up, or want to customize just a select part of the distribution--say, one or two of its features.

    If you're not going to stay on the distribution, you might ask, why bother using it at all? But there can still be compelling reasons. Building off of a distribution can give you a huge leg up, by providing functionality that would take you days, weeks, or months of work to reproduce.

    For example, at Chocolate Lily we recently produced Nature Explorers, a site for Nature Canada aimed at engaging Canadian youth in outdoors activities. We had less than two months from project start to site launch and a substantial list of functional requirements. A lot of this wasn't covered in Open Outreach, but a lot of it was. So we started with the Open Outreach code base, quickly customized some of the features by manually editing the code, and had a huge jump start on the site by the end of the first week.

    Still, if you fork the site you'll be on your own when it comes to future updates and upgrades, so you want to make this decision carefully.

    Staying on the distribution update path

    The first thing to keep in mind is: the closer you can stick to what the distribution offers, the more value you'll get from it. Take for example Open Atrium, which provides a lot of highly refined intranet functionality. Sure, there are things you might choose to do a bit differently from what OA does out of the box. But before you wade into making those changes, ask yourself: are they really necessary?

    Customizing on the distribution

    If you're planning to stay on the distribution's update path but still want to customize, here are some ideas and possibilities.

    • Selectively turn off UI feature modules, if the distribution includes them. Some distributions separate user interface configuration like menu items and block display from backend functionality like content types and fields, putting them into distinct feature modules. See these posts about Voicebox and NodeStream. In this case, you can turn off the UI modules and replace their functionality with your own site elements.
    • Make your module additions in sites/all/modules or sites/default/modules. The distribution's files, including modules and themes, will generally be in profiles/[profilename]/modules and profiles/[profilename]/themes, where [profilename] is the name of the profile that you installed. To make sure your customized code is distinct from what ships with the distribution, put yours in sites/all or sites/default. These locations will override the profiles code, meaning that if you need to e.g. use a different version of Views, you can put it in sites/all/modules/views and it will be used instead of the version that shipped with the distribution.
    • Isolate your changes. Whenever you can, try to add new components (views, fields, etc.) rather than changing existing ones. This is because the existing components that ship with the distribution may be changed in future versions of the distribution. For example, in a new release of a distribution a view of events might get enhancements to an existing calendar display. Say you had a need for a "past events" page. If you've edited your own copy of the events view to add your past events page display, it's now "overridden". When you update the distribution, you won't get changes to that view that came with the new version--at least, not without tossing out your own customization.
    • Carefully track any changes you make to components that ship with the distribution. If you really need to make changes to components that ship with the distribution, keep track of them. When it comes time to update, you'll need to make sure you don't "revert" those components (more on this below). Alternately, if the changes you're making are relatively simple and you're comfortable working with Drupal hooks, you could consider making your customizations in a custom module via an implementation of an alter hook for the specific component. For a context, for example, you would implement hook_context_default_contexts_alter(). Or you could try out the Features override module, which offers some limited ability to export overrides to a new feature.

    Taking the site its own way

    If you're planning to take the site off the distribution update path, here are some potential approaches.

    • Turn off everything you don't need. Take some time at the outset to strip the site down a bit. Consider each feature. Do you need it? Then look at dependencies that feature enabled. Are they needed? The more you can turn off, the less clutter you'll have as you customize.
    • Consider forking individual feature modules. Most distributions are built off the Features module, which enables capturing bundles of site configuration in code. If you're going to take a feature in your own direction, you might want to consider forking it--changing its machine name so that it's no longer linked to the original, which you can do with a bit of search and replace. By doing so, you can use and adapt the original code and at the same time avoid possible future confusion ("Should this feature be updated to the new release?").

    Doing updates

    With a typical Drupal site, you might be used to downloading and installing module or Drupal core updates one by one, when you receive update notifications.

    With a distribution, however, there are additional factors to consider:

    • To achieve its specific aims, a distribution may include "patches" (modifications) to the code it includes. If you update without applying these patches, critical features may break or disappear.
    • Because a distribution is often a complex and carefully designed whole, replacing a single part at a time, like updating a module, can have unexpected outcomes. Changes introduced in the new module version may well break existing functionality.

    Updating to the latest distribution release

    If you've decided to stay on the distribution update path, you'll probably want to ignore update notices for individual modules. Rather than updating one by one, you'll instead be updating when a new version of the distribution is available.

    Like any time you're updating, unless a new release is a designated security update, or contains fixes to problems on your site, you may choose to wait before updating. See the drupal.org documentation on updating and upgrading a Drupal site.

    If the distribution is in alpha or beta, you can expect some rough spots. Be sure you read release notes carefully, and follow any instructions they give.

    Gotchas to watch for include:

    • New modules: In some cases a distribution may have introduced new module dependencies between releases. One of the distribution's features may now require a helper module that wasn't required before. Dependencies like this one are automatically enabled when you first enable a module, but are not automatically enabled if the module (in this case, the feature that's part of the distribution) is already enabled on a site. So you may need to manually enable the dependency as part of update.
    • Field type changes: Occasionally a distribution may change the type of an existing field. We did this this between beta releases in Open Outreach, when the field type we were using, 'media', was deprecated and we switched to 'file'. Anyone trying to update will get a fatal error, since the type of an existing field can't be changed. The only solution is to first delete the field - and all its data - before updating.

    Basic steps to update your distribution-based site (adapted from Open Publish documentation) might be:

    • Make full backup of your installation.
    • Download the new release and unpack it on top of your old one.
    • Run update, via http://yourdomain.com/update.php
    • Do any special stuff specific to the update, like enabling required modules.
    • Clear caches.
    • Go to admin/structure/features if it's a Drupal 7 site or admin/build/features if it's Drupal 6. If you see any features showing "overriden" status, go into the feature with your list of intentionally overridden components in hand. Check overridden components - unless these are components that you've intentionally overridden - and click on "revert components". Repeat for all other overridden features.

    But, again, each release will have its quirks--look for release notes to guide you through the update or upgrade.

    Updating a "forked" site

    If you're updating bit by bit rather than by updating the distribution as a whole, here are some points to keep in mind.

    • Make sure you know what patches need to be applied. The most likely place to find these is in Drush Make (.make) files. Places to look for Drush Make files include:
      • profiles/[profilename] where [profilename] is the name of the profile you installed.
      • The directories of individual features included with the distribution.
      Depending on what code changes have happened meantime, the old version of the patch may not apply--you may need to look for a new one. While you can do this manually - download and apply the new modules plus their patches - another option is to continue to maintain and update the distribution's .make file, so that you continue to have a place where code is tracked.
    • Be careful about switching versions or branches. If the distribution ships with a nonstandard version of a particular module - e.g. the dev branch of views - there's going to be some reason for that choice. Maybe it's to get a patch that hasn't yet reached a stable release. Or maybe it's for compatibility with another module. For example, in Open Outreach we're using the dev branches of Views and Ctools because they're currently required for compatibility with Date and Calendar. In this case, if you're not using the event feature in this distribution, you could try switching to the stable releases of Views and Ctools--but if you are using event, switching would break the site.

    Summing up

    To restate the obvious, you'll get the biggest benefit from the distribution if there's a close fit between what you need and what it offers. If you can install the distribution and start entering content, fantastic. But if you go in with your eyes open, you can also get a lot of benefit from taking what the distribution has to offer and building from there.

    Got more tips or questions about building a site off a distribution? Successful case studies? Disaster stories? Please leave a comment.

    Oct 26 2011
    Oct 26

    Open Outreach is a Drupal 7 distribution designed to provide a web solution that small and medium sized nonprofits can just turn on and start using. Today we rolled the beta5 release, which takes some major steps forward in terms of usability while catching up with the latest progress in Media module. We also have a new project website at openoutreach.org, with a growing collection of screencast tutorials on installing and using the distribution.

    If you're looking for a leg up in building out nonprofit sites, now's a good time to download Open Outreach and run it through its paces. And if you're a Drupal shop or developer looking to partner in building a great distribution for nonprofits, now's a perfect time to join in.

    The whys and wherefores of a nonprofit distribution

    Efforts to build Drupal distributions for nonprofits have usually focused in on functionality specific to nonprofits, like volunteer management or donation handling. You can see why--many or most nonprofits will indeed want ways to work with volunteers or collect donations. And specialized applications like Donor Rally and Volunteer Rally are definitely valuable tools.

    But groups' main needs in a website are much more basic. They need a platform they can quickly understand, that's accessible to them, that allows them to post content and get their message out. And, usually, they don't have a lot of funds to throw around on the way.

    Probably the best researched and documented source on the needs of web nonprofits and current solutions is the 2010 Idealware comparison of open source CMSs. If you build sites for nonprofits, this is invaluable reading.

    The Idealware report highlights above all the importance of getting the basics right. Yes, having a powerful and flexible platform is great--but you have to be able to understand and use it. All the extensibility in the world isn't worth a lot if getting and learning to use even a basic website is way out of reach.

    The simple conclusion from the report might be: if you're a small group, use Wordpress. This was the basic theme of Jennifer Lampton's insightful and provocative Wordpress is better than Drupal talks: yes, Wordpress is limited in many ways, but it's intuitive and works out of the box.

    But as a grassroots organization, can you have it all? Can you get a web platform that just works for you, without dropping thousands of dollars on specialists to help you get going, but that can still be extended in any direction you can imagine if and when you have the need and the resources?

    Drupal has a reputation for being powerful but expensive. By building sites off a distribution, we aim to bring a powerful and fully featured site within the reach of the small, underresourced, grassroots groups that are at the heart of social change movements. Drupal + affordable = action!

    Getting off the configuration treadmill

    If you've built even a few sites for nonprofits, you'll know that a lot of the initial work is pretty much the same each time. By the time you've got the basics in place - social links, a half dozen content types and their fields, views and blocks, a bit of SEO functionality, a WYSIWYG editor and accompanying text formats - you're well into your site building budget--and you've only just got to the part that's really specific to the group you're working for.

    If we as Drupal shops or site developers end up in a rut of basically repeating a lot of work with small variations for each site we build, it's not a lot better for the organizations buying our services. They end up paying again and again for basically the same initial work.

    This is the basic motivation for building Open Outreach. Like Drupal and community open source in general, it's about building shared solutions to common needs, while allowing groups to focus their limited resources on the new functionality that's truly specific to their unique work and organizational culture.

    Is Open Outreach everything you could wish for? Not at all. If you're an experienced site builder, you'll immediately find functionality you'd like to add. You might decide to skip some of what comes with Open Outreach--not every project requires a forum or an event calendar. You might also decide to take some of the features and fork them for the specific needs of a given site. But even if you keep just 80% of what it provides and dig in from there, Open Outreach can save you a lot of rote work--and also provide a consistent framework to base new functionality on.

    A different kind of distribution

    There's a growing number of Drupal distributions out there, including several with a nonprofit focus. What's different about Open Outreach?

    Flexible

    Many distributions are built to a highly specific use case: an online fundraising campaign (Donor Rally), an organizational intranet (Open Atrium), and so on. In contrast, Open Outreach is intentionally designed for a broad range of organizational uses. As the distribution develops for sure we'll want to see more focused functionality added in as options, or else specialized distributions built off the same base (see below on Debut). But first things first. By focusing on universal or common needs, we're creating a distribution that can be used as either a stand alone site for a smaller group or the base for a customized build for a larger organization.

    Modular

    While many distributions are fairly monolithic, Open Outreach is scrupulously modular. The features that make up the distribution have almost no interdependencies. At install time, you can select exactly which features you want to enable, and then add new ones at any time.

    Open Outreach features are built to a common, documented standard: the Debut feature specification. Debut documentation includes notes on how to build out debut features as well as simple instructions for building your own distribution off of Debut.

    Get involved!

    We at Chocolate Lily have put in the legwork to get Open Outreach to a solid beta release, but we've never intended it as a one-shop project. With some basics now in place, we're actively seeking Drupal shops and developers to partner with. Do you work for nonprofits? Want to be part of planning and building out a great community-produced Drupal nonprofit platform? Talk to us.

    Sep 21 2011
    Sep 21

    Sooner or later most site developers are going to want to include related content--the articles most relevant to the content being viewed. Here are some quick notes on approaches to related content in Drupal 7.

    Inferred or curated relationships?

    The first distinction to make is between two types of related content.

    1. Curated relationships that are directly designated by site administrators.
    2. Inferred relationships that can be determined from available information. For example, a solution might use keyword analysis to determine content that has similar keywords and therefore is implicitly related.

    Curated relationships

    A curated relationship is an explicit link created between two pieces of content.

    Less precise than an explicit relationship is the taxonomy system, which allows content to be assigned to shared categories.

    Taxonomy

    The taxonomy system is used to classify content. A simple approach to related content might create a number of vocabularies, assign them terms, and then classify content with those terms. From there it's relatively straightforward to display, alongside a piece of content, other content tagged with the same term or terms. See e.g. these tips for creating a view of related content.

    Explicit relationships

    The traditional solution for mapping explicit content relationships in Drupal is Node reference. For Drupal 7 it's packaged in the References module. See this tutorial for tips on displaying node reference-based related content using Views and Panels.

    While it's tried and true, Node reference has a number of longstanding challenges. These include:

    1. The relationship mapped by node reference is necessarily one-way. Since the reference is a field that's added to a given content type, it's a property of only one of the pieces of content--the one that has the field. Depending on the type of relationship, this fact can create challenges when displaying related content; there's no easy way to mix "content this node references" and "content that references this node", even though the two may both be "content related to this node".
    2. The module doesn't allow a relationship to have any properties. For example, if an individual is associated with an organization, we can't store the date of joining as part of the relationship.
    3. The workflow of adding a reference can be cumbersome if the item you need to link to doesn't yet exist. For example, if you are adding an author and listing her publications, but have not yet entered the publications, by default you can't continue--you must first create the publications as separate content and then return to edit or create the user.

    In Drupal 7, the Relation module is being written to try to address some of these issues.

    Inferred relationships

    Many sites, especially larger ones, are being built on Solr as a search solution, using mainly the Apache Solr search integration module.

    While the main incentives for using Solr tend to be superior performance and advanced faceted search solutions, related content can be a valuable side benefit. The Apache Solr search integration module offers a related content block, with matches generated on the basis of complex analysis of the structured data sent to the Solr index.

    The Recommender API offers another algorithm-based approach to determining relevant content.

    More information

    There's a useful comparison page on some of the related content modules available on drupal.org.

    Sep 19 2011
    Sep 19

    The Block Cache Alter module for Drupal allows custom configuration of whether and how each block should be cached.

    Caching is a key component of Drupal site performance. While page-level caching can provide big performance gains for anonymous page views, for logged in users block level caching is important. Drupal core supports cache settings defined by modules that provide blocks, but doesn't provide a way for site administrators to customize settings per block.

    That's what Block Cache Alter is for.

    The basics

    For every block enabled on the site, you'll have the following questions to answer:

    1. Should the block be cached?
    2. If the block is cached:
      • With what specificity should it be cached--globally (sitewide), differently on each page? etc. This is the "Cache setting".
      • When should the cache be refreshed ("flushed"), triggering a refresh of the block? These are the "Cache refresh options".

    Deciding if a block should be cached.

    Caching blocks improves performance, but there are certain conditions in which it doesn't make sense to cache blocks.

    Do not enable caching for a block if any of the following is true.

    • Restricted content. The block could possibly contain any content that has restricted access. Caching might expose sensitive information to users who shouldn't see it.
    • Forms. The block contains one or more forms, e.g., a poll voting form. Caching will prevent the form from working properly.
    • Broken content. You turn on the caching for the block and it breaks in ways that aren't easy to pinpoint and fix.

    Setting cache specificity: selecting a "Cache setting"

    If you're looking at a block and it already has a cache specificity value, this might have been set by the module that provides the block. Consider leaving the value as it is.

    Blocks will often differ per section of the site. For example, if a block is on a taxonomy term page and should display content specific to that term, select "per page" so that you'll get a different version of the block on each page it appears.

    In terms of efficiency, the most desirable setting for block caching is to cache globally, because then there's only a single version of the block that needs to be generated and saved. If the block doesn't need to reflect anything per section, user, or role, select global.

    Setting when to refresh a cache

    There are three options here:

    • Only refresh manually, in which case "Cache refresh options" are left blank.
    • "Cache lifetime": Refresh automatically every X seconds.
    • "Drupal actions": Refresh when an event occurs, like when a user record is created.

    In general:

    • Use an event to refresh the block's cache if you can find an event that closely matches the block content.
    • Use a time period of 3600 (one hour) if a block's content mixes many different content types.
    • Don't use either of these if the block's content needs to be refreshed only on very rare occasion, or when the block itself is edited.

    Some examples

    • A block showing the most recently registered users.
      • Cache setting: Cache once for everything (global), since this block doesn't change per page, user, or role.
      • Refresh setting: This block can be refreshed whenever a user account is created/edited/deleted.
    • A block showing recent classified ads for books in a given category (term).
      • Cache setting: Per page, since this block is specific to a given term.
      • Refresh setting: Drupal event--This block can be refreshed whenever a node of type classified book is created/edited/deleted.
    • A custom block, produced by the core block module or the boxes module.
      • Cache setting: Cache once for everything (global), since the text doesn't change per page.
      • Refresh setting: none--This block changes only when it is directly edited.
    • A menu block, like one at the bottom of the page.
      • Cache setting: Per role, since the block could conceivable change per user permissions.
      • Refresh setting: none--This block changes infrequently, only when the corresponding menu is edited. An admin should manually refresh the block's cache whenever the menu is edited.
    • A block featuring a poll form that users can use to vote.
      • Cache setting: Do not cache, since the form will break.
      • Refresh setting: n/a.

    How to edit a block's cache settings

    1. Bring up the block's edit form.
    2. Near the bottom of the block configuration form you'll find a "Cache settings" section. Choose an appropriate value for "Cache setting".

    Manually refreshing a block's cache

    To manually refresh a block's cache:

    1. Bring up the block's edit form.
    2. Without changing anything, submit the block's settings form.

    Block Cache Alter and Context

    The Context module won't be aware of any settings made through Block Cache Alter. If you use Context to position blocks, you can try the Context blockcache alter sandbox project.

    Alternatives and next steps

    Selective use of Block Cache Alter can give you important new tools to improve performance for registered users.

    Additional approaches to look at include:

    • Use the Views module's built in support for setting the cache options for a particular view display.
    • If you're using the Panels module, use Panels' build in support for setting cache options for individual panels pages and mini-panels.
    • While Views and Panels themselves doesn't give very refined tools for controlling when to flush their caches, you can use Cache actions to achieve refined control over when individual caches are flushed.
    Sep 19 2011
    Sep 19

    When developing a Drupal site using the Features module, it's important that everyone is clear on what kinds of changes are done where.

    Features is used to store a website's configuration in code. Configuration is done on a development site, saved to features, and then commonly brought on to a staging site and finally the live site. Making this work requires careful documentation and education about which site specific types of changes should be made on.

    The workflow for configuration changes often looks like:

    1. Make configuration changes on dev, test, and confirm.
    2. Push these changes to features and commit to a versioning system like Git or SVN.
    3. Update the code on the dev site and revert the updated features on dev.
    4. Update the code on the staging site and revert the updated features on staging.
    5. Test stage, make any needed fixes, retest on stage.
    6. Update the code on the live site and revert the updated features on staging.

    For this system to work, some types of data input must be done on the development site and then pushed to features, but other work needs to be done directly on the live site and from there pulled by the dev site (via database syncing). What goes where?

    The general question to ask when deciding if something should be done on the development site or on live is: is this configuration, or is it content? Configuration goes on dev, so that it can be developed, added to features, pushed to stage for testing, and finally pushed to live. Content editing goes on live. But, like any guideline, there are exceptions, so here's some more detailed explanation.

    Specifics

    Configuration that goes in features and should be done on the dev site

    • Boxes (custom blocks)
    • Contexts
    • Content types
    • Menus and menu items
    • Panels and mini-panels
    • Settings for modules
    • Spaces presets, including:
      • Dashboard configuration
      • Space-specific overrides
    • Taxonomy vocabularies
    • Views

    Content editing that should be done on live

    • All content entry (creation of new nodes--pieces of content)
    • All taxonomy terms
    • Content administration--e.g., mass application of terms to content
    • Custom translations done through the translation interface
    • URL aliases

    Exceptions and special cases

    While most configuration is done on a development site and pushed to features, there are usually exceptions--either types of configuration that don't have Features support or configuration that isn't appropriate to Features.

    Custom block titles

    Overrides of block titles (click the link to edit a block and enter a custom title) cannot currently be saved to features. If these are used, they must be manually repeated on stage and live.

    Specific settings that should be excluded from Features

    There are certain cases where settings should not go into features and should instead be managed through manual configuration on each site. These are settings that either (a) need to be different in different environments or (b) are needed early in the page load process, before Strongarm (the module that handles variables in features) is available to set variables.

    For example, cache settings often need to be set differently on the live site vs. a development environment. So some variables that might be left out of features and instead handled manually might include:

    • block_cache
    • cache
    • cache_lifetime

    Summing up

    Features is a powerful tool but can blur some traditional divisions--between code and configuration or between geek and form submitter. Keeping clear on where to do what is a key part of effective features-based site development. Accomplishing this will require working with your whole team to make sure everyone's clear on protocols, so the configuration changes they make and the content they enter don't disappear into some netherworld of lost checkboxes and incorporeal node bodies.

    Sep 19 2011
    Sep 19

    The Features module for Drupal is a powerful tool for site development, but it's got its share of challenges, as Lullabot's James Sansbury noted in an insightful piece on Features including dos and don'ts of using Features as a deployment tool.

    Features lets you push configuration like content types, fields, blocks, and views from the database into code. At its best, it can provide a smooth and reliable means of building out configuration on a development site and then deploying it to production.

    But it can also have you tearing your hair out, trying to remember who did what, which changes have already been captured, where those supposed "conflicts" came from, and why everything's suddenly showing as "overridden".

    Here are some frequent problems and gotchas in features-based development and some brief tips about how to avoid them.

    1. Lumping disparate functionality together in a single feature. A feature is defined as "a collection of Drupal entities which taken together satisfy a certain use-case." The Features Plumber module aims to help with the task of breaking up poorly constructed features along more logical lines.
    2. Not tracking what changes have been made and need to be committed to features. This is especially true for variables, since what shows up in Strongarm for adding is a mishmash of every setting that's been saved, not in any way a list of what needs to be added to features. A good guideline is to add a variable to a feature only if the variable is (a) edited via a settings form, and (b) set to its non-default value. A good habit to get in is, when you are doing configuration, note the machine name of any variable you change from its default. (You can do this by, e.g., using Firebug to select the relevant form element and looking at its "name" attribute.) Then decide which feature that variable is relevant to and add it when you're regenerating the feature.
    3. Too infrequent commits of features changes. If you're looking at a huge diff with various and sundry changes, the tendency will be to commit without reviewing. Try to regenerate features often and make each new feature or bug fix its own commit.
    4. Making configuration edits on the wrong site (stage or live rather than dev). Doing so means the changes are lost, in that they won't reach features, and leads to overrides on the stage or live sites, meaning that further changes made on dev and pushed to features won't appear on the stage or live sites. Once you're working with features, you want to make all configuration changes captured in Features in a development environment, leaving your staging and live sites in a clean state, free of overrides.
    5. Creating or editing content, taxonomy terms, or other non-features-based data on the wrong site (dev rather than live). These data will be lost, and any references to them that make their way into features - e.g., a setting referencing a taxonomy term by ID - will be broken on stage and live. You need to make content changes on the live site and then pull a version of that database to dev--after all current features-based changes have been exported from the dev site, ensuring that you're not losing anything when you pull from live.
    6. Considering something "fixed" when it seems to work on the dev site. It's not fixed until it's been added to a feature, brought over to the stage site, and successfully tested.
    7. Not using a stage site. Pushing changes straight to live doesn't allow for essential QA testing.
    8. Blindly committing changes. It's important to look over the changes that have been made. Someone else on the site may have been messing around with configuration. If you're committing a change, it's your responsibility to understand it and know why it's been made.
    9. Letting dev get too far behind live. If the dev site isn't frequently synched with the live site's database, it can get too far behind to be an effective development ground. Required data like new taxonomy terms and content won't be available. Excess configuration may be in place on dev. Due to accumulating differences between dev and live, configuration pushed to features may be broken or look very different on live.
    10. Too frequent pushes to live. It's not necessary to be constantly pulling in changes from dev on the live site. Bundling changes and pulling on a schedule can allow for more control and rigour in the process.
    11. Lack of a branching strategy. It's essential to isolate live from major new development. It may be necessary to push minor fixes to live while bigger changes are in development. Branching can make this possible. In this case, it may be necessary to commit only partial features updates to a stable branch, while making larger changes in a development branch.
    12. Not reverting after regenerating. After each time a features is regenerated, its components should be reverted on all sites. Doing so ensures a clean database and helps prevent differences between different sites' states.
    13. Lack of coordination between different developers. It's difficult for more than one person to work on a given feature at a time, since changes may conflict and no one knows what all is being worked on or needs to be committed. A system like tracking who's "got" a particular feature at any given time can help.
    14. Not saving configuration in features-appropriate ways. Some configuration needs to be explicitly saved to a features-aware component. For example, if Spaces is in use, any overrides in a space must be explicitly saved to that space's preset before they will appear in a regenerated feature.
    15. Not regenerating before doing code-level edits. It's often handy to edit features at the code level, but as soon as you do so you'll need to revert them to get the changes recognized. Because of this, before you start to do code-level edits, make sure you have no uncommitted overrides, that would later be lost when you revert.
    16. Relying excessively on UI tools and manual workarounds. For day to day Features-based development, the Drush commands available for automated inter-site file and database syncing and features management are indispensable time savers.
    17. Using a version control system that doesn't work well with the Drush features commands. Because Drush backs up and then deletes feature directories before regenerating them, version control systems like SVN that add subdirectories to every directory are poorly suited. Moving to Git greatly facilitates Drush features management because Git doesn't save its information in each directory.

    If you find this list intimidating, well, I have to say that in some ways it should be. Pushing your configuration to features won't in itself solve your deployment headaches, and in fact will require a fair bit of new work you didn't have to do before. That said, if you go into it with open eyes and ready to roll up your sleeves, Features can be an effective tool and also have numerous advantages, enforcing a new level of rigour in feature planning and site development.

    P.S. This list assumes a typical dev > stage > live setup in which configuration changes are made on a development site, saved to features, and then brought to stage for testing and finally to live. Features can also be developed by individual developers working in their own sandbox sites, an approach that may have some advantages over a shared dev site in that it's easier to frequently update from the live database without affecting other developers' work.

    Jun 11 2010
    Jun 11

    Any way you do it, upgrading an organizational website to a new version of Drupal is a significant task. Before digging in and even before planning out the work, it's worth asking some larger questions: What do we hope to achieve by upgrading? If we were building the site from scratch now, how would we do it? How can we get the most benefit out of upgrading? What major improvements should we consider making in the process?

    Which version to upgrade to?

    An important factor to keep in mind is the current point in the Drupal release cycle. Now, for example, we're at the tail end of the Drupal 6 cycle. Coming in at the end of a cycle has some advantages: Drupal 6 is very stable and almost all important modules have been fully upgraded to Drupal 6. But it also has costs. Drupal 6 is already aging. Even before Drupal 7 was released in January 2011, the focus of innovation haed already moved to this new version. There are important modules that may be available only for Drupal 7--e.g., the media module, which promises to revolutionize image, audio, video, and other media handling and embedding in Drupal.

    Depending on the specific needs of the site, there might be reasons not to go to Drupal 7 yet. Maybe the modules you need for a key area of functionality aren't yet available in Drupal 7. If so, your best bet is to put off upgrading for a bit, until those modules are further along. Or if you really need to upgrade from Drupal 5 now, a key goal in upgrading to Drupal 6 should be to ease future upgrades, by carefully planning toward Drupal 7 solutions. By doing so, you can bring your site more and more tightly in line with the whole point of open source: shared solutions that are collectively developed and maintained, rather than private, custom code.

    Goals

    Some of the more obvious reasons to upgrade a Drupal site include:

    • Get access to new features. Many new features are available in the current Drupal version that weren’t available previously. By upgrading, you can get access to these features "for free".
    • Receive regular updates. Older versions (core and contributed modules) receives less and less attention and upgrading. Some modules you're currently using may no longer be maintained, or if they are maintained they may not be for long.

    But there are also less obvious but potentially more important benefits, including:

    • Streamlined site. By upgrading, you may be able to remove previous customizations, replacing them with standard solutions available in the new version.
    • New approaches. A lot of innovation happens between one Drupal version and the next, and sometimes this innovation opens up whole new vistas for site development. In Drupal 6, such innovation included: vastly improved theme system and stable and proven base themes; the Features system, which allows major components of a sites configuration to be managed in code, facilitating both upgrades and the sharing of features between different sites; and the Pressflow Drupal distribution for high performance sites. Drupal 7 innovation included improved file handling and native support for adding fields to users and taxonomy terms as well as content types.
    • Distributions. In building a new site, you may be able to take advantage of one of the expanding number of Drupal distributions, which provide at least a big leg up towards building a site to your needs.

    Together, this second set of factors can produce a series of very significant benefits.

    • Lower maintenance costs, as there is less custom code to maintain.
    • Better stability, as more of the site's code is publicly tested and maintained.
    • Better performance due to improvements in the code.
    • Better solutions, as the newly available solutions will in many cases be fuller or more feature rich, particularly if previous ones were built in-house.
    • Reduced reliance on a particular expertise, since the site uses more standard solutions and less custom code.
    • Easier upgrade path in future, as the site is now much less reliant on custom code.
    • Streamlined future development that can focus effort on just the specific new needs rather than carrying forward previous customizations.

    In short: if you're taking the time and energy to upgrade, you can and probably should also look at this as an opportunity to make significant improvements, ones that not only may benefit the site in the long term but may also prove to require less work and expense than a straight upgrade of the existing code.

    Approach: Where to Start?

    What’s the best place to start upgrading your site?

    The most obvious answer would be, with what you already have--the code of the current sites, along with their databases and files. Specifically, you could hire a developer or team to dig into the task of upgrading the code that drives the current site, including selecting new versions of the modules that were used and upgrading customizations that may have been made to Drupal core and contrib.

    But, as is often the case, the obvious answer is not necessarily the best.

    In an email thread about upgrading a highly customized site from Drupal 5 to Drupal 6, prominent Drupal contributor Laura Scott at PingV had this to say:

    This may be an approach that will not appeal to you and your team....

    Drupal 6 is in many ways profoundly different from Drupal 5, to the point that trying to reverse engineer D5 hacks to complement D6 may be a quite inefficient way to go. One thing to consider is that existing D6-based solutions may provide ready replacements for some of the D5 hacks. In some ways, you may end up "crossgrading" rather than "upgrading" much of your site.

    So consider wiping the code base entirely (not the database, though) and re-implementing in Drupal 6. If your D5 hacks did not result in alterations to the database structure, you can simply perform a stock D6 upgrade process and then implement through contributed modules and a new or refactored theme the various end results those hacks were trying to achieve. This way you can embrace what D6 offers first before resorting to coding up custom module implementations.

    In other words, start with the goals, the use cases, the desired results, and don't assume that the D5 hack logic will carry over into D6 modules. Your content lives in the database and uploaded files. Treat the rest of your site as disposable — to be used as notes but not necessarily as technical architecture.

    While it might seem unorthodox, Laura's advice is probably a really good starting point in planning a Drupal site upgrade. To get the most out of the process, it's important to keep an open mind in terms of the fine details of appearance or architecture. Your primary questions should be: for each problem, what are the standard, proven, available solutions? Do these meet our goals and use cases? If not fully, what small tweaks would be needed?

    I've written more on this subject in a couple of blog posts:

    Specific

    Theming: Selecting a base theme

    If you already have a custom theme, upgrading it to the new version will be a significant part of your upgrade task. So again, before digging straight into the work, it's worth pausing to look at how the theme is implemented and what improvements you might be able to make.

    Specifically, if the theme is not already based on one of the leading base theme systems, it's probably time to consider adopting one.

    A base theme system is a specialized Drupal theme designed not to be used directly on a site but rather to be a building block for other themes. Generally it itself has little or nothing in the way of styling, but provides all the flexibility and selectors that are needed to build out a site however you might want it to look.

    There are several proven theming systems for Drupal 6, with Zen and Fusion being the leaders in both site installs and documentation. By converting your custom theme to use one of the leading base theme systems, you might be able to reduce your custom theme work to one or two stylesheets and some images, avoiding partially or even completely the need to write and maintain custom templates.

    Leapfrog upgrades

    As Laura Scott suggested, the most effective way of upgrading from a previous version may be to build a new site from scratch and then migrate the old content. This is exactly what prominent Drupal contributor Nate Haug demonstrated in a blog post Direct Drupal 5 to Drupal 7 Migration in 24hrs. While he skipped a major version, the same approach could be taken when upgrading from Drupal 6 to 7.

    Migrating without upgrading site code

    If you've created a new site and need to migrate in your old content and media assets, you could end up with custom scripts that read in your old data and save it to the data structures of your new site. If you have more complex data structures, you might look at the Migrate module, which provides a set of tools to facilitate data migration into Drupal.

    Selecting a distribution

    There is a growing number of Drupal distributions to choose from. Open Outreach, produced by Chocolate Lily, is a Drupal 7 distribution specifically aimed at the needs of the nonprofit sector.

    Questions to look at in evaluating distributions include:

    • Does the distribution match our use case? E.g. for a publication-focused site, there are multiple news or publication distributions.
    • What version of Drupal is it in? You may wish to move to Drupal 7, but many distributions are still available only for Drupal 6.
    • What kinds of hosting and support options are available?

    Keep in mind that, if you have very specific needs, a distribution is likely to get you only part way and you may need to further customize it. Still, if you find a distribution that's a good match for your needs, you might be able to get up and going relatively quickly. From there, you can hive off the migration of your old content as a separate sub-project.

    Features

    Beyond the modules in use and the content, a Drupal site is built from components that are constructed through site configuration--views, content types, fields, panels, image style presets, etc. These components are stored in the site database, a fact that presents challenges when upgrading. If you've opted against the "leapfrog" approach, after completing the process of upgrading and slowly configuring components in the new version you'll face the issue of how to integrate the updated content that's still in the old version.

    In Drupal 6 and 7, the Features module addresses this problem and can be a key tool in upgrading. As an added bonus, it can help immensely in continuing to maintain and develop your site beyond the upgrade.

    Features allows automated conversion of a large amount of site configuration to code. For example, views, content types and their fields, roles, and permissions can all be pulled out of the database into code, while new areas of Features support are being added all the time.

    The process of upgrading then becomes:

    • Upgrade a copy of the site to the new version of Drupal.
    • In the new site, upgrade all the components that Features supports.
    • Create features including these components so that they are exported to code. You can now scrap the copy of the site you upgraded. Everything you need is in the features.
    • Once everything is ready, upgrade the live site again to the new version, and - presto! - switch on the features you built. You end up with the latest version of your content and your upgraded site components happily reunited.

    Features is valuable on a single site, where it can be used to facilitate staging. When a change needs to be made to e.g. a content type or view, the change can be made and tested on a development version of the site, then passed via Features to a QA version of the site for testing. Only when it's fully tested does it then get passed to the live site--all without any repeated manual configuration.

    Features are all the more valuable if you have a need of multiple related sites. While each site has its unique needs, much is shared in common. With features, launching a new site could be as easy as selecting the features to enable, enabling them, and then adding some small custom tweaks. Even better, from there on, most maintenance of the new site would be automatically received as updates from the main site's features.

    Performance

    The biggest performance advance in Drupal 6 actually comes from Drupal 7. It's the Pressflow distribution of Drupal 6, which includes "backports" (adaptations for an earlier version) of major performance improvements that are part of Drupal 7.

    Pressflow is designed to work seamlessly with "reverse proxy" caching software, providing potentially huge performance gains. The reverse proxy software most used with Pressflow is Varnish.

    The gains in performance, stability, and ease of maintenance from a base theme, Features, and Pressflow + Varnish are on their own great incentives to upgrade to a new version of Drupal. And they're only a few of the opportunities you'll identify as you dig into upgrade planning.

    About Drupal Sun

    Drupal Sun is an Evolving Web project. It allows you to:

    • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
    • Facet based on tags, author, or feed
    • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
    • View the entire article text inline, or in the context of the site where it was created

    See the blog post at Evolving Web

    Evolving Web