Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Nov 25 2021
Nov 25

If you're a seasoned Drupal module developer, or even a relatively new one, it's hard not to like the fact that, starting with Drupal 8.7.7, it's possible for a single version of a module to be compatible with multiple versions of Drupal core. Suddenly, maintaining your module became way easier. It's noteworthy enough that the process of making a module work with the Drupal 9 release was incomparably easier than any previous major version upgrade. But beyond that, you could actually maintain a single version for both Drupal 8 and 9, or both Drupal 9 and 10. How great is that?

But - and there always is a but, isn't there? - it's not quite so straightforward. There are some significant tradeoffs to sticking with a single release branch for two major versions of core. I'll look at a couple here - deferred refactoring and missed improvements - and ways to mitigate them.

Pitfall #1: deferred refactoring

One of the modules I've maintained for Drupal 8+ is Features. While not nearly as widely used as in Drupal 7, Features still has a reported install base of around 29,000 in Drupal 8 and 9, placing it somewhere in the top 100 or so modules by usage.

To get Features working in Drupal 9, I did what a lot of Drupal module maintainers did--the minimum.

I focused on what was strictly necessary to make the module work in the new version. Even that was a fair bit of work, spread over many issues and several releases. We had to upgrade the automated tests, fix up the way we generate new feature modules, and lots more.

Sure, at the same time I also took a pass or two through the issue queue and applied fixes where I could, addressing a few outstanding bugs not explicitly required for Drupal 9.

The combined result was a lot of improvements to the module, some of them addressing longstanding issues. What I didn't even begin to do, however, was ask: how would we do it now?

When we started work on the Drupal 8 version, keeping the same code and approaches we had in Drupal 7 wasn't remotely an option. Drupal had been rewritten from the ground up, and contributed modules had to keep pace.

Now, though, sticking with what we did in Drupal 8 definitely is an option, and it's an alluring one.

The bulk of work on the current Features version was completed early in the Drupal 8 cycle. At the risk of understatement, a lot has changed in the meantime. In Drupal core, there was a whole Configuration Management 2.0 initiative that introduced various improvements during the Drupal 8 cycle. In contrib there are some great modules now that hadn't so much as been imagined when we started in on Features.

Then there's what we've learned through using and developing Features itself.

How would I approach Features if I was writing it from scratch today? Very differently.

For starters, I'd make it a lot more modular. There are pieces of Features that duplicate what other modules do, sometimes better than Features does.

But I don't have to do any of that--and so I won't.

Don't get me wrong, I still think Features is a fine module for many uses. It just won't get the attention it might have if stasis wasn't an option.

Pitfall #2: missed improvements going forward

There's a flip side to the fact that a Drupal 8 module could run on Drupal 9. It's that a Drupal 9 module had to still run on Drupal 8.

There are workarounds that in some cases make it possible to take advantage of newer functionality while maintaining backward compatibility. But short of such additional work, a release that's compatible with both Drupal 8 and 9 is stuck in some version of the past. Module developers can't add anything that comes with Drupal 9--any of the improvements in Drupal 9.1 or later minor releases.

At the same time, they can't remove anything that's deprecated in the Drupal 9 cycle and slated to be removed in Drupal 10, because doing so would mean their code wouldn't work in Drupal 8.

And here's the clincher--by the time a lot of developers began making their modules compatible with both Drupal 8 and 9, Drupal 8 was already on its way out. Drupal 8 reached end of life this month. At this point, the ability for a module to work in both Drupal 8 and 9 is kinda academic.

Worse--all these still-maintained Drupal 8 compatible modules may contribute to a mistaken sense that it's safe to stay on Drupal 8. Compatibility with an obsolete core version looks less like a feature than a bug.

Conclusions?

It might sound like I'm tilting against multi-version releases. Calling for discipline--resist the temptation! Cut a new branch as soon as a new major version is available! Refactor to use the best of what's added throughout the new cycle! Remove deprecated code in each Drupal 9 minor release, so the code is already compatible with Drupal 10 when it's released!

But that's not really my point. Instead, I'm thinking there's probably a middle ground. Maybe, for some site and module maintainers, that looks like:

  • After the work of the a new version upgrade, ride for awhile the relative ease of multi-version compatibility.
  • But don't leave it forever. When you're ready, cut a new branch and see what improvements the new core version brings.
  • At the latest, cut the new release branch when the old major version of core is no longer supported. For Drupal 8, that time is now.

In sum? Multi-version compatibility makes the work of maintaining code more manageable. It gives Drupal users quicker access to the modules they know and rely on from past versions. We can appreciate it for what it is, while not losing sight of what it isn't: a magic cure to the problem of keeping software current.

Nov 23 2021
Nov 23

Many Drupal insiders recognize that data on Drupal usage that's collected and displayed on Drupal.org have their limitations. Since 2018 there's been a proposed Drupal core telemetry initiative to expand and improve the data collected. Meantime, though, the usage stats are widely referred to and cited. So it's worth spending a bit of time with them. What do they actually capture?

Before digging in, a disclaimer. Circa early 2007 I wrote the first rough prototypes of what became - with a lot of great work from Earl Miles, Derek Wright, and others - the client and server components of the Drupal core Update module. But I had little or nothing to do with any of that further work and I haven't done more than glance over the current Update module code, let alone the various pieces that run on Drupal.org to receive, log, and tally the data. So my notes here are more inference than studied conclusions.

To start off, a brief step back to look at where the stats come from.

How the stats are calculated

The Drupal.org data on project usage look simple enough at first glance, but like many statistics they have some non-obvious complexity and nuance.

Take the figures for Drupal core. There are date-specific tallies for how many sites report using Drupal core, broken down by major and minor release version and - if you scroll down the page - by individual release. The explanatory text states, "the figures show the number of sites that reported they are using a given version of the project."

Where to the data come from? A Drupal install using the "standard" install profile default installs a core module, Update, that checks periodically - by default, once daily - for available updates. When it does so, the instance sends data about itself to the Drupal servers, including data on all the projects it has installed. A "project" here is in the Drupal.org sense of something that's downloaded and installed as a package. It could be Drupal core, including all the core modules and themes, or a contributed project like Admin Toolbar, including submodules like Admin Toolbar Search.

To complicate things, the mechanism that triggers sending data to drupal.org - cron - relies by default on page visits, so installs with very low traffic may be missing from the data if they receive no page views at all in a given week. This fact may account for periodic dips in usage stats over periods that include holiday seasons, when certain kinds of sites receive fewer visits. Similarly, some Drupal sites won't have the Update module installed and so won't show up either.

On the flip side, it's always possible that some of what show up in the stats aren't Drupal installs at all but result from some attempt to game the system. Glancing over the graphs of Drupal core usage, it's hard not to notice some questionable spikes in the data, maybe most notably two in March and April of 2018 that showed supposed Drupal 7 usage jumps of nearly twenty percent from one week to the next, followed immediately by equally steep declines. So, yeah, some oddities, and over the years there have been discussions about spikes in the data and how to address them.

Once the installs have "called home," code on Drupal.org analyzes the logged data returned by all those Drupal installs in a given week, tallies it up, and presents the results in those handy tabular lists.

So does that mean for example that as of October 24, 2021 there were (doing some quick sums of the 9.0.x, 9.1.x, 9.2.x, 9.3.x, and 9.4.x numbers) at least 155,449 Drupal 9 sites?

The answer is, that depends on what you mean by "sites."

Installs vs. sites

A key thing to keep in mind when looking at the Drupal.org usage data is that they report Drupal installs--and these can be spun up for many different reasons. Some for sure are production websites. But other uses include:

  • Development environments. It's common for Drupal developers to spin up many installs they use purely for development purposes. Various dev tools have this kind of workflow baked in. For example, if you're using Pantheon to manage a Drupal site, you might spin up a different "multidev" environment for each new feature or bug you work on. If you have dozens of issues in a release, this might mean dozens of installs for each production website.
  • Code and configuration staging. Aside from one-off environments created for a particular issue, it's also common practice to use several permanent environments for staging code or configuration stages. Looking again to Pantheon, their platform features built-in support for Dev, Test, and Live environments. In other words, you might have two (or more) additional installs used for staging code and configuration changes on a single Drupal "site."
  • Evaluation. Some installs are created just to evaluate or try out Drupal or one or more of its extensions. For example, the simplytest.me service can be used to spin up short-lived Drupal installs for demo or testing purposes.
  • Continuous integration and automated testing. Drupal core and many Drupal extensions feature continuous integration including automated tests that are run on each proposed change to the software. There are many thousands of these issues open at any given time. Since tests are run for every new iteration of a proposed change classed as needing review, a given active issue can trigger multiple automated test runs in a given week, each involving a new Drupal install.

The main point: production websites are only a subset of Drupal installs.

Development installs

A sign that many reported installs are probably driven by automated testing is a data quirk that shows up whenever a new core minor version branch is opened up. For example, here's a chronological excerpt of the usage data from the weeks around the 9.2.x dev release.

Week 9.0.x 9.1.x 9.2.x September 27, 2020 24,405 29,493 0 October 4, 2020 23,753 26,411 0 October 11, 2020 25,506 28,148 345 October 18, 2020 24,558 10,723 19,926 October 25, 2020 26,288 16,150 18,461

What immediately stands out is the leap in usage numbers for 9.2.x. By the end of the week starting October 18, 2020 - just eight days after the 9.2.x dev release was cut - there were suddenly close to 20,000 installs reportedly running the release. Meanwhile, the usage of 9.1.x dropped by nearly as much.

Really? Did tens of thousands of site developers decide to switch their sites to the newest dev branch, practically overnight?

Less immediately obvious but equally striking are the high numbers for the two development Drupal 9 release branches compared to the stable one. During this period, combined usage numbers for the then-unsupported 9.1.x and 9.2.x development releases were around twenty percent higher than usage numbers for the stable 9.0 branch.

Huh? Were the majority of Drupal 9 sites really running the completely unsupported cutting edge releases? Where did these "sites" come from?

The answer is probably in the October 6, 2020 pre-announcement of the 9.1.0-alpha1 release, which noted that "all outstanding issues filed against 9.1.x" were to be "automatically migrated to 9.2.x"--meaning they'd now be tested against the 9.2.x version rather than 9.1.x.

And voila, all those 9.2.x installs appeared.

After shooting up to nearly 20,000 installs in eight days, the 9.2.x usage figures remained relatively static in the subsequent months, settling in for the most part at from 20,000-25,000. This pattern is consistent with usage driven mostly by automated tests.

Many higher-end Drupal sites feature their own flavour of continuous integration, with their own rules about which version or versions of core to test against--and their own potential footprint of test installs.

In short, many or most of these reported development-version installs are probably ephemeral software instances installed and then taken down by infrastructure scripts. "Sites" only in the most abstract sense. This probability means we should take care when basing conclusions on raw usage data, particularly early in a major version cycle.

Drupal 7 vs. 8+

Automated test installs, development environments, and the rest aren't new, but several of these types of install are likely to be on the increase in Drupal 8+.

In previous versions it was possible to stage configuration between multiple versions of the same site, but Drupal 8 was the first version to include explicit support for this workflow. "Reliable configuration management for safe and straightforward deployment of changes between environments" was a key selling point when Drupal 8 was announced. All things equal, we can expect a lot more usage of staging in Drupal 8+ than in earlier versions--and therefore a lot more installs per production site.

Automated testing too may have a bigger footprint in Drupal 8+ than it did in Drupal 7. It's widely accepted that Drupal 8+ has proportionally more high end, enterprise level sites, where dedicated development teams and continuous integration and automated testing are much more likely. Further, in Drupal 8+ the introduction of minor release cycles (for example, 9.0, 9.1) means at any given time there are more current or upcoming versions to test against and hence more potential test installs.

All of these factors mean among other things that the falling graph of Drupal core usage data probably understates the case if we're looking for data on production websites as opposed to installs.

Core vs. contrib

For core usage, there are other data sources available, like those from W3Techs, that for some questions may be a good corollary or alternative to drupal.org stats.

For contributed projects, there don't tend to be other sources. But Drupal site builders often want to know which are the most-used projects for a certain area of functionality and there, sinc the interest is in relative rank, whether the data do or don't include non-production sites is mostly irrelevant.

Refining the data

Is there anything we can do to exclude non-production install types from the core usage data and so more closely model numbers of production websites?

There is one easy adjustment we can make: filter out reported installs running development releases for future versions. These releases are, by definition, not suitable for production sites. For example, since at time of writing the latest stable release branch is 9.2.x, this would mean excluding data for the 9.3.x and 9.4.x development releases.

As of the October 31 data for Drupal 9, that adjustment brings us down from a gross figure of 155,449 reported installs to 130,772, a 16% reduction.

But that's probably as far as we can go without resorting to conjecture.

Could we reliably filter out installs done for the purpose of developing a module or theme? There are some relevant data sets, like the drupal.org commit log that lists all commits pushed to drupal.org core and contrib repositories, and shows up to hundreds of commits per day, but even with easy access to those data extrapolating from them to assume a number of development installs used in a given week would be mostly guesswork.

Similarly, it would be useful to be able to filter out duplicate environments of a given site. Every Drupal 8 or 9 site has a unique setting - system.site.uuid - that's shared across all environments and if it was possible to filter Drupal usage stats by unique site ID, doing so would remove the duplicate environments from the data. But the site's UUID doesn't figure in the key sent to Drupal.org to identify the particular install. Without that identifier, it's hard to see how we could distinguish duplicate environments--except, again, by making some fairly arbitrary assumptions. For example, as well as guessing how many additional environments exist per site, we'd also have to assume how likely it is that each environment is used or visited in a given week--which gets us solidly into the realm of speculation.

Conversely, is there anything we can do to include data for sites that didn't "call home" in a given week? Because data are tracked per install, drupal.org servers probably do have the needed data. For example, that could look like:

  • For sites that (a) haven't sent data in a given week, (b) have reported in non-consecutive weeks at least once in the past year, and (c) did report in one of the prior three weeks, use the most recent data for that site.

That approach would include a few sites past the date they go offline and could report non-current data for some sites, but overall could be useful for filling in missing data.

Summing up

  • Drupal usage data reflect the number of reporting installs, not production sites.
  • Though it still leaves various types of non-production installs such as additional site environments, the best adjustment that can be easily made is removing stats for development releases for future versions.
  • Some sites are missing from the data because they don't have the Update module installed or haven't "called home" in a given week.
  • The ratio of production sites to reported installs is likely lower for Drupal 8+ than for Drupal 7, meaning the falling graph of total core usage would likely be steeper if it captured production sites rather than all installs.

All of the above doesn't in any way mean the usage figures on drupal.org are without value. Just that they're like most statistics--best used with an eye to limitations and context.

Nov 16 2021
Nov 16

As Drupal 8 entered its last months of life, I found myself reflecting. What did version 8 mean for the project? What was lost? What was gained? What were the roads not taken? To try to puzzle out at least some preliminary answers, I read up on what others had shared and spent some time with various sources--Drupal core and contrib code bases, usage statistics, old conference presentations, release announcements and more. I started the exercise motivated by personal interest but now that I've wrapped it up I figure I may as well share my observations in case they're of interest to others.

There've been insightful one-off posts addressing such questions. Often these are reflections on problems with Drupal 8, sometimes coming from people partly or completely moving on from the project. One such was "Lessons Learned From The Stall of Drupal" by Matt Farina, which reads like an exit survey turned in by an exceptionally perceptive and frank respondent.

I wanted to take a deeper pass.

Version 8 of Drupal brought with it by far the most sweeping changes in the twenty year history of the software project. The broad outlines of the shift were discernible as early as 2011 when, in his DrupalCon Chicago keynote address, project lead/"Benevolent Dictator for Life" Dries Buytaert said that, based on consulting some twenty of the largest companies and organizations using Drupal about their pain points, he'd derived core "initiatives" for Drupal 8 - among them the Configuration Management Initiative (CMI) - designed to fix "the pains of large organizations adopting Drupal." Drupal would be fundamentally rewritten to suit the priorities and requirements of enterprise-level users.

Now Drupal 8 has reached end of life (EOL). Not only that, but unlike for prior versions there are no plans to offer commercial security support for Drupal 8. The contrast with Drupal 7, the previous major version, couldn't be more striking. Drupal 7 will be officially supported for a full year after Drupal 8's EOL, while "Vendor Extended Support" for Drupal 7 will be available for an additional three years beyond that, if not longer. When Drupal 8 reached EOL, there were still many more sites out there running Drupal 7 than Drupal 8 and 9 put together. For added symbolic weight, one of those sites still on Drupal 7 is drupal.org itself.

For all the announcements and initiatives that went into Drupal 8, there's been notably little in the way of qualitative or quantitative evaluation, at least in public. Various aims were announced and strategies pursued. Were the objectives achieved? If so, did they have the desired effects?

I was one of several people who raised concerns and flagged issues early on in the Drupal 8 cycle. One motivation in a retrospective was to see: to what extent were our various concerns validated by the process as it played out? To what extent, conversely, was the project able to avoid such issues, or to successfully address them when they arose?

But I had more immediate concerns as well. I've been "doing Drupal" for over 18 years. In many ways I've loved being part of the project. It's shaped my career, enabled me to work part time at home while my kids were young, allowed me to collaborate with organizations whose missions I deeply believe in, and brought me into contact with many interesting people. Early on I made contributions to core systems, some of which - the theme region system, the JavaScript behaviours system - are still around in some form. Over the years I've written or contributed to an embarrassingly long list of modules--embarrassing because the sheer number suggests a tendency to dabble with a problem before moving on to a new interest. But from the start there have been aspects of the project that didn't sit well with me. Now I'm in my fifties. Like many others before me, I'm taking a long look at whether this is still a project I want to be part of. Debriefing Drupal 8 is part of asking, is it time at last to move on? And if not, what exactly do I still want to do in and with the project?

As well as all that, I believe in the value of critique. A project is stronger for having members asking questions and pursuing diverse, even - or especially - unorthodox directions.

Beyond esoteric concerns of a particular software project, the story of Drupal 8 brings in issues of wider concern: voluntarism, conditions of employment, authority, technology and society, corporate influence, the potentials and pitfalls of collective endeavour.

A final motivation was pure curiosity. I found myself following up arcane tangents, exploring puzzling quirks I'd long meant to look into but had never found the time, pursuing answers that brought only further questions, surfacing results that at times felt novel and intriguing, even startling.

In a series of upcoming posts I'll share what I found. These are, of course, just my own reflections. Others will have different experiences, observations, questions, perspectives, insights.

You can't say very much, critical or otherwise, about a project without mentioning at least some initiatives, participants, or groups and things they've done or said. Just in case, I'll say in advance that my interest is the project as a whole; none of what I'm sharing is intended as personal criticism of any participant or contributor. If anyone has concerns or other feedback I'm very open to hearing about it and people may contact me via my drupal.org contact form (requires being logged in to drupal.org).

As a starting place, I'll look at some curious quirks in Drupal usage stats and implications for understanding Drupal 8+ adoption.

Aug 27 2018
Aug 27

At the Drutopia project, one of our big focuses has been improvements to configuration management in Drupal 8. In this series, I'll be covering our work to date along with related efforts and contributions.

Drutopia is a platform cooperative initiative, building out cooperatively owned and hosted Drupal distributions. In our 2016 white paper, we recognized that the Configuration Management Initiative (CMI) in Drupal 8 "produced a lot of improvements in configuration handling" while noting that these "mainly addressed the use case of 'staging' configuration from one version of a site to another, a site-building technique that lower budget sites often don’t have time or money for." We committed to focus on "the free software use case left out of Drupal core: reusable configuration that can be shared across multiple sites". For background, see Drupal 8 configuration management: what about small sites and distributions? and sections on Drupal 8, corporate influence, and the CMI in this interview.

There's a current initiative to improve configuration management in Drupal core. Dubbed "CMI 2.0", the effort comes out of a similar conclusion that limitations and missing use cases in configuration management are a major barrier to Drupal 8 adoption; see Angie Byron's post proposing the initiative.

In the past three years, we at Drutopia have contributed to a growing collection of Drupal plugins that together address some of the tricky problems involved in managing shared configuration. As well as in kind contributions by Chocolate Lily, some of our work was sponsored by Agaric and the National Institute for Children's Health Quality (NICHQ) to meet their needs for an in-house platform of community sites.

Just what do we mean by managing shared configuration?

Say I have a site built on a Drupal distribution that's for community organizing. I installed the site a month ago and got groups-related configuration such as a group type. Then I made some modifications of my own. I've just downloaded a new release of the distribution, including enhancements to the groups-related configuration. How can I update my site so that I have all the latest changes from the distribution--while still retaining any customizations I made? That's the key question we've tried to tackle.

A more abstract way of putting the problem is: how can we provide packages of shared configuration in a way that lets site administrators both customize their sites and merge in configuration updates?

This series will cover distinct aspects of the problem of managing shared configuration packages and, along the way, highlight specific solutions we at Drutopia have sketched in. Our efforts are very much works in progress. We're not sure we've even got all the problems right, let alone fully addressed them ;) But have we made progress? Yes, we have. By sharing it here, we hope to raise the profile of these problems and solutions and invite further perspectives and contributions.

Here are the aspects we'll cover, along with associated Drupal modules:

  1. Configuration Providers (Configuration Provider)
  2. Configuration Snapshots (Configuration Snapshot)
  3. Respecting Customizations (Configuration Merge)
  4. Configuration Alters (Config Actions, Config Actions Provider)
  5. Updating from Extensions (Configuration Distro, Configuration Synchronizer, Config Filter)
  6. Packaging Configuration (Features)
  7. Base Configuration (Configuration Share)
  8. Summary and Future Directions

As we go, we'll briefly introduce some relevant concepts in Drupal 8 development as they come up, while pointing to sources of further reading.

First up, configuration providers!

Updating configuration from extensions

When you install a Drupal extension (a module, theme, or install profile), you'll often get configuration installed on your site. That's because the extension provides that configuration. Technically, the configuration is sitting in files in one of two directories within the extension.

Files that are in config/install are installed when the extension is installed. There's also optional configuration: files in config/optional that are installed when certain conditions are met. For details about how this works, see the documentation on drupal.org.

This system works great for the use case Drupal core supports: one-time installation of configuration provided by extensions. But when it comes to updating extension-provided configuration, we run into problems.

In order to run configuration updates, we need to know what updates are available. For extensions, this comes down to the question: what configuration would be provided if the currently installed modules were reinstalled?

Because core is tightly tied to the model of one-time installation of extension-provided configuration, relevant code is not available in a form we can use. A key problem is with optional configuration. In core, the logic to determine what optional configuration will be installed is embedded in ConfigInstaller::installOptionalConfig(). But this method doesn't just answer the question of what will be installed--it also goes ahead and installs it. Ouch.

Beyond optional configuration, a second issue is that core's code is limited to the two use cases covered by config/install and config/optional. In practice, though, contributed modules define configuration-providing directories of their own to address additional use cases. Examples of such are the Configuration Share module, which provides a config/share directory, and the Config Actions module, which enables programmatic alteration of configuration through actions provided at config/actions. (More on both of these modules later in this series.)

So, as one part of the puzzle of how to update configuration from modules and themes, we need a solution that can support configuration providers, both those in core and any that contributed modules might provide.

Configuration Provider

The Drutopia-supported Configuration Provider module addresses this use case by providing a ConfigProvider plugin type.

Drupal 8 makes it possible for modules to define their own types of plugins using the plugin API. For example, an image effect like cropping or rotating can be provided as an ImageEffect plugin.

A plugin of the type ConfigProvider satisfies a given use case for providing configuration from an extension. Configuration Provider ships with plugins for core's two use cases, config/install and config/optional.

Any module can meet additional use cases not covered in core by providing a new plugin. Two such examples are configuration provider plugins provided by the Config Actions Provider and Configuration Share modules.

The nitty gritty

For those interested in the technical details, here you go!

Drupal 8 modules can provide services: objects that do work and allow other code such as modules to access the functionality they provide. Services are one of the main approaches Drupal gets by incorporating elements of a framework called Symfony. For more background, see the relevant Symfony and drupal.org documentation. One type of functionality that can be exposed in a service is a storage for configuration. For example, Drupal core exposes services for the site's 'active' configuration storage and for the 'sync' storage that's used to synchronize configuration between different environments.

Drupal's configuration storage API allows configuration to be stored in multiple ways, like in the database or in files. Each configuration storage service can specify what type of storage is used. The way this is done is that Drupal core provides a storage interface. An interface is like a blueprint or contract for what a particular class (a code object) needs to do. Interfaces make it possible to have a standard answer to what a class needs to do, while leaving the question of how to do that open.

A particular class might have any number of methods, functions that are called to do something. A particular method in a class has what's called visibility, which determines where the method can be called from. Many may be private (accessible only from the class itself) or protected (accessible from the class or any of its parents or children). But the methods defined in an interface are public, meaning they can be called from anywhere. This makes sense. The whole point of an interface is to define what a class can reliably be called upon to do. So at its most basic, an interface defines a specific set of public methods a class needs to have. To implement that interface, a class just needs to have all of those methods. For more on interfaces, see the PHP documentation and a relevant section of Drupal's coding standards.

In core, there are multiple classes that implement the StorageInterface. The active configuration service uses database storage by default, while the sync service uses files.

Okay, back to the example at hand. In Configuration Provider we don't need to permanently store the configuration, so we don't need a database or file storage. Instead, through our own StorageInterface implementation, we provide a custom storage type: InMemoryStorage. This does pretty much what its name suggests: hold configuration in memory. Then, using that storage type, we provide a storage service, config_provider.storage. This is used to store configuration during the process of determining what's available on the site.

In a second service, config_provider.collector, we provide a public method ::addInstallableConfig() that can be used to populate the config_provider.storage storage with configuration that would be installed if all currently installed extensions (modules, themes, and the install profile) were reinstalled. Here's a sample usage:

$collector = \Drupal::service('config_provider.collector');
// Add installable configuration to the `config_provider.storage` storage.
$config = $collector->addInstallableConfig();
// Get the service for the storage that contains the installable configuration items.
$provider_storage = \Drupal::service('config_provider.storage');
// List available items.
$item_names = $provider_storage->listAll();

This approach can be used as part of an update workflow, to determine what updates are available from installed extensions. Spoiler alert: that's what's done in Configuration Synchronizer, which we won't cover until much later in this series.

Okay, confession time: to get around a thorny issue, we've resorted to what can only be called a messy workaround.

Because we're making it possible for contributed modules to define their own logic for providing configuration, we need to allow them to act when extension configuration is installed. When Drupal core is installing configuration, we need to also call our provider types. That's the domain of core's config.installer service. So, to add our bit, we need to alter core's service.

On the plus side, the ability to tweak and refine the way a service works is built into services--in fact, that's a big part of why Symfony and by extension Drupal provides services in the first place.

When altering the default way an existing service works, there's a "right" way to do it. That's to use a pattern called "decoration"--see the relevant Symfony documentation. By decorating a service, we add just the specific customization we need, while leaving the way open for other modules to also make their own tweaks. For more on decorators, see the blog posts Drupal 8: Service Decorators and Get a decorator for your Drupal home.

But whether or not decorating a service works depends on what we were talking about before: the interface that backs a service. As noted above, an interface defines a set of public methods a class needs to have. When you're decorating a service, it's just the interface-defined public methods you've got to work with.

And there's the rub. In our case, the method we need to alter is ConfigInstaller::getConfigToCreate()--and it's not public but protected. Again, this situation probably reflects the use case core was designed to meet. Since the assumption was that extension-provided config is only installed once, there was no need to make the ConfigInstaller::getConfigToCreate() method public.

Since decoration is out, we need to resort to a more heavy-handed approach: a service alter.

In our alter, we swap in a custom class that extends the core ConfigInstaller class to provide our own ::getConfigToCreate() method. In it, we pass the request to all available configuration provider plugins. We also pass configuration data provided by previous provider plugins, allowing a plugin to accomplish tasks like (a) fulfill configuration entity dependency requirements on demand (as in Configuration Share) or (b) modify configuration prior to it being installed (as in Config Actions).

Altering the service in this way is heavy-handed because it can only be done once on a given install. If another module used the same trick, one of the alters would fail, leading to breakage. Hence the appropriate warning in the drupal.org documentation:

you should use caution when making use of this feature.

Potential enhancements

Configuration Provider kinda fills the need. But are there ways it could be improved? For sure.

Configuration Provider introduces plugins that allow altering of configuration storages. That's more than a little reminiscent of what Config Filter does in a much more general way. So one option would be switch to using Config Filter plugins instead.

But Fabian Bircher, the intrepid developer of Config Filter and the related Configuration Split module, is working up a new and improved architecture for a "configuration transformer API" in this core issue. The basic idea: instead of plugins, pass and transform configuration using Drupal's event and event subscriber pattern. So, likely, whether in Drupal core or in contrib, a future iteration of Configuration Provider should wait until that proposed API is ready to roll.

Or the need could be met directly in core through a switch to an event-driven architecture when marshalling extension-provided configuration to be installed--or updated.

Related core issue

Add a Config Filter storage and plugins for extension-provided configuration.

Next up

Stay tuned for the next post in this series: Configuration Snapshots.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web