Feeds

Apr 09 2019
Apr 09

CMI 2.0 at Drupalcon Seattle 2019

Fabian Bircher

9 Apr 2019

0 Comments

Fabian Bircher

9 Apr 2019

0 Comments

Fabian Bircher, 9 Apr 2019 - 0 Comments

Help us answer difficult questions about configuration management for Drupal core.

Tomorrow Wednesday at 13:45 (Pacific Daylight Time) in room 6C we have the CMI 2.0 updates session at the Drupalcon.

Together with Mike Potter, we will briefly present the work that has been done so far and lay out the road map for what is to come. The session will be recorded and the slides are attached to this post (subject to change).

I will not go more into details of the session in this post but I would like to mention a challenge for which we need to find a solution and I would like everyone to invite to join the discussion on drupal.org #3046903 or here in the comments.

The challenge is how to deal with update hooks that change configuration which run only in certain environments and thus make the configuration deployment more difficult.

CMI 2.0 DC Seattle 2019.pdf (1.28 MB)Download
Mar 06 2019
Mar 06

Help us set the stage for future configuration management improvements in core.

The configuration management initiative 2.0 needs your help.

I will be presenting updates of CMI 2.0 at the upcoming Drupalcon Seattle together with Mike Potter.

Some of the highlights of the CMI 2.0 road map for inclusion in Drupal core are an improved version of the concept of Config Filter (in the form of the config storage transformation api) and a simplified version of Config Split (in the form of config environments).
Unfortunately those big things will not make it into 8.7, but we could lay the ground work for 8.8.

But the deadline for some patches for Drupal 8.7 is this Friday!

It would be great if you could help us get the following issues to RTBC and committed:

#3036193: Add ExportStorageFactory to allow config export in third party tools
This would allow us to add a service to core that drush, drupal console and other tools and modules can use to export configuration. If this lands in 8.7 we will be able to patch Drush and Drupal Console between 8.7 and 8.8 and make improvements to configuration management such as adding a Config Environment module to core without then patching the cli tools again afterwards.

#3016429: Add a config storage copy utility trait
This adds a new utility trait that would make dealing with configuration storages easier. Currently there are a bunch of modules that implement this logic by themselves and not all of them do it correctly. This lead to bugs in Drush and Drupal Console and even Drupal core has a bug (which is fixed by this issue).

Thanks already in advance.

PS: If you are interested in more CMI 2.0 issues to review or work on check the CMI 2.0 candidate issue tag.

Apr 09 2018
Apr 09

A new workflow for distribution configuration management.

Drupal core workflow limitations

It is not a secret that the configuration management in Drupal 8 core was made for only one specific use case: move configuration from one environment to another for the same site. Everything else was left for contrib to find solutions for. Config Installer is necessary to set up a site starting from existing configuration and Config Split is needed to have environment specific configuration that go beyond simple configuration overrides, like development modules enabled only locally. But the whole workflow still assumes that developers - most often in the same team - have control over the environments and deployment of the site.

Distributions are not covered

Distributions are very different. When maintaining a distribution one doesn't develop a specific site but rather the template to build different sites with. Deploying configuration as we do it for a single site does not have any meaning because there is no one production site to deploy to. Each site based off a distribution is its own site with its own site uuid and its own possible customisations and possibly its own development and production environments. At the same time as a consumer of a distribution - a site owner of a site based off a distribution - one wants to update the distributions features while keeping the customisations.

Update hooks?

Different distributions handle this in different ways. A first approach to solve this issue is treating it the same way you would a schema update and change the configuration on a site with an update hook. The problem with this is that update hooks are meant to fix the database to be able to run with the version of code that is in use on the site. The first thing one needs to do after updating the code base is to run the update hooks to realign the database. Configuration management has different needs and when updating the configuration in an update hooks means that one has to be careful to check that the configuration is in the expected state. And the only recourse is to log and make sure the user is informed what manual actions he needs to do to update the configuration himself. As there is no place to involve the user with a choice while update hooks are run.

We propose a new kind of workflow, in essence it allows developers of a distribution to say how the "distributions configuration" is supposed to be updated and it allows site owners of a site based on a distribution to treat the distribution maintainers as a developer on their team. At the same time this does not interfere with the configuration management workflow for staging and deploying configuration to the production site.

The primary function of Config Distro is to allow updating a distributions configuration in a new workflow and give the site owners more control and insight on how the configuration of their site changes. It is a new way to imagine the configuration management system in a way that lets sites own their configuration but allowing distributions to provide updated configuration for existing sites.

A new dimension

In a blog post one year ago I talked about different dimensions of configuration management;

  • vertical: moving configuration between environments but for the same site
  • horizontal: moving configuration between different sites. for example for distribution updates.

Config Distro is a UI and a drush command for that workflow. It is intended to be used by site builders who base their site on a distribution and want to import configuration updates from the updated distribution. The UI is essentially a parallel to configuration import screen for the workflow supported by core. But instead of importing the configuration from the files used to move/deploy configuration between environments you import from a special configuration storage which starts out with the sites active configuration and has the distributions updates applied to it. This means you do not have problems with mismatched site uuids or things being removed that you added and were not part of the distribution. And updates can apply (or not) to translations even if the distribution did not have languages.

Where do updates come from

If you only install Config Distro, the import screen eternally shows that there is nothing to import. This is because the module only provides the framework for this to work, ie the UI and the drush command. The module is not opinionated on how or what should be updated. All the complexity can be addressed by a distribution maintainer with the help of a ConfigFilter plugin (the same way Config Split and Config Ignore work). One such module is Config Sync. All the complexity of finding out what what configuration a module has originally shipped with, what it does now and whether a user has changed the originally installed configuration is left to Config Sync and its dependencies.

Choice

Just like Config Ignore allows you to opt out of some of the configuration management workflows, Config Distro has a Config Distro Ignore module that lets you retain certain configuration from being changed when you hit the "import" button. The "Retain configuration" button is available right next to the "View differences" button.

Config Distro overview

Clicking it leads to a form that lets you choose to retain the configuration permanently or only for this specific update. It also allows you to ignore a update for a specific language.

Config Distro detail

Example

In our team we set up a site based on a distribution. We added our own login module for our single sign on and added a few webforms. Now there is a new version of the distribution with some new features and I would like upgrade the site to using the new features. I am tasked with updating the site, here is what I do:

  • I update the code of the distribution by specifying the new version in composer.json and do a composer update to get the updated code*
  • I run the database updates drush updb to align the database with the code*
  • I go to the distro update screen provided by Config Distro
    • This screen looks very familar, it looks the same way as when I import configuration changes my team mates made after I get their code with git pull.
    • I see a new module is going to be installed and a few views and settings from the distribution are updated.
    • Our webforms are not going to be removed and our custom modules not uninstalled.
  • I click "import all"
  • Now my site has the updated code and configuration so I export the configuration for deployment: drush cex and commit*
  • My colleges get the update the same way they get normal changes that happen during development*:
    • git pull get the code
    • composer install get the external code (vendor, contrib, etc)
    • drush updb align the database with the new code base.
    • drush cim import the sync configuration.

* This is the same procedure as for any module update on any site.
Distributions may add a "Auto upgrade" setting and then import the Config Distro in a post_update_hook to bypass the manual step required of site administrators upgrading the distribution.

Conclusion

Config Distro provides a new workflow for updating distributions with a familiar feel. It recognizes that update hooks are not an adequate solution for updating a sites configuration when a site owns the configuration and site administrators may have changed the initially provided configuration.
It allows developers of distributions to alter the sites configuration through ConfigFilter plugins and it gives site administrators a choice of what to import.
Config Distro is just the framework for extending cores configuration management to allow managing configuration changes to get into the site from a third party such as the distribution maintainers. It does not interfere with the traditional workflow and importing the configuration updates from a distribution should be seen as any other configuration change such as adding a new view or changing permissions: You go to an admin page and you click some things in a form, then you export the configuration and deploy it.

Future - CMI 2.0

While it already works, Config Distro is still in an alpha state. Config Distro is part of a larger effort to improve the configuration management in Drupal. You can find further information and participate in the discussion on drupal.org

Oct 13 2017
Oct 13

Config Split treats all the cli the same.

Drupal 8.4 and its upgrade to Symfony 3 has made the compatibility of the global Drush 8 a bit more challenging. Drush 9 works with Drupal 8.4 but it is not stable yet and the format of how third party Drush commands are made has changed significantly.

While Drush 9 comes with a command that helps porting Drush 8 commands you will still end up maintaining very similar code in two places one with calls to drush_confirm('...') and one with $this->io()->confirm('...'). If you decide to also provide your commands for Drupal console you now have three times the burden.

Because we tried to provide the commands for Config Split for both Drush and Drupal console early on we faced this problem already more than a year ago. And now it has paid off because porting the commands to Drush 9 was very quick.

The solution is actually really simple and brings the added benefit of being able to test the business logic of the commands in the absence of Drush or Drupal console. It is all about separating the command discovery from the command logic. Drush 8, 9 and Drupal console all have a bit different ways to discover and invoke commands, but the business logic you want to implement is the same so all we have to do is to extract a common "interface" our custom service can implement and then make the command definitions wrap that and keep things DRY.

The CliService

Config Split defines a config_split.cli service with the class ConfigSplitCliService with all its dependencies injected. It has the methods \Drupal\config_split\ConfigSplitCliService::ioExport and \Drupal\config_split\ConfigSplitCliService::ioImport that implement all the commands logic and delegate the actual importing and exporting to specific methods.

The method signature for both the export and import method are more or less the same: CliService::ioMethod($arguments, $io, callable $t).

  • $arguments: The arguments passed to the command.
  • $io: This is an object that interacts with the command line, in Drush 9 and Drupal console this comes from the Symfony console component, for Drush 8 we created a custom wrapper around drush_confirm and drush_log called ConfigSplitDrush8Io.
  • $t: This is essentially a t function akin to how Drupal translates strings. Because Drupal console translates things differently we had to be a bit creative with that by adding a t method to the command.

Commands wrap the service

The Drush 8 command is essentially:

<?php
function drush_config_split_export($split = NULL) {
 
// Make the magic happen.
 
\Drupal::service('config_split.cli')->ioExport($split, new ConfigSplitDrush8Io(), 'dt');
}
?>

For Drush 9 we can use dependency injection and the Drush 9 command becomes essentially:

<?php
class ConfigSplitCommands extends DrushCommands {
  public function
splitExport($split = NULL) {
   
$this->cliService->ioExport($split, $this->io(), 'dt');
  }
}
?>

And very similar the Drupal console command:

<?php
class ExportCommand extends SplitCommandBase {
  protected function
execute(InputInterface $input, OutputInterface $output) {
   
$this->setupIo($input, $output);
   
// Make the magic happen.
   
$this->cliService->ioExport($input->getOption('split'), $this->getIo(), [$this, 't']);
  }
}
?>

Testing

The ConfigSplitCliServiceTest is a KernelTest which asserts that the export works as expected by exporting to a virtual file system. The test coverage is not 100% (patches welcome) but the most important aspects for the complete and conditional splitting (blacklist/graylist) is thoroughly tested. There are no limitations on what or how you can test your CliService since it is self contained in your module and does not depend on Drush or the Drupal console. For example one could write a unit test with a mocked $io object that asserts that the messages printed to the cli are correct.

Aug 21 2017
Aug 21

Stable release for Config Split and Config Filter

Fabian Bircher

21 Aug 2017

1 Comment

Fabian Bircher

21 Aug 2017

1 Comment

Fabian Bircher, 21 Aug 2017 - 1 Comment

Celebrating the 8.x-1.0 release of our configuration management helper modules.

One year ago we released the first public version of Config Split with the goal to simplify working with Drupal cores configuration management. The main motivation was to find a solution for having development modules in local environments and avoiding to deploy their configuration. To achieve this in the cleanest way possible we decided to interact with Drupal only during the configuration import and export operations by altering what is read from and written to the \Drupal\Core\Config\StorageInterface.

We quickly realized that this is a powerful way to interact with how Drupal sees the configuration to import and so we split off the code that does the heavy lifting into its own module and now Config Ignore and Config Role Split use the same mechanism to do their thing.

Config Split now has documentation pages you are welcome to contribute to and there will be a session at DrupalCon in Vienna in which I will show how it can be used to manage a sites configuration in interesting ways.

If you were an early adopter (pre beta3) but have not updated to a recent version, you will need to install Config Filter first or patch core. The workaround and legacy code has now been removed and the current code is going to be backwards compatible in the future. So if you use the APC class loader, make sure to clear the apc/apcu cache or you might get an error after updating.

Apr 25 2017
Apr 25

Key points from our configuration management sessions.

Unfortunately none of us from Nuvole are attending DrupalCon Baltimore and can’t, therefore, attend the “hallway track” and join discussions in person. We have held presentations about advanced configuration management in Drupal 8 at all Drupal events we attended in the last year including the last DrupalCon in Dublin. So I’ll try to cover some concepts here that could help the discussion about how the configuration management can be improved.

To me there are at least two important dimensions to configuration management in Drupal 8 and different contrib project have sprouted to address:

Vertical: transfer configuration between different environments of the same site.
Horizontal: transfer configuration between different sites.

Following are a few contrib solutions and core issues that address the different itches. This is not meant to be an exhaustive or definitive list but highlight the different problem spaces. Many more contrib solutions address some issues in this space to accommodate various different workflows.

Vertical

Drupal 8 core only addresses this use case, however, there are some important cases not covered (yet). The management of configuration between the different environments is best taken care of by importing and exporting the whole sites configuration together.

Installing from existing configuration:

Contrib solution: Config installer
Core issues: Allow a site to be installed from existing configuration, Allow a profile to be installed from existing config

Environment specific configuration (ie devel only on develop)

Contrib solution: Config Split
Core issues: Allow exported configuration to be environment-specific, Allow development modules to opt out of config-export

Configuration with Content

Contrib solution: Default Content, Deploy

Horizontal

Due to the fact that the same tools can be used for both dimensions and the fact that in Drupal 7 features was abused for doing the vertical configuration management as well this concept may not be so clear. Ideally configuration between sites is shared from the development environment of one site to the development environment of another, and the vertical tools with drupal core are used for deployment. Moving configuration between different sites is done by moving a subset of configuration between environments.

Re-using a set of configuration in another site

Contrib Solution: Features
There are many more modules designed to deal with distributions and dealing with the paradigm that sites now own the configuration.

Inheriting installation profiles

Core issue: Allow profiles to provide a base/parent profile

Multisite with large portion of shared configuration

Contrib Solution: Config Split
While this is not the original problem it tries to solve, it is reported to be used for it..

Conclusion

I may have an obvious bias towards config split since we maintain that module but it is just one of many config related modules. I hope there is a fruitful discussion in Baltimore about configuration management in Drupal 8. BoF

Related blog posts:

Feb 27 2017
Feb 27

Our solution for advanced configuration management workflows has just become more powerful! The core of what makes Configuration Split work so nicely with drush and the Drupal UI has been split off in a new module: Config Filter! Config Filter exposes a ConfigFilter plugin and swaps the sync storage applying all the active ConfigFilters to it. This means it is no longer necessary to manually swap the service as we recommended to do in the past and it works also when installing a site, as the swapping happens only when the module is installed. As of drush 8.1.10 this works with the default config import and export commands.

Update Config Split

For current users of the Config Split module it means that they should remove the custom service swapping as part of the update and then apply the database updates which will install Config Filter (you already downloaded it with composer update, right?).

New command-line options and new workflows

With the re-factoring we also changed the options of the drush and console commands. We added a new option for specifying a split and then export to and import from only that split directory; this supersedes the previous options to specify separate export destinations and import sources - they are not needed for the simple work flow we advocate for, and the behaviour can easily be achieved by modifying settings.php as needed.

With the addition of the single split export, a new workflow becomes possible: export only the "production" split, before importing the whole configuration which will have it merged back in.

Improved Graylists

In addition to the new architecture the graylist feature has been improved so that dependencies can be graylisted as well and optionally only items that differ from the sync storage are split off. More details and more documentation will come in future.

Roadmap

The ultimate goal is to improve the export API in Drupal 8 core so that more advanced workflows are possible. Config Filter is our proposal for a solution in contrib, but we feel that this functionality should belong to Core.

On the way to Core, here are some more immediate steps:

  • Clean up after the big refactoring, adding automated tests for some special cases.
  • Improve the documentation.
  • Integrate with Config Readonly to allow locking the site except for the configuration defined in splits.
  • For those that want to do configuration management without seatbelts, integrate Config Ignore with Config Filter so that it can work smoothly next to Config Split.
  • ...and eventually propose Config Filter for inclusion in Core!

More information and acknowledgments

The re-factoring started during the Drupal Mountain Camp in Davos and I would like to thank the organizing committee for the great event. On Friday I gave a presentation about my favorite topic in Drupal: Configuration Management, attached here are the slides based on the Nuvole presentation at DrupalCon Dublin 2016.

For any further information see the module pages for Config Split and Config Filter and their issue queues.

Nov 28 2016
Nov 28

Solving one of Drupal 8 core's development and deployment work-flow issues.

One of the nice things that happened during Drupal Ironcamp in Prague last week was the first beta release of the Configuration Split that I started developing a few months ago.

As explained in the previous blog post about Configuration Split, the typical use case of Configuration Split is to be able to maintain a "development" configuration and a "production" configuration, where some of the modules or settings you use in development are not copied to production upon configuration export/import.

A typical use case

We assume that you have a production and development version of the same site and that they are aligned.

Step 1: Enable anything you need in development and work normally

Enabled UI modules In the development environment we enable Devel and UI modules enabled and we can create new content types, add fields and do all things we like to do while developing a Drupal site. Some of this configuration work is meant to go to the production site and some (e.g., the fact that Devel is enabled and related configuration) is not. We thus need to split our configuration.

Step 2: Create a split definition

Basic Split settings Choose a name for your "local" configuration and select all the configuration you don't want to have on the live site (even though, if you wish, you can still share it with colleagues). This can include modules you would not want to be enabled or configuration that is for development only. The path here is a folder next to the normal config sync directory outside of the web root. In the example above, we are splitting out to a configuration set named "Development" the Devel, Fields UI, Views UI modules and the system.menu.devel configuration.

Step 3: Export your configuration with config_split

$ drush config-split-export

In the most basic use case we just replace config-export with config-split-export in our workflow. This exports the elements we selected above to ../config/dev and the rest (the configuration we want to deplay) to our "standard" configuration folder, in this case ../config/sync.

Step 4: Import the "clean" configuration in production

On all environments we use the configuration import of config_split:

$ drush config-split-import

So in our usual workflow we replace config-import with config-split-import

In order not to have different workflows for development and production we simply deactivate the split on production by overriding the configuration in settings.php on production: $config['config_split.config_split.development']['status'] = FALSE;

This will result in the split definition to be deactivated. Deactivated Split settings And consequently the import will have the configuration blacklisted in the split removed and development modules deactivated.

What's new in the beta version

The following features are new or changed in the beta release:

  • The drush command works and has some text to warn users to which directories configuration will be written to.
  • The drupal console command is disabled, but there is a patch/branch you can use or help to make it functional again.
  • Split entities now have a status and the default command only uses the active splits. Of course you can override that in settings.php as with almost all configuration.
  • You can switch out the default config.storage.sync service in your services.yml file so that the default configuration sync UI uses the split import. Details are in the README.txt file.

More information and acknowledgments

On Friday I gave a presentation about my favorite topic in Drupal: Configuration Management, attached here are the slides.

I wish to thank IronCamp participants and especially Swentel and mr.baileys who worked on patches, and I'd like to thank IronCamp organizers for a very well organized event where I met a lot of old and new friends. Swentel also took care of documenting more advanced use cases of Configuration Split on his blog.

For any further information see the module page and its issue queue.

Aug 23 2016
Aug 23

Answering the most commonly asked question about Configuration Management in Drupal 8; and a preview of our DrupalCon training.

This post is an excerpt from the topics covered by our DrupalCon Dublin training: Drupal 8 Development - Workflows and Tools.

For the last two years we have been giving trainings and presentations at various Drupal events about configuration management and its new workflows in Drupal 8. One of the recurring questions has been:

How do I keep some configuration from being deployed? For example the devel module and its configuration?

Until now we have answered to use drush with the --skip-modules flag and then to gitignore the configuration for devel. But then you can't share the development configuration and the process is error prone when there are more modules and other configuration items that depend on the configuration you gitignore.

For some simpler cases where configuration needs to be different between environments (for example the error verbosity) the configuration override system allows to override the configuration in settings.php. This is a solution for some cases, however, you can not override which modules are enabled or override configuration that doesn't exist.

Enter: Configuration split!

Our new module Configuration split splits the configuration when exporting it with a special Drupal Console command. It can be configured to split out enabled modules and given blacklisted configuration and it will then separate all the configuration that is dependent on the listed modules or configuration. The module's settings also allow to set a folder to which the separated configuration will be exported to.

That way the configuration set which you use to deploy configuration between different environments is a subset of your development configuration and the trusty configuration system of Drupal 8 can be used unharmed. Of course when importing with the special command the split configuration is merged back, allowing you to keep your development configuration in place.

"To split" is a synonym for "to break" and as such "Configuration split" has a dangerous ring to it. This is on purpose because the exported subset is on purpose not what you have on your development site and not what you have locally tested. So you need to compensate that with a workflow that imports and verifies the configuration you are going to deploy. This is better than to import and export individual configuration because Drupal needs the whole set of configuration to do its checks.

How do I use it?

Download and enable the Configuration split module like any other module. Then configure it under admin/config/development/configuration/config_split. Set the split folder to a different folder than your normal config sync folder. If you want a prettier interface, consider using Chosen. Then use the Drupal Console command config_split:export and config_split:import to export and import respectively.

Let's look at the devel example from above. We typically version the configuration outside of the webroot in ../config/sync as seen by Drupal. (In the project root when starting a project with drupal-composer/drupal-project.) So the folder we specify for config_split would be ../config/dev. The module to filter would be Devel and the rest can be left empty. The devel.settings will be detected automatically. Note though that the system.menu.devel does not depend on the devel module and can not be detected automatically, but it is easy to add it to the blacklisted config and it could theoretically be deployed without breaking anything.

Configuration Split settings form with chosen

The resulting config_split.settings.yml could look something like this when exporting it:

folder: ../config/dev
module:
  devel: 0
theme: {  }
blacklist:
  - system.menu.devel

Finally the following command will export the configuration to the default sync directory without the devel module enabled and export the devel configuration into the dev directory.
web$ ../vendor/bin/drupal config_split:export

Now if you would import the configuration through the UI or with drush cim the devel module would be un-installed, and you can do that to see the site without its development configuration. However, If you want the development configuration to stay or become active and the devel module installed use the following command:
web$ ../vendor/bin/drupal config_split:import

How does it work internally?

We implemented a StorageWrapper that allows filters to interact with the configuration after it has been read and before it is written to the wrapped FileStorage during the import and export operation. The SplitFilter has a secondary storage and decides where to read from or write to. This is a very similar concept to what drush does with its --skip-modules flag since we will want to easily integrate this with drush in the future.

What comes next?

The module is still in an early development stage and some more additions in the scope of splitting configuration could be added. The subset without the split configuration could be verified after exporting it and we could warn the user if it couldn't be imported. Or for example we currently use only config_split.settings but if the need arises we could support multiple split configurations. Or we could add a "gray list" to ignore some configuration that exists rather than removing it when splitting, essentially making it a configuration override outside of the scope of Drupal's runtime. This could be useful when maintaining several sites that are almost the same but all have their little "special snowflake" configuration which in turn could be synchronized with the normal workflow.

It is important to understand that the configuration system of Drupal has limitations that are there for a good reason. Most of them are to ensure data integrity and robustness and reliability of the synchronisation and so on. In other words measures to protect you from accidentally breaking your site. Using tools like config_split or drush --skip-modules you circumvent some of these security and integrity checks so use them with caution.

Since the module is not required for the regular functioning of a Drupal 8 site (it can even be used to blacklist itself) it can already be used in the development process of your current projects already. Your feedback is welcome, see you in the issue queue.

Aug 10 2015
Aug 10

Drupal 8 Configuration Management: beware of pitfalls

Fabian Bircher

10 Aug 2015

0 Comments

Fabian Bircher

10 Aug 2015

0 Comments

Fabian Bircher, 10 Aug 2015 - 0 Comments

Nuvole's Configuration Management workshop at Drupalaton 2015 in Hungary

Drupalaton in Hungary last weekend gave the opportunity for a 90 minutes workshop about Configuration Management (CMI) and how it changes a Drupal developers work flow with the coming Drupal 8.

CMI allows you to have the whole site configuration under control and that is a great improvement. But it also comes with its pitfalls. Developers have to be more vigilant when sharing configuration with each other: mindless exporting configuration may override the configuration shared by others, mindless importing may override one's own work and forgetting to import merged configuration leads to ignoring the work of your co-developers at a later stage.

CMI, while it is much more awesome than anything we have in Drupal 7, is also much more unforgiving. So it would be all too easy to not use it properly and what it was designed for and then wonder why you can't have nice things.

Fortunately, a methodical git workflow can help recover from mistakes (except importing configuration before exporting it first). So don't forget the order of the steps:

  1. config export
  2. git commit
  3. git merge
  4. config import

And do it often to keep the merges and failures simple and easy to resolve.

Please find the full Drupalaton 2015 workshop slides attached below.

CMI-drupalaton2015.pdf (745.87 KB)Download
Apr 27 2015
Apr 27

Use Features together with a configuration management workflow and get the best of both worlds.

This is a preview of Nuvole's training at DrupalCon Los Angeles: "An Effective Development Workflow in Drupal 8".

Features for Drupal 8 will exist and it is already in alpha state. This may be surprising to those who wrongly assumed that Configuration Management was going to supersede Features; but, as we explained before, it all comes down to understanding how to use Features the right way in Drupal 8.

If you are using Features in Drupal 8 for deployment of configuration, you are doing it wrong!™

The role of Features

While the configuration management was built to keep configuration consistent and enable deploying it between different environments of the same site it was not built for sharing configuration between different sites. This is where Features comes in. Features allows you to easily bundle related configuration and re-use it on another site.

For re-using configuration and building starter profiles the current alpha version is already great. It works just like the re-usable “feature” we were blogging about last year, except it comes with a familiar UI and lets you do everything as a site builder. It even comes with a smart system that auto detects configuration that belongs together. Features-overview

Features-export

Configuration changes are development

Developers and site builders working with configuration in Drupal 8 need to understand that changing configuration needs to be treated equally with development. While this may seem trivial at first, the consequences are a fundamental shift to the approach of building a Drupal 8 site.

Drupal 8 comes with the configuration management system that was built to deploy configuration between different environments of the same site. It allows the whole configuration of a site to be synchronized both through the UI for “site builders” and through drush and git for “developers”. While the system allows individual configuration objects to be exported and imported, deployment and synchronization is always done with the entire set to ensure it is consistent.

This means that when you change configuration on a production site you either opt out of using the configuration management to deploy configuration or you need to go through a more elaborate work flow to synchronize the configuration changes with the development environments similar to what you would need to do if you were to change code directly on the production server.

For the math-oriented,

Δ config ⊆ development

What if you want to be free of the temptation to edit configuration in production? Luckily there is a module for that! It is called Configuration read-only mode and allows to lock the forms where you change the configuration, thus enforcing its role as a "developers only" tool.

Of course some configuration will differ between environments. Just as the database credentials are kept separate from the rest of the code, specific configuration can be overridden in the instances’ settings.php or services.yml. The rest of the code is usually treated as a whole for consistency reasons, the same should be true for configuration.

What about not using configuration management?

Features and packaging related modules should not be regarded as a solution to deploy only partial configuration. Packaged configuration by definition is never aware of the whole site and the possible inter-dependencies of the site's particular configuration. Deploying only partial configuration circumvents the safeguards Drupal put in place to make the deployment more robust. Of course nobody is forced to use the new tools for configuration management and you can easily opt out of using it by just ignoring that option. It is also still possible to develop a Drupal 8 site by sharing a database dump just as with Drupal 5. But remember that a Drupal 7 approach will yield the Drupal 7 headaches. We would recommend re-evaluating the deployment strategies when starting to use Drupal 8.

Managing distributions in Drupal 8

The more complicated scenario which is yet to be tackled is a feature for a richer distribution which will update over time. For example a newer version of a feature could come with an updated view or additional or updated fields for a content type.

First steps in that direction have already been taken by Configuration Update Manager (a dependency of features) and Configuration Synchronizer (a sandbox module). Configuration Update Manager compares the configuration in use on the site with the configuration which was provided by a module or installation profile. Configuration Synchronizer goes a step further and keeps a snapshot of the configuration when it was installed in order to determine whether the configuration has been customized by an administrator or whether it can safely be updated to the one provided in the new version of the module/feature.

All the modules mentioned above are used in a development environment of a particular site. Or in other words in an environment where configuration changes are expected and part of the process of updating a site.

For more, see our presentation from the Drupal Developer Days (embedded below); thanks to all the people there for the fruitful discussions we had on this topic during the event.

Dec 08 2014
Dec 08

Nuvole's files and documents management feature is now available for the latest version of Open Atrium

We received many requests to make an updated version of our Atrium Folders feature available for the latest version of Open Atrium, the excellent Drupal-based solution for Intranets developed by Phase2.

OECD sponsored the development of the new version as an open source project, in order to add a file management functionality to the Innovation Policy Platform site that it manages together with the World Bank. Atrium Folders for Open Atrium 2 is thus now available to everybody.

The usual features, a new way

Open Atrium changed completely and so did Atrium Folders. There are many differences under the hood, with a complete code rewrite, but the familiar user experience is still there.

Uploading and downloading files

Creating folders and adding files to the folders is as easy as creating any other content in Open Atrium 2: it is enough to create "Files sections". When you are viewing a folder, specific buttons allow to create subfolders, upload files and directly download any file.

Access management

The access management works like for other nodes in Open Atrium. Access to the folder can be restricted for both viewing and editing separately, and it can be determined at a folder level.

Notifications

The notification system of Open Atrium can also be used for folders. Users can be informed when new files are added, with the same interface used by other Open Atrium features.

And much more

Media module support

The files are attached with the Media widget and it is thus possible to manage not only files, but everything Media supports, like for example YouTube videos or files attached to other content.

Multiple uploads, with drag and drop support

The multi-upload feature of Open Atrium 2 can also be used with Folders to upload several files at the same time. Drag and drop uploads are supported too.

Download folder as ZIP file

The download button for files exists also for folders and it allows to download a folder with its subfolders and all included files as a ZIP archive. This functionality is available as a submodule, bundled with Atrium Folders.

File and folder revisions

Atrium Folders supports history and revisions both for folders and files. You can view previous versions of a file and optionally restore an older version. This functionality is available as a submodule, bundled with Atrium Folders.

Download, installation and support

The Open Atrium Folders feature can be downloaded and installed like any other module.

It is available on drupal.org at https://www.drupal.org/project/oa_folders/

Please report any issues in the module's issue queue at drupal.org

Aug 20 2014
Aug 20

The D8 way to replace "features-update" and "features-revert-all".

This is a preview of Nuvole's training at DrupalCon Amsterdam: An Effective Development Workflow in Drupal 8.

One of the new key features of Drupal 8 is the possibility to deal with configuration in code. Since configuration is now in text files, we want to put it under version control in Git to enjoy the many advantages this brings: comparing configuration states, keeping a history of configuration changes and even moving configuration between sites.

Setup

We will assume that you have a development version of Drupal 8, git and drush available on your system. You can set up your Drupal git repository in several ways. One of them is outlined in Building a Drupal site with Git on drupal.org. The document is written for Drupal 7, but can easily be adapted for Drupal 8.
Another, probably simpler method is to simply download a Drupal 8 (alpha) release and initialise a new repository with it.

In either case you should copy example.gitignore to .gitignore and adapt it to your needs to prevent settings.php and the files directory from being versioned.

The next step is to make sure that a configuration directory is versionable. By default Drupal 8 will place the staging directory under sites/default/files and it is considered a good practice to not version that location, but an alternative location can easily be specified in settings.php:

<?php
$config_directories
['staging'] = 'config/staging';
?>

It is also possible and even advisable to specify a directory outside of the web root of course. In that case you would put the parent directory of your web root where drupal is under version control and use ../config/staging. We will later see that it is also possible to add more directories and keys to the $config_directories variable.

Because the configuration management of Drupal 8 only works between different instances of the same site, the different instances of the site need to be cloned. Cloning a Drupal 8 site is done the same way as cloning a Drupal 7 site. Just dump the database of the site to clone and import it in the other environment.

Development

After cloning your site you can go ahead and start configuring your site.
Once the part of the configuration you were working on is done the whole configuration of the site needs to be exported.

local$ drush config-export staging
The current contents of your export directory (config/staging) will be deleted. (y/n): y
Configuration successfully exported to config/staging.

Next, you need to merge the work of other developers. In some cases it may be enough to simply use git pull, otherwise the configuration has to be merged after it has been committed:

Git can merge changes in text files quite well, but git does not know about Drupal and its yaml format for configuration. It is, therefore, important to verify that the merged configuration makes sense and is valid. In most cases it will probably not be an issue and just work, but it is always better to be vigilant and be on the safe side. So, after merging, you should always run:

local$ drush config-import staging

If the import went smooth you can push the configuration to the remote repository. Otherwise the configuration needs to be fixed first.

Deployment

The simplest case is when the configuration on the production site has not been changed. There is an interesting Configuration Read-only mode module that can enforce this.

If the configuration did not change deploying the new configuration is simply:

remote$ git pull
remote$ drush config-import staging

If the configuration changes on the production site, it is best to frequently export the live configuration into a dedicated directory.
Add a new config directory in settings.php:

<?php
$config_directories
['to_dev'] = 'config/to_dev';
?>

remote$ drush config-export to_dev -y

Add, commit and push it to the production branch so that the developers can deal with it and integrate the changes into the configuration which will be deployed next. Exporting the configuration into a dedicated directory rather than the staging directory avoids the danger that merge conflicts happen on the production site. The deployment to the production site should be kept hassle free, so it should always be safe to pull from git and import the configuration without the risk of a conflict.

Important notes

It is important to first export the configuration changes and then pull changes from collaborators because the exporting action wipes the directory and re-populates it with the active configuration. Since everything is in git, you can recover from such a mistake without much difficulty but why make your life complicated.

Import the configuration before pushing it to the remote repository. Broken configuration breaks the site, be a nice co-worker.

Git doesn't solve everything! Imagine Alice and Bob start with the same site, it has one content type and among others an "attachment" field. Alice deletes the attachment field, exports the configuration and pushes it to git. In the meantime, Bob creates a new content type and adds the attachment field to it. Bob exports his configuration, merges Alice's configuration changes without a problem (the changes are separate files) and imports the merged configuration. The attentive reader sees where this leads. The commit of Alice deletes the field storage for the attachment field, but Bob added a field instance which depends on the field storage. The exported configuration now contains a field instance that can't be imported.
At the time of writing, drush will signal a successful import but doesn't actually import it while the UI is more helpful and complains that the attachment field instance was not imported due to the missing field storage.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web