Feeds

Author

Jan 21 2019
Jan 21

As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those without previous experience working with command line, and can still be a confusing experience for those who do.

This is the fourth post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:

If you have not yet read part 1 and part 2, then before reading through this post, it is probably best to ensure you understand the concepts outlined in the summaries of those articles before moving forward.

Composer for Drupal Developers

The final section in this series addresses how Drupal developers can integrate profiles, modules and themes with Composer, to manage 3rd party libraries and/or other Drupal modules used by their code. To understand when and why a developer would wish to do this, let's have a look at a use case.

Use Case 1: Instagram Integration

Imagine a developer building a module to integrate with Instagram, pulling the most recent images from an Instagram account and displaying them on the Drupal site. Instagram provides an API for retrieving this information, but before Instagram provides this data, the Drupal site is required to authenticate itself to Instagram, using the OAuth2 protocol. After authenticating, various API calls can be made to retrieve data from the API using the the OAuth2 authentication token.

Up to Drupal 7, the Drupal way to do this would be to write a custom OAuth 2 integration (or use the OAuth2 Client module), then write custom functions that made the API calls to Instagram, retrieving the data.

Drupal 8 however, using Composer, acts like a hub, allowing for the inclusion of 3rd party libraries. This saves developers from having to re-create code within Drupal (aka rebuilding the wheel). For example, Instagram integration can be handled with the Instagram-PHP-API library. This library provides OAuth2 integration specific to Instagram, as well as various PHP wrappers that interact with Instagram's API to handle the retrieval of data from the API. Using Composer, our developer can integrate this 3rd party library into Drupal, and use it in their Drupal code. This saves our developer the time and effort of creating and testing the code that integrates with Instagram, as these tasks are handled by the library maintainers. If the library is well maintained, it will be updated in line with updates to Instagram's API, saving developer time (and therefore client money) in maintenance as well. And finally, as the library is open source, it will have the benefit of having eyes on it by PHP developers from various walks, rather than just Drupal developers, making for a stronger, more secure library.

Use Case 2: Drupal Module Dependency

Imagine that it is a requirement that the images retrieved from Instagram be shown in a popup (aka modal or lightbox). This can be handled using the Drupal Colorbox module. As such, the Colorbox module will be set as a dependency in the Instagram module's .info.yml file. When a site builder managing their Drupal project with Composer downloads/manages our developer's Instagram module with Composer, then tries to enable the Instagram module in Drupal, they will get an error from Drupal that the Colorbox module is missing. The problem here is that Drupal is being told the Colorbox module is a dependency, but we are not managing that dependency with Composer, and therefore the code has not been downloaded and does not exist.

At this point, it is easy to think that site builders could just add the Colorbox module to Composer with a require command. This would work, and after doing so, they would be able to enable the Instagram module, since the Colorbox module now exists. This opens up a problem however; imagine the site builder later decides to remove our developer's Instagram module. They disable/uninstall it in Drupal, then use composer remove to remove it. Everything looks good - the module has been uninstalled from Drupal, and the code has been removed from the codebase. However, there is one more step to return the system to its original state; disabling/uninstalling the now unnecessary Colorbox module from Drupal and removing the code using Composer. The necessity of this additional step opens up situations where the module will be left on the system due to forgetfulness, a lack of documentation, or changing site builders, creating site bloat due to unnecessary code existing and being executed.

The solution to this is to also manage Drupal profile/module/theme dependencies with Composer. In our use case, our developer will list the Colorbox module as a dependency of the Instagram module not just in the Instagram module's .info.yml file, but also as a Composer dependency in composer.json. This way, when the Instagram module is added to a project using Composer, Composer will also download and manage the Colorbox module. And when the site builder removes the Instagram module, Composer will remove the Colorbox module as well (if no other libraries list it as a dependency).

Integrating Drupal Code with Composer

Step 1: Create a composer.json file

Integrating with Composer requires that a composer.json file be created in the root of the profile/module/theme (from here on out referred to as the Drupal library) folder. This is explained in detail on the Drupal.org documentation page Add a composer.json file. Once this file has been added to the Drupal Library, it should be validated. This can be done by running composer validate on the command line in the same directory that the composer.json file lives in.

Note that Drupal.org automatically adds a composer.json file to projects that do not have one. This is so that Composer is able to manage all projects on Drupal.org, not just projects to which maintainers have explicitly added a composer.json file.

Step 2: Declare your dependencies to Composer

All library dependencies, whether 3rd party libraries, or contributed Drupal libraries (modules etc), need to be declared as dependencies to Composer. This is done by calling composer require [LIBRARY NAME], from within the folder of the Drupal library. This means that if you are adding dependencies for a module, you will navigate to the module folder, and add your dependencies. If you are adding dependencies for a theme, navigate to that folder, and add the dependencies there. The key point here is to not add your dependencies from the wrong folder, as they will be be added to the composer.json file of the wrong package.

Adding Remote Library Dependencies

In the use case example for this article, the Instagram-PHP-API library was suggested for integration with Instagram. This library has the Composer key cosenary/instagram. To set this library as a dependency of a module, navigate to the root of the module, and run the following:

composer require cosenary/instagram

Running this code results in the following:

  1. The cosenary/instagram library is added as a dependency to the composer.json file, so that Composer knows the package is managed by Composer. The composer.json file will contain something like the following:
     

    "require": {
      ...
      "cosenary/instagram": "^2.3",
      ...
    }

    Note that composer.json is committed to Git.

  2. The [MODULE ROOT]/vendor folder is created, and the Instagram-PHP-API library is downloaded into this folder. Composer by default creates the vendor folder in the same directory as the composer.json file. 

    Note that once the Drupal library has been pushed to a repository where it can be managed by Composer, installing the Drupal library with Composer will install the Instagram-PHP-API library to the project's vendor folder. As such, it's a good idea to delete the vendor folder in the module after the new composer.json and composer.lock files have been committed to the remote repository, before requiring the module with Composer.

    As a best practice, it is a good idea to create a .gitignore file in the module root, and include the vendor directory to ensure that it is never accidentally committed. 

  3. The composer.lock file is created, locking the installed version of the Instagram-PHP-API library to the Instagram module. This ensures that users of the Instagram library are working with a compatible version of the Instagram-PHP-API library. Note that this file is also committed to Git.

  4. Any dependencies of the Instagram-PHP-API library, declared in its own composer.json file, are downloaded.

  5. All libraries and dependencies are checked for version conflicts.

Adding Drupal Library Dependencies

In the use case example for the article, the Drupal Colorbox module (aka library) was mentioned to be a dependency of the Instagram module, and therefore is to be managed with Composer. Adding Drupal modules as dependencies requires an additional step, due to their being hosted on a non-default repository. To add a Drupal library as a dependency:

  1. Edit the composer.json file and add the Drupal repository to it, so that Composer knows where to look for the Drupal libraries, including the Colorbox module:

    "repositories": [
      {
        "type": "composer",
        "url": "https://packages.drupal.org/8"
      }
    ]

    Now Composer knows where to look for the Colorbox module when the Instagram module is installed.  If developers try to declare Colorbox module as a dependency before this step, they will get an error that Composer cannot find the drupal/colorbox library.
     

  2. Add the module with composer require:

    composer require drupal/colorbox

    Note that this will download the Colorbox module to [MODULE ROOT]/vendor/drupal/colorbox. This is NOT a good thing, for while Drupal will be able to find the module in this location, it is not the standard location, and if another copy of the module ends up in other directories that Drupal scans when looking for modules this will create hard to debug issues. So either delete the Colorbox module in the vendor folder right away, or immediately move it to the location where the rest of your contributed modules are located, so it's very clear where it is at.
     

After this, both the composer.json file and the the composer.lock file should be committed to Git. The module now has its dependencies declared on both the Instagram-PHP-API library and the Drupal Colorbox module. When site builders install the module to their Drupal project using Composer, the Instagram library will be downloaded to the project (site) vendor folder, and the Drupal Colorbox module will be installed to the web/modules/contrib folder.

Developing With Composer-Managed Drupal Libraries

Now that the module has been integrated with Composer, Composer can be used to manage the module during development. Let's imagine that the Instagram module being developed has the key drupal/insta, and is on Version 8.x-1.x. Downloading the -dev version of the module can be done by appending :1.x-dev when requiring the module:

composer require drupal/insta:1.x-dev

This will download the 8.x-1.x-dev version of module to the web/modules/contrib folder, where it can be worked with. A .git folder will be included in the module root, so that changes can be committed to Git. Sounds great, but when our developer tries to push their commits, they will get an error, as the remote URL of the repository is not the repository for maintainers of the module and does not allow commits. As such, we need to change the Git repository's remote URL to the URL of the repository for maintainers, allowing for code to be pushed, and releases to be made.

To find the correct Git repository URL, first go to the module's download page on Drupal.org. Make sure you are logged in as a maintainer of the module. If you are a maintainer and have the correct permissions, you will see a tab called version control. Click this tab. Next, copy the Git URL on the page. It will look something like this:

[email protected]:project/[MODULE NAME].git

Next, navigate to your module folder, and run the following commands:

git remote rm origin
git remote add [email protected]:project/
[MODULE NAME].git

The first line of this code removes the incorrect remote Git URL for non-maintainers, and the second line adds the correct Git remote URL for maintainers. After this, you will be able to push changes, including tags (releases).

You can also then easily switch between your development and release versions of the module by alternating the following:

composer require drupal/insta

composer require drupal/insta:1.x-dev

Note however that when doing any commits, you will need to switch up the remote URL each time you switch to the dev version of the module.

Summary

In this final part of the series on using Composer with Drupal, we have looked at how Drupal developers can integrate their custom code with Composer, to ensure that dependencies of the module are correctly managed. We have also looked at how to develop modules that are being managed with Composer. Using these techniques will allow site builders to keep a clean codebase that manages library conflicts.

And this concludes my four-part series on Drupal and Composer. I hope you have enjoyed it, and that it shows just how powerful Composer is, and how it can be effectively used to manage Drupal 8 sites. Happy Composing and Happy Drupaling!

Jan 03 2019
Jan 03

As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those without previous experience working with command line, and can still be a confusing experience for those who do.

This is the third post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:

If you have not yet read part 1 and part 2, then before reading through this post, it is probably best to ensure you understand the concepts outlined in the summaries of those articles.

Switching Management of your Drupal site to Composer

So you’ve worked your way through parts one and two of this series, and you now understand what Composer is, how it can be used to work with Drupal 8, and how to start a new Drupal 8 project using Composer. But, you started your current project without using Composer, and want to switch to managing your project using Composer. Where do you start? This article will discusses a few strategies behind converting your existing system to Drupal. Fortunately some automated tools exist for converting existing sites to Composer, and in the situation that neither of these tools work, an overview is provided on how to manually convert an existing site 

But, before moving on to any of these methods...

Take a backup! As this process will be destructive to your system, make sure you take a backup of your file system, and take a backup of your database. Then go and check to ensure that you have a full backup of the file system, and a full back up of the database.

If you skip this step, or do not do it properly, you may end up with an entirely broken system, so don’t skip this step.

Method 1: Composerize (Composer plugin)

Composerize Drupal is a Composer plugin that has been built to convert existing Drupal installations to use Composer. Instructions are on the download page. If this method doesn't work, you can try the Drupal Composerize module:

Method 2: Composerize (Drupal module)

The Composerize module is a Drupal module that is built to convert existing Drupal installations to use Composer. At the time of writing, this module has the following disclaimer on the page:

This module is still in development. It supports very basic Drupal 8 setups, but there are many necessary features it still lacks (e.g., support for patches, JavaScript libraries, distributions, etc). We're working on all that stuff, but this module is definitely not ready for prime time.

If this method doesn't work, you'll likely have to manually convert your site.

Method 3: Manual method

If the above steps fail, your last option is to convert your installation to using the Drupal Composer Template manually. Note that pretty much every system will be different, so these instructions are an overview of the end goal, rather than a complete set of steps that will convert your system. There is a good chance you’ll run into issues along the way that are not covered here, so make sure you took that backup!

Converting a system to the template requires achieving the following goals:

  • Setting up the file structure of the system to match the Drupal Composer Template
  • Delete old Composer files
  • Add the Template file system to your Drupal system
  • Set up the configuration directory and the private files directory
  • Set up your profiles, modules and themes to be managed by Composer

Step 1: Convert your file structure to match the Drupal Composer Template file structure

The Drupal Composer Template is set up to manage the directory above the webroot, as well as the webroot. The webroot is located in the [PROJECT ROOT]/web folder. Therefore you will need this structure:

  • / - Project root. Contains various scaffolding files such as composer.json and composer.lock, as well as the configuration export folder, and the webroot folder
    • /web - The webroot. Contains all Drupal core, profile, module and theme files.

You'll need to set up your Drupal installation to match this structure, and make sure that the server is set up so that the web root is at [PROJECT ROOT]/web.

Note: Some servers require the webroot to be in a directory with a specific name, such as public_html, or www. In this case, you can try setting up symlinks from the required directory name to the /web directory, so that, for example, [PROJECT ROOT]/public_html is a symlink pointing at [PROJECT ROOT]/web. Your Drupal files will then reside in the /web directory (satisfying the Drupal template), and also be accessible at /public_html (satisfying the server requirements). If this does not work, another option would be to edit the composer.json file (which is added in step 3) and change any paths that point at the /web directory, to point at the directory name you are actually using.

Step 2: Delete old Composer files

I'll say it again, because it needs to be said, make sure you took that backup in step 0!

If any of the following files or folders exist in your installation, delete them. Note however that you may want to save the composer.json file for reference if you've manually added any libraries into your existing installation using Composer.

  • [PROJECT ROOT]/vendor directory
  • [PROJECT ROOT]/composer.json file
  • [PROJECT ROOT]/composer.lock file
  • [PROJECT ROOT]/web/vendor directory
  • [PROJECT ROOT]/web/composer.json file
  • [PROJECT ROOT]/web/composer.lock file.

Step 3: Add the Template file system to your Drupal system

Go here, click 'clone or download' and download the .zip file (note - or clone the Git repository if you prefer)

  1. Save/move the zip file into the project root folder (the directory above the 'web' folder you created above). You can then unpack it using the following command. Before this step, the file /composer.json should not exist. After this step, if you've done it correctly, this file will exist.
    tar -zxvf [FILE] --strip-components=1
  2. Run the following command from the project root folder. This command will Install the Composer dependencies as well as create the /vendor directory.
    composer install
  3. Run the following to ensure your database is up to date, and caches are cleared.
    drush updb; drush cr;

Step 4: Set up the configuration directory and the private files directory (Optional)

This next step is optional, however it will make for a more secure system. First, the following directories need to be created if they don't already exist:

  • [PROJECT ROOT]/config/sync
  • [PROJECT_ROOT]/private

The first folder is where exports of the Drupal 8 configuration system will be exported to. The second folder is the private files folder. Creating both of these directories as siblings to the webroot adds security, as the files are not in a web-accessible location. The next thing to do is tell the system of the location of these files. This is done by declaring the folder paths in settings.php. You can do this by adding the following two lines to the bottom of settings.php:

$config_directories['sync'] = '../config/sync';
$settings['file_private_path'] = '../private';

After this, clear the registry (drush cr;). You can confirm that the configuration directory was properly set by running drush cex sync, and then checking that there are .yml files in the [PROJECT ROOT]/config/sync directory. You can confirm that the private files folder was properly set by going to Admin -> Configuration -> Media -> File System, and confirming that the private files directory is listed as ../private.

Step 5: Set up your profiles, modules and themes to be managed by Composer

The final step is to set up Composer to manage Drupal profiles, modules and themes to be managed by Composer. The Drupal Composer Template tracks Core by default, but needs to be informed of the rest of your code. Note that if you do not want to use the most recent version of these profiles/modules/themes, you will need to alter the commands below to set the version you want to install.

Drupal profiles, modules and themes can be installed with the following command:

composer require drupal/[PACKAGE NAME]

For example, if you were using the Administration Toolbar module (admin_toolbar), you would run:

composer require drupal/admin_toolbar

After you have done this, ensure you are up to date with a DB update and cache clear:

drush updb; drush cr;

At this point, your system should be converted to the Drupal Composer Template, with contributed code being managed by Composer.

Summary

This article looks at converting exiting Drupal 8 sites to being managed by the Drupal Composer Template. Doing this can potentially be automated using the Composerize Composer plugin or the Composerize Drupal module. In situations where this does not work, the manual directions in this article can be used as an alternative.

In the next and final part of this series, we'll look at how Drupal developers can integrate 3rd party libraries into their custom Drupal profiles, modules and themes.

Dec 03 2018
Dec 03

As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those who don't have previous experience working with the command line, and can still be a confusing experience for those who do. This is the second post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:

This article will be difficult to understand without first understanding the concepts explained in part 1, so If you have not read it, it would probably be worth your while to ensure you understand the concepts outlined in the summary of that article, before proceeding with this one.

Beginning a New Project

Fortunately a lot of work has been put into creating a Composer base (called a template) for Drupal projects. This Drupal Composer template can be found on Github at: https://github.com/drupal-composer/drupal-project. Instructions on how to use the template can be found there, and the same instructions are found in the README.md file that comes with the project when installed.

Starting a new project with the Drupal Composer template can be done with the following line of code:

composer create-project drupal-composer/drupal-project:8.x-dev some-dir --stability dev --no-interaction

This command does a number of things, many of which will be addressed below.

1) A new Composer project is created

The first thing the installation does is to create the directory specified as some-dir in the command, and initializes a Composer project in that directory. Change this to the appropriate name for your project. This is now your new project. The project contains the composer.json and composer.lock files that will be used to manage the code base of the project.

Note: The command provided lists the --stability as dev. This can be confusing as developers may think this means they are getting the -dev version of Drupal core. Don't worry, the command given above will always install the current full release of Drupal core, whatever it is at the time when the command is run.

2) Library dependencies (as well as their dependencies) are installed

The Composer template has a number of dependencies included by default, some of which we will take a look at here. These libraries are set by default as requirements in composer.json, and therefore are included when running the install command given earlier.

  • Drush: Drush is cool. If you don’t know what it is, it’s worth some Google-fu. Anything to be written about Drush has already been written somewhere else, so check it out - it will be worth your while!
  • Drupal Console: Drupal Console is also really cool. See comments on Drush.
  • Composer Patches: This one is very cool. This library in and of itself is worth using Composer to manage Drupal projects in my eyes. Even if Composer had no other benefits, this one would be great. First, an explanation of a patch is necessary. A patch is kind of like a band-aid that can be applied to code. Patches allow developers to submit changes, be they bug fixes or new functionality, to the library maintainer. The library maintainer may or may not add the patch to the source code, but in the meantime, other developers can apply the patch to their own systems, both to test if it works, as well as use it if it does. However, when the library the patch has been applied to is updated to a newer version, the patches have to be re-applied. What Composer Patches does is allow developers to track patches applied to the project, and have them applied automatically during the update process. This ensures that bugs don't arise from forgetting to re-apply patches after the update. Patches are tracked by adding them to composer.json. Here is an example:

    "extra": {
      "patches": {
        "drupal/core”: {
          “Patch description”: "https://www.drupal.org/files/issues/someissue-1543858-30.patch"
        }
      }
    }

     

  • With the above code, the next time composer update drupal/core is run, Composer will attempt to apply the patch found at https://www.drupal.org/files/issues/someissue-1543858-30.patch to Drupal core. Note that the description of the patch, given above as "Patch description", is arbitrary, and should be something descriptive. If there is an issue for the patch, a link to the issue is good to add to the description, so developers can quickly look into the status of the patch, see if any updated patches have been released, and check if the patch has been incorporated into the library, rendering the patch unnecessary.

  • And Others: There are many other libraries that are included as part of the Drupal Composer template, but the truth is that I haven’t looked into them. Also note that Drupal core alone has multiple dependencies which have their own dependencies. At the time of writing, 123 libraries are installed into the project with the install command.

But wait - I don’t use [fill in library here]

The Composer template is just that - a template. Some people don’t use Drush, some don’t use Drupal console. If these are not needed, they can be removed from a project in the same manner as any Composer library. Example:

composer remove drush/drush

The above command will remove Drush from the code managed by the Composer template.

3) The system folder architecture is created

The file system created by the Composer template deserves a close look.

  • /web: The first thing to notice is that Drupal is installed into the /web directory in the root of the project. This means that the root of the project created with the Composer template is one level above the webroot. When configuring the server, the domain for the server will need to point at the /web directory.
  • /config/sync: The Composer template sets up the project to store Drupal 8 configuration .yml files in this folder. When running drush cex sync to export configuration, the entire site configuration will be exported to this folder. This is folder is best kept out of the webroot for security purposes.
  • /drush: This folder holds a few Drush specific items in it. If multiple environments exist for your project, Drush aliases can be set in /drush/sites/self.site.yml, allowing for interaction with your various environments from anywhere within the project.
  • /scripts: At the time of writing, this folder contains only a single file, /scripts/composer/ScriptHandler.php. This is a really cool file that contains code run by Composer during various processes.

    The composer.json file in the Drupal Composer template contains the following:
     

    "scripts": {
      "drupal-scaffold": "DrupalComposer\\DrupalScaffold\\Plugin::scaffold",
      "pre-install-cmd": [
        "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
      ],
      "pre-update-cmd": [
        "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
      ],
      "post-install-cmd": [
        "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
      ],
      "post-update-cmd": [
        "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
      ]
    },

    The code above executes the stated code on pre-install, pre-update, post-install and post-update. Any time either composer install or composer update are executed on the system, the pre and post hook for that call are executed, calling the relevant functions above. Developers can create their own pre/post install and update hooks following the examples shown above.

  • /vendor: The first thing to note is that the location of this file differs from a vanilla Drupal 8 installation. When installing Drupal manually, the vendor folder is part of the webroot by default. This however could lead to security issues, which is why the Composer template installs it above the webroot. The vendor folder contains most of the libraries that Composer manages for your project. Drupal core, modules, themes and profiles however are saved to other locations (to be discussed in the next section). Everything else is saved to the /vendor folder.

4) Drupal File/Folder Installation Locations are set

As mentioned above, Drupal core is installed into the /web folder. The Composer template also sets up installation locations (directories) for Drupal libraries, modules, themes and profiles so that when Composer installs these, they are put in the appropriate Drupal folder locations. This is the code in composer.json that handles the installation locations:

"extra": {
  "installer-paths": {
    "web/core": [
      "type:drupal-core"
    ],
    "web/libraries/{$name}": [
      "type:drupal-library"
    ],
    "web/modules/contrib/{$name}": [
      "type:drupal-module"
    ],
    "web/profiles/contrib/{$name}": [
      "type:drupal-profile"
    ],
    "web/themes/contrib/{$name}": [
      "type:drupal-theme"
    ],
    "drush/contrib/{$name}": [
      "type:drupal-drush"
    ]
  }
}

The most common library type that a Drupal developer will install will be Drupal modules. So let’s look at the line of code specific to the module installation location:

"web/modules/contrib/{$name}": [
  "type:drupal-module"
],

This line of code says that if the type of Composer library is drupal-module, then install it to the /web/modules/contrib/[MODULE MACHINE NAME] folder.

But how does Composer know that the library being downloaded is type drupal-module? Well, the key to this is in how Composer libraries are managed in the first place. Throughout this article and the one that precedes it, we have repeatedly looked at the composer.json file that defines this project. Well, every Composer library contains a composer.json file, and every composer.json file that comes with packaged with Drupal modules contains a type declaration as follows:

"type": "drupal-module",

When Composer is installing libraries, it looks for a type declaration, and when it finds one, if there is a custom install location set in composer.json for that type, the library is installed to the declared folder. Drupal themes are of type drupal-theme, Drupal profiles are of type drupal-profile, and so on, and they are installed to the folders declared in composer.json.

Managing Custom Drupal Code with Composer

Custom code for a project can be managed with Composer as mentioned in Part 1 of this series. Drupal convention generally separates contributed (3rd party) modules and custom modules into separate folders. To install custom code in the locations according to this convention, the following lines should be added to the installer-paths declaration in composer.json:

"web/modules/custom/{$name}": [
  "type:drupal-custom-module"]
,
"web/profiles/custom/{$name}": [
  "type:drupal-custom-profile"
],
"web/themes/custom/{$name}": [
  "type:drupal-custom-theme"
],

This code adds three additional library types for custom modules, custom profiles, and custom themes.

Next, you’ll need to add a composer.json file to the custom module/theme/profile. Directions for this can be seen here: https://www.drupal.org/docs/8/creating-custom-modules/add-a-composerjson-file. Note that for the step named Define your module as a PHP package, you should set the type as drupal-custom-[TYPE], where [TYPE] is one of: module, theme, or profile.

Continuing on, make sure the composer.json file containing the type declaration has been pushed to the remote private repository.

The last step step is to add your private repository to you project’s composer.json, so that when running composer require my/privatelibrary, Composer knows in which repository to look for the library. Declaring private repositories in composer.json is explained here: https://getcomposer.org/doc/05-repositories.md#using-private-repositories.

With the above steps, when running composer install my/library, Composer will find the private repository declared in composer.json, search that repository for my/library, and download it. The composer.json file in my/library tells Composer that it’s of type drupal-custom-[TYPE], so Drupal will install it into the directory specified for Drupal custom [TYPE].

If using Git for version control on your system, you'll probably want to alter the .gitignore file in the Composer project root to ignore the custom folder locations. If you have created a custom module, and will be managing all custom modules with Composer and private repositories, you should probably add the /web/modules/custom folder to .gitignore. If you will be managing some custom modules with Git and not Composer, then you should probably add the custom module you have created to .gitignore as /web/modules/custom/[MODULE NAME].

Managing site settings: settings.php and settings.local.php

This section isn’t actually directly related to Composer and Drupal, but it’s a good step for setting up a project, and we can use the methodology to work with Composer template and Git.

Each Drupal installation depends on the file settings.php. This file is loaded as part of Drupal’s bootstrap process on every page load. Site-specific settings are added into this file, such as database connection information.

Towards the bottom of settings.php, the following lines can be found:

# if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) {
# include $app_root . '/' . $site_path . '/settings.local.php';
# }

These lines are commented out by the # symbol at the start of each line. This means that the code is not executed. If these lines are uncommented, by removing the hash symbol at the start of each line, this code is executed. The code looks for a file named settings.local.php, and if it exists, it includes that file. This means any settings in settings.local.php become part of the bootstrap process and are available to Drupal. After uncommenting the code, it will look like this:

if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) {
  include $app_root . '/' . $site_path . '/settings.local.php';
}

Why do this? This allows settings to be split into two types: settings that will be the same across all environments (eg. production, staging, local installations etc), and local settings that are only relevant to the current Drupal environment in which this file exists. This is done by committing settings.php to Git so it is shared amongst every environment (note - settings.local.php is NOT committed to Git, as it should not be shared). For example, the majority of Drupal installations have a separate database for each environment, meaning that database connection details will be specific to each environment. Therefore, database settings would be put into settings.local.php, as they are not shared between environments. The connection details to a remote API however may be the same regardless of environment, and therefore would be put into settings.php. This ensures any developer working on the system has access to the API details.

After splitting up the settings this way, settings.php is committed to Git so it is tracked and shared between environments. 

The full process is as follows:

  1. Install the Drupal Composer template as described earlier in this article
  2. Uncomment the code in settings.php as explained above
  3. Install Drupal as normal
  4. Run git diff settings.php to see what has been added to settings.php as part of the installation process. Anything that shows up should be added to settings.local.php, and removed from settings.php. This will definitely be the database connection, but could be other files as well.
  5. Edit .gitignore in the Composer project root, and remove this line:

    /web/sites/*/settings.php

    You can now add settings for all environments to settings.php and settings specific to the local environment to settings.local.php.

Setting Up the Private Files Directory

After setting up settings.php and settings.local.php as described above, you can now add the following to settings.php:

$settings['file_private_path'] = ‘../private’;

Next, create the folder /private in the root of your Composer project. Finally clear the Drupal cache, which will create the file /private/.htaccess. At this point you can now add settings.php and the /private folder to Git. Finally, edit .gitignore and add the following:

# Ignore private files /private/

This sets up the private file directories across all installations, saving developers having to set it up for each installation. Note that the .gitignore setting will ensure the contents of this folder are ignored by Git, as they should be.

What should I add to Git?

The Drupal Composer template project page states:

You should create a new git repository, and commit all files not excluded by the .gitignore file.

In particular, you will need to ensure you commit composer.json and composer.lock any time composer changes are made.

Is It Safe to Use Composer on a Production Server?

The actual answer to this question is not one I have. I am not a server guy overall, and for that matter, I’m not even that much of a Composer expert. It may be that Composer is entirely safe on a production server, but personally my thoughts are that having a program that can write files to the server from remote servers would seem to open up a potential security risk, and therefore it’s likely better to NOT have Composer on a production server. This comment may lead you to question why I would have wasted the time to write so much on Composer if it’s better to not use it in the first place. But not so fast partner! It really depends on the system architecture and the server setup. Some servers have been set up so that as part of the deployment process, the codebase is built using Composer, but then set up as a read-only file system or a Docker container or some other process ensuring security. This however is a particularly complex server set up. Fortunately there is an alternative for developers who are not working with servers configured in this fashion, which we'll look at next.

Using Composer, without having Composer installed on the production server

There is an in-between solution that allows us to use Composer to manage our projects, even with multiple developers, while not having Composer installed on the production server. In this case, we can use a hybrid of a Composer managed project and the old style of using pure Git for deployment.

First, we need to edit the .gitignore folder in the root of our Composer installation. In particular, we need to remove the following code:

# Ignore directories generated by Composer
/drush/contrib/
/vendor/
/web/core/
/web/modules/contrib/
/web/themes/contrib/
/web/profiles/contrib/
/web/libraries/

The above directories are all created by Composer and managed by Composer, which is why they were originally ignored by Git. However, we will not be managing our production server using Composer, and therefore we want to include these folders into the Git repository rather than ignoring them.

After setting up your project, commit the folders listed above to Git. This ensures all the files that are managed by Composer will be part of Git. That waycomposer install never needs to be run on the production server, since any code that command would download will already be part of Git.

What this means now is that any time a developer on the project add/updates code by running either composer update or composer install, they will need to commit not just the composer.json and composer.lock files, but also the added/updated source files that Composer manages, so that all code be available on the production server when checking out the code from Git.

Updating Drupal using Composer

In part one of this series, I discussed library versions. I am not going to go deep into how the versioning works internally, but I’ll explain how updating works specific to Drupal core. At the time of writing, the current version of Drupal is 8.6.3. The dependency set in composer.json is for Drupal 8.6.*. The * at the end of this means that your project uses upon any version of Drupal 8.6, so when a minor version update comes out for 8.6, for example 8.6.4, Drupal core will be updated to Drupal 8.6.4 when composer update drupal\core is run.

However, when Drupal 8.7 is released, it will not be automatically installed, since it does not fit the pattern 8.6.*. To upgrade to Drupal 8.7, the following command is run:

composer update drupal/core:~8.7

The ~/8.7 part of the above command tells Composer to use any version of Drupal 8.7. This means that in the future, when you run composer update drupal/core, minor releases of the 8.7 branch of Drupal core will be installed.

Summary

In this article, we have gone over how Composer is used with Drupal to make project deployment a little smoother, more stable, and consistent between environments. You should have an understanding of:

  • How to set up a new Drupal project using Composer
  • How the following folders relate to the project:
    • /config
    • /drush
    • /scripts
    • /web
    • /vendor
  • How Composer adds Drupal modules and/or themes
  • Drupal library dependencies
  • Managing custom Drupal code with Composer
  • Which items should be committed to Git
  • How to use Composer to manage Drupal projects where the production server does not have Composer
  • How to update Drupal using Composer

In the next post, coming soon, we'll look at how to convert an existing Drupal project to being managed by Composer.

Nov 25 2018
Nov 25

As any developer working with Drupal 8 knows, working with Composer has become an integral part of working with Drupal. This can be daunting for those without previous experience working with the command line, and can still be a confusing experience for those who do. This is the first post in an explorative series of blog posts on Drupal and Composer, hopefully clearing up some of the confusion. The four blog posts on this topic will be as follows:

So without further ado, let’s get started.

Composer: What is it?

The Wikipedia page (https://en.wikipedia.org/wiki/Composer_(software)) describes Composer as follows:

Composer is an application-level package manager for the PHP programming language that provides a standard format for managing dependencies of PHP software and required libraries.

That’s an accurate description, though a little wordy. So let’s break it down a little further to understand what it means.

Programmers like to use the term DRY - Don’t Repeat Yourself. This means that whenever possible, code should be re-used, rather than re-written. Traditionally, this referred to code within the codebase of a single application, but with Composer, code can now be shared between applications as well. DRY is another way of saying don’t re-invent the wheel; if someone else has already written code that does what you want to do, rather than writing code that does the same thing, it’s better to re-use the code that that has already been written. For example, the current standard for authentication (aka logging in) to remote systems is the OAuth 2 protocol. This is a secure protocol that allows sites or applications to authenticate with other sites, such as Facebook, Google, Twitter, Instagram, and countless others. Writing OAuth 2 integrations is tricky, as the authentication process is somewhat complex. However, other developers have written code that handles OAuth 2 integration, and they have released this code on the internet in the form of a library. A library is basically a set of code that can be re-used by other sites. Using Composer, developers can include this library in a project, and use it to authenticate to the remote API, saving the developer from having to write that code.

Composer allows developers to do the following:

  • Download and include a library into a project, with a single command
  • Download and include any libraries that library is dependent upon
  • Check that system requirements are met before installing the library
  • Ensure there are no version conflicts between libraries
  • Update the library and its dependencies with a single command 

So how does Composer work?

Composer itself is a software/program. After a user has installed Composer, they can then say ‘Composer: download Library A to my system’. Composer searches remote repositories for libraries. A repository is a server that provides a collection of libraries for download. When Composer finds Library A in a repository, it downloads the library, as well as any libraries that Library A is dependent upon.

A note on terminology

In this article, the term Library is used. Libraries are also known as Packages, and referred to as such on https://getcomposer.org/

A project is the codebase, generally for a website or application, that is being managed by Composer. 

By default, the main repository Composer looks at is https://packagist.org/. This is a site that has been set up specifically for Composer, and contains thousands of public libraries that developers have provided for use. When a user says ‘Composer download Library A’, the Composer program looks for Library A on https://packagist.org/, the main public Composer repository, and if it finds the Library, it downloads it to your system. If Library A depends upon (aka requires) Library B, then it will also download Library B to your system, and so on. It also checks to make sure that your system has the minimum requirements to handle both Library A and Library B and any other dependencies, and also checks if either of these packages have any conflicts with any other libraries you've installed. If any conflicts are found, Composer shows an error and will not install the libraries until the conflicts have been resolved.

While packagist.org is the default repository Composer searches, projects can also define custom repositories that Composer will search for libraries. For example, many developers use Github or Bitbucket, popular services that provide code storage, to store their code in the cloud. A project owner can set up Composer to look for projects in their private Github, Bitbucket, or other repositories, and download libraries from these repositories. This allows for both the public and private code of a project to be managed using Composer.

What happens when I install a library?

Composer manages projects on a technical level using two files: compser.json and composer.lock.  First we’ll look at the composer.json file. This file describes the project. If a developer is using private repositories, the repositories will be declared in this file. Any libraries that the project depends on are written in this file. This file can also be used to set specific folder locations into which libraries should be installed, or set up scripts that are executed as part of the Composer install process. It’s the outline of the entire project.

Each library has a name. The name is combined of two parts, first a namespace, which is an arbitrary string that can be anything but is often a company name, or a Github user name etc. The second part is the library name. The two parts are separated by a forward slash, and contain only lower case letters. Drupal modules are all part of the drupal namespace. Libraries are installed using Composer’s require command. Drupal modules can be installed with commands like:

// Drupal core:
composer require drupal/core
// Drupal module:
composer require drupal/rules
// Drupal theme:
composer require drupal/bootstrap

When the above commands are run, Composer downloads the library and its dependencies, and adds the library to the composer.json file to indicate that your project uses the library. This means that composer.json is essentially a metadata file describing the codebase of your project, where to get that code, and how to assemble it.

Composer and Git, Multiple Environments and Multiple Developers

Composer and Git work really well with each other. To understand how, let’s first look at traditional site management using Git. Developer A is creating a new Drupal project, purely managed with Git:

  1. Developer A downloads Drupal core
  2. Developer A creates a new Git repository for the code they have downloaded, and commits the code to the repository
  3. Developer A pushes the code to a central repository (often Github or Bitbucket)
  4. Developer A checks out (aka pulls) the code to this server.

This all sounds good, and it actually works very well. Now let’s imagine that Developer B comes onto the project. Developer B uses Git to download the code from the central repository. At this point, the codebase in Git exists in four locations:

  • Developer A’s computer
  • Developer B’s computer
  • The central repository
  • The production server 

At the moment, the codebase only consists of Drupal core. The Drupal core code is being managed through Git, which would allow for changes to be tracked in the code, yet it’s very unlikely that either Developer A or Developer B, or indeed any other developers that come on the project, will actually ever edit any of these Drupal core files, as it is a bad practice to edit Drupal core. Drupal core only needs to be tracked by developers who are developing Drupal core, not by projects that are simply using it. So the above setup results in sharing and tracking a bunch of code that is already shared and tracked somewhere else (on Drupal.org).

Let’s look at how to start and use Composer to manage a project. Note that this is NOT the best way to use Composer to manage a Drupal site, and is simply an example to show how to use Composer (see part 2 of this series for specifics on how to use Composer to manage a Drupal site).

  1. Developer A creates a new project folder and navigates into it.
  2. Developer A initializes the project with composer init, which creates a composer.json file in the project folder
  3. Developer A adds the Drupal repository at https://packages.drupal.org/8 to composer.json, so that Drupal core, modules and themes can be installed using Composer
  4. Developer A runs composer require drupal/core, which installs Drupal core to the system, as well as any dependencies. It also creates composer.lock (which we'll look at further down the article)
  5. Developer A creates a new Git repository, and adds composer.json and composer.lock to the Git repository
  6. Developer A pushes composer.json and composer.lock to the central repository
  7. Developer A sets up the production server, and checks out the code to this server. At this point, the code consists only of the composer.json and composer.lock files. Additional servers can be set up by checking out the code to any server.
  8. Developer A runs composer install on the production server. This pulls all the requirements and dependencies for the project as they are defined in composer.json

Now when Developer B comes on the project, Developer B uses Git to download the codebase to their local computer. This codebase contains only composer.json and composer.lock. However, when they run composer install they will end up with the exact same codebase as the production server and on Developer A’s machine.

Now the codebase exists in the same four locations, however the only code being tracked in the Git repository is the two files used to define the Composer managed project. When an update is made to the project, it is handled by running composer update drupal/core, which will update both composer.json and composer.lock. These files are then updated in the Git repository, as they are the files specific to our project.

The difference between the traditional Git method, and the above method using Composer, is that now Drupal core is considered to be an external library, and is not taking up space unnecessarily in our project's Git repository.

Project Versions

Projects can, and pretty much always do, have versions. Drupal 8 uses semantic versioning, meaning that it goes through versions 8.1, 8.2, 8.3… and so on. At the time of writing the current version is 8.6.3. If a new security fix is released, it will be 8.6.4. In time, 8.7.0 will be released.  Composer allows us to work with different versions of libraries. This is a good thing, however it opens up the risk of developers on a project working with different versions of a library, which in turn opens up possibility of bugs. Composer fortunately is built to deal with versions, as we will look at next.

Tracking Project Versions

So how does Composer handle versions, allowing developers to ensure they are always using the same library versions? Welcome the composer.lock file. The composer.lock file essentially acts as a snapshot of the all the versions of all the libraries managed by composer.json. Again, I’ll refer back to the Composer managed site described above. When we first run composer require drupal/core in our project, a few things happen:

  1. The current (most recent) version of Drupal is downloaded to the system
  2. All libraries that Drupal depends on are also downloaded to the system
  3. composer.json is updated to show that Drupal is now a dependency of your project
  4. composer.lock is created/updated to reflect the current versions of all Composer managed libraries

So composer.json tracks which libraries are used, and composer.lock is a snapshot tracking which versions of those libraries are currently being used on the project. 

Synchronizing Project Versions

The problem with developers using different versions of libraries is that developers may write code that only works on the version of the library that they have, and other developers either don’t yet have, or maybe they are using an outdated version of the library and other developers have updated. Composer projects manage library versions using the commands composer install and composer update. These commands do different things, so next we'll look at the differences between them.

Composer Install and Composer Update

Imagine that Composer didn’t track versions. The following situation would happen (again, this is NOT how it actually works):

  1. Drupal 8.5.6 is released.
  2. Developer A creates a new project, and sets Drupal core as dependency in composer.json. Developer A has Drupal 8.5.6
  3. Drupal 8.6.0 is released
  4. Developer B clones the Git project, and installs the codebase using composer install. Composer downloads Drupal core. Developer B has Drupal 8.6.0

The two developers are now working on different versions of Drupal. This is dangerous, as any code they write/add may not be compatible with each other's code. Fortunately Composer can track libraries. When a user runs composer install, the versions defined in composer.lock are installed. So when Developer B runs composer install, Drupal 8.5.6 is installed, even though Drupal 8.6.0 has been released, because 8.5.6 is listed as the version being used by the project in composer.json. As such, developers working on Composer managed projects should run composer install each time they pull updates from remote Git repositories containing Composer managed projects.

Updating versions

As has been discussed, the composer.lock file tracks the versions of libraries currently used on the project. This is where the composer update command comes in. Let’s review how to manage version changes for a given library (this is how it actually works):

  1. Drupal 8.5.6 is released.
  2. Developer A creates a new project, and sets Drupal core as dependency. The composer.lock file records the version of Drupal core used by the project as 8.5.6.
  3. Drupal 8.6.0 is released
  4. Developer B clones the Git project, and installs the codebase using composer install. The composer.lock file lists the version of Drupal core being used on the project as 8.5.6, so it downloads that version.
  5. Developer A sees that a new version of Drupal has been released. Developer A runs composer update drupal/core. Composer installs Drupal 8.6.0 to their system, and updates composer.lock to show the version of Drupal core in use as 8.6.0.
  6. Developer A commits this updated composer.lock to Git, and pushes it to the remote repository. 
  7. Developer B pulls the Git repository, and gets the updated composer.lock file. Developer B then runs composer install, and since the version of Drupal core in registered as being used is now 8.6.0, Composer updates the code to Drupal 8.6.0.

Now Developer A and Developer B both have the exact same versions of Drupal on their system. And still the only files managed by Git at this point are composer.json and composer.lock.

Tying it all together

Developers should always run composer.install any time they see that a commit has made changes in the composer.lock file, to ensure that they are on the same codebase as all other developers. Developers should also always run composer.install anytime they switch Git branches, such as between a production and a staging branch. The dependencies of these branches may be very different, and running composer install will update all dependencies to match the current composer.lock snapshot. The composer update command should only be used to update to new versions of libraries, and the composer.lock file should always be committed after running composer update. Finally, any time a developer adds a new dependency to the project, they need to commit both the composer.json file and the composer.lock file to Git.

Summary

Before moving on to the next blog post in this series, you should understand the following:

  • What the composer.json file does
  • What the composer.lock file does
  • When to use composer install
  • When to use composer update
  • How Git and Composer interact with each other

In Part 2, we'll look specifically at building and managing a Drupal project using composer.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web