Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jan 07 2021
Jan 07

In the last article, we discussed the changes required to get Drupal 9.1 running on PHP 8. At that time, we got the Drupal 9.1 dev release working on PHP 8.0.0 RC4 with a few patches. Since then, a lot has changed with many of those patches being committed and Drupal 9.2 dev open for development. But we’ll talk about all of that at a later date. Today, let’s look at getting some of the common PHP extensions and configure it to run with Drupal.

We left off at a point where we have plain Drupal 9.1 running on a plain PHP 8 RC4 setup. Drupal doesn’t require any extensions, not in PHP core, and that means we only had to enable extensions like gd, MySQL, and others to have Drupal 9.1 running. With that, we were able to install Umami and use the site without any problems at all. To enable those extensions, we only needed our docker-php-ext-enable script, which is part of the PHP base Docker imageSee the Dockerfile in the reference repository for the source code (lines 41-52). Installing other extensions that are not part of the PHP core is not quite that simple. Think of it this way: if a module is present in Drupal core, you can install it right after downloading Drupal. But if it is a contrib module, you have to download and install it separately. It’s the same thing with PHP extensions.

Why test with extensions?

Just as you probably wouldn’t have a Drupal site with at least one contrib module, you probably wouldn’t have a PHP installation without a few of the common extensions. Drupal core utilizes some of these extensions when they are available (such as APCu and YAML), which yields better performance. This means that even though the extensions are not technically required, you would most likely have them.

I started with extensions, which I almost always install on sites I work. These are APCu, YAML, and Redis. Drupal core doesn’t use Redis, but I almost always install the Redis module for caching, which requires this module. It made sense to test if it worked on PHP 8 (both the module and the extension). As for the other two extensions, Drupal core uses APCu and YAML extensions for better performance if they are available. Again, it is a good idea to test Drupal with these extensions installed.

Installing extensions

Typically, we would use PECL to install any extensions we needed. PECL is a repository for PHP extensions, very much like a composer for PHP packages. With PECL, we would just need to run a command such as pecl install Redis to install the extension. You can see this being used in lines 53-58 in the Dockerfile.

pecl install apcu redis yaml

This is not as simple with PHP 8. PHP 7.4 removed default support for PECL and the official Docker image removed the command in PHP 8 images (it applied an explicit option to keep it for PHP 7.4).

Alternative tool to build extensions

I found another tool called pickle, which was intended to replace PECL but became dormant as well. I noticed some activity on the project, including a relatively recent release, and I tried that first.

The tool worked very well for APCu and Redis extensions. However, for YAML, it failed because it could not parse YAML's beta version number (2.2.0b2). I found that this was fixed in a recent commit but that would have meant that I would need to build pickle in my Docker image rather than just downloading and using it. I was not looking to go that route.

Building the extension manually

This left me with only one option: building the extensions myself. Fortunately, this turned out to be much simpler than I thought. You can see the steps required for each extension in lines 54-67 in the reference repository’s Dockerfile. For each extension, there are essentially just two steps:

  1. Clone their source code of the extension
  2. Run phpize, make, and make install to build the extension

We need the PHP source available to use the above tools and this is easily achieved using a helper script in the Docker image. You can see it being used in line 39 in the reference repository. Once we build the extensions, we clean up the PHP source to keep our Docker image size small. This is what the complete step looks like:

docker-php-source extract;
git clone https://github.com/krakjoe/apcu.git; cd apcu;
phpize; make; make install; cd ..;
# ... Install other extensions same way
docker-php-source delete;

Enabling extensions

Now that the extensions are installed, we can use the same script (docker-php-ext-enable) as earlier to enable the extensions. In our reference repository, you can see this done on lines 69-72. Thanks to these helper scripts, we now have our extensions enabled and configured for both the PHP module (for Apache) and the CLI. This can be verified by running the following command:

php -m

The above command will list all the enabled PHP extensions (internal ones as well) and you should be able to find apcu, redis, and yaml in the list.

Bringing it together

Now, we need to make sure that Drupal works with the above extensions. Since APCu and YAML extensions are used by the core, we should see any issues immediately. We can even verify that Redis is connected and APCu is being used by looking at the status report page, as shown in the following screenshot. 

Tweet from Hussainweb

For Redis, we need to install the Drupal module as well as the Drupal core doesn’t use it directly. We will discuss installing modules in another post.

PHP 8 remains an important milestone in PHP history, not just because of cool new features but also because it established a trusted release cycle promised at the release of PHP 7. A predictable release cycle helps build trust and also consistently brings new features and innovation to the product. We saw that with Drupal 8’s regular six-monthly release cycle and we will see that with PHP as well.

Jan 07 2021
Jan 07

Drupal is a popular web-based content management system designed for small to large enterprises with needs such as complex workflows, multilingual content, and enterprise integrations. An increasing number of organizations move to Drupal from their current systems every year and with richer features being added to Drupal 9 and planned for 10, the growth will only accelerate. This means that migrations to Drupal remain an ever-popular topic.

Drupal provides a powerful and flexible migration framework that allows us to “write” migrations in a declarative fashion.

The migration framework supports a variety of sources and the ability to specify custom sources and destinations. Furthermore, the framework provides a powerful pipelined transformation process that allows us to map source content to destination fields declaratively.

Thanks to this framework, migration is more of a business challenge rather than a technical one. The overall process (or workflow) of the migration may differ depending on various business needs and attributes of the current (source) system. Depending on the type of migration, we may plan to reuse in-built migrations (in core or contrib), selectively adapt migrations from different sources, or entirely write new migrations. Further, depending on the source, we may choose to migrate incrementally or at one-time.

Many similar decisions go into planning an overall migration strategy and we’ll talk about the following here:
 

01. Migration Concepts

02.Understanding the content

03. Drupal to Drupal migration

04. Migration to Drupal from another system

05. Migration from unconventional sources
 

Migration Concepts

The Drupal migration framework is composable, which is why it can be used flexibly in many scenarios. The basic building entity (not to be confused with Drupal entities) is called, migration. Each migration is responsible for bringing over one discrete piece of content from the source to the destination. This definition is more technical than a business one as a “discrete piece” of content is determined by Drupal’s internal content model and may not match what you might expect as an editor.

For example, an editor may see a page as a discrete piece of content, but the page content type may have multiple files or paragraph fields or term references, each of which has to be migrated separately. In this case, we would have a separate migration for files, paragraph fields, and so on, and then eventually for the page itself. The benefit of defining migrations this way is that it allows the migrate framework to handle each of these pieces of the content itself, providing features like mapping IDs and handling rollbacks.

Correspondingly, a migration specifies a source, a destination, and a series of process mappings that define the transformations that a piece of content may go through while being mapped from a source field to a destination field. These are called plugins (because of their internal implementation). We might use different source plugins depending on the source system with the common ones provided by Drupal core (for sources such as Drupal, SQL databases, etc.).

There are dozens of contributed modules available for other systems such as WordPress, CSV files, etc. Similarly, process plugins are diverse and influential in allowing a variety of transformations on the content within the declarative framework. On the other hand, destination plugins are limited because they only deal with Drupal entities.

Incremental Migrations

The Drupal migrate framework supports incremental migrations as long as the source system can identify a certain “highwater mark” which indicates if the content has changed since a recent migration.

A common “highwater mark” is a timestamp indicating when the content was last updated.

If such a field is not present in the source, we could devise another such field as long as it indicates (linearly) that a source content has changed. If such a field cannot be found, then the migration cannot be run incrementally, but other optimizations are still available to avoid a repeat migration.

Handling dependencies and updates

The migrate framework does support dependencies between different migrations, but there are instances where there might be dependencies between two content entities in the same migration. In most cases, the migrate framework can transparently handle this by creating what are known as “stubs.” In more complex cases, we can override this behavior to gain more adequate control on stub creation.

As discussed in the previous section, it is better to use “highwater marks” to handle updates but may not be available in some cases. For these, the migrate framework stores a hash of the source content to track if the migration should be run. Again, this is handled transparently in most cases but can be overridden when required.

Rollbacks and error management

As long as we follow the defined best practices for the migrate framework, it handles fancier features such as rollbacks, migration lookups, and error handling. Migrate maintains a record of each content piece migrated for every migration, its status, hashes, and highwater marks. It uses this information to direct future migrations (and updates) and even allow rollbacks of migrations.
 

Understanding the content

Another important part of the equation is the way content is generated. Is it multilingual? Is it user-generated content? Can content be frozen/paused during migration? Do we need to migrate the revision history, if available? Should we be cleaning up the content? Should we ignore certain content?

Most of these requirements may not be simple to implement, depending on the content source. For example, the source content may not have any field to indicate how the content is updated and in those cases, an incremental migration may not be possible. Further, if it’s impossible to track updates to source content using simple hashes, we may have to either ignore updates or update all content on every migration. Depending on the size of the source and transformations on the content, this may not be possible and we have to fall back to a one-time migration.

The capabilities of the source dictate the overall migration strategy.

Filtering content is relatively easy. Regardless of the source, we can filter or restructure the content within the migration process or in a custom wrapper on a source plugin. These requirements may not significantly impact the migration strategy.

Refactoring the content structure

A migration can, of course, be a straightforward activity where we map the source content to the destination content types. However, a migration is often a wonderful opportunity to rethink the content strategy and information flow from the perspective of end-users, editors, and other stakeholders. As business needs change, there is a good chance that the current representation of the content may not provide for an ideal workflow for editors and publishers.

Therefore, it is essential to look at the intent of the site and user experience it provides to redefine what content types make sense then and in the near future. At this stage, we also look at common traits that distinguish the content we want to refactor and write mappings accordingly. With this, we can alter the content structure to split or combine content types, split or combine fields, transform free-flowing content to have more structure, and so on. The possibilities are endless, and most of these are simple to implement.

Furthermore, in many cases, the effort involved in actually writing the migration is not significantly different.
 

Drupal to Drupal migration

This is usually the most straightforward scenario. The core Drupal migrate framework already includes source plugins necessary for reading the database of an older version of Drupal (6 or 7). In fact, if the intention is to upgrade to Drupal 8 or 9 from Drupal 6 or 7, then the core provides migrations to migrate configuration as well as content. This means that we don’t even need to build a new Drupal site in many simple cases. It is simply a question of setting up a new Drupal 8 (or 9) website and running the upgrade.

Drupal is often not used for simple cases or for any non-trivial site needs rebuilding. 

A typical example is “views,” which are not covered by migrations. Similarly, page manager pages, panels, etc., need to be rebuilt as they cannot be migrated. Further, Drupal 8 has brought improvements, and updated techniques to build sites and the only option, in that case, is to build the functionality with the new techniques.

In some cases, it is possible to bring over the configuration selectively and remove the features you want to rebuild using a different system (or remove them altogether). This mix-and-match approach enables us to rebuild the Drupal site rapidly and also use the migrations provided in core to migrate the content itself. Furthermore, many contributed modules augment or support the core migration, which means that Drupal core can transparently migrate certain content belonging to contributed modules as well (this often happens in the case of field migrations). If the modules don’t support a migration path at all, this would need to be considered separately, similar to migration from another system (as explained in the next section).

Incremental migrations are simpler to write in case of Drupal-to-Drupal migration as the source system is Drupal and it supports all the metadata fields such as timestamps of content creation or updates. This information is available to the migrate framework, which can use it to enable incremental migrations. If the content is stored in a custom source within the legacy Drupal system and it does not have timestamps, a one-time migration may have to be written in that case. See the previous section on incremental migrations for more details.

While Drupal-to-Drupal migrations can be very straightforward and even simple, it is worth looking into refactoring the content structure to reflect the current business needs and editorial workflow in a better way. See the section on “Refactoring the content structure” for more details.
 

Migration to Drupal from another system

Migrating from another popular system (such as WordPress) is often accomplished by using popular contrib modules. For instance, there are multiple contrib modules for migrating from WordPress, each of which migrates from a different source or provides different functionalities. Similarly, contrib modules for other systems may offer a simple way to define migration sources.

Alternatively, the migrate framework can directly use the database source to retrieve the content. Drupal core provides a source that can read all MySQL compatible sources and there are contributed modules that allow reading from other databases such as MSSQL.

Similar to the Drupal migration source, features such as incremental migrations, dependencies, and update tracking may be used here as long as their conditions are satisfied. These are covered in detail in earlier sections. 

Check out the case study that outlines migrating millions of content items to Drupal from another system
 

Migration from unconventional sources

Some systems are complex enough to present a challenge during migration, even with the sophistication of source plugins and custom queries. Or there may be times when the content is not conventionally accessible. In such scenarios, it may be more convenient to have an intermediate format for content such as a CSV file, XML file, or similar formats. These source plugins may not be as flexible as a SQL source plugin (as advanced queries or filtering may not be possible over a CSV data source). However, with migrate’s other features, it is still possible to write powerful migrations.

Due to limitations of such sources, some of the strategies such as incremental migration may not be as seamless; nevertheless, in most cases, they are still possible with some work and automation.

An extreme case is when content is not available in any structured format at all, even as CSVs. One common scenario is when the source is not a system per se, but just a collection of HTML files or an existing web site. These cases are even more challenging as extracting the content from the HTML could be difficult. These migrations need higher resilience and extensive testing. In fact, if the HTML files vary significantly in their markup (it’s expected when the files are hand-edited), it may not be worth trying to automate this. Most people prefer manual migration in this case.
 

Picking a strategy

Wherever possible, we would like to pick all the bells and whistles afforded to us by the migrate framework, but, as discussed previously, a lot of it depends on the source. We would like every migration to be discrete, efficient with incremental migration support, track updates, and directly use the actual source of the content. However, for all of this to be possible, certain attributes of the source content system must be met as explained in the “Understanding the content” section.

The good news is that we often find a way to solve the problem and it is almost always possible to find workarounds using the Drupal migrate framework.

Dec 21 2020
Dec 21

PHP 8 beta 4 is out. In fact, the chances are that by the time you read this, we might even have the first RC.

PHP 8 adds a lot of exciting new features, but at the same time, being a major version, it breaks a lot of previous behaviors and functionalities.

Getting Drupal to work on PHP 8 is not as simple as getting it to work on a new minor release such as PHP 7.4.

The Drupal community began planning to fix the compatibility issues early on. And as releases started rolling out, there were individual issues to address each deprecation, changed method signatures, and other breaking changes. These fixes went into a single issue so that we could run a single test against PHP 8. That is the patch I started with when I wanted to test Drupal 9 with PHP 8. To make it even more fun, I also used Composer 2 for all of these steps.

Why am I writing this?

It’s clear that this article may not have any value at all in some time when Drupal 9 officially supports PHP 8, along with all of its dependencies. Why am I writing this then? For one, I believe writing down things helps clarify the ideas and goals. It serves as documentation that can help throughout the process of experimentation. Secondly, I hope that parts of this article will be useful to people who are trying to upgrade their own complex applications to work with PHP 8.

The challenges I describe here are more relevant to applications that need to support a spectrum of PHP versions, not just one or two.

Drupal 8 supports PHP 7.0 to 7.4 right now and the issue I mentioned earlier also tries to add support for PHP 8 to Drupal 8.9 as well (it looks like it might just happen as well). This makes the challenge of supporting PHP 8 in Drupal even bigger as we have to support several breaking changes simultaneously.

Also, many of the problems may not be relevant to applications that need to run on one version of PHP as they just have to change code to match the changes in PHP 8.

I will also not try to explain all the changes that have gone in to support PHP 8. I’ll just talk about the parts that I analyzed, reviewed, or changed myself. With all that said, let’s begin.

Problem 1: Environment and initial setup

Docker is great for setting up quick environments for testing and development. At Axelerant, we usually use Lando for setting up a project (in fact, our project template tool supports generating a default scaffold for Lando). But that wouldn’t fit my needs here because Lando doesn’t support PHP 8 yet. Anyway, docker-compose is much simpler for something like this. I only need two services to begin with–a web server container (with PHP) and a database.

The Docker and PHP community maintain a great starting point in the form of official PHP Docker images in a variety of flavors: CLI, FPM, and with Apache on Buster. We use the last one here and add various PHP extensions and settings optimized for Drupal. I already maintain a collection of Drupal optimized PHP images and I only adapted that to work with the PHP 8 beta 4 image. The only difference is that as pecl is no longer included with PHP (as of PHP 8), I just removed those lines. That meant that the common PHP extensions such as APCu and YAML wouldn’t be available, but that’s okay for the first attempt. (I eventually added them in the image anyway.)

The other service in the docker-compose file is for MariaDB and I use Bitnami’s Docker version here. There’s an official one, but I am more used to the Bitnami one as Lando uses it. I don’t do any fancy setup with MariaDB as this is just for experimentation. You can see the docker-compose.yml and Dockerfile on Github. Apart from the Docker environment, we also need a Drupal site setup. Fortunately, this was very easy with the templating tool I mentioned earlier. Once the axl-template tool is installed, I just run this command:

init-drupal hussainweb/test-d9p8c2 --core-version "^[email protected]"

Typically, this would have been enough for a good starting point, but the template is optimized for composer 1. It includes certain packages that improve the composer’s performance with Drupal. However, I want to use composer 2 and those packages are not required. In fact, they don’t even work. So, after the init-drupal command above, I remove that package before upgrading the composer to version 2. I also update composer-patches to the latest dev release, which has the updated “require” statements to work with composer 2.

composer remove zaporylie/composer-drupal-optimizations
composer require cweagans/composer-patches:"^[email protected]"

Now, I’m good to update composer to the latest version (2.0 RC1 as of this writing):

composer self-update --2

At the time of this writing, the composer-patches plugin doesn’t work with composer 2. I have a PR open for the final fix (as of now) and I just used my fork as the repository for the package. Normally, in the Drupal world, I would have tried to apply a patch, but the plugin responsible for applying patches is broken here. Anyway, using forks is better. This commit shows how I used my fork, which works properly with composer 2. By the time you’re reading this, you might not have to do this at all.

Another thing to note is that my local machine is still running PHP 7.4. This is important because many of Drupal’s dependencies do not support PHP 8 and cannot be installed (unless we use the --ignore-platform-requirements flag).

This is good enough to get a basic environment running. Spin up the docker containers and find the port that the container exposes (look at docker ps -a). Access the site, and you might see an error message.

Problem with dependencies

Drupal is built on top of many packages and components in the PHP world and they have to support PHP 8 as well. As of right now, most of these components are not easily installable on PHP 8 due to the requirements in their composer.json. However, I ran composer on my local machine, which still runs PHP 7.4 and accordingly, it didn’t complain about PHP 8. This was obviously a risk and we should never do this on a production site (you shouldn’t be using PHP 8 beta on production right now anyway). But for this experiment, I wanted to try running those components on PHP 8 despite their composer.json restrictions. The good news is that none of those components caused any problems in my tests so far.

Of course, this is a blocker to add PHP 8 support for Drupal and is being tracked in this issue.

Problem 2: Fix Drupal errors

First, I faced problems with incompatible method declarations in doctrine/reflection. The problems were in methods getConstants and newInstance and I manually fixed both those instances (it was trivial) and it moved ahead. It turned out that they had already been fixed in the patch and I need not have worried but I moved on.

This fix at least got Drupal’s installation pages loaded and I went as far as the page where we enter the database details. (I eventually configured the .env file and never saw that page again, but that’s beside the point.) At this step, I saw an error related to an invalid method signature for a PDO method. In PHP 8, the method signatures of PDOStatement::fetchAll and PDOStatement::fetch have changed. Unfortunately, Drupal 8 and 9 have a wrapper on this method with the old signature. This now needs to change for PHP 8. But changing it will break support for PHP 7 and this creates our complication.

The solution is rather brilliant hackery by Alex Pott where we introduce two interfaces–one for PHP 7 and the other for PHP 8. Depending on the PHP version, we alias the relevant interface which gets used by the actual class. Then, to handle both method signatures, we have two different traits–again one for PHP 7 and the other for PHP 8. We again alias the relevant trait depending on the PHP version, which gets used in the class. This looks something like this:

if (PHP_VERSION_ID >= 80000) {
  class_alias('\Drupal\Core\Database\Php8StatementInterface',  '\Drupal\Core\Database\StatementInterfaceBase');
}
else {
  class_alias('\Drupal\Core\Database\Php7StatementInterface',   '\Drupal\Core\Database\StatementInterfaceBase');
}
interface StatementInterface extends StatementInterfaceBase, \Traversable {
  // ..
}

Similarly, the traits are aliased and each of the traits calls a new helper method in the actual Statement class (they are just renamed from the previous method). For example, the erstwhile fetchAll method would now become doFetchAll and since it is a different name, it doesn’t matter what signature it has. The fetchAll method would now reside in the relevant trait with the appropriate signature depending on the PHP version and would just call the doFetchAll method. This way, we have two different method signatures in the interfaces and traits depending on the PHP version!

The above changes can be reviewed in the patch at the issue where this is being worked at the time of this writing. When I tested, the patch only contained support for the differing signatures for the fetch method. Subsequently, the signature for the fetchAll method had changed as well and I added support for that in the patch. This solved the problem with installing Drupal. I was surprised and happy that there were no more errors during the rest of the installation and even when I was greeted with my new site’s homepage running Drupal 9.1 and PHP 8!

Before I go on to the next problem, I should note that the issue here with PDOStatement is really because of a mistake in the design.

As a developer, we should never directly depend on the API of something we can’t directly control. PHP version is something that we, as Drupal core developers, can’t control. We can certainly require a minimum version of PHP but we can’t control PHP to control other dependencies.

Intertwining our business logic with PHP’s API brings risks such as this. Like Alex Pott says in a comment in that issue, “These are not our methods. These are from \PDOStatement and their signature is owned by PHP and not by Drupal.”

Fortunately, Drupal boasts a high code coverage in the automated tests and refactoring could be as safe as we can hope. There might still be problems with contributed modules that might have extended this method and rely on Drupal’s implementation on top of the PHP wrapper. In a complex product like Drupal, it is precarious to refactor code like this.

Problem 3: Dynamic routes

I was elated with a working install on PHP 8 and wanted to test further, all the while keeping an eye on the error log. I found a few niche errors in how Drupal behaved on dynamic routes but that turned out to be a problem with the changes in PDOStatement::fetchAll and how it behaves with optional parameters. It was very messy and I am glad that PHP 8 changed the method signature to use variadics. The problem here was how the traits wrapped the call to the relevant Statement class's actual method. I had to use an ugly switch..case block to account for the number of parameters similar to how it was already handled in Drupal core. You can see the changes in this patch.

Problem 4: CKEditor warnings

I noticed multiple warnings being logged by CKEditor when I opened the node add/edit form. The warnings from a method called CKEditorPluginManager::getEnabledButtons and it was due to how a certain parameter was passed into the callback for array_reduce. Fortunately, this turned out to be an easy fix because there was no need at all to pass that parameter by reference and the patch was quickly committed. The issue contains more details and sample code for reproducing the issue.

Next steps

If you want to test this yourself, find my code here and set it up yourself locally. You would need composer and Docker and hope it is self-explanatory, but I will soon add documentation to the repository.

I was able to run Drupal 9 and perform many actions on PHP 8. But not many people need just the Drupal core. To test complex sites, I needed to test as many features as possible in core and also contrib modules. I will write about this in a subsequent post and maybe even do a simple benchmark comparing PHP 7.4 and 8.0 performance.

Check out the next part of this series - Upgrade Drupal To PHP 8: Compiling Extensions and watch out for more! 

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web