Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Feb 26 2024
Feb 26

In my previous article I outlined that I really needed to get Drupal 10 sites running in Aegir 3. I had no time to wait around for other solutions, and I was going to try to decouple Aegir from running Drupal sites, so that it wouldn't be tied to Drupal versions.

Since writing that article and actually sitting down to do this work, I realised that it's quite an undertaking, and the approach outlined by Omega8cc in various github issues was worth a second look.

It works!

But it's wild. It comprises three main things:

  1. A custom/forked version of Drush 8.
  2. A custom/forked version of Provision 3.
  3. A custom/forked version of Drupal 10.
  4. (A sneaky fourth is that you absolutely do not want to have site-local Drush)

So...yeah...it's not going to be for the faint-hearted, but I'm going to detail out those three things a little, and how you might want to apply them to your situation.

Custom Drush 8

So the official Drush 8 doesn't support Drupal 10. But, it's actually not that much work to get this up and running, since Drush has pluggable 'engines' that load code based on the Drupal version that's detected when running commands.

Omega8cc has done a great job of doing the leg-work and providing a forked version of Drush, you can see the changes that have been made to support Drupal 10 here:
 

https://github.com/omega8cc/drush/compare/8.4.12...8-boa-micro

Although note, that there are some Aegir specific and non Drush 10 supporting changes in there too, so what I did was to make my own (private) fork of Drush and go through and apply the relevant changes to the files. If you don't know what the relevant changes are, then you probably are better off simply replacing your Drush with the Omega8cc fork, and it'll probably be fine.

Essentially this step is needed so that Drush sees the Drupal 10 codebase, and can interact with it correctly. It's likely that in Drupal 10's lifetime something will change, and Drush 8 will need some more work to be compatible with 10.3 and 10.4 etc.

If you were to run Drush 8 commands at this point, they'd start to bootstrap the site, but would fail with fatal errors. You need changes in Drupal 10 core too...

Custom Provision 3

Provision also makes use of Drush engines to load the correct code for the correct situation, so that if you ask Aegir to install a new site on a platform it'll load up the correct bit of code for the relevant Drupal version.

Again Omega8cc has done all the hard work here and provided the changes you need, but they are a bit harder to pick out of the repo. Try searching for files that contain '_10':

https://github.com/omega8cc/provision

And that should get you the files you need. I will note that unlike the Drush fork above, this really is a heavy fork. You are getting a lot more besides Drupal 10 support, if you can, I'd go for patching this into your existing Provision codebase.

I could add these to official provision 3 on Drupal.org, but without the other pieces they are useless and won't help you. Aegir 3 is essentially unmaintained, so while I've committed some PHP 8 fixes in the last few weeks, I'm not going to commit huge things like this.
I suppose we could have some code maintained in an issue fork or a patch.

This step is need so that Provision will call the correct bits of code at the right time, as far as I can see, these are largely the same as Drupal 9 versions of the engines.

Custom Drupal 10

Now for the big compromise.

The main problem here is that Drupal 10 uses Symfony 6, which has type hints on various interfaces and methods, and Drush 8 uses Symfony 2, which does not.

So, if you load one before the other, then as soon as PHP tries to load the second, it'll die because either the types are there, or they aren't.

This is a bit of a showstopper and there's no decent way to get it to work nicely. Instead, Omega8cc has discovered some band-aids you can slap on and while it'll work, it's brittle!

Omega8cc's fork of Provision automatically tries to apply some of these patches to Drupal 10 platforms, and gives you a nice way to add/remove them. But again, also has a lot of other opinionated changes (to provision), so if you want to know what to do to Drupal 10 core...keep reading:

We need to:

  1. Remove a bunch of typehints from Drupal core's logging mechanisms.
  2. Downgrade psr/log
  3. Patch symfony/console

Removing the typehints

Omega8cc have got a nice patch that does this:

https://raw.githubusercontent.com/omega8cc/boa/5.x-dev/aegir/patches/drupal-ten-aegir-01.patch

And you can apply that using composer patches if you want.

Note that if you have any classes that extend these patched classes, they'll need patching too, so if you have any custom or contrib loggers, you'll need to go sort those out.

Downgrading psr/log

Composer makes this one super easy:

composer require 'psr/log:1.1.4 as 3.0.0'

Will do what you want. It's the companion to the first step, those classes implement these interfaces, so they need to get downgraded too. This might need to get adjusted if Drupal 10 core requires psr/log 3.1.0 or something like that, but the principles should still work.

If your Drupal 10 codebase isn't managed by Composer...well...getta out here and get that sorted first.

Patching symfony/console

Drush 8 only uses part of this package, so we can get away with patching just a tiny bit of it. Essentially you want to remove the 'Input' directory within the package. If you have a build system you could delete the files as part of your build process. Omega8cc's approach is to make the directory essentially unreadable.

Either way the result is that when PHP tries to load some code that uses version 6 of Symfony\console\Input it'll fail to find the code in Drupal core, and instead fallback to the version that came with Drush 8. Luckily this code isn't used by Drupal core, but you'll need to check that your custom and contrib code doesn't use it either, or handles the case when it magically loads code from Symfony 2.

For me and my codebase this was fine. None of our custom or contrib code used the classes or interfaces, so all good.

That's it

For Drupal 10 support in Aegir, that should be it. You can run Aegir 3 in PHP 8.1 and Drupal 7 and Drupal 10 on the same box. Magic.

Make sure you don't have a site-local Drush in your composer require, otherwise, Drush 8 being the great tool that it is will find it, and pass execution over to it, but it's not the same Drush that Aegir has all it's claws/hooks in, so things will go badly quite quickly.

Gotchas

Probably many, we've not actually rolled this out into our production environment yet, but we're close.

The main thing I can think of is that Drupal 10 contrib modules are very much going to assume that there's a modern version of Drush running commands, and not Drush 8, so it's likely that they aren't going to provide Drush 8 commands. You might need to write your own shims that copy/paste large lumps of code, or call through the new style Drush commandfiles, but that means that Drush command hooks won't be working quite right etc.

So yeah, danger if you're doing anything interesting using Drush commands on your sites.

Oh, and I guess it's possible that this will all break hard in the Drupal 10 lifecycle. I mean, it probably won't because it'll always be a foundation of Symfony 6, but you never know, the Drop is always moving and all that.

Thanks to Omega8cc

This was all shown to be possible, and documented in code by the great team at Omega8cc, so thanks very much for the work they are doing. I have nothing against there heavy forks of Aegir, they just aren't for me, hence why I've tried to pull out just the bits you need into this post.

Future

I still think that Aegir 3 could be decoupled from hosting Drupal sites. So that it was hosting 'sites' and then one could have some well-defined way to interact with those sites via some command line tool depending on what they were. However, that's a big effort, and Aegir 3 is essentially unmaintained now, so that's not going to happen. 

A number of others in the #aegir Drupal.org Slack channel have mentioned other alternatives, such as Aegir 5, or getting Drush 12 running Provision commands and running everything with Drush 12 instead of Drush 8, or using some other tool as the outer runner.
I think they are all going to run into the same issues/have to deal with the same thing: abstracting the notion of the 'site'. Provision 3 takes a number of shortcuts to make it really easy to pass data around between itself and the site, and those shortcuts simply aren't possible in Modern Drupal.

However, in case there's someone out there with a big ol' budget, and a desire to keep Aegir 3 going, I think the 'decoupled' Aegir would look something like:

  • A provision where the places that it calls through to functions executed in the Drupal context abstracted out into some kind of site specific 'plugin'. Instead of trying to use the nice Drush functions for calling a 'inner' Drush command, instead the usual command line interface would be used and data simply encoded on stdin/stdout/stderr in a format not tightly coupled to Drush 8 or Drush 12/13 etc.
  • Provision calls into the Drupal site a lot and because it's never needed to be explicit about doing it, it's sometimes hard to spot where this is happening.
  • Some kind of shim package that can be composer required into sites hosted on Aegir that would provide some way for the Provision plugins to call into the Drupal site and get the response it needs. Maybe this would merely provide some Drush commands.

Good luck if anyone ever embarks on such a project! Or contact us if you want a shoulder to cry on to hire us for a project!

Jan 26 2024
Jan 26

Aegir is a hosting system built in Drupal, for Drupal.

It lets you easily create new Drupal sites and create databases, filesystems, virtual hosts etc. for the sites. You can manage hundreds or thousands of sites using a simple Drupal based UI. As simple as you would manage a list of 100 blog posts, you can manage 100 Drupal websites.

Currently the latest released version of Aegir is: Aegir 3.

Aegir 3 relies on Drush 8, which means it can work with: Drupal 7; Drupal 8 and Drupal 9. But not Drupal 10. Oh.

I have need for it to work with Drupal 10. Specifically we have a large collection of Drupal sites, that currently Drupal 9, but ready and waiting to jump to Drupal 10. Yes, we've left it too late, we should have done this a long time ago, but we are where we are.

I've spent the day looking into various options, they are roughly:

  • Move to BOA
  • Get Aegir 3 to work with Drupal 10
  • Move to Aegir 4
  • Move to Aegir 5
  • Move to something else

My terms of reference

I'm looking for a solution that changes as little as possible in the current stack. We've got a big, beefy server, and lots and lots of Drupal sites on it. The Drupal sites themselves run fine and we're after the management of them really. We don't need to scale out particularly, and if we do, we've got options on the hardware side vertically. We also don't need some fancy multi-datacenter approach, and ideally keep with our Nginx, Varnish and Apache sandwich.

I do want to manage things in the old school way, but I need a few, select, non-technical users to be able to manage the sites in a friendly UI like Aegir's hosting system.

Lots of the new hotness out there are essentially orchestrating Kubernetes, and that's great, but I don't actually want 100 containers all running PHP which is how I understand all of these systems would essentially function.

I used to be an Aegir maintainer, and I have a deep knowledge of how Aegir works and even helped write some of the inter-process communications stuff in Drush 8. 

Move to BOA

The first option is one that has shown to be working, and requires a custom fork of Drush 8, some core patches and using BOA. BOA is quite an aggressive fork of Aegir, and I'm not sure I want all the changes and whatnot that comes with it. I think I have to discount it, because the last time I looked there would simply be too much change on the nginx side of things at least, and it was doing all kinds of things I don't simply understand and thus won't be able to reason about.

The main inspiration I take from BOA is that they've got it working! Well, they've got something working. It looks like they've essentially forked Drush 8, applied some Drupal core and Aegir patches. This makes it so that Aegir can still use it's aliases, code and talk to Drupal 10 sites. How they've been able to do this is very impressive indeed. I feel like it's just a matter of time before it breaks though, surely Drupal 10 is going to change something in it's lifecycle and cause issues. I don't quite see how this can work with Drush commands in contrib modules. For example, during deployments on our sites we revert features, but that's a features module provided Drush command, which won't be callable from Drush 8 in Drupal 10.

Get Aegir 3 to work with Drupal 10

This I feel is the least effort option. Find some way to get standard Aegir 3 to communicate with Drupal 10. Now, Drush 8 can't directly bootstrap a Drupal 10 site. Aegir 3 uses Drush 8 to be both the backend storage for all the data about the sites to host, and to bootstrap the sites it manages.

This worked really well in the Drupal 5/6/7 days, and just about coped with Drupal 8/9, but hasn't for Drupal 10. Think of it this way, what if instead of hosting a Drupal site, it was a Wordpress site? Aegir simply wouldn't be able to neatly bootstrap into a Drupal site and do it's stuff. It's the same with Drupal 10.

How much stuff does it really do though? I'm not 100% sure. I know that it gets the currently installed packages for example, but what else does it do. It looks to me like it often hands off a Drush subprocess do the heavy lifting. Maybe we can hand off to another subprocess, one that runs a site-local Drush 12+ instance, passing in all the options it needs to bootstrap and run the actual commands on the site.

Please, if this is a completely crazy way to go, someone please comment and let me know!

Move to Aegir 4

So there is a 4.x branch of Aegir, and apparently it can work with Drupal 10 sites. It does this by replicating the aliases to YAML files and then running a global Drush 11, which can then bootstrap the Drupal sites. Apparently it also has replaced some of the Drush invocations with standard process invocations. I guess it had to do this because the Drush to Drush process communication stuff was in Drush 8 and got removed.

It seems like Aegir 4 also brought in a lot of other changes too, and that's not particularly something I want or need. Also, it's never been officially released/tested by the community etc.

Move to Aegir 5

This got announced in a few blog posts a while back, and I've not heard anything since. It's possible that it's there and ready to go, but as far as I understand it, it's a whole change to how the sites would be managed and hosted, and while an import from Aegir 3 is on the roadmap, it seems like it would import the sites into something completely different, not some simple vhosts and a DB server.

Also, even if it does exist, it'll have had minimal testing and eyes on it.

Move to something else

I think this is my preferred long-term option. These sites don't really need Drupal. They could actually be static sites plugged into a central content repository. That would drastically simplify lots and lots of things. Or we keep the Drupal sites, but move them to Kubernetes and using something like Amazee's Lagoon to host them etc. This would be very cool, but probably quite expensive in terms of having enough resources to host all the containers.

One for the long term future I think.

Get Aegir 3 to work with Drupal 10

I might give this a go in that I think what I can do is this:

  1. Get Aegir running normally.
  2. Install a Drupal 9 site.
  3. Get a single Aegir task running that doesn't bootstrap the Drupal 9 site with Drush 8. I'm thinking something like getting the one-time login link. This has data flowing in both directions, and I think, Drush 8 trying to bootstrap Drupal to the run the provision command. Getting this working would involve calling a site local Drush based on the data from the alias, but not actually using the alias.
  4. If that works, put something into settings.php that throws an exception if Drush 8 is detected, and then try to get everything else working for my use-cases.
  5. Then try and set up a Drupal 10 site.

I'd love to know your thoughts on this. Do you have an old Aegir running sites, and you don't know what to do with it? Or do you think I'm crazy and want to offer me a better way to go, use the comments below, please!

Updates/notes

So looking at the provision commands, the actual provision command to say, get the one-time login link bootstraps the Drupal site. So that would need to change that command so that it no-longer bootstrapped Drupal at all, but instead changed to be a pure Drush command. Then it could load the data from the alias etc.
I'm tempted to go ahead and get a dev instance of Aegir running and the add a new Drush command, that doesn't bootstrap Drupal, but does attempt to grab data from an alias and see what I can do. That possibly an even simpler proof of concept, as it's an entirely new Drush command, but one I then know could be called by provision's task runner instead of the current one.

Further updates

So that approach did seem sort of promising, but also, a decent amount of work. I decided to re-investigate the approach that Omega8cc advised, a custom version of Drupal 8 and then a platform of Drupal 10 code with some modifications shall we say. It worked. It has some rough edges, but I'm going to see if we can get the required bits and pieces into the mainline of Aegir 3.

Feb 25 2023
Feb 25

Let's define the scope and goals of our project to upgrade this very website to Drupal 10.

Essentially, that's it: we want to upgrade this website to Drupal 10 so that we can benefit from security releases etc.
At the moment we want to do so with the minimum of effort, so I don't want to have to be writing lots and lots of code or changing fundamentally how the site works, but I am up for simplifying things if it gets us to a point where we have to maintain less code.

Since Drupal 9, major version upgrades now take this basic form:

  • Update your code to be fully compatible with the last version of Drupal, removing all deprecations: hard.
  • Upgrade to the new version of Drupal: easy!

I'm going to install and use the fantastic Upgrade Status module to get a detailed handle on what we need to change, upgrade and rewrite to get the site working in Drupal 9, but ready for Drupal 10. We'll use that as a basis to see what we need to upgrade, the best plan for each component and go from there.

Upgrade status - First pass

We previously have composer require'd the upgrade status module into our codebase, so after enabling and running the report, here are the major findings that concern us for this series:

Environment

  • We'll need to upgrade to PHP 8.x, the site is currently running on PHP 7.4.
  • We're using deprecated or obsolete modules that come with core and will be removed in Drupal 10. This is a rather scarily long list for us:
    • CKEditor
    • Color
    • RDF
    • Seven
    • Stable

But other than that, we're good to go from an environment point of view.

Contrib projects

Upgrade status breaks the list of contributed projects down into a few sections, those are:

  • Projects that need an upgrade that might make them Drupal 10 compatible:
    • Better exposed filters
    • Components
    • Disqus
    • Advanced link
    • Entity browser
    • jQuery UI Slider
    • Scheduler
    • Simple XML Sitemap
    • Twig Tweak
    • Webform
  • Projects that don't have Drupal 10 releases yet, so either require patches or work to get them to Drupal 10:
    • Entity Embed
    • jQuery UI Sortable
    • Kraken
    • Markdown
    • Social media share
    • Term Reference Change
    • Unified Twig Extensions
    • Video Embed HTML5
    • Weight
  • Projects that are compatible with Drupal 10 already, I'll not list those, but there are plenty already, it's great to see community support for Drupal 10.

Custom code

Upgrade status will scan your code and tell you if there are problems that can be spotted that will stop the code working with Drupal 10. This is static analysis, so isn't perfect, but is a really good start.
We have a few custom modules doing very specific things on our site, but we have a custom theme, doing quite a lot of custom things, and that's where the main bulk of the issues the scanner found are, so we're going to need to set aside some time for that.

Simplifications

This site was built in the early Drupal 8 days, and we've not actually made too many changes since, specifically when we upgraded to Drupal 9 we basically did the smallest amount of work to get it there. How you'd typically handle media on a Drupal site has fundamentally changed since we built this site, in that you'd likely use the core Media module and add entity reference fields to your entities rather than adding image/file fields directly. However, we never had that luxury and never got around to changing our approach to use the core Media framework.

So, we're going to allow ourselves a bit of scope creep to do this 'sub project' given that the benefits are that we're going to be able to remove a bunch of modules: entity browser, file browser, etc. that will then mean that we don't need to upgrade those modules and our dependencies will be better supported: since they'll be in Drupal core. It's no slight against those modules, it's just that we don't need the functionality they bring, for our site today.

The scope/plan

So roughly the scope/plan is shaping up to be:

  1. Convert our file/image fields to core media, and remove entity browser, file browser, etc.
  2. Update our custom code
  3. Evaluate the remaining upgradeable contrib projects to see if we can remove them, and if not, upgrade them.
  4. Evaluate the remaining non-upgradeable contrib projects to see if we can remove them, and if not, work with maintainers to get them upgraded.
  5. Handle the core modules that have been marked as deprecated or obsolete.
  6. Upgrade the PHP version we use to run the site
  7. Get the site running in tip-top condition with the latest Drupal 9 etc.
  8. Do the Drupal 10 upgrade.

Then we'll have a shiny Drupal 10 install, ready for the next few years of security patching.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web