Apr 07 2020
Apr 07

In the official Coder Sniffer install guide on Drupal.org, it recommends installing Coder and the Drupal code sniffs globally using the command:

composer global require drupal/coder

I don't particularly like doing that, because I try to encapsulate all project requirements within that project, especially since I'm often working on numerous projects, some on different versions of PHP or Drupal, and installing things globally can cause things to break.

So instead, I've done the following (see this issue) for my new JeffGeerling.com Drupal 8 site codebase (in case you haven't seen, I'm live-streaming the entire migration process!):

  1. Install the Coder module as a dev requirement, along with the Composer installer for PHP_CodeSniffer coding standards: composer require --dev drupal/coder dealerdirect/phpcodesniffer-composer-installer
  2. Verify the installation is working: ./vendor/bin/phpcs -i (should list Drupal and DrupalPractice in its output).

Once it's working, you can start using phpcs which was installed into the vendor/bin directory:

./vendor/bin/phpcs \
  --standard="Drupal,DrupalPractice" -n \
  --extensions="php,module,inc,install,test,profile,theme" \
  web/themes/jeffgeerling \

GitHub Actions - PHPCS coding standards Coder review for Drupal

I also added a build stage to my site's GitHub Actions CI workflow which runs phpcs using the above command, to make sure my code is always up to snuff. You can see my phpcs GitHub Actions configuration here.

Happy coding!

Mar 13 2020
Mar 13

tl;dr: Docker's default bind mount performance for projects requiring lots of I/O on macOS is abysmal. It's acceptable (but still very slow) if you use the cached or delegated option. But it's actually fairly performant using the barely-documented NFS option!

Ever since Docker for Mac was released, shared volume performance has been a major pain point. It was painfully slow, and the community finally got a cached mode that offered a 20-30x speedup for common disk access patterns around 2017. Since then, the File system performance improvements issue has been a common place to gripe about the lack of improvements to the underlying osxfs filesystem.

Since around 2016, support has been around (albeit barely documented) for NFS volumes in Docker (see Docker local volume driver-specific options).

As part of my site migration project, I've been testing different local development environments, and as a subset of that testing, I decided to test different volume/sync options for Docker to see what's the fastest—and easiest—to configure and maintain.

Before I drone on any further, here are some benchmarks:

Time to install Drupal 8 - different Docker volume sync methods

Time to first Drupal 8 page load - different Docker volume sync methods

Benchmarks explained

The first benchmark installs Drupal, using the JeffGeerling.com codebase. The operation requires loading thousands of code files from the shared volume, writes a number of files back to the filesystem (code, generated templates, and some media assets), and does a decent amount of database work. The database is stored on a separate Docker volume, and not shared, so it is plenty fast on its own (and doesn't affect the results).

The second benchmark loads the home page (/) immediately after the installation; this page load is entirely uncached, so Drupal again reads all the thousands of files from the filesystem and loads them into PHP's opcache, then finishes its operations.

Both benchmarks were run four times, and nothing else was open on my 2016 MacBook Pro while running the benchmarks.

Using the different sync methods


To use NFS, I had to do the following (note: this was on macOS Catalina—other systems and macOS major versions may require modifications):

I edited my Mac's NFS exports file (which was initially empty):

sudo nano /etc/exports

I added the following line (to allow sharing any directories in the Home directory—under older macOS versions, this would be /Users instead):

/System/Volumes/Data -alldirs -mapall=501:20 localhost

(When I saved the file macOS popped a permissions prompt which I had to accept to allow Terminal access to write to this file.)

I also edited my NFS config file:

sudo nano /etc/nfs.conf

I added the following line (to tell the NFS daemon to allow connections from any port—this is required otherwise Docker's NFS connections may be blocked):

nfs.server.mount.require_resv_port = 0

Then I restarted nfsd so the changes would take effect:

sudo nfsd restart

Then, to make sure my Docker Compose service could use an NFS-mounted volume, I added the following to my docker-compose.yml:

version: '3'

      - 'nfsmount:/var/www/html'

    driver: local
      type: nfs
      o: addr=host.docker.internal,rw,nolock,hard,nointr,nfsvers=3
      device: ":${PWD}"

Note that I have my project in ~/Sites, which is covered under the /System/Volumes/Data umbrella... for older macOS versions you would use /Users instead, and for locations outside of your home directory, you have to grant 'Full Disk Access' in the privacy system preference pane to nfsd.

Some of this info I picked up from this gist and it's comments, especially the comment from egobude about the changes required for Catalina.

So, for NFS, there are a few annoying steps, like having to manually add an entry to your /etc/exports, modify the NFS configuration, and restart NFS. But at least on macOS, everything is built-in, and you don't have to install anything extra, or run any extra containers to be able to get the performance benefit.


docker-sync is a Ruby gem (installed via gem install docker-sync) which requires an additional configuration file (docker-sync.yml) alongside your docker-compose.yml file, which then requires you to start docker-sync prior to starting your docker-compose setup. It also ships with extra wrapper functions that can do it all for you, but overall, it felt a bit annoying to have to manage a 2nd tool on top of Docker itself in order to get syncing working.

It also took almost two minutes (with CPU at full bore) the first time I started the environment for an initial sync of all my local files into the volume docker-sync created that was mounted into my Drupal container.

It was faster for most operations (sometimes by 2x) than NFS (which was 2-3x faster than :cached/:delegated), but for some reason the initial Drupal install was actually a few seconds slower than NFS. Not sure the reason, but might have to do with the way unison sync works.

docker bg-sync

bg-sync is a container that syncs files between two directories. For my Drupal site, since there are almost 40,000 files (I know... that's Drupal for you), I had to give this container privileged access (which I'm leery of doing in general, even though I trust bg-sync's maintainer).

It works with a volume shared from your Mac to it, then it syncs the data from there into your destination container using a separate (faster) local volume. The configuration is a little clunky (IMO), and requires some differences between Compose v2 and v3 formats, but it felt a little cleaner to manage than docker-sync, because I didn't have to install a rubygem and start a separate process—instead, all the configuration is managed inside my docker-compose.yml file.

bg-sync offered around the same performance as docker-sync (they both use the same unison-based sync method, so that's not a surprise), though for some reason, the initial sync took closer to three minutes, which was a bit annoying.


I wanted to write this post after spending a few hours testing all these different volume mount and sync tools, because so many of the guides I've found online are either written for older macOS versions or are otherwise unreliable.

In the end, I've decided to stick to using an NFS volume for my personal project, because it offers nearly native performance (certainly a major improvement over the Docker for Mac osxfs filesystem), is not difficult to configure, and doesn't require any extra utilities or major configuration changes in my project.

What about Linux?

I'm glad you asked! I use the exact same Docker Compose config for Linux—all the NFS configuration is stored in a docker-compose.override.yml file I use for my Mac. For Linux, since normal bind mounts offer native performance already (Docker for Linux doesn't use a slow translation layer like osxfs on macOS), I have a separate docker-compose.override.yml file which configures a standard shared volume.

And in production, I bake my complete Docker image (with the codebase inside the image)—I don't use a shared volume at all.

Mar 11 2020
Mar 11

DrupalCon Minneapolis is two months away, and that means it's time for the 2020 Drupal Local Development Survey.

2019 results - Local Drupal development environments
Local development environment usage results from 2019's survey.

If you do any Drupal development work, no matter how much or how little, we would love to hear from you. This survey is not attached to any Drupal organization, it is simply a community survey to help highlight some of the most widely-used tools that Drupalists use for their projects.

Take the 2020 Drupal Local Development Survey

The results will be released around DrupalCon Minneapolis and discussed at the panel: Local development environments panel discussion.

See results from previous years:

Feb 04 2020
Feb 04

Drupal 8 Live migration YouTube series image for JeffGeerling.com

This website is currently (as of February 2020) running on Drupal 7. Drupal 8 was released in November 2015—half a decade ago. Drupal 7 support has been extremely long-lived, as it will not be end-of-life'd until November 2021. As with all software, once it is out of date, and security patches are no longer provided, it becomes harder to ensure the software is secure, much less running well on the latest servers and PHP versions!

Therefore, I decided it was time to start migrating JeffGeerling.com to Drupal 8. And I figured instead of fumbling through the process all by myself, and maybe posting a couple blog posts about the process at the end, I'd adopt a new mantra: Let's fail together! (Just kidding—sorta.)

So for this migration, I'm going to be live-streaming the entire process, end-to-end! Every Tuesday (starting today, February 4, 2020), at 10 a.m. US Central time (4 p.m. UTC), I will be live streaming part of the migration process for one hour. The videos will all be visible after the fact on my YouTube account.

There are a few caveats to get out of the way going into this project:

  • This is the first time doing any Drupal 8 migration work since Drupal 8.2, in 2016. A lot has changed in Drupal 8 since then, so I'll be learning some of this as I go—I figure some people might enjoy seeing how I learn new processes and techniques (hint: lots of Google).
  • This is the first Drupal 8 work I've worked on since upgrading the theme of the Raspberry Pi Dramble website in late 2018, so my Drupal 8 is a little rusty as well.
  • This is the first time I've ever tried live-streaming, and I'm doing it on a four year old 13" MacBook Pro; the CPU might not be up to the task at times, but we'll see how it holds up!

If you want to follow along as it happens, please subscribe to my YouTube channel and sign up for notifications—that way YouTube can notify you when a live stream is about to begin (I will try to remember to have them pre-scheduled every Tuesday). Otherwise, I will also be embedding all the live streams after the fact on this blog post (so check back again later to see them if you miss a week!).

Supporting this project

I'm not being paid or sponsored for any of the work involved in producing these videos; if you like them, and have the ability, please consider sponsoring my work on one of the following platforms:

If I get enough, maybe I can afford a faster computer and stream with a better frame rate :-)

Episode 1 - Setting up a new Drupal project, installing Drupal the first time

Streamed on: February 4, 2020

[embedded content]

Summary: I created a new GitHub repository called jeffgeerling-com, then I created a new Drupal 8 codebase. I added a .gitignore file since Drupal's composer template doesn't currently ship with one, and made sure to not add the vendor directory or other Composer-managed files to the Git codebase. I committed all the code, and pushed it to the master branch of the jeffgeerling-com repository on GitHub. Then I created a simple Docker image for local development, brought up a local environment using Docker Compose (with a Drupal/Apache/PHP container and a MySQL container), and finally installed Drupal using drush site:install. I logged in and verified the site is working correctly (if a bit sparse—I installed the minimal profile!).


Episode 2 - Getting the Configuration and Content Migration Ready

Streamed on: February 11, 2020

[embedded content]

Summary: I downloaded the Upgrade Status module and went through which modules did or did not have upgrade paths for Drupal 8. I found that most modules I use are now in Drupal core, and most of the contributed modules I use have a stable Drupal 8 version available. There are a couple modules that might need some special tweaks to make sure I don't lose data—like making sure Redirects are migrated correctly from Drupal 7 to Drupal 8—but most things should just work in Drupal 8 without much effort. I then read through the Upgrading from Drupal 6 or 7 to Drupal 8 documentation guide, and chose to use Drush for the migration. So I installed the appropriate modules, and tried connecting my site to the legacy Drupal 7 database by adding a database to the $databases array in the Drupal 8 site's settings.php file. Unfortunately, I was unable to get the Drupal 8 site to see the Drupal 7 database before the episode's time ran out, so we'll get that working in the next episode, and run the drush migrate-upgrade command for the first time!


Episode 3 - Fixing bugs, exporting configuration, and running the first migrations

Streamed on: February 18, 2020

[embedded content]

Summary: After figuring out we had to use docker.for.mac.localhost as the hostname to connect our new Drupal 8 site to the old Drupal 7 site's MySQL database, we were able to run the drush migrate-upgrade command! But then it failed, because Drush 10 is currently incompatible with migrate-upgrade. So we downgraded to Drush 9, then got migrate-upgrade to work! Once it was done, we were able to see all the available Drupal 7 to 8 migrations, and run them, but we ran into a few errors. We used these instructions to export the site's configuration to the codebase, and pushed the changes to the site's git repository. Then we reinstalled the site from scratch, to prove the configuration export and import worked well, and ran the migrations again, running into a new error, "Field discovery failed for Drupal core version 7," which we'll debug on next week's episode!

Episode 4 - Setting up CI with GitHub Actions and Overcoming Field Migration Issues

Streamed on: February 25, 2020

[embedded content]

Summary: We started off by setting up GitHub Actions for some basic Continuous Integration (CI) tests for Drupal, but quickly ran into a wall as GitHub experienced an outage and the Issue queue and Actions stopped working. So we switched back over to the migrations and found that all the issues like Attempt to create a field storage field_project_images with no type. meant that a field-related module was not enabled in Drupal 8. So we enabled the right modules (e.g. Date, Link, Taxonomy, Comment, Options, etc.), and ran the migrate-upgrade command again... which started generating duplicate migration entities, named like upgrade_upgrade_upgrade_d7_thing.yml. So to prevent confusion, I deleted all migration configuration, reinstalled the site, re-exported the configuration, then ran migrate-upgrade yet again. It now created the migration entities with a single upgrade_ prefix, which is much nicer to manage. I kicked off the migrations, and it got further along, and started migrating images (which took some time, as it had to download each image, save it to the Docker shared filesystem (which is slow) and create the file entity in the database (which is slow). So we cut off the episode there and will get back into running the migration again next week!

Episode 5 - Migrations (mostly) working, setting up CI on GitHub Actions

Streamed on: March 3, 2020

[embedded content]

Summary: We installed a few new modules and re-ran a few migrations to try to get content to work correctly in the Drupal 8 site after it was migrated. GitHub was actually working today, so we were able to work on the CI process to install the site from configuration on every commit, and we now have some content ready for theming next week! Tune in next week to see how we start upgrading the theme from Drupal 7 to 8 with Twig, and see if Jeff could ever figure out what was causing the Drupal install to fail on GitHub Actions!


Episode 6 - Setting up admin theme and jeffgeerling theme

Streamed on: March 10, 2020

[embedded content]

Summary: We set up an admin theme after discussing Seven and Claro. We added the Admin Toolbar module for a nicer admin UI experience. Then we copied the 'jeffgeerling' theme over from my Drupal 7 site to Drupal 8, and started updating the theme components like the .info.yml and CSS libraries to work with Drupal 8! GitHub Actions is running now, so every time we push a new commit it is tested in a CI environment, yay!


Episode 7 - Faster local dev environment, coronavirus, and theming the blog

Streaming on: March 17, 2020

[embedded content]

Summary: We mentioned the importance of coming together during the strange (and sometimes lonely!) times we live in with coronavirus and quarantines. We summarized a few things worked on between last episode and today, like a faster local development environment (trying Drupal VM, DDEV Local, and Symfony Local Server too!). We also ran through some pragmatic choices with regard to switching modules in Drupal 8 (and why I chose to stick with the older but working code filter module). Finally, we spent a half an hour working on making blog posts look the same on the Drupal 8 site as the Drupal 7 site.


Episode 8 - Migrating the Blog view from Drupal 7 to Drupal 8

Streaming on: March 24, 2020

[embedded content]

Summary: Following the weekly livestream agenda, we manually migrated the Blog view (which included a Blog landing page, a blog.xml RSS feed, and a 'Recent Blog Posts' block) from Drupal 7 to Drupal 8. Then we worked on styling and theming the Blog landing page, incorporating a theme hook suggestion to add a 'visually-hidden' class to the Blog landing page's title so it would not display but would still be present for those using screen readers.

we upgraded Drupal core from 8.8.2 to 8.8.4 using Composer. We also discussed some migration struggles with the naming of the database connection key (hint: it should be 'migrate'). Then we worked on theming the blog comments, and upgrading a template_preprocess_hook() function from Drupal 7 to Drupal 8.

Episode 9 - TBD

Streaming on: March 31, 2020

[embedded content]

Summary: Following the weekly livestream agenda, we upgraded Drupal core from 8.8.2 to 8.8.4 using Composer. We also discussed some migration struggles with the naming of the database connection key (hint: it should be 'migrate'). Then we worked on theming the blog comments, and upgrading a template_preprocess_hook() function from Drupal 7 to Drupal 8.


Episode 10 - Projects view and home page blocks migration

Streaming on: April 7, 2020

[embedded content]

Summary: We mentioned the #DrupalCares effort to sustain the Drupal Association, we showed a brief clip of the 2007 rendering of the Drupal Song (https://www.youtube.com/watch?v=lZ-s3DRZJKY), we migrated the Projects listing, block, and feed from a View in Drupal 7 to Drupal 8, and we worked on finishing the Home Page migration from Drupal 7 to Drupal 8.


Episode 11 - TBD

Streaming on: April 14, 2020

[embedded content]

Summary: TBD.

Jan 29 2020
Jan 29

tl;dr: Subscribe to my YouTube Channel; I'm going to start migrating this website to Drupal 8 on a livestream every Tuesday at 10 a.m. US Central (3 p.m. UTC).

Ever since Drupal 8 was released, I've been waffling on the decision to migrate/upgrade this website (JeffGeerling.com) to Drupal 8. The site started off years ago as a static HTML site generated by Thingamablog, a really old Java-based static blog generator.

In the years since, I migrated from Thingamablog to Drupal 6, and from Drupal 6 to Drupal 7. Each of these migrations also incorporated a complete redesign, and I did another semi-redesign halfway through the Drupal 7 lifecycle, to the design you see today:

JeffGeerling.com - dark mode in 2020 in Drupal 7
Dark mode ftw!

I've written a bit about Drupal 8's successes and failures on this blog, and I'm still undecided about how I see Drupal's long-term success as a platform—especially for simpler sites like mine, which are not über-huge monstrosities with hundreds of content types, taxonomies, media styles, etc.

But I do see this site as a decent fit for Drupal 8's content model, and there is an upgrade path for every feature I currently use—though some might be a little different in implementation than Drupal 7!

Starting on Tuesday, February 4th, at 10 a.m. US Central (3 p.m. UTC), I will be live-streaming the migration of JeffGeerling.com from Drupal 7 to Drupal 8. Make sure you subscribe to my YouTube channel so you can get notified when the streams begin. I don't know how long the whole process will take, but hopefully after a couple months of hour-long sessions I'll have something to show for it :)

The first stream, my plan is to get the new Drupal 8 project set up on GitHub, build out the very rough initial Drupal 8 site, set up some form of CI so I can have at least basic tests for the migration, and then see where to go from there. Future streams will work on building the new site's structure (content types, mostly), writing and testing the content migrations, upgrading the theme from PHPTemplate to Twig, and incorporating Hosted Apache Solr for site search.

The first live stream will be on YouTube, or you can watch it here:

[embedded content]

Dec 03 2019
Dec 03

PHP 7.4.0 running on Drupal VM with Drupal 8's status report page

Drupal VM 5.1.0 was just released (release name Recognizer), and the main feature is PHP 7.4 support; you can now begin running and testing your Drupal sites under PHP 7.4 to check for any incompatibilities.

PHP 7.4 includes some new features like typed properties, arrow functions, and opcache preloading which could help with certain types of code or site deployments (I'm interested to see if opcache preloading could help the startup time of Drupal inside container environments like Kubernetes!).

And with the release of PHP 7.4, PHP 7.1 support was dropped—thus Drupal VM 5.1 is the first version to completely drop support for PHP 5.6, 7.0, and 7.1. In the past, it was possible (though not recommended or supported) to install these older PHP versions. As of Drupal VM 5.1 it is no longer possible. So if you need to still migrate some code from an ancient codebase running on PHP 5.6 or some other unsupported version of PHP, stick with Drupal VM 5.0 until that's done.

There are a few other small bugfixes and compatibility updates in Drupal VM 5.1 (see the CHANGELOG for details), but the headline feature is PHP 7.4 support. Go check it out at DrupalVM.com!

Nov 26 2019
Nov 26

I realized I haven't posted about my DrupalCon Seattle 2019 session titled Everything I know about Kubernetes I learned from a cluster of Raspberry Pis, so I thought I'd remedy that. First, here's a video of the recorded session:

[embedded content]

The original Raspberry Pi Dramble Cluster
The original Pi Dramble 6-node cluster, running the LAMP stack.

I started running the Raspberry Pi Dramble in 2014, after I realized I could automate the setup of everything in a LAMP stack on a set of Raspberry Pi 2s using Ansible (one Pi for an HTTP load balancer/reverse proxy, two for PHP app backends, and two for MySQL redundancy. Kubernetes was the logical next step, so I moved things towards Kubernetes in 2017, but running Kubernetes was a lesson in pain due to the Pi's limited memory (1 GB maximum) at the time.

When the Raspberry Pi 4 came around, I acquired some 2 GB models as quickly as I could, and redeployed onto them. Gone were the restrictions that were causing Kubernetes' API to be flaky with the older Pis, and now the Pi 4 cluster is extremely reliable. Early on, cooling was an issue, but the recent firmware update has made that less problematic. I now power the cluster using the official PoE HAT, which means I only have one plug for each Pi, and everything fits nicely into a small travel case (if I want to bring the cluster with me anywhere).

Raspberry Pi Dramble 4 Kubernetes edition cluster
The current Pi Dramble, running Kubernetes and sporting BlinkStick Nanos.

I even got it all to run off a 10,000 mAh battery pack with a bunch of USB splitters... but it did not stay powered long, and kept giving low power warnings—so I'll have to consider other options for a highly-mobile four-node Kubernetes bare-metal cluster.

I've continually updated the cluster so it is tested in a Docker-based Kubernetes environment, a Vagrant-based Kubernetes local development environment, and of course, the Pi environment. The latter environment causes much consternation, as many common container images are not maintained in an armv6 or linux/arm format, which is required when running on the 32-bit ARM OS the Pi uses, Raspbian. But the Pi Dramble abides, and it quietly goes on, serving up traffic for https://www.pidramble.com through the years.

The official Pi Dramble Wiki has all the instructions for building your own Pi Kubernetes cluster, with links to buy all the parts I have, along with every step to get it running using open source Ansible roles to install Kubernetes and Docker for ARM, then configure a new four-node cluster.

Nov 07 2019
Nov 07

tl;dr: You can now sponsor my open source development work via GitHub Sponsors.

GitHub sponsors geerlingguy

GitHub Sponsors is the latest foray into building a more sustainable future for open source software development. There have been many attempts before, a few of which I tried (Gratipay, Patreon, etc.), but most of them never reached a critical mass, and at most you'd end up getting maybe $20-50/month out of the platform. Another prolific open source contributor I've long followed wrote about the topic of open source support and developer burnout in a post this year, Webform, Drupal, and Open Source...Where are we going?.

For a job (software development) where that's what someone makes in one hour, it's not worth putting in the hours to try to promote ongoing sponsorship when you'd get back only a few hours' worth of pay in a year. I also don't have an explicit goal of monetizing my work, so I often feel guilty even asking about monetary support. On the flip side, I would be more likely to persist through some of the stress of maintainership if I knew it was helping support my growing family.

I have long been involved in the Drupal and Ansible communities, and have submitted PRs, documentation fixes, and code reviews to hundreds of open source projects my work has touched. And if I could find a way to sustain some of my financial needs through open source work, I'd love to devote more time to:

  • Maintaining and improving Drupal VM, and modules like Honeypot (both of which are still seeing increased usage, but have been more or less in maintenance mode of late).
  • Improving my suite of Ansible roles, and building new Collections, to help solve many automation pain points I've encountered with Kubernetes, Drupal, PHP, and other applications.
  • Continuing revisions of my book Ansible for DevOps for another four years (at least!).
  • Attending more Drupal events in person (I've had to skip most of the camps I could've attended this year since I'm no longer sponsored by a particularly-large Drupal company).

I've had a Patreon account for some time, and it's good enough for a fast food meal or two every month, but I think Patreon is geared more towards the 'creative' community (YouTubers, podcasting, art, etc.). Maybe GitHub Sponsors will do better for open source contributors... and maybe not.

I'd be humbled and grateful if you could support me (geerlingguy) on GitHub Sponsors.

Aug 09 2019
Aug 09

Kubernetes is taking over the world of infrastructure management, at least for larger-scale operations, and best practices have started to solidify. One of those best practices is the cultivation of Custom Resource Definitions (CRDs) to describe your applications in a Kubernetes-native way, and Operators to manage your the Custom Resources running on your Kubernetes clusters.

In the Drupal community, Kubernetes uptake has been somewhat slow, but is on the rise. Just like with Docker adoption for local development, the tooling and documentation has been slowly percolating. For example, Tess Flynn from TEN7 has been boldly going where no one has gone before (oops, wrong scifi series!) using the Force to promote Drupal usage in a Kubernetes environment.

I've also moved my Raspberry Pi Dramble (a cluster of Raspberry Pi computers running a Drupal 8 site) from a standard LAMP stack cluster to Kubernetes, and I've been blogging about that journey here and there, even taking the Dramble on a tour, presenting Everything I know about Kubernetes I learned from a cluster of Raspberry Pis.

But so far, all the Drupal organizations using Kubernetes are 'out there', doing their own thing, often using a more traditional approach to the management of Drupal containers in a Kubernetes cluster.

In mid-2018, after some informal meetings at DrupalCon and between various interested parties online, a few of the kind folks at Drud Technologies (behind DDEV) helped form the sig-drupal Drupal on Kubernetes Special Interest Group. We sometimes chat in the #kubernetes in the Drupal Slack, and there are biweekly meetings on Wednesdays.

One of the things that a few different members of the community have been working on (mostly informally, for now) is a Drupal Operator for Kubernetes.

I'm Jeff, I'll be your operator

Tank being the operator in the Matrix

First, even if you've been using Kubernetes for a while, you might not know much about Operators. You might use one to manage something that you copied and pasted into your cluster for logging, metrics, or a service mesh component, but what is an Operator, and why would you be interested in making your own?

And, to me, the million dollar question is: will you need to know Go to write an Operator? After all, most things in the Kubernetes ecosystem seem to be Go-based...

Well, first, an Operator is a Kubernetes 'controller' that watches over specific Custom Resources (which follow your Custom Resource Definition), and makes sure the Custom Resources are configured and running as they should be, based on the Custom Resource metadata and spec.

So if you want a widget with a fizz equal to buzz:

     apiVersion: widget.example.com/v1alpha1
     kind: Widget
       name: my-special-widget
       fizz: buzz

Then it is the responsibility of the operator to make it so. It could do that in a variety of ways. You could write some Go to interact with the Kubernetes API and run the proper containers with the proper configuration (be that a ConfigMap with a Deployment, or a StatefulSet, or whatever it needs to be).

So your Operator might say "ah, the fizz for my-special-widget is currently set to foo, but it should be buzz. Now I will change that foo to a buzz, update my application's database schema to reflect the change, and notify some external service of the fizz change."

Once the change is made, the Operator goes back into it's ever-watchful state, waiting for another change, and also ensuring the Custom Resource it's watching is running as it should be (with the help of the Kubernetes API).

When you delete the Custom Resource, the Operator can even take a final backup of your special widget, and maybe notify an external service that your special widget is no longer very special.

Ansible Operator SDK

There are a number of different ways you can build and maintain Operators; many of them require you (and anyone you ever want to help maintain the operator) to program in Go. Go is a great programming language, and it's not very hard to learn. But it is usually not the same programming language you use for the applications you regularly develop, so it would be nice to have an alternative option besides learning an entire new programming language.

That's where Ansible Operator SDK comes in! (See also: the full developer guide.)

Ansible has had great support for managing Kubernetes cluster resources for some time, mostly via its k8s and k8s_facts modules. For example, in an Ansible playbook, you can add or modify a Kubernetes resource with the following:

- name: Create a Service object from an inline definition
    state: present
      apiVersion: v1
      kind: Service
        name: mysite
        namespace: mysite
          app: mysite
          service: drupal
          app: mysite
          service: drupal
        - protocol: TCP
          targetPort: 80
          name: port-80-tcp
          port: 80

You can also use powerful Jinja2 templating and Ansible variables along with separate Kubernetes YAML files to manage cluster resources. And Ansible can go much further, because you have thousands of other modules which can be used to integrate external services (e.g. you can have the same Ansible operator playbook manage an external AWS RDS/Aurora database cluster, interact with a chat client like Slack, and update your website monitoring service, all with very easy-to-define syntax.

Drupal Operator, built with the Operator SDK

To explore how Ansible Operator SDK can make managing a Drupal Operator easy, I forked Thom Toogood's original PoC Drupal Operator, then rebuilt it with the latest version of the Operator SDK. My fork of the Drupal Operator is currently structured to run a Drupal site with a codebase like the one in the official Docker library image for Drupal, but that is subject to change, because the way that image is built is not (IMHO) the way you should build a modern best-practices Drupal site codebase (for more on that, see my Drupal for Kubernetes project!).

If you have a Kubernetes cluster running (you can get one quickly by Installing minikube and running minikube start), it's easy to get started:

  1. Deploy Drupal Operator into your cluster:

     kubectl apply -f https://raw.githubusercontent.com/geerlingguy/drupal-operator/master/deploy/drupal-operator.yaml
  2. Create a file named my-drupal-site.yml with the following contents:

     apiVersion: drupal.drupal.org/v1alpha1
     kind: Drupal
       name: my-drupal-site
       namespace: default
       drupal_image: 'drupal:8.7-apache'
       # You should generate your own hash salt, e.g. `Crypt::randomBytesBase64(55)`.
       drupal_hash_salt: 'provide hash_salt here'
       drupal_trusted_host_patterns: 'provide trusted_host_patterns here'
       files_pvc_size: 1Gi
       manage_database: true
       database_image: mariadb:10
       database_pvc_size: 1Gi
  3. Use kubectl to create the site in your cluster:

     kubectl apply -f my-drupal-site.yml

You should now have a Drupal site running inside the cluster, accessible via a NodePort on any of your Kubernetes servers' IP address (use kubectl get service my-drupal-site to see details). In the future, this Drupal Operator will also support Ingress (instead of access via NodePort), among many other features.

Since the Operator uses a Persistent Volume (PV) for the Database and Drupal files directory, you can upgrade container image versions (to run the latest version of Drupal core) without losing your Drupal site data. Currently, upgrading the site or performing other types of site maintenance is done manually via the web interface (e.g. via /update.php), but eventually the Operator will manage many site maintenance tasks automatically via Drush.

Building your own Operator

This Drupal Operator is still a 'proof of concept', and will likely see many changes over the next year or two before it is more of a stable solution for running your sites in production.

But that doesn't mean you shouldn't start building your own Operators to manage your Kubernetes resources right now! You can build your own operator quickly and easily with the Operator SDK and ansible-operator following the Operator SDK Quick Start guide. My directions assume you're using a Mac:

  1. Install Operator SDK CLI: brew install operator-sdk
  2. Activate Go's module mode: export GO111MODULE=on
  3. Create the new Operator: operator-sdk new my-app --api-version=myapp.example.com/v1alpha1 --kind=MyApp --type=ansible
  4. Change into the Operator directory and try testing the basic components using Molecule:
    1. pip install docker molecule openshift (if you don't already have these installed)
    2. molecule test -s test-local
  5. Those tests should pass, and at this point you have a barebones, but working, Operator.

Note: If you want to be able to create MyApp instances in any namespace in your cluster, you should make sure the Operator is cluster-scoped (see the Operator scope documentation). Otherwise the Operator can only be deployed and used in one particular namespace at a time. This could be desireable in some scenarios.

At this point, I would commit all the files to a newly-initialized Git repository for the Operator project (git init && git add -A && git commit -m "Initial commit."), then start working on making the Operator do exactly what I want.

I plan on writing more on Operators and Kubernetes cluster management in my upcoming book, Ansible for Kubernetes. You can sign up to be notified when the book is released on LeanPub.

Jun 27 2019
Jun 27

Since 2014, I've been working for Acquia, doing some fun work with a great team in Professional Services. I started out managing some huge Drupal site builds for Acquia clients, and ended up devoting all my time for the past couple years to some major infrastructure projects, diving deeper into operations work, Ansible, AWS, Docker, and Kubernetes in production.

In that same time period, I began work on my second book, Ansible for Kubernetes, but have not had the dedicated time to get too deep into writing—especially now that I have three young kids. When I started writing Ansible for DevOps, I had one newborn!

Also, during my time at Acquia I encountered major health issues relating to my Crohn's disease, culminating in a major surgery and a new permanent friend. I am extremely grateful to have had great benefits and understanding colleagues and managers at Acquia, in addition to a remote job that granted me flexibility during some of the worst weeks, health-wise.

For the past year, since having that surgery, my health has improved dramatically, and for the first time in years, I started considering diving into a more entrepreneurial career path. I've been running Midwestern Mac, LLC since 2008, managing SaaS products like Hosted Apache Solr and Server Check.in (both sites are still running on Drupal 7, with tons of backend microservices!), and I've always wanted to grow those products.

But doing so—and writing a book, and managing a growing family—with a full time job has been nearly impossible. Oh, also managing over 200 open source projects with over 10k forks and 20k stars (I'm sorry if I've neglected reviewing your PR for a while!).

The point of this is, I'm going to devote myself to some new ventures, focusing on a few consulting projects (mostly infrastructure-related, probably lots of Kubernetes and Ansible—more to come ?), my writing, and Midwestern Mac's SaaS products.

What does this mean for my Drupal work and contributions? Well, that's mostly TBD; currently all the sites I run personally (including this blog) are running on Drupal 7. It might or might not make sense to upgrade to Drupal 8 or Drupal 9, and depending on that I'm not sure how much involvement I'll continue to have in the Drupal community. A lot depends on where my future projects take me!

One thing is certain, though: I'm going to make an effort to get at least the first few chapters of Ansible for Kubernetes published and available on LeanPub by the end of July, and I hope to attend AnsibleFest Atlanta this September!

Apr 16 2019
Apr 16

I'm on the flight home from this year's North American DrupalCon. Couldn't sleep, so thought I'd jot down a few words after a great experience in Seattle.

Last year, some remember seeing me walking the halls in Nashville akin to a zombie. But not the hungry, flesh-eating kind... more like the thin, scraggly, zoned-out kind. Last year my health was very poor. I went to DrupalCon mostly because it was the first DrupalCon within driving distance of St. Louis since DrupalCon Chicago several years ago. In hindsight it might not have been the best idea, and I had to skip a number of events due to my health.

Since that time, I experienced a grueling surgery and recovery, and learned to live with my new friend, the stoma. (Warning: scatalogical humor ahead—hey, it's my coping mechanism!).

After surgery, my health has rebounded. For the first time in 16 years, I am not taking any medications (that, after years of taking medications that would be billed to insurance sometimes at over a hundred thousand dollars per year)! I went from a paltry 144 lbs to a healthy new normal of 165. (I used to hover around 170... apparently a colon weighs something like five pounds ?).

This year I wanted to spend more time than ever focusing on people. One of my little hobbies at DrupalCon—mostly since my face and name are often recognized and so it's hard to do—is to try to talk with at least two or three people who have never heard of me each day, and spread some of the infectious joy and passion that I see in good open source communities.

Drupal's community is not perfect. Far from it. We've been bruised, beat up, and we're still flailing about here and there as we resolve internal and external conflicts.

But there's a name for an imperfect group of people who come together and work things out, and still show compassion and kindness for each other (even in cases of severe disagreement): a family. And Drupal being the first major open source community I encountered—really, the reason why I dove headfirst into open sourcing practically everything I do—DrupalCon is kind of like my family reunion.

So this year I started asking people who have made some personal impact on my Drupal journey to take a picture with me. There are not nearly enough hours in the day to get everyone who has helped me in the Drupal community to stand for a few seconds for a photo, but I did overcome my shyness to ask a few people through the week:

Jeff with People at DrupalCon Seattle 2019

In the images above (again, these are just a few of the people who have had a personal impact on my Drupal community life in one way or another—I am indebted to so many more!):

Everyone I know in the Drupal community (including those who never get to show up at an event like DrupalCon) has stories of 'that time so-and-so helped me get out of a bind' or when 'maintainer such-and-such was extra patient and helped me resolve this issue'. There's a reason our unofficial slogan is come for the code, stay for the community. Even in the midst of the sea change brought on by Drupal 8, the community (whether local, national, or international) persists. But this community is fragile, and fractures have had a chance to form over the past few years. I really hope that those fractures can be patched over soon.

Rejoining the Photography Team

2019 marks my return to joining the volunteer photography team, led by Susanne Coates (susannecoates), Paul Johnston (pdjohnson), and Michael Cannon (comprock). Last year there was no chance I could lug around even a small lens for a day. This year I changed that dramatically by lugging both my monster 14-24mm f/2.8 lens (used to take the 2019 DrupalCon Seattle group photo) and my 70-200mm f/2.8. Oh, and also my little Sony a6000 which I used for more candid shots!

Nikon D750 with 70-200mm 2.8 VR and Sigma Art 14-24 2.8 and Sony a6000 with 30mm 1.4

Taking photos is a fun (but crazy expensive ?) hobby for me, and I am happy to share my photo equipment and time to help contribute images the Drupal community can use to remember the event forever. Heck, this year I even smiled sometimes!

Jeff Geerling directing DrupalCon attendees to move to their right for the group photo

(Special thanks to Hussain Abbas for taking the above photo!)

And the final group photo was well worth lugging the 5 lb 14-24 lens around for the entire trip:

DrupalCon Seattle 2019 Group Photo

Through some stroke of luck, I was also able to try out a camera that I'm strongly considering purchasing in the future—at least one variant of it—the Nikon Z7. And not just any Z7, but Dries' Z7, at the Acquia pre-Con party at Spin Seattle:

Dries' Nikon Z7 with 24-70mm Nikon 2.8

He mounted the venerable Nikon 24-70mm f/2.8 G lens by way of a Z-to-F mount adapter, and though it masks the compact nature of the Z-series, a good lens like that showcases the excellent quality of the new mirrorless cameras. It looks like Dries has been making good use of the camera too, at least if these few samples are an indication.

I remember images vividly, and they are the best reminder of the times I've had at an event like DrupalCon. I hope to continue contributing photos at DrupalCon and regional Camps I attend for a very long time! Thanks to all the other members of the photography team, especially the 'shepherds' listed above who helped coordinate a complex, multi-location, multi-day event.

Sessions - Kubernetes, Drupal on Raspberry Pi, and Local Development, oh my!

Being on the photography team was not enough, though... I decided to help with not one but two sessions this year. Lucky for me, Chris Urban took the lead on the now-running-for-two-years What Should I Use? 2019 Local Development Survey Results session. The results were a little surprising, to say the least! Speaking of surprise, I was actually late to that session because of calendar issues on my part, so a triple thanks to Chris for taking charge on that and covering for me while I ran up the escalators.

The other session was actually the first time I was able to present on a topic near and dear to my heart at DrupalCon. I chose a 30 minute session because 90 minutes felt too long, though I really wish I could've had 45 or 60 minutes to go into a smidge more depth. But I was able to cram a fun demo and a few gags into my session, and I think people enjoyed it. Well, at least they were laughing at something ?.

You can check that session out here: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis.

DrupalCon Seattle Wednesday Afternoon Schedule with Jeff Geerling's session

I usually have a policy of only doing one session per conference, but as Chris was able to take lead on getting the survey up and running, publicizing it, etc., I was willing to expand that to two. I'm glad I did, it was a lot of fun to work with Chris on that 2nd session!

I also attended two BoFs, one on Acquia BLT, and another on Kubernetes. Last year, my DrupalCon focus was improving the initial onboarding experience in Drupal core. This year I switched tracks and wanted to kick off some new discussion about Drupal, Kubernetes, and containers in production. This is an area that there's not a whole lot of prior knowledge in, and it seems most of the Drupal companies exploring Kubernetes are doing so somewhat independently. Also, there is little standardization in the Docker scene in terms of Drupal (how many CLIs do I have to use now, for various projects I work on? lando, fin, ddev, in addition to drush and all the other CLIs common in web development...), and I hope that we can do better there in the future.

Thoughts on DrupalCon North America

The Drupal Association recently announced that DrupalCon will become a coastal-city-centric event, alternating between the West and East coasts (and the middle of America will simply become 'flyover country' again, at least in terms of DrupalCon). I'm not super pleased about this decision, though I understand the financial and resource-allocation motivations.

But it's one of those things that seems (to me, at least?) like it will reinforce the lack of cultural diversity in the community's geography. Yes, there is a statistically lower attendance when you go to a city where Drupal is not already 'hot'. But there are various cities in Canada and the many, many US states besides MA, CA, WA, VA, and NY which are up-and-coming in the tech world and could be rich new centers of Drupal development. Alas, it is not to be. At least not until 2025 or later, since we already have the schedule for 2020 (Minneapolis), 2021 (Boston), Portland (2022), Pittsburgh (2023), and Portland (2024).

Worse than that, DrupalCon's shift towards more-expensive coastal cities further cuts down on diversity by excluding those who can't afford the monetary cost of attending in these cities. I could not personally justify attending DrupalCon were it not for the generous help of my company, Acquia. And in the past five years I've seen fewer and fewer independent Drupal developers (or developers who primarily work for non-profits and smaller agencies) at DrupalCon.

I've tweeted about this before, but basically, DrupalCon NA seems to have 'jumped the shark' towards an enterprise focus in the past few years. Much of the community seems to have moved on to only attending more intimate (and infinitely more affordable) local or regional Camps. But this has implications for the community, as often (besides sporadic 'dev days' events) the nexus of Drupal Core community work and contribution mentorship—at least in the Americas—is DrupalCon.

And if we exclude more and more groups of people (besides marketers, enterprise, and others who can expense everything on their trip), the Drupal ecosystem will suffer. When I started with Drupal, there were numerous prolific core contributors who worked independently or as part of some non-Drupal-specific entity. Nowadays, it seems the majority of Drupal core work is done by those who are fully sponsored by large Drupal agencies.

While the latter can lead to more focus on specific large-scale initiatives, the former allowed Drupal to flourish more in a variety of use cases, as many other voices (which are absent now) were heard. Maybe I'm being a bit zany here, but I still think Drupal 8/9 has (or at least had?) a shot at being good at more than just 'ambitious digital experiences'.

Anyways, let's move on towards this year's scenic location: Seattle!


Some DrupalCon locations are great for the city's atmosphere (NOLA, Nashville). Others for their geography, or the culture of the area. Seattle has a lot of all these attributes. It's a bit expensive, but unlike some other locations, the geography and natural beauty are worth it! I stayed with relatives in a nice area across Lake Washington, and driving to and from DrupalCon, I was mesmerized by the lakeside geography, and one sneak peek (it was mostly foggy and cloudy... on clearer days it would've been more majestic) at Mt. Rainier.

If I go back, I'm definitely going to make some time to visit Mt. Rainier; I just didn't have the time this trip.

Also, there's plenty of interesting architecture when strolling through parts of downtown. And then there are fun and interesting places like the Pike Place Market, which is kind of a farmers' market on steroids. I know it puts the few we have in St. Louis to shame—though to be fair, we don't have fresh fish we can throw about here... unless you count fresh, muddy catfish!

[embedded content]

And there are some other iconic places in Seattle, including pretty much everything about Elliott Bay:

Ferris wheel at Elliot Bay and docks in Seattle WA

And, of course, the Space Needle:

Seattle WA skyline with space needle, taken at Kerry Park

I tried waiting for a nice day without tons of rain and cloudiness to explore and take more photos... but that wasn't in the cards. Every day was a gloomy mix of half-rain-half-mist. I only barely caught a glimpse of Mt. Rainier on the way to the airport because visibility was not that great in the few daylight hours I was outdoors.

In the end, DrupalCon was a positive experience, and I enjoyed the community most of all. If they could just have Trivia Night and full-scale Contribution days all DrupalCon, I'd do it twice a year—I guess that's kind of how 'Dev Days' are structured, but then I'd also feel obligated to push more code, which feels more like work, rather than enjoy the company of the hundreds of community members I have come to know online.

I hope to be back at DrupalCon next year, in Minneapolis; see you there!

Apr 12 2019
Apr 12

Update: Since posting this, there have been some interesting new developments in this area, for example:

Since 2014, I've been working on various projects which containerized Drupal in a production environment. There have always been a few growing pains—there will for some time, as there are so few places actually using Docker or containers in a production environment (at least in a 'cloud native' way, without tons of volume mounts), though this is changing. It was slow at first, but it's becoming much more rapid.

You might think that Drupal and Docker work together nicely. They definitely can and do, in many cases, as we see with local development environments built around Docker, like Docksal, Ddev, Lando, and even Drupal VM. But local development environments, where the Drupal codebase is basically mounted as a volume into a Docker container that runs the code, differ radically from production, where the goal is to 'contain' as much of production into a stateless container image as possible, so you can scale up, deploy, and debug most efficiently.

If you have to have your Drupal code in one place, and a docker container running elsewhere, you have to manage both separately, and rolling out updates can be challenging (and requires most of the same tools you'd use in old-style Drupal-on-a-normal-server approaches, so there's no streamlining this way).

Throw Kubernetes into the mix—where anything and everything is almost forced to operate as a 12-factor app, and you really have to be on your game in the world of containerized-Drupal-in-production!

12 Factor Apps

You may wonder, "What is a 12 Factor App?" The link to the website above has all the details, but on a very basic level, a 12-factor app is an application which is easy to deploy into scalable, robust, cloud-based environments.

Many applications need things like environment-specific database credentials, a shared filesystem for things like uploaded or generated files shared between multiple instances, and other things which may make deploying in a scalable, flexible way more difficult.

But a 12 Factor App is an application which can do some fancy things which make it easier to deploy and manage:

  • It stores all its configuration in some sort of declarative format, so there aren't manual or non-automatable setup steps.
  • It is not tied too tightly to one type of operating system or a complex tangle of language and environment configurations.
  • It can scale from one to many instances with minimal (ideally zero) intervention.
  • It is identical in almost every way between production and non-production environments.

A 12 Factor App can require database credentials, shared filesystems, etc.—but ideally it can work in ways that minimizes the pain when dealing with such things.

Challenges with Drupal

There are five major challenges to turning a Drupal application or website into a more 'stateless' 12-factor app:

  1. The database must be persistent.
  2. The Drupal files directory must be persistent and shared among all web containers.
  3. Drupal installation and features like CSS and JS aggregation needs to be able to persist data into certain directories inside the Drupal codebase.
  4. Drupal's settings.php and other important configuration is challenging to configure via environment variables.
  5. Drupal's cron job(s) should be run via Drush if you would like to be able to run it in the most efficient, scalable fashion.

With each one of these challenges, you can choose whether to use an easy-to-implement but hard-to-scale solution, or a hard-to-implement but easy-to-scale solution. And in the case of Drupal installation, you can even avoid the problem altogether if you're only ever going to run one Drupal site, and never reinstall it—you could install outside of production, then move some things around, and copy the database into production. (Though you can solve the 'how do I install Drupal in a mostly-stateless environment' problem too; it's just not easy-to-implement).

Meeting the Challenges with Kubernetes and Drupal

I'll walk you through some solutions to these problems, drawing from my experience building the Raspberry Pi Dramble (a 4-node Kubernetes cluster running Drupal), the Drupal website that runs on it (Drupal for Kubernetes) and also running a fairly large Drupal web application for an internal project I work on at Acquia inside an Amazon EKS Kubernetes cluster.

Solving Database Persistence

There are a number of ways you can solve the problem of database persistence when running in Kubernetes—and this is not just a challenge with Drupal, but with any database-backed web service. One answer the cloud providers have, and which is often the easiest option, is to use a 'managed database', for example, RDS or Aurora in AWS, or Cloud SQL in Google Cloud. You can connect your Drupal sites directly to the managed database backend.

The downside to a managed service approach is that it is often more expensive than a self-managed solution. Also, it requires the opening of a communications channel from inside your Kubernetes cluster to an outside service (which increases your security attack surface).

The upside to a managed service is that managed databases can be a lot faster, better optimized, and more robust than a self-managed solution. Consider Aurora, which I have personally witnessed handling thousands of writes per second, over 400,000 iops, on a large Magento site I manage. It is set up to be geographically redundant, snapshotted nightly (and even more frequently if desired), can scale up or down very easily, and is automatically patched and upgraded.

Where cost is a major concern, or cloud portability, you can still run databases inside a Kubernetes cluster. I do this quite a bit, and the default mysql Docker library image actually runs smaller sites and apps quite well, sometimes with a few small tweaks. But if you want to be able to have highly-available, writer/reader replicated databases inside Kubernetes, be prepared to do a lot more work. There are initiatives like the MySQL Operator which aim to make this process simpler, but they are still not as stable as I would like for production usage.

When you run a database like MySQL or PostgreSQL in Kubernetes, you will need to persist the actual data somewhere, and the most common way is to attach a Kubernetes PV (PersistentVolume) to the MySQL container, usually backed by something like AWS EBS volumes (io1 storage type for better speed), or Google Cloud Persistent Disks. You can—but probably should not—run a MySQL database backed by an NFS PV... like I do on the Raspberry Pi Dramble cluster. But you want the database filesystem to be fast, easy to attach and detach to the MySQL Pod, and robust (make sure you enable backups / snapshots with something like Heptio Velero!).

Looking towards the future, some database systems are actually 12 Factor Apps themselves; for example, CockroachDB, which bills itself as a 'cloud-native database'. CockroachDB doesn't actually require PVs in certain configurations, and each CockroachDB instance would be able to be fully stateless, using local (and much faster, if using NVMe disks) disk access.

However, for the shorter term, Drupal works best with MySQL or MariaDB, and in most cases PostgreSQL. So for now, you will have to solve the problem of how to run a persistent database in your Kubernetes cluster if you want to go fully cloud-native.

Making Drupal's files directory persistent and scalable

Drupal needs to be able to write to the filesystem inside the codebase in a few instances:

  • During installation, Drupal writes over the settings.php file (if it needs to).
  • During installation, Drupal fails the installer preflight checks if the site directory (e.g. sites/default) is not writeable.
  • When building theme caches, Drupal needs write access to the public files directory (by default; this can be configured post-installation) so it can store generated Twig and PHP files.
  • If using CSS and JS aggregation, Drupal needs write access to the public files directory (by default; this can be configured post-installation) so it can store generated .js and .css files.

The easiest solution to all these problems—while allowing Drupal to scale up and run more than one Drupal instance—is to use a shared filesystem (usually NFS, Gluster, or Ceph) for the Drupal files folder (e.g. sites/default/files). Easiest does not mean best, however. Shared filesystems always have their share of troubles, for example:

  • Gluster and Ceph can sometimes become 'split-brained' or have strange failure modes which require manual intervention.
  • NFS can have issues with file lock performance at certain scales.
  • NFS can be slow for certain types of file access.

For all of these reasons, many people decide to use Amazon S3 as the public filesystem for their Drupal site, using something like the S3FS module. But in most cases (and for a generalized approach that doesn't require more cloud services to be used), an approach using NFS (e.g. EFS in AWS) or Rook to manage Ceph filesystems is used.

There's an open Drupal.org issue Make the public file system an optional configuration which can help make the public filesystem a little more flexible in a container-based environment.

Making the settings.php file writeable during installation is a more tricky proposition—especially if you need to install Drupal, then deploy a new container image (what happens to all the values written to settings.php? They go poof!—which is why it gets special treatment as its own major challenge in running Drupal in Kubernetes:

Making Drupal Installable inside a Kubernetes Cluster

Drupal installation is an interesting process. In some cases, you may want to allow people to install Drupal via the Install wizard UI, on the frontend. In other cases, you want to automatically install via Drush.

In either case, Drupal needs to do a few things during installation; one of those things—if you don't already have all the correct variables set in settings.php—is appending some extra configuration to the sites/default/settings.php file (or creating that file if it didn't already exist).

Through a bit of trial and error on the Drupal for Kubernetes project, I found that the following settings must be pre-set and correct in settings.php if you want Drupal to automatically skip the installation screen that asks for DB connection details, and it also conveniently doesn't require writing to the settings.php file if you have these things set:

// Config sync directory.
$config_directories['sync'] = '../config/sync';

// Hash salt.
$settings['hash_salt'] = getenv('DRUPAL_HASH_SALT');

// Disallow access to update.php by anonymous users.
$settings['update_free_access'] = FALSE;

// Other helpful settings.
$settings['container_yamls'][] = $app_root . '/' . $site_path . '/services.yml';

// Database connection.
$databases['default']['default'] = [
  'database' => getenv('DRUPAL_DATABASE_NAME'),
  'username' => getenv('DRUPAL_DATABASE_USERNAME'),
  'password' => getenv('DRUPAL_DATABASE_PASSWORD'),
  'prefix' => '',
  'host' => getenv('DRUPAL_DATABASE_HOST'),
  'port' => getenv('DRUPAL_DATABASE_PORT'),
  'namespace' => 'Drupal\Core\Database\Driver\mysql',
  'driver' => 'mysql',

Notice how I used environment variables (like DRUPAL_HASH_SALT) with PHP's getenv() function instead of hard-coding values in the file. This allows me to control the settings Drupal uses both for installation and for new containers that are started after Drupal is installed.

In a Docker Compose file, when I want to pass in the variables, I structure the web / Drupal container like so:

    image: geerlingguy/drupal-for-kubernetes:latest
    container_name: drupal-for-kubernetes
      DRUPAL_HASH_SALT: 'fe918c992fb1bcfa01f32303c8b21f3d0a0'

And in Kubernetes, when you create a Drupal Deployment, you pass in environment variables either using envFrom and a ConfigMap (preferred), and/or you can directly pass environment variables in the container spec:

apiVersion: extensions/v1beta1
kind: Deployment
  name: drupal
        - image: {{ drupal_docker_image }}
          name: drupal
          - configMapRef:
              name: drupal-config
            - name: DRUPAL_DATABASE_PASSWORD
                  name: mysql-pass
                  key: drupal-mysql-pass

The above manifest snippet presumes you have a ConfigMap named drupal-config containing all the DRUPAL_* variables besides the database password, as well as a Secret named mysql-pass with the password in the key drupal-mysql-pass.

At this point, if you build the environment fresh using Docker Compose locally, or in a Kubernetes cluster, you could visit the UI installer or use Drush, and not have to provide any database credentials since they (and other required vars) are all pre-seeded by the environment configuration.

Environment-driven Drupal Configuration

Expanding on the previous challenge, there may be a number of other configuration settings you would want to be able to manipulate using environment variables. Unfortunately, Drupal itself doesn't yet have a mechanism for doing this, so you would need to either build these kind of settings into your own code using something like getenv(), or you would need to do something like configure a Config Split that manipulates certain Drupal configuration based on an environment variable that you pick up on during a configuration import process.

There is a Drupal.org issue, Provide better support for environment variables, which will track the work to make environment-variable-driven configuration easier to do.

Many other PHP projects use something like dotenv—which allows variables to be read from a .env file locally (e.g. while in development), or your application can use environment variables directly. This is often helpful when you want to give developers more flexibility in managing multiple projects locally, and is kind of a prerequisite to adding first-class environment variable support to Drupal.

Running Drupal's Cron Job(s) in Kubernetes

TODO: Working on this section. See also, for a hacky-but-workable solution: Running Drupal Cron Jobs in Kubernetes.

Conclusion and More Resources

Besides my internal project at Acquia, I also maintain three open source projects which I hope will be immensely useful to others who would like to run Drupal, containerized in production, in Kubernetes:

I welcome conversation in those projects' issue queues (and in the comments below!), and we had a good discussion about the above topics at the DrupalCon Seattle BoF: Drupal, Contained in Production - Docker and Kubernetes. If you're also generally interested in Drupal and Kubernetes, I had a fun presentation on how I run Drupal on Kubernetes on a cluster of Raspberry Pis at DrupalCon Seattle, just a couple days ago—the video and slides are both available if you click through the link: Everything I know about Kubernetes I learned from a cluster of Raspberry Pis.

You can join the #kubernetes channel on the Drupal Slack to follow some of the discussion around Kubernetes. You can also browse issues tagged with 'kubernetes' in the Drupal.org issue queue to try to make some of these challenges easier to meet.

Apr 09 2019
Apr 09

It's been five years since Drupal VM's first release, and to celebrate, it's time to release Drupal VM 5.0 "Flynn Lives"! This update is not a major architectural shift, but instead, a new major version that updates many defaults to use the latest versions of the base VM OS and application software. Some of the new default versions include:

  • Ubuntu 18.04 'Bionic' LTS (was Ubuntu 16.04)
  • PHP 7.2 (was PHP 7.1)
  • Node.js 10.x (was Node.js 6.x)

See the full release notes here: Drupal VM 5.0.0 "Flynn Lives"

There are also a number of other small improvements (as always), and ever-increasing test coverage for all the Ansible roles that power Drupal VM. And in the Drupal VM 4.x release lifecycle, a new official pre-baked Drupal VM base box was added, the geerlingguy/drupal-vm Vagrant base box. Using that base box can speed up new VM builds by 50% or more!

And, as always, it's easy to override any of these settings and versions by adjusting variables in your config.yml. Current supported VM OSes include Ubuntu 18.04 or 16.04, Debian 9, and CentOS 7. Current supported PHP versions include PHP 7.1, 7.2, or 7.3 (note that 5.6 still works but is not officially supported—make sure you upgrade your sites soon and stop using this unsupported PHP version!).

Many people have asked if I'm going to turn Drupal VM entirely into a Docker-backed tool, instead of using Vagrant and VirtualBox; I have decided over the past year that I would rather keep the basic architecture I have now, as it's extremely mature, supports a lot of custom use cases, and has some benefits over a Docker-based architecture, especially for Mac or Windows developers. Also, if you do wish to use a Docker-based tool, there are now multiple mature projects to choose from like Docksal, Ddev, or Lando. (I personally use a mixture of Drupal VM and custom docker-compose-based local development environments for my own projects.)

Finally, if you're interested in the current state of local development tools used by the Drupal community, be sure to check out a session I'm co-presenting with Chris Urban at DrupalCon Seattle: What should I use? 2019 Developer Tool Survey Results.

Apr 08 2019
Apr 08

Thoughts about Drupal 8, Drupal 7, Backdrop, the Drupal Community, DrupalCon's meteoric price increases, DrupalCamps, and the future of the framework/CMS/enterprise experience engine that is Drupal have been bubbling up in the back of my mind for, well, years now.

I am almost always an optimist about the future, and Drupal 8 promised (and usually delivered) on many things:

  • Vastly improved content administration
  • Views in core, and even better than ever
  • Media in core
  • Layouts in core
  • Modern programming paradigms (fewer #DrupalWTFs)
  • 'Getting off the island' and becoming more of a normal PHP application (kinda the opposite of something like Wordpress)

But one thing that has always been annoying, and now is probably to the state of alarming, for some, is the fact that Drupal 8 adoption has still not hit a level of growth which will put it ahead of Drupal 7 adoption any time soon.

This fact has been beaten to death, so you can read more about that elsewhere, or see the current usage graph here and yes, not every site reports back to Drupal.org, so those numbers are not perfect... but it's the best data we have to work with.

So people are asking questions:

  • Should I stick with D7 LTS support for years and years, and re-platform on something else if I ever get the budget?
  • Should I upgrade to Drupal 8 now (or soon) since Drupal 8 to 9 will supposedly be a painless upgrade?
  • When will [insert critical module like Rules here] be ready for Drupal 8?
  • Is Drupal dead?
  • With 'ambitious digital experiences' being the new market Drupal targets, should I still build [insert any kind of non-enterprise type of website here] on it?

And the main driver for this post, in particular, was the following tweet by @webchick (Angie Byron), who coincidentally is probably the main reason I dove headfirst into the Drupal community many years ago after she mentored me through my first core patch (hi Angie!):

Time for my semi-annual pre-#DrupalCon informal poll: If you're still on #Drupal 7 and haven't moved to 8 yet, what's holding you back? (Please RT.)

— webchick (@webchick) April 2, 2019

I won't answer all the questions above—there are a lot of nuances to each that I could not possibly answer in a blog post—but I do want to jot down a number of areas where I have seen pain (and usually experienced on my own) and which are still holding back widespread adoption of Drupal 8 by those who used to default to Drupal for 'all the things'.

Drupal 8 Failures

Caveat to those who read on—you may think I'm trying to disparage Drupal through the rest of this post. I'm not. I'm exposing the dark side of a major open source project's decision to radically re-architect it's core software on an entirely new foundation. It's helpful to know these things so we can figure out ways to avoid hitting all the same pain points in the future, and also as a sort of 'call to action' in case anyone reading this thinks they can push some initiative forward in one area or another (it's no coincidence I'm finishing this post on the flight to DrupalCon Seattle!).

The web has changed

So... a lot of people mention that because more people build custom Node.js-based single page apps using the MEAN stack, or now do hip and trendy 'full stack development', and Drupal is some old monolith, Drupal has been left in the dust. I don't buy that argument, because otherwise we'd see similar attrition in pretty much all the other PHP CMS communities... and we don't.

Sure, there are use cases where someone would consider either Drupal or a hip trendy decoupled web framework backend. But Drupal 8 is actually a really good choice for those who build decoupled architectures using JSON:API, or GraphQL, or whatever other fancy decoupled framework and need a reliable content backend. To be honest, though, it seems that those who do 'decoupled' with Drupal are often people who started with Drupal (or something like it), and then get into the decoupled game. And that's a pretty small slice of the market. Drupal is a hard sell if you have a team of non-PHP developers (whether they do Node, Ruby, Python, Go, or whatever) and are looking into decoupled or otherwise buzzwordy architectures.

Migrations instead of Updates for modules

With Drupal 4, 5, 6, and 7, modules could define upgrade paths from one major version to the next through Drupal's normal update.php mechanism, and while the entire update mechanism was a very Drupal-centric oddity, it worked. And most modules would, as part of the general upgrade process, write an update path so those using the Drupal 6 version would have all that module's configuration make its way to Drupal 7, as long as they were running the latest and greatest Drupal 6 module version when they upgraded.

Things became a bit harder in Drupal 8, because of two things:

  1. New API architecture often required full module rewrites.
  2. The traditional update.php process was abandoned for major version upgrades (see: Drupal 7 sites can no longer be upgraded to Drupal 8 with update.php).

And these two problems kind of fed into each other—not only did module authors had to often rewrite (or at least radically alter) large swaths of code to support the new Drupal 8 APIs, but they also had to scrap any hook_update() upgrade implementation they may have worked on once that change record was published. I'm not speaking in the hypothetical here; this is exactly what happened with the Honeypot module. In fact, I still have not had time to work on writing a module migration for Honeypot for Drupal 8, even though I had a fully working and tested upgrade path using the old update.php method years ago.

For a simpler module like Honeypot, this isn't a major issue, because site builders can reconfigure Honeypot pretty quickly. But for more complex modules that are not in core, like Rules, XML Sitemaps, Google Analytics, etc., there are a ton of configuration options, and without a reliable configuration upgrade path, site owners literally have to re-do all the original configuration work they did when they built out their current Drupal site. Even if the module has the exact same features and functionality. Sometimes one little checkbox buried in the third page of a module's settings is the difference between some site feature working correctly or not—and when you have hundreds or thousands of said checkboxes to check on the new site... the upgrade becomes a much more risky proposition.

There is an open issue in the Drupal project issue queue to help improve this situation, but we're far from the end game here: [Meta] Better support for D7 -> D8 contrib migrate.

Composer is still not a first-class citizen... but is often necessary

One of the largest 'get off the island' tasks the Drupal community checked off early was 'start using Composer for dependency management'. However, Drupal is... weird. You can't resolve two decades worth of architectural assumptions and dependency cruft in one major release.

Especially since until Drupal 8, it was not even really possible (except if you were willing to do some really wacky stuff) to manage a Drupal codebase using Composer.

I've spilled enough ink on these pages over the years over Composer and Drupal 8 (2019, 2018, 2018-2, 2017, 2017-2, 2017-3, etc.), and most of my older postings still highlight current problems in using Drupal and Composer together.

There's a massive initiative to make things better: [META] Improve Drupal's use of Composer. It will still take time, and maybe even cause a little more strife in the end, as some more old Drupalisms may need to be put to rest. But having one standard way to build and maintain Drupal codebases will be better in the end, because right now it can be quite messy, especially for those who downloaded Drupal as a tarball and use no CI system.

OOP and Symfony are the future

Along the same theme as the previous topic, Drupal rearchitected most of the foundational bits of code (the menu routing system, the HTTP request system, the Block system, the Entity system—pretty much everything except maybe Forms API) on top of Symfony, a very robust and widely-used PHP Framework.

But this meant that large swaths of Drupal experience were thrown out the window. A 'Block' was still a 'Block'... but the way they are built changed from weird-but-conventional Drupal hooks to using 'Plugins'. And menus used to be defined in a hook, but now there are sometimes multiple new YAML files you have to add to a module to get Drupal's menu system to pick up a new menu item—and you have to know how to wire up a menu item to a route, and what a route is, etc. In the past many of these things were kind of papered over by Drupal's simple-but-good-enough menu system, but now you have to be more formal about everything.

Speaking of which, a solid understanding of OOP-style programming is basically required in Drupal 8, whereas weekend hackers could kind of cobble together things in Drupal <= 7 using some hooks and copied-and-pasted code from Stack Exchange. Debugging—for those not used to a full-fledged debugger—is also a lot different. Simple print() statements or dsm() don't always cut it anymore. And debugging things on the frontend—well, I'll get to that soon.

The overarching issue is that the Drupal community was sold on the idea that moving Drupal to Symfony would pull in thousands of other PHP developers who would flock to Drupal once it started using a more modern code architecture. That promise really never panned out, as if anything, it seems harder to find solid senior-level Drupal engineers nowadays (at least in my experience—am I wrong here?).

Some Drupal developers who are not classically-trained (like me! I never took a comp sci class in my life) chose to expand their knowledge and grow with Drupal 8's new architecture. Others chose more familiar pastures and either moved on to some other PHP-based CMS or switched to some other ecosystem. I don't blame them, not at all; it's a tough decision you have to make to balance your career desires and opportunities, and everyone has to make their own decision.

In any case, the new architecture has more complexity than the old; and because of this, it's almost a necessity to adopt the following:

  • Use an IDE like PHPStorm, or lots of plugins with other editors, to be able to code efficiently across sometimes multiple files.
  • Integrate some sort of linting framework lest you hit deprecated code and weird syntax issues.
  • Have a CI/build process because a modern Drupal site can't usually be managed and run in one Git codebase and branch, checked out on a production server.

Themes have to be rebuilt

Along with all the other changes, Drupal's theme system was completely swapped out—it went from using the unholy monster that was PHPTemplate to a clean, new, standard system from Symfony, Twig. I'm one of the first to admit that this was probably one of the best and most necessary architecture changes in Drupal. The theme system was dangerous, messy, and difficult to work with on the best days.

But this is in many cases the straw that breaks the camel's back. In addition to the revamped architecture, new required build processes, and upgrade difficulties, almost every Drupal site has to completely rewrite its theme. And for many of the Drupal 7 sites I've built and worked on, this is probably where the majority of the effort would need to happen.

Try as we might (as a general web development community), the number of sites using a strict frontend design system where the design is decoupled from the theme itself, and can evolve and be migrated from one system to another, is vanishingly small. In 99% of the sites I've seen, very little of the frontend code from Drupal 7 could be quickly moved to Drupal 8. Maybe a few theme and form hooks, and a few CSS files, but the theme is usually very deep and complex, and most organizations use an upgrade as an opportunity to sink another chunk of money into refreshing their site's themes anyways.

But at least with Drupal 5 to 6 or 6 to 7, that was a choice, and you could upgrade the underlying system without also upgrading the theme. In Drupal 8, you kind of have to rebuild your theme or build an entirely new theme.

Custom code requires a comprehensive rewrite

I admit that I am guilty of running two Drupal 7 sites with a very large amount of custom code. Hosted Apache Solr and Server Check.in are both currently running on Drupal 7 (well, the frontend parts at least), and I have tens of thousands of lines of custom code which integrates with backend APIs (using things like Drupal's Entity API, Form API, Block API, Queue API, etc.).

Some of this code will not need to be rewritten completely (thankfully!), but there is enough that I will have to schedule a substantial chunk of time—which I could devote to features, bugfixes, or improving the platform in other ways—to upgrade to Drupal 8 (or 9) when the time comes.

There is always technical debt associated with custom code. And I always try to manage that by adding separate Behat tests which test the frontend functionality in as generic a way as possible (that way I can at least upgrade against a set of critical feature tests). But I'm not alone in facing this problem. Thousands of project managers (some of whom have precious little budget to work with) have to decide whether to allocate time and money to a Drupal site rebuild. The more custom code, the more difficult the decision.

Multisite is... interesting

One of the few huge differentiators between Drupal and most other CMSes has always been the ability to run 'multisite' installations. That is, you have one codebase, maybe even on one server, and you can run many Drupal websites (each with its own database, set of modules, unique files directory, theme, etc.).

Many multisite detractors are quick to point out that this is kind of an abomination and is architecturally impure. However, the site you're reading right now (assuming I haven't yet upgraded it to something else) is actually a multisite—I run six different Drupal 7 sites off one codebase, and there's no way I could've justified building each of these sites in Drupal at all if I wasn't able to build one build pipeline, one production server, and one development workflow that literally does all six sites. The dollar cost alone from running 1 Drupal production server to 6 prevents me from even considering it (most of these sites are maintained by me gratis).

There are a lot of massive Drupal multisite installations, especially in education and non-profits, where the cost benefit of not having to manage tens or hundreds (or in some cases thousands) of Drupal codebases, CI workflows, and many more production servers (since you can no longer share PHP's opcache between sites, besides some other things) necessitates multisite installation.

Here's the rub: Multisite architecture is kind of in conflict with some of the core ways Composer works. So trying to manage a modern Drupal 8 codebase with Composer and having the ability to have different copies/versions of different modules inside the codebase is... not quite impossible, but can be very close to that. Especially if you are not a Composer whiz.

Yes, yes, there are a thousand other arguments against multisite... but the fact is, there are a number of organizations—usually some of the orgs with hundreds or thousands of the sites that show up in the Drupal project usage statistics—who are holding off upgrading to Drupal 8 because multisite is harder, and the future of multisite is still fuzzy.


Drupal 8 was a radical re-architecture of a widely-used CMS platform. Many developers made their careers through the Drupal 6 and 7 development lifecycle, and were sideswiped by what happened when Drupal 8 was released. There's no doubt Drupal 8 has a great feature set, a thoroughly-tested core codebase, is excellent as a general site-building tool, and is primed for the building great (and 'ambitious') digital experiences.

But do I recommend Drupal 8 in all the same kinds of situations where I used to recommend Drupal 7 in the past? Definitely not.

Drupal 8 is a very different framework and platform than Drupal 6 or 7 was. There are some massive benefits, like the fact that it is easier to use modern programming paradigms, dependency management tools, and site architecture. And these benefits are massive for new site builds or migrations from outside the Drupal ecosystem into Drupal. But there are many tradeoffs for older Drupal sites; many users (and developers) have been left with a dilemma as they face re-building an entire site, in light of the fact that upgrades are more time-consuming and difficult than they had been in the past.

It may be noted that many of the more 'ambitious' Drupal 6 sites also needed a full migration to Drupal 7 and couldn't be directly upgraded—but for the long tail of smaller sites which usually used core modules and a smattering of contrib modules, and had little if any custom code, the upgrade.php process worked quite well, and resulted in hundreds of thousands of site upgrades that I don't believe we will see with Drupal 7 to Drupal 8.

Architecturally, almost every major change that resulted in the Drupal 8 we know and love (and sometimes shake our fists at!) is sound. But when taken as a whole, I do not begrudge the project managers who have to decide if and when to upgrade to Drupal 8—or sit tight on Drupal 7 LTS, or move to Backdrop, or re-platform to some other system.

I am still optimistic about Drupal's future, especially as the plan seems to be to not make such a massive set of architecture changes in a major version again, but instead to upgrade subsystems here and there through point releases. But I think the usage pattern and value proposition for Drupal has changed. I definitely think there are classes of websites that are more ideally situated on some other platform now, and I also think there will be a large set of organizations willing to stick it out on Drupal 7 LTS for as long as there is some form of commercial support available.

But I think the moral of Drupal's saga is if you revamp many major portions of an ecosystem's architecture in one release, you have to accept the attrition that comes with such a refactoring. The radical alternative is to kind of stick your head in the sand like Wordpress seems to be doing (with regard to modern best practices and the PHP community), but I'm not sure if I like that solution much, either ?.

Feb 28 2019
Feb 28

It's that time of year again! Leading up to DrupalCon Seattle, Chris Urban and I are working on a presentation on Local Development environments for Drupal, and we have just opened up the 2019 Drupal Local Development Survey.

Local development environments - 2018 usage stats
Local development environment usage results from 2018's survey.

If you do any Drupal development work, no matter how much or how little, we would love to hear from you. This survey is not attached to any Drupal organization, it is simply a community survey to help highlight some of the most widely-used tools that Drupalists use for their projects.

Take the 2019 Drupal Local Development Survey

Chris and I will present the results of the survey at our DrupalCon Seattle session What Should I Use? 2019 Developer Tool Survey Results.

We will also be comparing this year's results to those from last year—see our presentation from MidCamp 2018, Local Dev Environments for Dummies.

Feb 20 2019
Feb 20

Over the years, as Drupal has evolved, the upgrade process has become a bit more involved; as with most web applications, Drupal's increasing complexity extends to deployment, and whether you end up running Drupal on a VPS, a bare metal server, in Docker containers, or in a Kubernetes cluster, you should formalize an update process to make sure upgrades are as close to non-events as possible.

Gone are the days (at least for most sites) where you could just download a 'tarball' (.tar.gz) from Drupal.org, expand it, then upload it via SFTP to a server and run Drupal's update.php. That workflow (and even a workflow like drush up of old) might still work for some sites, but it is fragile and prone to cause issues whether you notice them or not. Plus if you're using Drush to do this, it's no longer supported in modern versions of Drush!

So without further ado, here is the process I've settled on for all the Drupal 8 sites I currently manage (note that I've converted all my non-Composer Drupal codebases to Composer at this point):

  1. Make sure you local codebase is up to date with what's currently in production (e.g. git pull master).
  2. Reinstall your local site in your local environment so it is completely reset (e.g. blt setup or drush site-install --existing-config). I usually use a local environment like Drupal VM or a Docker Compose environment, so I can usually just log in and run one command to reinstall Drupal from scratch.
  3. Make sure the local site is running well. Consider running behat and/or phpunit tests to confirm they're working (if you have any).
  4. Run composer update (or composer update [specific packages]).
  5. On your local site, run database updates (e.g. drush updb -y or go to /update.php). _This is important because the next step—exporting config—can cause problems if you're dealing with an outdated schema.
  6. Make sure the local site is still running well after updates complete. Run behat and/or phpunit tests again (if you have any).
  7. If everything passed muster, export your configuration (e.g. drush cex -y if using core configuration management, drush csex -y if using Config Split).
  8. (Optional but recommended for completeness) Reinstall the local site again, and run any tests again, to confirm the fresh install with the new config works perfectly.
  9. If everything looks good, it's time to commit all the changes to composer.lock and any other changed config files, and push it up to master!
  10. Run your normal deployment process to deploy the code to production.

All done!

That last step ("Run your normal deployment process") might be a little painful too, and I conveniently don't discuss it in this post. Don't worry, I'm working on a few future blog posts on that very topic!

For now, I'd encourage you to look into how Acquia BLT builds shippable 'build artifacts', as that's by far the most reliable way to ship your code to production if you care about stability! Note that for a few of my sites, I use a more simplistic "pull from master, run composer install, and run drush updb -y workflow for deployments. But that's for my smaller sites where I don't need any extra process and a few minutes' outage won't hurt!

Jan 28 2019
Jan 28

From time to time, I have the need to take a Twig template and a set of variables, render the template, replacing all the variables within, and then get the output as a string. For example, if I want to have a really simple email template in a custom module which has a variable for first_name, so I can customize the email before sending it via Drupal or PHP, I could do the following in Drupal 7:

= theme_render_template(drupal_get_path('module', 'my_module') . '/templates/email-body.tpl.php', array(
'first_name' => 'Jane',
send_email($from, $to, $subject, $body);

In Drupal 8, there is no theme_render_template() function, since the template engine was switched to Twig in this issue. And until today, there was no change record indicating the fact that the handy theme_render_template() had been replaced by a new, equally-handy twig_render_template() function! Thanks to some help from Tim Plunkett, I was able to find this new function, and after he pushed me to do it, I created a new change record to help future-me next time I go looking for theme_render_template() in Drupal 8: theme_render_template changed to twig_render_template.

In Drupal 8, it's extremely similar to Drupal 7, although there are two additions I made to make it functionally equivalent:

= twig_render_template(drupal_get_path('module', 'my_module') . '/templates/email-body.html.twig', array(
'my-variable' => 'value',
// Needed to prevent notices when Twig debugging is enabled.
'theme_hook_original' => 'not-applicable',
// Cast to string since twig_render_template returns a Markup object.
$body = (string) $markup;
send_email($from, $to, $subject, $body);

If you are rendering a template outside of a normal page request (e.g. in a cron job, queue worker, Drush command, etc.) the Twig theme engine might not be loaded. If that's the case, you'll need to manually load the Twig engine using:

// Load the Twig theme engine so we can use twig_render_template().
include_once \Drupal::root() . '/core/themes/engines/twig/twig.engine';

I shall go forth templating ALL THE THINGS now!

Jan 11 2019
Jan 11

I've been going kind of crazy covering a particular Drupal site I'm building in Behat tests—testing every bit of core functionality on the site. In this particular case, a feature I'm testing allows users to upload arbitrary files to an SFTP server, then Drupal shows those filenames in a streamlined UI.

I needed to be able to test the user action of "I'm a user, I upload a file to this directory, then I see the file listed in a certain place on the site."

These files are not managed by Drupal (e.g. they're not file field uploads), but if they were, I'd invest some time in resolving this issue in the drupalextension project: "When I attach the file" and Drupal temporary files.

Since they are just random files dropped on the filesystem, I needed to:

  1. Create a new step definition
  2. Track files that are created using that step definition
  3. Add code to make sure files that were created are cleaned up

If I just added a new step definition in my FeatureContext which creates the new files, then subsequent test runs on the same machine would likely fail, because the test files I created are still present.

Luckily, Behat has a mechanism that allows me to track created resources and clean up after the scenario runs (even if it fails), and those in Drupal-land may be familiar with the naming convention—they're called hooks.

In this case, I want to add an @AfterScenario hook which runs after any scenario that creates a file, but I'm getting a little ahead of myself here.

Create a new step definition

Whenever I want to create a new step definition, I start by writing out the step as I want it, in my feature file:

When I add file "test.txt" to the "example" folder

Now I run the scenario using Behat, and Behat is nice enough to generate the stub function I need to add to my FeatureContext in it's output:

--- Drupal\FeatureContext has missing steps. Define them with these snippets:

     * @When I add file :arg1 to the :arg2 folder
    public function iAddFileToTheFolder($arg1, $arg2)
        throw new PendingException();

I copy that code out, drop it into my FeatureContext, then change things to do what I want:

   * @When I add file :file_name to the :folder_name folder
  public function iAddFileToTheFolder($file_name, $folder_name) {
    $file_path = '/some/system/directory/' . $folder_name . '/' . $file_name;
    $file = fopen($file_path, 'w');
    fwrite($file, '');

Yay, a working Behat test step! If I run it, it passes, and the file is dropped into that folder.

But if I run it again, the file was already there and the rest of my tests may also be affected by this rogue testing file.

So next step is I need to track the files I create, and make sure they are cleaned up in an @AfterScenario.

Track files created during test steps

At the top of my FeatureContext, I added:

   * Keep track of files added by tests so they can be cleaned up.
   * @var array
  public $files = [];

This array tracks a list of file paths, quite simply.

And then inside my test step, at the end of the function, I can add any file that is created to that array:

   * @When I add file :file_name to the :folder_name folder
  public function iAddFileToTheFolder($file_name, $folder_name) {
    $file_path = '/some/system/directory/' . $folder_name . '/' . $file_name;
    $file = fopen($file_path, 'w');
    fwrite($file, '');
    $this->files[] = $file_path;

That's great, but next we need to add an @AfterScenario hook to clean up the files.

Make sure the created files are cleaned up

At the end of my feature context, I'll add a cleanUpFiles() function:

   * Cleans up files after every scenario.
   * @AfterScenario @file
  public function cleanUpFiles($event) {
    // Delete each file in the array.
    foreach ($this->files as $file_path) {

    // Reset the files array.
    $this->files = [];

This @AfterScenario is tagged with @file, so any scenario where I want the files to be tracked and cleaned up, I just need to add the @file tag, like so:

Feature: MyFeature

  @api @authenticated @javascript @file
  Scenario: Show changed files in selection form using Git on Site page.
    Given I am logged in as a user with the "file_manager" role
    When I am on "/directory/example"
    Then I should see the text "There are no files present in the example folder."
    And I should not see the text "test.txt"
    When I add file "test.txt" to the "example" folder
    And I am on "/directory/example"
    Then I should see the text "text.txt"

And that is how you do it. Now no matter whether I create one file or a thousand, any scenario tagged with @file will get all its generated test files cleaned up afterwards!

Dec 31 2018
Dec 31

tl;dr: Run composer require zaporylie/composer-drupal-optimizations:^1.0 in your Drupal codebase to halve Composer's RAM usage and make operations like require and update 3-4x faster.

A few weeks ago, I noticed Drupal VM's PHP 5.6 automated test suite started failing on the step that runs composer require drupal/drush. (PSA: PHP 5.6 is officially dead. Don't use it anymore. If you're still using it, upgrade to a supported version ASAP!). This was the error message I was getting from Travis CI:

PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 32 bytes) in phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleWatchNode.php on line 40

I ran the test suite locally, and didn't have the same issue (locally I have PHP's CLI memory limit set to -1 so it never runs out of RAM unless I do insane-crazy things.

So then I ran the same test but with PHP's memory_limit set to 2G—yeah, that's two gigabytes of RAM—and it failed! So I ran the command again using Composer's --profile option and -vv to see exactly what was happening:

# Run PHP 5.6 in a container.
$ docker run --rm -it drupaldocker/php-dev:5.6-cli /bin/bash
# php -v     
PHP 5.6.36 (cli) (built: Jun 20 2018 23:33:51)

# composer create-project drupal-composer/drupal-project:8.x-dev composer-test --prefer-dist --no-interaction

# Install Devel module.
# cd composer-test
# composer require drupal/devel:^1.2 -vv --profile
Do not run Composer as root/super user! See https://getcomposer.org/root for details
[126.7MB/5.04s] ./composer.json has been updated
[129.6MB/6.08s] > pre-update-cmd: DrupalProject\composer\ScriptHandler::checkComposerVersion
[131.5MB/6.10s] Loading composer repositories with package information
[131.9MB/6.52s] Updating dependencies (including require-dev)
[2054.4MB/58.32s] Dependency resolution completed in 3.713 seconds
[2054.9MB/61.89s] Analyzed 18867 packages to resolve dependencies
[2054.9MB/61.89s] Analyzed 1577311 rules to resolve dependencies
[2056.9MB/62.68s] Dependency resolution completed in 0.002 seconds
[2055.5MB/62.69s] Package operations: 1 install, 0 updates, 0 removals
[2055.5MB/62.69s] Installs: drupal/devel:1.2.0
[2055.5MB/62.70s] Patching is disabled. Skipping.
[2055.5MB/62.80s]   - Installing drupal/devel (1.2.0): [2055.6MB/62.83s] [2055.6MB/63.02s] Downloading (0%)[2055.6MB/63.02s]                   [2[2055.6MB/63.04s] Downloading (5%)[2[2055.6MB/63.06s] Downloading (15%)[[2055.7MB/63.08s] Downloading (30%)[[2055.7MB/63.10s] Downloading (40%)[[2055.8MB/63.12s] Downloading (55%)[[2055.8MB/63.14s] Downloading (65%)[[2055.9MB/63.15s] Downloading (75%)[[2055.9MB/63.15s] Downloading (80%)[[2055.9MB/63.17s] Downloading (90%)[[2056.0MB/63.18s] Downloading (100%)[2055.5MB/63.19s]
[2055.5MB/63.19s]  Extracting archive[2055.6MB/63.57s]     REASON: Required by the root package: Install command rule (install drupal/devel 1.x-dev|install drupal/devel 1.2.0)
[2055.6MB/63.57s] No patches found for drupal/devel.
[731.5MB/71.30s] Writing lock file
[731.5MB/71.30s] Generating autoload files
[731.8MB/73.01s] > post-update-cmd: DrupalProject\composer\ScriptHandler::createRequiredFiles
[731.6MB/78.82s] Memory usage: 731.61MB (peak: 2057.24MB), time: 78.82s

So... when it started looking through Drupal's full stack of dependencies—some 18,867 packages and 1,577,311 rules—it gobbled up over 2 GB of RAM. No wonder it failed when memory_limit was 2G!

That seems pretty insane, so I started digging a bit more, and found that the PHP 7.1 and 7.2 builds were not failing; they peaked around 1.2 GB of RAM usage (yet another reason you should be running PHP 7.x—it uses way less RAM for so many different operations!).

Then I found a really neat package which had some outlandish promises: composer-drupal-optimizations mentioned in the README:

Before: 876 MB RAM, 17s; After: 250 MB RAM, 5s

I went ahead and added the package to a fresh new Drupal project with:

composer require zaporylie/composer-drupal-optimizations:^1.0

(Note that this operations still uses the same huge amount of memory and time, because the package to optimize things is being installed!)

And then I ran all the tests on PHP 5.6, 7.1, and 7.2 again. Instead of spelling out the gory details (they're all documented in this issue in the Drupal VM issue queue), here is a table of the amazing results:

PHP Version Before After Difference 5.6 2057.24 MB 540.02 MB -1.5 GB 7.1 1124.52 MB 426.64 MB -800 MB 7.2 1190.94 MB 423.93 MB -767 MB

You don't have to be on ancient-and-unsupported PHP 5.6 to benefit from the speedup afforded by ignoring unused/really old Symfony packages!

Next Steps

You should immediately add this package to your Drupal site (if you're using Composer to manage it) if you run Drupal 8.5 or later. And if you use a newer version of Acquia BLT, you're already covered! I'm hoping this package will be added upstream to drupal-project as well (there's sort-of an issue for that), and maybe even something could be done on the Drupal level.

Requiring over 1 GB of RAM to do even a simple composer require for a barebones Drupal site is kind-of insane, IMO.

Dec 19 2018
Dec 19

BLT to Kubernetes

Wait... what? If you're reading the title of this post, and are familiar with Acquia BLT, you might be wondering:

  • Why are you using Acquia BLT with a project that's not running in Acquia Cloud?
  • You can deploy a project built with Acquia BLT to Kubernetes?
  • Don't you, like, have to use Docker instead of Drupal VM? And aren't you [Jeff Geerling] the maintainer of Drupal VM?

Well, the answers are pretty simple:

  • Acquia BLT is not just for Acquia Cloud sites; it is a great way to kickstart and supercharge any large-scale Drupal 8 site. It has built-in testing integration with PHPUnit and Behat. It has default configurations and packages to help make complex Drupal sites easier to maintain and deploy. And it's not directly tied to Acquia Cloud, so you can use it with any kind of hosting!
  • Yes, Acquia BLT is a great tool to use for building and deploying Drupal to Kubernetes!
  • Why, yes! And while I use, maintain, and love Drupal VM—even for this particular project, for the main local dev environment—it is just not economical to deploy and maintain a highly-scalable production environment using something like Drupal VM... not to mention it would make Kubernetes barf!

Anyways, I figured I'd jump right in and document how I'm doing this, and not get too deep into the whys.

Generating a build artifact using only Docker

The first problem: BLT has built-in tooling to deploy new code to environments in traditional cloud environments, in a separate Git repository. This works great for hosting in Acquia Cloud, Pantheon, or other similar environments, but if you need to deploy your BLT site into a container-only environment like Kubernetes, you need to generate a BLT deployment artifact, then get that into a deployable Docker container.

Along that theme, my first goal was to make it so I could build the deployment artifact in a perfectly clean environment—no PHP, no Composer, no Node.js, no nothing except for Docker. So I built a shell script that does the following:

  1. Starts a build container (uses the PHP 7.2 CLI Docker image from Docker Hub).
  2. Installs the required dependencies for PHP.
  3. Installs Node.js.
  4. Installs Composer.
  5. Runs blt artifact:build to build a deployment artifact in the deploy/ directory.
  6. Deletes the build container.

Here's the script:

All you need is to be in a BLT project directory, and run ./blt-artifactory. Now, I could optimize this build process further by building my own image which already has everything set up so I can just run blt artifact:build, but for now I'm a little lazy and don't want to maintain that. Maybe I will at some point. That would cut out 3-5 minutes from the build process.

Building a Docker container with the build artifact

So after we have a deployment artifact in the deploy/ directory, we need to stick that into a Docker container and then push that container into a Docker registry so we can use it as the Image in a Kubernetes Deployment for Drupal.

Here's the Dockerfile I am using to do that:

I put that Dockerfile into my BLT project root, and run docker build -t my/site-name . to build the Docker image. Note that this Dockerfile is meant to build from a PHP container image which already has all the required PHP extensions for your project. In my case, I have a preconfigured PHP base image (based off php:7.2-apache-stretch) which installs extensions like gd, pdo_mysql, simplexml, zip, opcache, etc.

Once I have the my/site-name image, I tag it appropriately and—in this case—push it up to a repository I have set up in AWS ECR. It's best to have a private Docker registry running somewhere for your projects, because you wouldn't want to push your site's production Docker image to a public registry like Docker Hub!

Running Drupal in Kubernetes

Now that I have a Docker image available in AWS ECR, and assuming I have a Kubernetes cluster running inside AWS EKS (though you could be using any kind of Kubernetes cluster—you'd just have to configure it to be able to pull images from whatever private Docker registry you're using), I can create a set of namespaced manifests in Kubernetes for my Drupal 8 site.

For this site, it doesn't need anything fancy—no Solr, no Redis or Memcached, no Varnish—it just needs a horizontally-scalable Drupal deployment, a MySQL deployment, and ingress so people can reach it from the Internet.

Unfortunately for this blog post, I will not dive into the details of the process of getting this particular site running inside Kubernetes, but the process and Kubernetes manifests used for doing so is extremely similar to the ones I am building and maintaining for my Raspberry Pi Dramble project. What's that?, you ask? Well, it's a cluster of Raspberry Pis running Drupal on top of Kubernetes!

If you want to find out more about that, please attend my session at DrupalCon Seattle in April 2019, Everything I know about Kubernetes I learned from a cluster of Raspberry Pis (or view the video/slides after the fact).

Dec 19 2018
Dec 19

Earlier this year, I completely revamped Hosted Apache Solr's architecture, making it more resilient, more scalable, and better able to support having different Solr versions and configurations per customer.

Today I'm happy to officially announce support for Solr 7.x (in addition to 4.x). This means that no matter what version of Drupal you're on (6, 7, or 8), and no matter what Solr module/version you use (Apache Solr Search or Search API Solr 1.x or 2.x branches), Hosted Apache Solr is optimized for your Drupal search!

Hosted Apache Solr - version selection

This post isn't just a marketing post, though—I am also officially announcing that the actual Docker container images used to run your search cores are free and open source, and available for anyone to use (yes, even if you don't pay for a Hosted Apache Solr subscription!). I maintain a variety of Solr versions, from 3.6.x (I still use it to support some annoyingly-outdated Magento 1.x sites which only work with 3.6.x) to 7.x and everything in between, and there are instructions for using the Solr containers with your own projects (even in production if you'd like!) in the source code repository:

You can add a subscription to supercharge your Drupal site's search—no matter what version you want—over at hostedapachesolr.com.

(Aside: Within the first quarter of 2018, we will also add support for changing Solr versions at-will!)

Dec 03 2018
Dec 03

On a recent project, I needed to add some behavioral tests to cover the functionality of the Password Policy module. I seem to be a sucker for pain, because often I choose to test the things it seems there's no documentation on—like testing the functionality of the partially-Javascript-powered password fields on the user account forms.

In this case, I was presented with two challenges:

  • I needed to run one scenario where a user edits his/her own password, and must follow the site's configured password policy.
  • I needed to run another scenario where an admin creates a new user account, and must follow the site's configured password policy for the created user's password.

So I came up with the following scenarios:

Feature: Password Policy
  In order to verify that password policies are working
  As a user
  I should not be able to use a password
  Unless it meets the minimum site password policy constraints

  @api @authenticated
  Scenario: Users must meet password policy constraints
    Given users:
    | name                 | status | uid    | mail                             | pass         |
    | test.password.policy |      1 | 999999 | [email protected] | fEzHZ3ru9pce |
    When I am logged in as user "test.password.policy"
    And I am on "/user/999999/edit"
    And I fill in "edit-current-pass" with "fEzHZ3ru9pce"
    And I fill in "edit-pass-pass1" with "abc123"
    And I fill in "edit-pass-pass2" with "abc123"
    And I press "edit-submit"
    Then I should see "The password does not satisfy the password policies."
    And I should see "Fail - Password length must be at least 12 characters."

  @api @authenticated @javascript
  Scenario: Password policy constraints are enforced when creating new users
    Given I am logged in as user "administrator_account"
    When I am on "/admin/people/create"
    And I fill in "mail" with "[email protected]"
    And I fill in "name" with "test.create.user"
    And I fill in "edit-pass-pass1" with "test.create.userABC123"
    And I fill in "edit-pass-pass2" with "test.create.userABC123"
    And I pause for "1" seconds
    And I press "edit-submit"
    Then I should see "The password does not satisfy the password policies."

Now, there are a couple annoying/special things I'm doing here:

  • For the first scenario, I had trouble making it work without specifying the uid of the new user, because I needed to get to the user edit page (user/[id]/edit), but for some reason trying a step like And I click "Edit" was not working for me.
  • The first scenario doesn't seem to have any trouble with the process of clicking submit then seeing the password policy validation error message—hold onto that thought for a second.
  • The second scenario uses @javascript to indicate this test should be run in a browser environment with javascript running. Apparently this means there is some tiny amount of delay between the time the 'edit-pass-passX' fields are filled in and the drupal password validation javascript does whatever it does—any time I would submit without a pause, I would get the error "The specified passwords do not match." Infuriating!

To resolve the third problem listed above, I added a custom step definition to my project's FeatureContext:

   * @When I pause for :seconds seconds
  public function iPauseForSeconds($seconds) {

And the way I finally figured out that it was a timing issue was because I stuck in a Behat breakpoint (e.g. And I break) in different points in the scenario, and found it would work if I paused between tasks.

Sometimes testing can be a bit infuriating :P

I'm guessing there are a few slightly-better ways to get this done, but it works for me, and a 1s pause two times in my test suite isn't so bad.

Nov 29 2018
Nov 29

There are times when you may notice your MySQL or MariaDB database server getting very slow. Usually, it's a very stressful time, as it means your site or application is also getting very slow since the underlying database is slow. And then when you dig in, you notice that logs are filling up—and in MySQL's case, the slow query log is often a canary in a coal mine which can indicate potential performance issues (or highlight active performance issues).

But—assuming you have the slow query log enabled—have you ever grabbed a copy of the log and dug into it? It can be extremely daunting. It's literally a list of query metrics (time, how long the query took, how long it locked the table), then the raw slow query itself. How do you know which query takes the longest time? And is there one sort-of slow query that is actually the worst, just because it's being run hundreds of times per minute?

You need a tool to sift through the slow query log to get those statistics, and Percona has just the tool for it: pt-query-digest. This tool has many other tricks up its sleeve, but for this post, I just want to cover how it helps me analyze and summarize slow query logs so I can quickly dig into the worst queries that might be bringing down my production application or Drupal or other PHP-based website.

I'm doing this on my Mac, but the process should be similar for most any linux or unix-y environment (including the WSL on Windows 10):

  1. Create a directory to work in: mkdir db-analysis && cd db-analysis
  2. Download pt-query-digest: curl -LO https://percona.com/get/pt-query-digest
  3. Make it executable: chmod +x pt-query-digest
  4. Download your slow-query.log file from the database server (or if using something like AWS RDS/Aurora, download it from AWS Console).
  5. Run pt-query-digest over the log file: ./pt-query-digest slow-query.log

At this point, you should see a full report with a summary of the worst queries at the top (along with stats about how many times they were invoked, the average amount of time they took, rows examined and sent, etc.

$ ./pt-query-digest slowquery.log

# 4.3s user time, 200ms system time, 39.12M rss, 4.12G vsz
# Current date: Thu Nov 29 10:02:45 2018
# Hostname: JJG.local
# Files: slowquery.log
# Overall: 4.51k total, 36 unique, 1.27 QPS, 15.65x concurrency __________
# Time range: 2018-11-27 21:00:23 to 21:59:38
# Attribute          total     min     max     avg     95%  stddev  median
# ============     ======= ======= ======= ======= ======= ======= =======
# Exec time         55640s      5s    118s     12s     16s      4s     13s
# Lock time        18446744073714s    34us 18446744073710s 4085657602s   260us 271453769812s   194us
# Rows sent          2.95M       0 103.99k  684.27       0   8.14k       0
# Rows examine     293.63M       0  18.67M  66.59k    0.99 583.21k    0.99
# Query size        44.63M      79   1.22M  10.12k   2.16k  97.85k   2.06k

# Profile
# Rank Query ID                      Response time    Calls R/Call  V/M  
# ==== ============================= ================ ===== ======= =====
#    1 0x5AE6E128A4790517E5CFFD03... 52666.0213 94.7%  4363 12.0711  0.86 UPDATE some_table
#    2 0x222A6FC43B63B119D2C46918...   618.3909  1.1%    29 21.3238  1.91 UPDATE some_table
#    3 0xD217B90797E8F34704CF55AF...   463.7665  0.8%    29 15.9919  0.07 SELECT some_other_table some_other_other_table

And then the rest of the report shows the queries in ranked order from worst to least offensive.

At this point, I'll grab the worst offender (usually there's only one that is taking up 90% or more of the slow query time) and run it using EXPLAIN on my MySQL server. This gives me more details about whether indexes would be used, whether temporary tables would be created, etc. And from that point, it's a matter of working with the application developers (or in many cases, my own dumb code!) to improve the query, or if that's not possible or the root cause, working on the MySQL configuration to ensure all the tuning parameters are adequate for the queries being run and the database being used.

Nov 28 2018
Nov 28

There are a number of things you have to do to make Drupal a first-class citizen inside a Kubernetes cluster, like adding a shared filesystem (e.g. PV/PVC over networked file share) for the files directory (which can contain generated files like image derivatives, generated PHP, and twig template caches), and setting up containers to use environment variables for connection details (instead of hard-coding things in settings.php).

But another thing which you should do for better performance and traceability is run Drupal cron via an external process. Drupal's cron is essential to many site operations, like cleaning up old files, cleaning out certain system tables (flood, history, logs, etc.), running queued jobs, etc. And if your site is especially reliant on timely cron runs, you probably also use something like Ultimate Cron to manage the cron jobs more efficiently (it makes Drupal cron work much like the extensive job scheduler in a more complicated system like Magento).

The reason you want to run cron via an external process—instead of having it be triggered by front-end page visits (this is how Drupal sets up cron by default, to run every few hours based on someone hitting the site)—is so you can make sure cron job runs don't interfere with any normal page visits, and so you can trace any issues with cron separately from normal web traffic.

Inside of Kubernetes, you can't (well, at least you shouldn't) have a crontab set up on any of your Kubernetes nodes running against your Drupal site(s). And while you could have an external system like Jenkins run cron jobs against your Drupal site, it's much easier (and simpler) to just run Drupal cron as a Kubernetes CronJob, ideally within the same Kubernetes namespace as your Drupal site.

The most robust way to run Drupal cron is via Drush, but running a separate Drush container via CronJob means that the CronJob must schedule a beefy container running at least PHP and Drush, and likely also your app codebase if you run Drush as a project dependency. CronJob Pods should be as lightweight as possible so they could be scheduled on any system node and run very quickly (even if your Drupal container hasn't been pulled on that particular Kubernetes node yet).

Drupal's cron supports being run from outside the site by hitting a URL, and as long as your cron runs can complete before PHP/Apache/Nginx's timeout, this is the simplest option for working with Kubernetes (IMO). For my Drupal sites running in Kubernetes, I configure a CronJob similar to the following:

apiVersion: batch/v1beta1
kind: CronJob
  name: drupal-cron
  namespace: {{ k8s_resource_namespace }}
  schedule: "*/1 * * * *"
  concurrencyPolicy: Forbid
          - name: drupal-cron
            image: {{ curl_image }}
            - -s
            - {{ drupal_cron_url }}
          restartPolicy: OnFailure

In my case, I'm templating the Kubernetes manifest using Ansible and Jinja2 (deployed via my K8s Manifests role), so I have Ansible replace the three variables with values like:

k8s_resource_namespace: my-drupal-site
curl_image: byrnedo/alpine-curl:0.1
drupal_cron_url: http://www.my-drupal-site.com/cron/cron-url-token

The drupal_cron_url is the URL specific to your site, which you can find by visiting /admin/config/system/cron. Make sure you also have "Run cron every" set to "Never" under the cron settings, so that cron is only triggered via Kubernetes' CronJob.

I use the byrnedo/alpine-curl docker image, which is extremely lightweight—only 5 or 6 MB in total—since it's based on Alpine Linux. Most of the other containers I've seen base on Ubuntu or Debian and are at least 30-40 MB (so they'll take that much longer to download the first time the CronJob is run on a new node).

You can check on the status of the CronJob with:

kubectl describe cronjob drupal-cron -n my-drupal-site

Nov 17 2018
Nov 17

This blog post contains a written transcript of my NEDCamp 2018 keynote, Real World DevOps, edited to match the style of this blog. Accompanying resources: presentation slides, video (coming soon).

Jeff Geerling at NEDCamp 2018 - New England Drupal Camp

I'm Jeff Geerling; you probably know that because my name appears in huge letters at the top of every page on this site, including the post you're reading right now. I currently work at Acquia as a Senior Technical Architect, building hosting infrastructure projects using some buzzword-worthy tech like Kubernetes, AWS, and Cloud.

I also maintain Drupal VM, the most popular local development environment for the Drupal open source CMS. And I run two SaaS products with hundreds of happy customers, Hosted Apache Solr and Server Check.in, both of which have had over 99.99% uptime since their inception for a combined 15 years. I also write (and continuously update) a best-selling book on Ansible, Ansible for DevOps, and a companion book about Kubernetes. Finally, I maintain a large ecosystem of Ansible roles and automation projects on GitHub which have amassed over 17,000 stars and 8,000 forks.

Oh, I also have three children under the age of six, have a strong passion for photography (see my Flickr), maintain four Drupal websites for local non-profit organizations, and love spending time with my wife.

You might be thinking: this guy probably never spends time with his family.

And, if you're speaking of this weekend, sadly, you'd be correct—because I'm here in Rhode Island with all of you!

But on a typical weeknight, I'm headed upstairs around 5-6 p.m., spend time with my family for dinner, after-meal activities, prayers, and bedtime. And on weekends, it's fairly rare I'll need to do any work. We go to the zoo, we go on family trips, we go to museums, and we generally spend the entire weekend growing together as a family.

Some nights, after the kids are settled in bed, I'll spend an hour or two jumping through issue queues, updating a section of my book—or, as is the case right now, writing this blog post.

How do I do it?

CNCF Cloud Native Landscape

Well I apply complex self-healing, highly-scalable DevOps architectures to all my projects using all the tools shown in this diagram! I'm kidding, that would be insane. But have you seen this graphic before? It's the Cloud Native Landscape, published by the Cloud Native Computing Foundation.

The reason I show this picture is because I expect everyone reading this to memorize all these tools so you know how to implement DevOps by next week.

Just kidding again! Some people think the mastery of some tools in this diagram means they're doing 'DevOps'. To be honest, you might be practicing DevOps better than someone who integrates fifty of these tools using nothing but Apache and Drupal—neither of which are listed in this infographic!

What is DevOps?

The framework I use is what I call 'Real World DevOps'. But before I get into my definition, I think it's important we understand what the buzzword 'DevOps' means, according to our industry:

Azure DevOps

Microsoft, apparently, packaged up DevOps and sells it as part of Azure's cloud services. So you can put a price on it, apparently, get a purchase order, and have it! Right?


And I see a lot of DevOps people talk about how Docker transformed their developers into amazing coding ninjas who can deploy their code a thousand times faster. So Docker is part of DevOps, right?

There is no cloud sticker

And to do DevOps, you have to be in the cloud, because that's where all DevOps happens, right?

Agile Alliance

And DevOps requires you to strictly follow Agile methodologies, like sprints, kanban, scrums, pair programming, and pointing poker, right?

Well, let's go a little further, and see what some big-wigs in the industry have to say:

"People working together to build, deliver, and run resilient software at the speed of their particular business."

So it sounds like there's a people component, and some sort of correlation between speed and DevOps.

Okay, how about Atlassian?

DevOps "help[s] development and operations teams be more efficient, innovate faster, and deliver higher value"

So it sounds like it's all about making teams better. Okay...

"Rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach"

(Oh... that's funny, this quote is also in one of O'Reilly's books on DevOps, in a post from Ensono, and in basically every cookie-cutter Medium post about DevOps.)

In Gartner's case, they seem to focus strongly on methodology and service delivery—but that's probably because their bread and butter is reviewing products which purportedly help people track methodology and service delivery! It's interesting (and telling) there's no mention about people or teams!

But what do I say about DevOps?

But what about me? I just claimed to practice DevOps in my work—heck, my book has the word DevOps in the title! Surely I can't just be shilling for the buzzword profit multiplier by throwing the word 'DevOps' in my book title... right?

Ansible for DevOps - is Jeff Geerling just riding the wave of the buzzword?

Well, to be honest, I did use the word to increase visibility a bit. Why else do you think my second book has the word 'Kubernetes' in it!?

But my definition of DevOps is a bit simpler:

Jeff Geerling's DevOps Definition - Making people happier while making apps better

"Making people happier while making apps better."
—Jeff Geerling (Photo above by Kevin Thull)

I think this captures the essence of real world, non-cargo-cult DevOps, and that's because it contains the two most important elements:

Making people happier

DevOps is primarily about people: every team, no matter the size, has to figure out a way to work together to make users happy, and not burn out in the process. And what are some of the things I see in teams that are implementing DevOps successfully?

  • Reduced friction between Operations/sysadmins, Developers, Project Management, InfoSec, and QA. People don't feel like it's 'us against them', or 'we will loop them in after we finish our part'. Instead, everyone talks, everyone has open communication lines in email or Slack, and requirements and testing are built up throughout the life of the project.
  • Reduced burnout, because soul-sucking problems and frustrating communications blockades are almost non-existent.
  • Frequent code deploys, and almost always in the middle of the workday—and this also feeds back into reduced burnout, because nobody's pulling all-nighters fixing a bad deploy and wrangling sleepy developers to implement hotfixes.
  • Stable teams that stay together and grow into a real team, not just a 'project team'; note that this can sometimes be impossible (e.g. in some agency models), but it does make it easier to iteratively improve if you're working with the same people for a long period of time.
  • There are no heroes! Nobody has to be a rockstar ninja, working through the weekend getting a release ready, because DevOps processes emphasize stability, enabling a better work-life balance for everyone on the team.

How many times have you seen an email praising the heroic efforts of the developer who fixed some last-minute major issues in a huge new feature that were discovered in final user acceptance testing? This should not be seen as a heroic deed—rather it should be seen as a tragic failure. Not a failure of the developer, but as a failure of the system that enabled this to happen in the first place!

DevOps is about making people happier.

Making apps better

Devops is also about apps: you can't afford to develop at a glacial pace in the modern world, and when you make changes, you should be confident they'll work. Some of the things I see in the apps that are built with a DevOps mentality include:

  • Continuous delivery: a project's master (or production) code branch is always deployable, and passes all automated tests—and there are automated tests, at least covering happy paths.
  • Thorough monitoring: teams know when deployments affect performance. They know whether their users are having a slow or poor experience. They get alerts when systems are impaired but not down.
  • Problems are fixed as they occur. Bugfixes and maintenance are part of the regular workflow, and project planning gives equal importance to these issues as it does features.
  • Features are delivered frequently, and usually in small increments. Branches or unmerged pull requests rarely last more than a few days, and never more than a sprint.

Small but frequent deployments are one of the most important ways to make your apps better, because it also makes it easier to fix things as problems occur. Instead of dropping an emergent bug into a backlog, and letting it fester for weeks or months before someone tries to figure out how to reproduce the bug, DevOps-empowered teams 'swarm' the bug, and prevent similar bugs from ever happening again by adding a new test, correcting their process, or improving their monitoring.

DevOps is about making apps better.

DevOps Prerequisites

So we know that DevOps is about people and apps, and we know some of the traits of a team that's doing DevOps well, but are there some fundamental tools or processes essential to making DevOps work? Looking around online, I've found most DevOps articles mention these prerequisites:

  • Automation
  • CI/CD
  • Monitoring
  • Collaboration

I tend to agree that these four traits are essential to implementing DevOps well. But I think we can distill the list even further—and in some cases, some prerequisites might not be as important as the others.

I think the list should be a lot simpler. To do DevOps right, it should be:

  • Easy to make changes
  • Easy to fix and prevent problems (and prevent them from happening again)

Easy to make changes

I'm just wondering: have you ever timed how long it takes for a developer completely new to your project to get up and running? From getting access to your project codebase and being able to make a change to it locally? If not, it might be a good idea to find out. Or just try deleting your local environment and codebase entirely, and starting from scratch. It should be very quick.

If it's not easy and fast to start working on your project locally, it's hard to make changes.

Once you've made some changes, how do you know you won't break any existing functionality on your site? Do you have behavioral testing that you can easily run, and doesn't take very long to run, and doesn't require hours of setup work or a dedicated QA team? Do you have visual regression tests which verify that the code you just changed won't completely break the home page of your site?

If you can't be sure your changes won't break things, it's scary to make changes.

Once you deploy changes to production, how hard is it to revert back if you find out the changes did break something badly? Have you practiced your rollback procedure? Do you even have a process for rollbacks? Have you tested your backups and have confidence you could restore your production system to a known good state if you totally mess it up?

If you can't back out of broken changes, it's scary to make changes.

The easier and less stressful it is to make changes, the more willing you'll be to make them, and the more often you'll make them. Not only that, with more confidence in your disaster recovery and testing, you'll also be more confident and less stressed.

"High performers deployed code 30x more frequently, and the time required to go from “code committed” to “successfully running in production” was 200x faster."
—The DevOps Handbook

While you might not be deploying code 300 times a day, you'll be happy to deploy code whenever you want, in the middle of the workday, if you can make changes easy.

Easy to fix and prevent problems

Making changes has to be easy, otherwise it's hard to fix and prevent problems. But that's not all that's required.

Are developers able to deploy their changes to production? Or is there a long, drawn out process to get a change deployed to production? If you can build the confidence that at least the home page still loads before the code is deployed, then you'll be more likely to make small but frequent changes—which are a lot easier to fix than huge batches of changes!

Developers should be able to deploy to production after their code passes tests.

Once you deploy code, how do you know if it's helping or hurting your site's performance? Do you have detailed metrics for things like average end-user page load times (Application Performance Monitoring, or APM), CPU usage, memory usage, and logs? Without these metrics you can't make informed decisions about what's broken, or whether a particular problem is fixed.

Detailed system metrics and logging is essential to fix and prevent problems.

When something goes wrong, does everyone duck and cover, finding ways to avoid being blamed for the incident? Or does everyone come together to figure out what went wrong, why it went wrong, and how to prevent it from happening in the future? It's important that people realize when something goes wrong, it's rarely the fault of the person who wrote the code or pressed the 'go' button—it's the fault of the process. Better tests, better requirements, more thorough reviews would prevent most issues from ever happening.

'Blameless postmortems' prevent the same failure from happening twice while keeping people happy.

DevOps Tools

But what about tools?

"It's a poor craftsman that blames his tools."
—An old saying

Earlier in this post I mentioned that you could be doing DevOps even if you don't use any of the tools in the Cloud Native Landscape. That may be true, but you should also avoid falling into the trap of having one of these:

Golden hammer driving a screw into a board

A golden hammer is a tool that someone loves so much, they use it for purposes for which it isn't intended. Sometimes it can work... but the results and experience are not as good as you'd get if you used the right tool for the job. I really like this quote I found on a Hacker News post:

"Part of being an expert craftsman is having the experience and skills to select excellent tools, and the experience and skills to drive those excellent tools to produce excellent results."
jerf, HN commenter

So a good DevOps practitioner knows when it's worth spending the time learning how to use a new tool, and when to stick with the tools they know.

So now that we know something about DevOps, here's a project for you: build some infrastructure for a low-profile Drupal blog-style site for a budget-conscious client with around 10,000 visitors a day. Most of the traffic comes from Google searches, and there is little authenticated traffic. What would you build?

Complex Drupal hosting architecture in AWS VPC with Kubernetes

Wow! That looks great! And it uses like 20 CNL projects, so it's definitely DevOps, right?

Great idea, terrible execution.

Simple Drupal hosting architecture with a LAMP server and CloudFlare

Just because you know how to produce excellent results with excellent tools doesn't mean you always have to use the 'best' and most powerful tools. You should also know when to use a simple hammer to nail in a few nails! This second architecture is better for this client, because it will cost less, be easier to maintain long-term, and won't require a full-time development team maintaining the infrastructure!

Jeff Geerling holding a golden hammer

So know yourself. Learn and use new tools, but don't become an architecturenaut, always dreaming up and trying to build over-engineered solutions to simple problems!

That being said, not all the tools you'll need appear in the Cloud Native Landscape. Some of the tools I have in my toolbelt include:


I don't know how many times I've had to invoke YAGNI. That is, "You Ain't Gonna Need It!" It's great that you aspire to have your site get as much traffic as Facebook. But that doesn't mean you should architect it like Facebook does. Don't build fancy, complex automations and flexible architectures until you really need them. It saves you money, time, and sometimes it can even save a project from going completely off the rails!

Much like the gold plating on the hammer I was holding earlier, extra features that you don't need are a waste of resources, and may actually make your project worse off.

Andon board

In researching motivations behind some Agile practices, I came across an interesting book about lean manufacturing, The Machine that Changed the World. A lot of the ideas you may hear and even groan about in Agile methodology, and even DevOps, come from the idea of lean manufacturing.

One of the more interesting ideas is the andon board, a set of displays visible to every single worker in Toyota's manufacturing plant. If there's ever a problem or blockage, it is displayed on that board, and workers are encouraged to 'swarm the problem' until it is fixed—even if it's in a different part of the plant. The key is understanding that problems should not be swept aside to be dealt with when you have more time. Instead, everyone on the team must be proactive in fixing the problem before it causes a plant-wide failure to produce.

Time to Drupal

I did a blog post after DrupalCon last year discussing how different local Drupal development environments have dramatically different results in my measurement of "Time to Drupal". That is, from not having it downloaded on your computer, to having a functional Drupal environment you can play around with, how long does it take?

If it takes you more than 10 minutes to bring up your local environment, you should consider ways to make that process much faster. Unless you have a multi-gigabyte database that's absolutely essential for all development work (and this should be an exceedingly rare scenario), there's no excuse to spend hours or days onboarding a new developer, or setting up a new computer when your old one dies!

Dev to Prod

Similarly, how long does it take, once a feature or bugfix has been deployed somewhere and approved, for it to be deployed to production? Does this process take more than a day? Why? Are you trying to batch multiple changes together into one larger deployment?

The DevOps Handbook has some good advice about this:

"one of the best predictors of short lead times was small batch sizes of work"
—The DevOps Handbook

And wouldn't you know, there's a lean term along this theme: Takt time, or the average amount of time it takes between delivering units of work.

If you batch a bunch of deployments together instead of delivering them to production as they're ready, you'll have a large Takt time, and this means you can't quickly deliver value to your end users. You want to reduce that time by speeding up your process for getting working code to production.


Those tools might not be the tools you were thinking I'd mention, like DevShop, Drupal VM, Lando, Docker, or Composer. But in my mind, if you want to implement DevOps in the real world, those tools might be helpful as implementation details, but you should spend more time thinking about real world DevOps tools: better process, better communication, and better relationships.

If you do that, you will truly end up making people happier while making apps better.

Thank you.

Resources mentioned in the presentation

Nov 03 2018
Nov 03

Lately I've been spending a lot of time working with Drupal in Kubernetes and other containerized environments; one problem that's bothered me lately is the fact that when autoscaling Drupal, it always takes at least a few seconds to get a new Drupal instance running. Not installing Drupal, configuring the database, building caches; none of that. I'm just talking about having a Drupal site that's already operational, and scaling by adding an additional Drupal instance or container.

One of the principles of the 12 Factor App is:

IX. Disposability

Maximize robustness with fast startup and graceful shutdown.

Disposability is important because it enables things like easy, fast code deployments, easy, fast autoscaling, and high availability. It also forces you to make your code stateless and efficient, so it starts up fast even with a cold cache. Read more about the disposability factor on the 12factor site.

Before diving into the details of how I'm working to get my Drupal-in-K8s instances faster to start, I wanted to discuss one of the primary optimizations, opcache...

Measuring opcache's impact

I first wanted to see how fast page loads were when they used PHP's opcache (which basically stores an optimized copy of all the PHP code that runs Drupal in memory, so individual requests don't have to read in all the PHP files and compile them on every request.

  1. On a fresh Acquia BLT installation running in Drupal VM, I uninstalled the Internal Dynamic Page Cache and Internal Page Cache modules.
  2. I also copied the codebase from the shared NFS directory /var/www/[mysite] into /var/www/localsite and updated Apache's virtualhost to point to the local directory (/var/www/[mysite], is, by default, an NFS shared mount to the host machine) to eliminate NFS filesystem variability from the testing.
  3. In Drupal VM, run the command while true; do echo 1 > /proc/sys/vm/drop_caches; sleep 1; done to effectively disable the linux filesystem cache (keep this running in the background while you run all these tests).
  4. I logged into the site in my browser (using drush uli to get a user 1 login), and grabbed the session cookie, then stored that as export cookie="KEY=VALUE" in my Terminal session.
  5. In Terminal, run time curl -b $cookie http://local.example.test/admin/modules three times to warm up the PHP caches and see page load times for a quick baseline.
  6. In Terminal, run ab -n 25 -c 1 -C $cookie http://local.example.test/admin/modules (requires apachebench to be installed).

At this point, I could see that with PHP's opcache enabled and Drupal's page caches disabled, the page loads took on average 688 ms. A caching proxy and/or requesting cached pages as an anonymous user would dramatically improve that (the anonymous user/login page takes 160 ms in this test setup), but for a heavy PHP application like Drupal, < 700 ms to load every code path on the filesystem and deliver a generated page is not bad.

Next, I set opcache.enable=0 (was 1) in the configuration file /etc/php/7.1/fpm/conf.d10-opcache.ini, restarted PHP-FPM (sudo systemctl restart php7.1-fpm), and confirmed in Drupal's status report page that opcache was disabled (Drupal shows a warning if opcache is disabled). Then I ran another set of tests:

  1. In Terminal, run time curl -b $cookie http://local.example.test/admin/modules three times.
  2. In Terminal, run ab -n 25 -c 1 -C $cookie http://local.example.test/admin/modules

With opcache disabled, average page load time was up to 1464 ms. So in comparison:

Opcache status Average page load time Difference Enabled 688 ms baseline Disabled 1464 ms 776 ms (72%, or 2.1x slower)

Note: Exact timings are unimportant in this comparison; the delta between different scenarios what's important. Always run benchmarks on your own systems for the most accurate results.

Going further - simulating real-world disk I/O in VirtualBox

So, now that we know a fresh Drupal page load is almost 4x slower than one with the code precompiled in opcache, what if the disk access were slower? I'm running these tests on a 2016 MacBook Pro with an insanely-fast local NVMe drive, which can pump through many gigabytes per second sequentially, or hundreds of megabytes per second random access. Most cloud servers have disk I/O which is much more limited, even if they say they are 'SSD-backed' on the tin.

Since Drupal VM uses VirtualBox, I can limit the VM's disk bandwidth using the VBoxManage CLI (see Limiting bandwidth for disk images):

# Stop Drupal VM.
vagrant halt

# Add a disk bandwidth limit to the VM, 1 MB/sec.
VBoxManage bandwidthctl "VirtualBox-VM-Name-Here" add Limit --type disk --limit 5M

# Get the name of the disk image (vmdk) corresponding to the VM.
VBoxManage list hdds

# Apply the limit to the VM's disk.
VBoxManage storageattach "VirtualBox-VM-Name-Here" --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium "full-path-to-vmdk-from-above-command" --bandwidthgroup Limit

# Start Drupal VM.
vagrant up

# (You can update the limit in real time once the VM's running with the command below)
# VBoxManage bandwidthctl "VirtualBox-VM-Name-Here" set Limit --limit 800K

I re-ran the tests above, and the average page load time was now 2171 ms. Adding that to the test results above, we get:

Opcache status Average page load time Difference Enabled 688 ms baseline Disabled 1464 ms 776 ms (72%, or 2.1x slower) Disabled (slow I/O) 2171 ms 1483 ms (104%, or 3.2x slower)

Not every cloud VM has that slow of disk I/O... but I've seen many situations where I/O gets severely limited, especially in cases where you have multiple volumes mounted per VM (e.g. maximum EC2 instance EBS bandwidth per instance) and they're all getting hit pretty hard. So it's good to test for these kinds of worst-case scenarios. In fact, last year I found that a hard outage was caused by an E_F_S volume hitting a burst throughput limit, and bandwidth went down to 100 Kbps. This caused so many issues, so I had to architect around that potential issue to prevent it from happening in the future.

The point is, if you need fast PHP startup times, slow disk IO can be a very real problem. This could be especially troublesome if trying to run Drupal in environments like Lambda or other Serverless environments, where disk I/O is usually the lowest priority—especially if you choose to allocate a smaller portion of memory to your function! Cutting down the initial request compile time could be immensely helpful for serverless, microservices, etc.

Finding the largest bottlenecks

Now that we know the delta for opcache vs. not-opcache, and vs. not-opcache on a very slow disk, it's important to realize that compilation is just one in a series of many different operations which occurs when you start up a new Drupal container:

  • If using Kubernetes, the container image might need to be pulled (therefore network bandwidth and image size may have a great affect on startup time)
  • The amount of time Docker spends allocating resources for the new container, creating volume mounts (e.g. for a shared files directory) can differ depending on system resources
  • The latency between the container and the database (whether in a container or in some external system like Amazon RDS or Aurora) can cause tens or even hundreds of ms of time during startup

However, at least in this particular site's case—assuming the container image is already pulled on the node where the new container is being started—the time spent reading in code into the opcache is by far the longest amount of time (~700ms) spent waiting for a fresh Drupal Docker container to serve its first web request.

Can you precompile Drupal for faster startup?

Well... not really, at least not with any reasonable sanity, currently.

But there is hope on the horizon: There's a possibility PHP 7.4 could add a cool new feature, Preloading! You can read the gory details in the RFC link, but the gist of it is: when you are building your container image, you could precompile all of your application code (or at least the hot code paths) so when the container starts, it only takes a couple ms instead of hundreds of ms to get your application's code compiled into opcache.

We'll see if this RFC gets some uptake; in the meantime, there's not really much you can do to mitigate the opcache warming problem.


With Preloading, we might be able to pre-compile our PHP applications—notably beefy ones like Drupal or Magento—so they can start up much more quickly in lightweight environments like Kubernetes clusters, Lambda functions, and production-ready docker containers. Until that time, if it's important to have Drupal serve its first request as quickly as possible, consider finding ways to trim your codebase so it doesn't take half a second (or longer) to compile into the opcache!

Nov 02 2018
Nov 02

I am currently building a Drupal 8 application which is running outside Acquia Cloud, and I noticed there are a few 'magic' settings I'm used to working on Acquia Cloud which don't work if you aren't inside an Acquia or Pantheon environment; most notably, the automatic Configuration Split settings choice (for environments like local, dev, and prod) don't work if you're in a custom hosting environment.

You have to basically reset the settings BLT provides, and tell Drupal which config split should be active based on your own logic. In my case, I have a site which only has a local, ci, and prod environment. To override the settings defined in BLT's included config.settings.php file, I created a config.settings.php file in my site in the path docroot/sites/settings/config.settings.php, and I put in the following contents:

* Settings overrides for configuration management.

// Disable all splits which may have been enabled by BLT's configuration.
foreach ($split_envs as $split_env) {
  $config["$split_filename_prefix.$split_env"]['status'] = FALSE;

$split = 'none';

// Local env.
if ($is_local_env) {
  $split = 'local';
// CI env.
if ($is_ci_env) {
  $split = 'ci';
// Prod env.
if (getenv('K8S_ENVIRONMENT') == 'prod') {
  $split = 'prod';

// Enable the environment split only if it exists.
if ($split != 'none') {
  $config["$split_filename_prefix.$split"]['status'] = TRUE;

The K8S_ENVIRONMENT refers to an environment variable I have set up in the production Kubernetes cluster where the BLT Drupal 8 codebase is running. There are a few other little tweaks I've made to make this BLT project build and run inside a Kubernetes cluster, but I'll leave those for another blog post and another day :)

Aug 02 2018
Aug 02

Over the past decade, I've enjoyed presenting sessions at many DrupalCamps, DrupalCon, and other tech conferences. The conferences are some of the highlights of my year (at least discounting all the family things I do!), and lately I've been appreciative of the local communities I meet and get to be a part of (even if for a very short time) at Drupal Camps.

The St. Louis Drupal Users Group has chosen to put off it's annual Camp to 2019, so we're guiding people to DrupalCorn Camp, which is only a little bit north of us, in Iowa.

NEDCamp New England Drupal Camp logo

And I was extremely excited to receive an invitation to present the keynote at NEDCamp (New England Drupal Camp) 2018 in Rhode Island! Not only that, the keynote (and indeed the entire camp this year) is focused on DevOps—a topic/philosophy/role very near and dear to my heart.

I maintain Drupal VM, wrote Ansible for DevOps, and maintain hundreds of open source projects used by thousands of organizations to make their DevOps dreams a reality. In the keynote, I will be focusing a little more on the philosophy of DevOps, and how adopting this philosophy can accelerate your development and site building process, while reducing pain (outages, bugs, angry users and clients).

I'd love to see you at NEDCamp—or DrupalCorn (fingers crossed I'll be able to make it!)—later this year!

May 21 2018
May 21

I started Hosted Apache Solr almost 10 years ago, in late 2008, so I could more easily host Apache Solr search indexes for my Drupal websites. I realized I could also host search indexes for other Drupal websites too, if I added some basic account management features and a PayPal subscription plan—so I built a small subscription management service on top of my then-Drupal 6-based Midwestern Mac website and started selling a few Solr subscriptions.

Back then, the latest and greatest Solr version was 1.4, and now-popular automation tools like Chef and Ansible didn't even exist. So when a customer signed up for a new subscription, the pipeline for building and managing the customer's search index went like this:

Hosted Apache Solr original architecture

Original Hosted Apache Solr architecture, circa 2009.

Hosted Apache Solr was run entirely on three servers for the first few years—a server running hostedapachesolr.com, and two regional Apache Solr servers running Tomcat and Apache Solr 1.4. With three servers and a few dozen customers, managing everything using what I'll call GaaS ("Geerlingguy as a Service") was fairly manageable.

But fast forward a few years, and organic growth meant Hosted Apache Solr was now spread across more than 10 servers, and Apache Solr needed to be upgraded to 3.6.x. This was a lot more daunting of an upgrade, especially since many new customers had more demanding needs in terms of uptime (and a lot more production search traffic!). This upgrade was managed via shell scripts and lots of testing on a staging server, but it was a little bit of a nail-biter to be sure. A few customers had major issues after the upgrade, and I learned a lot from the experience—most especially the importance of automated and fast backup-and-restore automation (it's not good enough to just have reliable backups!).

It was around the time of Apache Solr 4.6's release when I discovered Ansible and started adopting it for all my infrastructure automation (so much so that I eventually wrote a book on Ansible!). For the Apache Solr 3.6 to 4.6 upgrade, I used an Ansible playbook which allowed me better control over the rollout process, and also made it easier to iterate on testing everything in an identical non-production environment.

But at that time, there were still a number of infrastructure operations which could be classified as 'GaaS', and took up some of my time. Wanting to optimize the infrastructure operations even further meant I needed to automate more, and start changing some architecture to make things more automate-able!

So in 2015 or so, I conceived an entirely new architecture for Hosted Apache Solr:

Hosted Apache Solr Docker Jenkins and Ansible-based Architecture

Docker was still new and changing rapidly, but I saw a lot of promise in terms of managing multiple search subscriptions with more isolation, and especially with the ability to move search indexes between servers more efficiently (in a more 'clustered' environment).

Unfortunately, life got in the way as I have multiple health-related problems that robbed me of virtually all my spare time (Hosted Apache Solr is one of many side projects, in addition to my book (Ansible for DevOps), Server Check.in, and hundreds of open source projects).

But the architecture was sound in 2015, even though—at the time—Docker was still a little unstable. In 2015, I had a proof of concept running which allowed me to run multiple Docker containers on a single server, but I was having trouble getting requests routed to the containers, since each one was running a separate search index for a different client.

Request routing problems

One major problem I had to contend with was a legacy architecture design where each client's Drupal site would connect directly to one of the servers by hostname, e.g. "nyc1.hostedapachesolr.com", or "nyc2.hostedapachesolr.com". Originally this wasn't a major issue as I had few servers and would scale individual 'pet' servers up and down as capacity dictated. But as the number of servers increases, and capacity needs fluctuate more and more, this has become a real pain point; mostly, it makes it hard for me to move a client from one server to another, because the move needs to be coordinated with the client and can't be dynamic.

To resolve this problem, I decided to integrate Hosted Apache Solr's automation with AWS' Route53 DNS service, which allows me to dynamically assign a new CNAME record to each customer's search index—e.g. instead of nyc1.hostedapachesolr.com, a customer uses the domain customer-subscription.hostedapachesolr.com to connect to Hosted Apache Solr. If I need to move the search index to another server, I just need to re-point customer-subscription.hostedapachesolr.com to another server after moving the search index data. Problem solved!

Another problem had to do with authentication: Hosted Apache Solr has used IP-based authentication for search index access since the beginning. Each client's search index is firewalled and only accessible by one or more IP addresses, configured by the client. This is a fairly secure and recommended way to configure access for Apache Solr search, but there are three major problems with this approach:

  1. Usability: Clients had to discover their servers' IP address(es), and enter them into their subscription details. This creates friction in the onboarding experience, and can also cause confusion if—and this happens a lot—the actual IP address of the server differs from the IP address their hosting provider says is correct.
  2. Security: A malicious user with a valid subscription could conceivably spoof IP addresses to gain access to other clients search indexes, provided they're located on the same server (since multiple clients are hosted on a single server, and the firewall was on the server level).
  3. Cloud Compatibility: Many dynamic Cloud-based hosting providers like AWS, Pantheon, and Platform.sh allocate IP addresses to servers dynamically, and a client can't use one stable IP address—or even a range of IP addresses—to connect to Hosted Apache Solr's servers.

Luckily, there's a second way to provide an authentication layer for Apache Solr indexes, and that's HTTP Basic Authentication. And HTTP Authentication is well supported by all versions of Drupal's Apache Solr modules, as well as almost all web software in existence, and it resolves all three of the above issues!

I settled on using Nginx to proxy all web requests to the Solr backends, because using server directives, I could support both dynamic per-customer hostnames, as well as HTTP Authentication, using Nginx configuration like:

server {
  listen 80;
  server_name customer-subscription.hostedapachesolr.com;

  auth_basic "Connection requires authentication.";
  auth_basic_user_file /path/to/htpasswd-file;

  location / {

Each customer gets a Solr container running on a specific port (solr-port), with a unique hostname and a username and password stored on the server using htpasswd.

Good migrations

This new architecture worked great in my testing, and was a lot easier to automate (it's always easier to automate something when you have a decade of experience with it and work on a greenfield rewrite of the entire thing...)—but I needed to support a large number of existing customers, ideally with no changes required on their end.

That's a tall order!

But after some more medically-induced delays in the project in 2017 and early 2018, I finally had the time to work on the migration/transition plan. I basically had to:

  1. Build an Ansible playbook to backup a Solr server, delete it, build a new one with the same hostname, then restore all the existing client Solr index data, with as little downtime as possible.
  2. Support IP-based authentication for individual search indexes during a transition period.
  3. Build functionality into the Drupal hostedapachesolr.com website which allows the customer to choose when to transition from IP-based authentication to HTTP Basic Authentication.

The first task was the most straightforward, but took the longest; migrating all the data for dozens of servers with minimal downtime while completely rebuilding them all is a bit of a process, especially since the entire architecture for the new version was different. It required a lot of testing, and I tested against an automated test bed of 8 Drupal sites, across Drupal 6, 7, and 8, running all the popular Apache Solr connection modules with a variety of configurations.

For IP-based authentication, I wrote some custom rules in an Nginx server directive that would route requests from certain IP addresses or IP ranges to a particular Solr container; this was fairly straightforward, but I spent a bit of time making sure that having a large number of these legacy routing rules wouldn't slow down Nginx or cause any memory contention (luckily, they didn't, even when there were hundreds of them!).

Finally, after I completed the infrastructure upgrade process, I worked on the end-user-facing functionality for upgrading to the new authentication method.

Letting the customer choose when to transition

Learning from past experience, it's never nice to automatically upgrade clients to something newer, even after doing all the testing you could possibly do to ensure a smooth upgrade. Sometimes it may be necessary to force an upgrade... but if not, let your customers decide when they want to click a button to upgrade.

I built and documented the simple upgrade process (basically, edit your subscription and check a box that says "Use HTTP Basic Authentication"), and it goes something like:

  1. User updates subscription node in Drupal.
  2. Drupal triggers Jenkins job to update the search server configuration.
  3. Jenkins runs Ansible playbook.
  4. Ansible adds a Route53 domain for the subscription, updates the server's Nginx configuration, and restarts Nginx.

The process again worked flawlessly in my testing, with all the different Drupal site configurations... but in any operation which involves DNS changes, there's always a bit of an unknown. A few customers have reported issues with the length of time it takes for their Drupal site to re-connect after updating the server hostname, but most customers who have made the upgrade have had no issues and are happy to stop managing server IP addresses!

Why not Kubernetes?

A few people who I've discussed architecture with almost immediately suggested using Kubernetes (or possibly some other similarly-featured container orchestration/scheduler layer). There are a couple major reason I have decided to stick with vanilla docker containers and a much more traditional scheduling approach (assign a customer to a particular server):

  1. Legacy requirements: Currently, most clients are still using the IP-based authentication and pointing their sites at the servers directly. Outside of building a complex load balancing/routing layer on top of everything else, this is not that easy to support with Kubernetes, at least not for a side project.
  2. Complexity: I'm working on my first Kubernetes-in-production project right now (but for pay, not for a side hustle), and building, managing, and monitoring a functional Kubernetes environment is still a lot of work. Way too much for something that I manage in my spare time, and expect to maintain 99.9%+ uptime through a decade!

Especially considering many Kubernetes APIs on which I'd be reliant are still in alpha or beta status (though many use them in production), the time's not yet ripe for moving a service like Hosted Apache Solr over to it.

Summary and Future plans

Sometimes, as a developer, working on new 'greenfield' projects can be very rewarding. No technical debt to manage, no existing code, tests, or baggage to deal with. You just pick what you think is the best solution and soldier on!

But upgrading projects like Hosted Apache Solr—with ten years of baggage and an architecture foundation in need of a major overhaul—can be just as rewarding. The challenge is different, in terms of taking an existing piece of software and transforming it (versus building something new). The feeling of satisfaction after a successful major upgrade is the same as I get when I launch something new... but in some ways it's even better because an existing audience (the current users) will immediately reap the benefits!

The new Hosted Apache Solr infrastructure architecture has already made automation and maintenance easier, but there are some major client benefits in the works now, too, like allowing clients to choose any Solr version they want, better management of search indexes, better statistics and monitoring, and in the future even the ability to self-manage configuration changes!

Finally... if you need to supercharge your Drupal site search, consider giving Hosted Apache Solr a try today :)

May 03 2018
May 03

A question which I see quite often in response to posts like A modern way to build and develop Drupal 8 sites, using Composer is: "I want to start using Composer... but my current Drupal 8 site wasn't built with Composer. Is there an easy way to convert my codebase to use Composer?"

Convert a tarball Drupal codebase to a Composer Drupal codebase

Unfortunately, the answer to that is a little complicated. The problem is the switch to managing your codebase with Composer is an all-or-nothing affair... there's no middle ground where you can manage a couple modules with Composer, and core with Drush, and something else with manual downloads. (Well, technically this is possible, but it would be immensely painful and error-prone, so don't try it!).

But since this question comes up so often, and since I have a Drupal 8 codebase that I built that doesn't currently use Composer, I thought I'd record the process of converting this codebase from tarball-download-management (where I download core, then drag it into the codebase, download a module, drag it into modules/, etc.) to being managed with Composer. This blog post contains the step-by-step guide using the method I recommend (basically, rebuilding your codebase from scratch with modern Composer/Drupal best practices), as well as a video of the process (coming soon - will be on my YouTube channel!).

Note: There are a few tools that attempt to convert your existing Drupal codebase to a Composer-managed codebase (e.g. composerize-drupal, or Drupal Console's composerize command), but I have found them to be a little more trouble than they are worth. I recommend rebuilding the codebase from scratch, like I do in this guide.

Getting started - taking an inventory of the codebase

The first thing you need to do is take an inventory of all the stuff that makes up your Drupal codebase. Hopefully the codebase is well-organized and doesn't contain a bunch of random files thrown wily-nily throughout. And hopefully you didn't hack core or contrib modules (though if you do, as long as you did so using patches, you'll be okay—more on that later).

I'm going to work on converting the codebase behind my Raspberry Pi Dramble website. Admittedly, this is a very small and tidy codebase, but it's a good one to get started with. Here's what it looks like right now:

Drupal codebase before Composer conversion

Note: I use Git to manage my codebase, so any changes I make can be undone if I completely break my site. If you're not using Git or some other VCS to version control your changes... you should make sure you have a backup of the current working codebase—and start using version control!

The most important parts are:

  • Drupal Core
  • Modules (/modules): I have one contrib module, admin_toolbar
  • Install profiles (/profiles): mine is empty
  • Themes (/themes): I have aone custom theme,pidramble`

For this particular site, I don't customize the robots.txt or .htaccess files, though I sometimes do for other sites. So as far as an inventory of my current codebase goes, I have:

  • Drupal Core
  • Admin Toolbar
  • pidramble (custom theme)
  • No modifications to core files in the docroot

Now that I know what I'm working with, I'm ready to get started switching to Composer.

Rebuilding with Composer

I'm about to obliterate my codebase as I know it, but before doing that, I need to temporarily copy out only my custom code and files (in my case, just the pidramble theme) into a folder somewhere else on my drive.

Next up, the scariest part of this whole process: delete everything in the codebase. The easiest way to do this, and include invisible files like the .htaccess file, is to use the Terminal/CLI and in the project root directory, run the commands:

# First command deletes everything besides the .git directory:
find . -path ./.git -prune -o -exec rm -rf {} \; 2> /dev/null

# Second command stages the changes to your repository:
git add -A

# Third command commits the changes to your repository:
git commit -m "Remove all files and folders in preparation for Composer."

At this point, the codebase is looking a little barren:

Drupal codebase is empty before converting to Composer

Now we need to rebuild it with Composer—and the first step is to set up a new codebase based on the Composer template for Drupal projects. Run the following command in your now-empty codebase directory:

composer create-project drupal-composer/drupal-project:8.x-dev new-drupal-project --stability dev --no-interaction

After a few minutes, that command should complete, and you'll have a fresh new Composer-managed codebase at your disposal, inside the new-drupal-project directory! We need to move that codebase into the project root directory and delete the then-empty new-drupal-project directory, so run:

mv new-drupal-project/* ./
mv new-drupal-project/.* ./
rm -rf new-drupal-project

And now you should have a Drupal project codebase that looks like this:

Drupal codebase after building a new Composer Template for Drupal

But wait! The old codebase had Drupal's docroot in the project root directory... where did my Drupal docroot go? The Composer template for Drupal projects places the Drupal docroot in a subdirectory of the project instead of the project root—in this case, a web/ subdirectory:

Drupal codebase - web docroot subdirectory from Composer Template

Note: You might also see a vendor/ directory, and maybe some other directories; note that those are installed locally but won't be committed to your Git codebase—at least not by default. I'll mention a few different implications of how this works later!

There are a few good reasons for putting Drupal's actual docroot in a subdirectory:

  1. You can store other files besides the Drupal codebase in your project repository, and they won't be in a public web directory where anyone can access them by default.
  2. Composer can manage dependencies outside of the docroot, which is useful for development and ops tools (e.g. Drush), or for files which shouldn't generally be served in a public web directory.
  3. Your project can be organized a little better (e.g. you can just have a project-specific README and a few scaffold directories in the project root, instead of a ton of random-looking Drupal core files like index.php, CHANGELOG.txt, etc. which have nothing to do with your specific Drupal project).

Now that we have the base Composer project set up, it's time to add in all the things that make our site work. Before doing that, though, you should commit all the core files and Drupal project scaffolding that was just created:

git add -A
git commit -m "Recreate project from Composer template for Drupal projects."

Adding contrib modules, themes, and profiles

For each contributed module, theme, or install profile your site uses, you need to require it using Composer to get it added to your codebase. For example, since I only use one contributed module, I run the following command:

composer require drupal/admin_toolbar:~1.0

Parsing this out, we are telling Composer to require (basically, add to our project) the admin_toolbar project, from the Drupal.org packagist repository. If you look on Drupal.org at the Admin Toolbar project page, you'll notice the latest version is something like 8.x-1.23... so where's this weird ~1.0 version coming from? Well, Drupal's packagist repository translates versions from the traditional Drupal style (e.g. 8.x-1.0) to a Composer-compatible style. And we want the latest stable version in the 1.x series of releases, so we say ~1.0, which tells Composer to grab whatever is the latest 1.x release. If you visit a project's release page on Drupal.org (e.g. Admin Toolbar 8.x-1.23), there's even a handy command you can copy out to get the latest version of the module (see the 'Install with Composer' section under the download links):

Install with Composer on Drupal.org project release page

You could also specify a specific version of a module (if you're not running the latest version currently) using composer require drupal/admin_toolbar:1.22. It's preferred to not require specific versions of modules... but if you're used to using Drupal's update manager and upgrading one module at a time to a specific newer version, you can still work that way if you need to. But Composer makes it easier to manage updating modules without even having to use Drupal's update manager, so I opt to use a more generic version like ~1.0.

Note: If you look in the docroot after adding a contrib module, you might notice the module is now inside modules/contrib/admin_toolbar. In my original codebase, the module was located in the path modules/admin_toolbar. When you move a module to another directory like this, you need to make sure you clear all caches on your Drupal site after deploying the change, and either restart your webserver or manually flush your PHP opcache/APC (otherwise there could be weird errors when PHP looks in the wrong directory for module files!).

After you run a composer require for each contrib project (you can also run composer require drupal/project_one drupal/project_two etc. if you want to add them all in one go), it's time to commit all the contrib projects to your codebase (git add -A, then git commit -m "Add contrib projects to codebase."), then move on to restoring custom modules, themes, and profiles (if you have any).

Adding custom modules, themes, and profiles

Following the convention of having 'contrib' modules/themes/profiles in a special 'contrib' subdirectory, a lot of people (myself included) use the convention of placing all custom projects into a 'custom' subdirectory.

I have a custom theme, pidramble, so I created a custom directory inside themes/, and placed the pidramble theme directory inside there:

Drupal codebase - pidramble theme inside custom themes subdirectory

For any of your custom code:

  • Place modules in web/modules/custom/
  • Place themes in web/themes/custom/
  • Place profiles in web/profiles/custom/

Note: If you use Drupal's multisite capabilities (where you have one codebase but many websites running off of it), then site-specific custom modules, themes, and profiles can also be placed inside site-specific folders (e.g. inside web/sites/[sitename]/modules/custom).

Adding libraries

Libraries are a little different, especially since right now there are a few different ways people manage third party libraries (e.g. Javascript libraries) in Drupal 8. It seems most people have settled on using Asset Packagist to bundle up npm dependencies, but there is still active work in the Drupal community to standardize third party library management in this still-nascent world of managing Drupal sites with Composer.

If you have a bunch of libraries you need to add to your codebase, please read through these issues for some ideas for how to work with Asset Packagist:

Adding customizations to .htaccess, robots.txt, etc.

You might need to customize certain files that are included with Drupal core for your site—like adding exclusions to robots.txt or adding a redirection to .htaccess. If so, make sure you make those changes and commit them to your git repository. The Composer template project recommends you do a git diff on any customized files any time you update Drupal core.

Note: There are more advanced ways of managing changes to these 'scaffold' files if you want take out the human review part from Drupal core updates, but they require a bit more work to make them run smoothly, or may require you to manually apply changes to your customized files whenever Drupal core updates those files in a new release.

Adding patches to core and contrib projects

One of the best things about using the Composer template project is that it automatically sets up the composer-patches project, which allows you to apply patches directly to Drupal core and contrib projects. This is a little more of an advanced topic, and most sites I use don't need this feature, but it's very nice to have when you do need it!

Add a local development environment

While you're revamping your codebase, why not also revamp your local development process, and add a local environment for easy testing and development of your code? In my blog post A modern way to build and develop Drupal 8 sites, using Composer, I showed how easy it is to get your codebase up and running with Drupal VM's Docker container:

composer require --dev geerlingguy/drupal-vm-docker
docker-compose up -d

Then visit http://localhost/ in your browser, and you can install a new Drupal site locally using your codebase (or you can connect to the Docker container's MySQL instance and import a database from your production site for local testing—always a good idea to validate locally before you push a major change like this to production!).

Drupal VM Docker container running a new Drupal Composer template project

Note: The quickest way to import a production database locally with this setup is to do the following:

  1. Drop a .sql dump file into the project root (e.g. sql-dump.sql).
  2. Import the database: docker exec drupal_vm_docker bash -c "mysql drupal < /var/www/drupalvm/drupal/sql-dump.sql"
  3. Clear caches: docker exec drupal_vm_docker bash -c "drush --root=/var/www/drupalvm/drupal/web cr

Deploying the new codebase

One major change—at least in my Raspberry Pi Dramble codebase—is the transition from having the Drupal docroot be in the project root to having the docroot be in the web/ subdirectory. If I were to git push this to my production web server (which happens to be a little Raspberry Pi on my desk, in this case!), then the site would break because docroot is in a new location.

So the first step is to make sure your webserver knows to look in project-old-location/web for the docroot in your Apache, Nginx, etc. configuration. In my case, I updated the docroot in my Apache configuration, switching it from /var/www/drupal to /var/www/drupal/web.

Then, to deploy to production:

  1. Take a backup of your site's database and codebase (it never hurts, especially if you're about to make a major change like this!)
  2. Stop Apache/Nginx (whatever your server is running) so no web traffic is served during this docroot transition.
  3. Deploy the updated code (I used git, so did a git pull on my production web server).
  4. Run composer install --no-dev inside the project root, so the production server has all the right code in all the right places (--no-dev means 'don't install development tools that aren't needed in production').
  5. Start Apache or Nginx so it starts serving up the new docroot subdirectory.

One more caveat: Since you moved the docroot to a new directory, the public files directory might need to also be moved and/or have permissions changed so things like CSS and JS aggregation work correctly. If this is the case, make sure the sites/default/files directory has the correct file ownership and permissions (usually something like www-data and 775), and if your git pull wiped out the existing files directory, make sure your restore the rest of the contents from the backup you took in step 1!

Note: There are many reasons not to run composer install on your production server—and in some cases, it might not even work! It's best to 'build' your codebase for production separately, on a CI server or using a service like Circle CI, Travis CI, etc., then to deploy the built codebase to the server... but this requires another set of infrastructure management and deployment processes, so for most of my smaller projects I just have one Git codebase I pull from and run composer install on the production web server.

Another option is to commit everything to your codebase, including your vendor directory, the web/core directory, all the contrib modules, etc. This isn't ideal, but it works and might be a better option if you can't get composer install working in production for one reason or another.

Managing everything with Composer

One of the main drivers for this blog post was the questions Matthew Grasmick and I got after our session at DrupalCon Nashville 2018, How to build a Drupal site with Composer AND keep all of your hair. Even before then, though, I have regularly heard from people who are interested in starting to use Composer, but have no clue where to start, since their current codebase is managed via tarball, or via Drush.

And this is an especially pressing issue for those using Drush, since Drush 9 doesn't even support downloading Drupal or contributed projects anymore—from the Drush 9 documentation:

Drush 9 only supports one install method. It requires that your Drupal 8 site be built with Composer and Drush be listed as a dependency.

So while you can continue managing your codebase using the tarball-download method for the foreseeable future, I would highly recommend you consider moving your codebase to Composer, since a lot of the tooling, tutorials, and the rest of the Drupal ecosystem is already in the middle of a move in that direction. There are growing pains, to be sure, but there are also a lot of benefits, many of which are identified in the DrupalCon Nashville presentation I mentioned earlier.

Finally, once you are using Composer, make sure you always run composer install after updating your codebase (whether it's a git pull on a production server, or even on your workstation when doing local development!

Apr 26 2018
Apr 26

The St. Louis Drupal Users Group has hosted a Drupal Camp in the 'Gateway to the West' for four years (since 2014), but this year, the organizers have decided to take a year off, for various reasons. Our camp has grown a little every year, and last year we even increased the scope and usefulness of the camp even more by adding a well-attended training day—but life and work have taken precedence this year, and nobody is able to take on the role of 'chief organizer'.

Meet me in Des Moines St. Louis Drupal Camp goes to DrupalCorn in Iowa

All is not lost, however! There are other great camps around the Midwest, and this year we're directing everyone to our northern neighbors, in Iowa: DrupalCorn Camp is going to be held in Des Moines, Iowa, from September 27-30, 2018!

We'll still be hosting our monthly Drupal meetups throughout the year (we've had an ongoing monthly meetup in the St. Louis area for more than 10 years now!), and you can follow along with the St. Louis Drupal community (and find out about next year's Camp!) via our many social media channels.

Apr 23 2018
Apr 23

Mollom End of Life Announcement from their homepage

Earlier this month, Mollom was officially discontinued. If you still have the Mollom module installed on some of your Drupal sites, form submissions that were previously protected by Mollom will behave as if Mollom was offline completely, meaning any spam Mollom would've prevented will be passed through.

For many Drupal sites, especially smaller sites that deal mostly with bot spam, there are a number of great modules that will help prevent 90% or more of all spam submissions, for example:

  • Honeypot: One of the most popular and effective bot spam prevention modules, and it doesn't harm the user experience for normal users (disclaimer: I maintain the Honeypot module).
  • CAPTCHA or reCAPTCHA: Modules that use a sort of 'test' to verify a human is submitting the form. Some tradeoffs in either case, but they do a decent job of deterring bots.
  • Antibot: An even simpler method than what Honeypot or CAPTCHA uses... but might not be adequate for some types of bot spam.

There are many other modules that use similar tricks as the above, but many modules only exist for Drupal 7 (or 8, and not 7), and many don't have the long-term track record and maintenance history that these other more popular modules have.

But there's a problem—all the above solutions are primarily for mitigating bot spam, not human spam. Once your Drupal site grows beyond a certain level of popularity, you'll notice more and more spam submissions coming through even if you have all the above tools installed!

This is the case with this website, JeffGeerling.com. The site has many articles which have become popular resources for all things Ansible, Drupal, photography, etc., and I get enough traffic on this site that I get somewhere between 5,000-10,000 spam comment submissions per day. Most of these are bots, and are caught by Honeypot. But 2% or so of the spam is submitted by humans, and they go right through Honeypot's bot-targeted filtering!

Filtering Human Spam with CleanTalk

Mollom did a respectable job of cleaning up those 100 or so daily human spam submissions. A few would get through every day, but I have a custom module whipped up that shoots me an email with links to approve or deny comments, so it's not a large burden to deal with less than 10 spam submissions a day.

The day Mollom reached EOL, the number of emails started spiking to 50-100 every day... and that was a large burden!

So I started looking for new solutions, fast. The first I looked into was Akismet, ostensibly 'for Wordpress', but it works with many different CMSes via connector modules. For Drupal, there's the AntiSpam module, but it has no Drupal 8 release yet, so it was kind of a non-starter. The Akismet module has been unmaintained for years, so that's a non-starter too...

Looking around at other similar services, I found CleanTalk, which has an officially-supported CleanTalk module for both Drupal 7 and Drupal 8, and it checks all the boxes I needed for my site:

  • Well-maintained module for Drupal 7 and 8
  • Easy SaaS interface for managing settings (better and more featureful than Mollom or Akismet, IMO)
  • Free trial so I could see how well it worked for my site (which I used, then opted for a paid plan after 3 days)
  • Blacklist and IP-based filtering features for more advanced use cases (but only if you want to use them)

I installed CleanTalk in the first week of April, and so far have only had 5 actual human spam comment submissions make it through the CleanTalk filter; 4 of them were marked as 'possible spam', and one was approved (there's a setting in the module to auto-approve comments that look legit, or have all comments go into an approval queue using Drupal's built-in comment approval system).

So CleanTalk worked better than Mollom did, and it was a little simpler to get up and running. The one tradeoff is that CleanTalk's Drupal module isn't quite as 'Drupally' as Mollom or Honeypot. By that, I mean it's not built to be a flexible "turn this on for any kind of form on your site" solution, but it's more tailored for things like:

  • Drupal comments and comment entities
  • User registration
  • Webform

To be fair, those are probably the top three use cases for spam prevention—but as of right now, CleanTalk can't easily be made to work with generic entity submissions (e.g. forum nodes, custom node types, etc.), so it works best on sites with simpler needs.

CleanTalk's pricing is pretty simple (and IMO pretty cheap) too: for one website, it's currently $8/year, or cheaper if you pay for multiple years in advance.

Disclaimer: CleanTalk offers a free year of service for those who post a review of CleanTalk on their websites, and I'll probably take them up on that offer... but I had actually written the first draft of this blog post over a month ago, before I found out about this offer, when I was initially investigating using CleanTalk for my DrupalCon Nashville session submission Mollom is gone. Now what?. I would still write the review exactly the same with or without their offer—it's been that good in my testing! Heck, I already paid for 3 years of service!


If you have a Drupal site and are disappointed by the uptick in user registration, webform, and comment spam since Mollom's end of life, check out CleanTalk for a potentially better solution!

Edit: Also see some excellent suggestions and ongoing work in integrating other spam prevention services into Drupal in the comments below!

Apr 19 2018
Apr 19
Fellow Acquian Matthew Grasmick and I just presented How to build a Drupal site with Composer AND keep all of your hair at DrupalCon Nashville, and though the session wasn't recorded, we posted the slides and hands-on guide to using Composer to manage your Drupal 8 sites to the linked session page.
Apr 13 2018
Apr 13

At DrupalCon Nashville 2018, I became deeply interested in the realm of first-time Drupal experiences, specifically around technical evaluation, and how people would get their feet wet with Drupal. There were two great BoFs related to the topic which I attended, and which I hope will bear some fruits over the next year in making Drupal easier for newcomers:

There are a number of different tools people can use to run a new Drupal installation, but documentation and ease of use for beginners is all over the place. The intention of this project is to highlight the most stable, simple, and popular ways to get a Drupal site installed and running for testing or site building, and measure a few benchmarks to help determine which one(s) might be best for Drupal newcomers.

For reference, here's a spreadsheet I maintain of all the community-maintained local Drupal development environment tools I've heard of.

Throughout the week at DrupalCon, I've been adding automated scripts to build new Drupal environments, seeing what makes different development tools like Lando, Drupal VM, Docksal, DDEV, SimplyTest.me, and even Drupal core (using code from the active issue Provide a single command to install and run Drupal) tick. And now I've compiled some benchmark data to help give an overview of what's involved with the different tools, for someone who's never used the tool (or Drupal) before.

All the code for these benchmarks is available under an open source license in the Drupal, the Fastest project on GitHub.

Time to Drupal

One of my favorite metrics is "time to Drupal": basically, how long does it take, at minimum, for someone who just discovered a new tool to install a new Drupal website and have it running (front page accessible via the browser) locally?

Time to Drupal - how long it takes different development environments to go from nothing to running Drupal

The new drupal quick-start command that will be included with Drupal core once this patch is merged is by far the fastest way to go from "I don't have any local Drupal environment" to "I'm running Drupal and can start playing around with a fresh new site." And being that it's included with Drupal core (and doesn't even require something like Drush to run), I think it will become the most widely used way to quickly test out and even build simple Drupal sites, just because it's easy and fast!

But the drupal quick-start environment doesn't have much in the way of extra tools, like an email catching system (to prevent your local environment from accidentally sending emails to people), a debugger (like XDebug), a code profiler (like Blackfire or Tideways), etc. So most people, once they get into more advanced usage, would prefer a more fully-featured local environment.

There's an obvious trend in this graph: Docker-based environments are generally faster to get up and running than a Vagrant-based environment like Drupal VM, mostly because the Docker environments use pre-compiled/pre-installed Docker images, instead of installing and configuring everything (like PHP, MySQL, and the like) inside an empty VirtualBox VM.

Now, 'time to Drupal' isn't the only metric you should care about—there are some good reasons people may choose something like Drupal VM over a Docker-based tool—but it is helpful to know that some tools are more than twice as fast as others when it comes to getting Drupal up and running.

Required Dependencies

Another important aspect of choosing one of these tools is realizing what you will need to have installed on your Computer to make that tool work. All options require you to have something installed, whether it be just PHP and Composer, or a virtualization environment like VirtualBox or Docker.

How many dependencies are required per development environment

Almost all Drupal local development environment tools are settling on either requiring Docker CE, or requiring Vagrant and VirtualBox. The two exceptions here are:

  1. SimplyTest.me runs in the cloud, so it doesn't require any dependencies locally. But you can't run the site you create in SimplyTest.me locally either, so that's kind of a moot point!
  2. Drupal's quick-start command requires PHP and Composer, which in some sense are less heavyweight to install locally (than Docker or VirtualBox)—but in another sense can be a little more brittle (e.g. you can only easily install and one version of PHP at a time—a limitation you can bypass easily by having multiple PHP Docker or VM environments).

Overall, though, the number of required dependencies shouldn't turn you off from any one of these tools, unless there are corporate policies that restrict you from installing certain software on your workstation. Docker, VirtualBox, PHP, Composer, Git and the like are pretty common tools for any developer to have running and updated on their computer.

Number of setup steps

While the raw number of steps involved in setting up a local environment is not a perfect proxy for how complex it is to use, it can be frustrating when it takes many commands just to get a new thing working. Consider, for example, most Node.js projects: you usually install or git clone a project, then run npm install, and npm start. The fewer steps involved, the less can go wrong!

How many steps are involved in the setup of a Drupal development tool

The number of individual steps required for each environment varies pretty wildly, and some tools are a little easier to start using than others; whereas Drupal VM relies only on vagrant, and doesn't require any custom command line utility or setup to get started, tools like Lando (lando), Docksal (fin), and DDEV (ddev) each require a couple extra steps to first add a CLI helper utility, then to initialize the overall Docker-based environment—then your project runs inside the environment.

In reality, the tradeoff is that since Docker doesn't include as many frills and plugins as something like Vagrant, Docker-based environments usually require some form of wrapper or helper tool to make managing multiple environments easier (setting up DNS, hostnames, http proxy containers and the like).


In the end, these particular benchmarks don't paint a perfect picture of why any individual developer should choose one local development environment over another. All the environments tested have their strengths and weaknesses, and it's best to try one or two of the popular tools before you settle on one for your project.

But I'm most excited for the new drupal quick-start command that is going to be in core soon—it will make it a lot faster and easier to do something like clone drupal and test a core patch or new module locally within a minute or two, tops! It's not a feature-filled local development environment, but it's fast, it only requires people have PHP (and maybe Composer or Git) installed, and it works great on Mac, Windows (7, 8, or 10!), and Linux.

Apr 11 2018
Apr 11

Note: If you want to install and use PHP 7 and Composer within Windows 10 natively, I wrote a guide for that, too!

[embedded content]

Since Windows 10 introduced the Windows Subsystem for Linux (WSL), it has become far easier to work on Linux-centric software, like most PHP projects, within Windows.

To get the WSL, and in our case, Ubuntu, running in Windows 10, follow the directions in Microsoft's documentation: Install the Windows Subsystem for Linux on Windows 10, and download and launch the Ubuntu installer from the Windows Store.

Once it's installed, open an Ubuntu command line, and let's get started:

Install PHP 7 inside Ubuntu in WSL

Ubuntu has packages for PHP 7 already available, so it's just a matter of installing them with apt:

  1. Update the apt cache with sudo apt-get update
  2. Install PHP and commonly-required extensions: sudo apt-get install -y git php7.0 php7.0-curl php7.0-xml php7.0-mbstring php7.0-gd php7.0-sqlite3 php7.0-mysql.
  3. Verify PHP 7 is working: php -v.

If it's working, you should get output like:

PHP 7 running under Ubuntu under WSL on Windows 10

Install Composer inside Ubuntu in WSL

Following the official instructions for downloading and installing Composer, copy and paste this command into the CLI:

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" && \
php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;" && \
php composer-setup.php && \
php -r "unlink('composer-setup.php');"

To make Composer easier to use, run the following command to move Composer into your global path:

sudo mv composer.phar /usr/local/bin/composer

Now you can run composer, and you should get the output:

Composer running under Ubuntu under WSL on Windows 10

That's it! Now you have PHP 7 and Composer running inside Ubuntu in WSL on your Windows 10 PC. Next up, dominate the world with some new PHP projects!

Apr 10 2018
Apr 10

The Drupal community has been on an interesting journey since the launch of Drupal 8 in 2015. In the past three years, as the community has started to get its sea legs 'off the island' (using tools, libraries, and techniques used widely in the general PHP community), there have been growing pains.

One area where the pains have been (and sometimes still are) highly visible is in how Drupal and Composer work together. I've written posts like Composer and Drupal are still strange bedfellows in the past, and while in some ways that's still the case, we as a community are getting closer and closer to a nirvana with modern Drupal site building and project management.

For example, in preparing a hands-on portion of my and Matthew Grasmick's upcoming DrupalCon Nashville lab session on Composer and Drupal, I found that we're already to the point where you can go from literally zero to a fully functional and complete Drupal site codebase—along with a functional local development environment—in about 10 or 15 minutes:

  1. Make sure you have PHP, Composer, and Docker CE installed (Windows users, look here).
  2. Create a Drupal codebase using the Drupal Composer Project: composer create-project drupal-composer/drupal-project:8.x-dev drupal8 --stability dev --no-interaction
  3. Open the project directory: cd drupal8
  4. Add a plugin to build a quick and simple local dev environment using Drupal VM Docker Composer Plugin: composer require --dev geerlingguy/drupal-vm-docker
  5. Start the local dev environment: docker-compose up -d
  6. Open a browser and visit http://localhost/

I'm not arguing that Drupal VM for Docker is the ideal local development environment—but accounting for about one hour's work last night, I think this shows the direction our community can start moving once we iron out a few more bumps in our Composer-y/Drupal-y road. Local development environments as Composer plugins. Development tools that automatically configure themselves. Cloud deployments to any hosting provider made easy.

Right now a few of these things are possible. And a few are kind of pipe dreams of mine. But I think this year's DrupalCon (and the follow-up discussions and issues that will result) will be a catalyst for making Drupal and Composer start to go from being often-frustrating to being extremely slick!

If you want to follow along at home, follow this core proposal: Proposal: Composer Support in Core initiative. Basically, we might be able to make it so Drupal core's own Composer usage is good enough to not need a shim like drupal-composer/drupal-project, or a bunch of custom tweaks to a core Composer configuration to build new Drupal projects!

Also, I will be working this DrupalCon to help figure out new and easier ways to make local development easier and faster, across Mac, Linux and Windows (I even brought my clunky old Windows 10/Fedora 26 laptop with me!). There are a number of related BoFs and sessions if you're here (or to watch post-conference), for example:

  • Tuesday
  • Wednesday
Apr 09 2018
Apr 09

Note: If you want to install and use PHP 7 and Composer within the Windows Subsystem for Linux (WSL) using Ubuntu, I wrote a guide for that, too!

[embedded content]

I am working a lot on Composer-based Drupal projects lately (especially gearing up for DrupalCon Nashville and my joint workshop on Drupal and Composer with Matthew Grasmick), and have been trying to come up with the simplest solutions that work across macOS, Linux, and Windows. For macOS and Linux, getting PHP and Composer installed is fairly quick and easy. However, on Windows there seem to crop up little issues here and there.

Since I finally spent a little time getting the official version of PHP for native Windows installed, I figured I'd document the process here. Note that many parts of this process were learned from the concise article Install PHP7 and Composer on Windows 10 from the website KIZU 514.

Install PHP 7 on Windows 10

PHP 7 running in Windows 10 in PowerShell

  1. Install the Visual C++ Redistributable for Visual Studio 2015—this is linked in the sidebar of the PHP for Windows Download page, but it's kind of hidden. If you don't do this, you'll run into a rather cryptic error message, VCRUNTIME140.DLL was not found, and php commands won't work.
  2. Download PHP for Windows. I prefer to use 7.1.x (current release - 1), so I downloaded the latest Non-thread-safe 64-bit version of 7.1.x. I downloaded the .zip file version of the VC14 x64 Non Thread Safe edition, under the PHP 7.1 heading.
  3. Expand the zip file into the path C:\PHP7.
  4. Configure PHP to run correctly on your system:
    1. In the C:\PHP7 folder, rename the file php.ini-development to php.ini.
    2. Edit the php.ini file in a text editor (e.g. Notepad++, Atom, or Sublime Text).
    3. Change the following settings in the file and save the file:
      1. Change memory_limit from 128M to 1G (because Composer can use lots of memory!)
      2. Uncomment the line that reads ; extension_dir = "ext" (remove the ; so the line is just extension_dir = "ext").
      3. In the section where there are a bunch of extension= lines, uncomment the following lines:
        1. extension=php_gd2.dll
        2. extension=php_curl.dll
        3. extension=php_mbstring.dll
        4. extension=php_openssl.dll
        5. extension=php_pdo_mysql.dll
        6. extension=php_pdo_sqlite.dll
        7. extension=php_sockets.dll
  5. Add C:\PHP7 to your Windows system path:
    1. Open the System Control Panel.
    2. Click 'Advanced System Settings'.
    3. Click the 'Environment Variables...' button.
    4. Click on the Path row under 'System variables', and click 'Edit...'
    5. Click 'New' and add the row C:\PHP7.
    6. Click OK, then OK, then OK, and close out of the System Control Panel.
  6. Open PowerShell or another terminal emulator (I generally prefer cmder), and type in php -v to verify PHP is working.

At this point, you should see output like:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\jgeerling> php -v
PHP 7.0.29 (cli) (built: Mar 27 2018 15:23:04) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies

This means PHP is working, yay!

Install Composer on Windows 10

Composer running in Windows 10 in PowerShell

Next, we're going to install Composer by downloading it and moving it into place so we can run it with just the composer command:

  1. Download the Windows Installer for Composer and run it.
  2. Note that the Windows Installer for Composer might ask to make changes to your php.ini file. That's okay; allow it and continue through the setup wizard.
  3. Close out of any open PowerShell or other terminal windows, and then open a new one.
  4. Run the composer command, and verify you get a listing of the Composer help and available commands.

That's it! Now you have PHP 7 and Composer running natively on your Windows 10 PC. Next up, dominate the world with some new PHP projects!

Mar 22 2018
Mar 22

For the past two minor release Drupal core upgrades, I've had major problems trying to get some of my Composer-based Drupal codebases upgraded. For both 8.3.x to 8.4.0, and now 8.4.x to 8.5.0, I've had the following issue:

  1. I have the version constraint for drupal/core set to ~8.0 or ~8.4 in my composer.json.
  2. I run composer update drupal/core --with-dependencies (as recommended in Drupal.org's Composer documentation).
  3. Composer does its thing.
  4. A few things get updated... but not drupal/core. It remains stubbornly on the previous minor release.

Looking around the web, it seems this is a very common problem, and a lot of people soon go for the nuclear (or thermonuclear1) option:

  1. Run composer update (updating everything in the entire project, contrib modules, core, other dependencies, etc.).
  2. Profit?

This works, but it's definitely not ideal. If you have a site that uses a number of contrib modules, and maybe even depends on some of their APIs in custom code or in a custom theme... you don't want to be upgrading core and all contrib modules all in one go. You want to update each thing independently so you can test and make sure things don't break.

So, I was searching around for 'how do I figure out why updating something with Composer doesn't update that thing?', and I got a few good answers. The most important is the command composer prohibits.

Use composer prohibits to figure out what's blocking an update

composer prohibits allows you to see exactly what is preventing a package from being updated. For example, on this codebase, I know I want to end up with drupal/core:8.5.0, so I can run:

composer prohibits drupal/core:8.5.0

This gave me a list of a ton of different Symfony components that seemed to be holding back the upgrade, for example:

drupal/core                     8.5.0       requires          symfony/class-loader (~3.4.0)
drupal-composer/drupal-project  dev-master  does not require  symfony/class-loader (but v3.2.14 is installed)
drupal/core                     8.5.0       requires          symfony/console (~3.4.0)
drupal-composer/drupal-project  dev-master  does not require  symfony/console (but v3.2.14 is installed)
drupal/core                     8.5.0       requires          symfony/dependency-injection (~3.4.0)

Add any blocking dependencies to composer update

So, knowing this, one quick way I can get around this problem is to include symfony/* to update the symfony components at the same time as drupal/core:

composer update drupal/core symfony/* --with-dependencies

Unfortunately, it's more difficult to figure out which dependencies exactly are blocking the update. You'd think all of the ones listed by composer prohibits are blocking the upgrade, but as it turns out, only the symfony/config dependency (which is a dependency of Drush and/or Drupal Console, but not Drupal core) was blocking the upgrade on this particular site!

I learned a bit following the discussion in the Drupal.org issue composer fail to upgrade from 8.4.4 to 8.5.0-alpha1, and I also contributed a new answer to the Drupal Answers question Updating packages with composer and knowing what to update. Finally, I updated the Drupal.org documentation page Update core via Composer (option 4) and added a little bit of the information above in the troubleshooting section, because I'm certain I'm not the only one hitting these issues time and again, every time I try to upgrade Drupal core using Composer!

1The thermonuclear option with Composer is to delete your vendor directory, delete your lock file, hand edit your composer.json with newer package versions, then basically start over from scratch. IMO, this is always a bad idea unless you feel safe upgrading all the things all the time (for some simple sites, this might not be the worst idea, but it still removes all the dependency management control you get when using Composer properly).

Mar 12 2018
Mar 12

Umami demo profile running on Lando for Drupal 8
Testing out the new Umami demo profile in Drupal 8.6.x.

I wanted to post a quick guide here for the benefit of anyone else just wanting to test out how Lando works or how it integrates with a Drupal project, since the official documentation kind of jumps you around different places and doesn't have any instructions for "Help! I don't already have a working Drupal codebase!":

  1. Install Docker for Mac / Docker for Windows / Docker CE (if it's not already installed).
  2. Install Lando (on Mac, brew cask install lando, otherwise, download the .dmg, .exe., .deb., or .rpm).
  3. You'll need a Drupal codebase, so go somewhere on your computer and use Git to clone it: git clone --branch 8.6.x https://git.drupal.org/project/drupal.git lando-d8
  4. Change into the Drupal directory: cd lando-d8
  5. Run lando init, answering drupal8, ., and Lando D8.
  6. Run lando start, and wait while all the Docker containers are set up.
  7. Run lando composer install (this will use Composer/PHP inside the Docker container to build Drupal's Composer dependencies).
  8. Go to the site's URL in your web browser, and complete the Drupal install wizard with these options:
    1. Database host: database
    2. Database name, username, password: drupal8

At the end of the lando start command, you'll get a report of 'Appserver URLs', like:

APPSERVER URLS  https://localhost:32771                        

You can also get this info (and some other info, like DB connection details) by running lando info. And if you want to install Drupal without using the browser, you could run the command lando drush site-install -y [options] (this requires Drush in your Drupal project, which can be installed via composer require drush/drush).

It looks like Lando has a CloudFlare rule set up that redirects *.lndo.site to, and the https version uses a bare root certificate, so if you want to access the HTTPS version of the default site Lando creates, you need to add an exception to your browser when prompted.

Note that Little Snitch reported once or twice that the lando cli utility was calling out to a metrics site, likely with some sort of information about your environment for their own metrics and tracking purposes. I decided to block the traffic, and Lando still worked fine, albeit with a few EHOSTDOWN errors. It looks like the data it tried sending was:


Nothing pernicious; I just don't like my desktop apps sending metrics back to a central server.

Docker Notes

For those who also use many other dev environments (some of us are crazy like that!), I wanted to note a few things specific to Lando's Docker use:

  • Lando starts 3 containers by default: one for MySQL, one for Apache/PHP, and one for Traefik. The Traefik container grabs host ports 80, 443, and 58086.
  • Running lando stop doesn't stop the Traefik container, so whatever ports it holds won't be freed up until you run lando poweroff.
  • If another running container (or some other service) is already binding to port 80, lando start will find a different free port and use that instead (e.g. http://lando-d-8.lndo.site:8000).
Aug 31 2017
Aug 31
In part one of this two-part series, I explained why my Hackathon team wanted to build an open source photo gallery in Drupal 8, and integrate it with Amazon S3, Rekognition, and Lambda for face and object recognition.In this post, I'll detail how we built it, then how you can set it up, too!


About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web