Jan 16 2017
Jan 16

I recently needed to re-save all the nodes of a particular content type (after I had added some fields and default configuration) as part of a Drupal 8 site update and deployment. I could go in after deploying the new code and configuration, and manually re-save all content using the built-in bulk operation available on the /admin/content page, but that would not be ideal, because there would be a period of time where the content isn't updated on the live site—plus, manual processes are fragile and prone to failure, so I avoid them at all costs.

In my Drupal 8 module, called custom, I added the following update hook, inside custom.install:

<?php
// Add this line at the top of the .install file.
use Drupal\node\Entity\Node;/**
* Re-save all Article content.
*/
function custom_update_8002() {
// Get an array of all 'article' node ids.
$article_nids = \Drupal::entityQuery('node')
->
condition('type', 'article')
->
execute();// Load all the articles.
$articles = Node::loadMultiple($article_nids);
foreach (
$articles as $article) {
$article->save();
}
}
?>

Though Drupal 8's configuration management system allows almost any config changes to be made without update hooks nowadays... I find I still need to use update hooks on many sites to deploy updates that affect the way a theme or a view displays content on the site (especially when adding new fields to existing content types).

Jan 14 2017
Jan 14

After migrating an older Drupal 6 site with 20,000 media items to Drupal 8, I found a strange problem with image uploads. On the Drupal site, using Image FUpload and Adobe Flash, I could upload up to 99 images in one go. On the new Drupal 8 site, I was only able to upload 20 images, even though I didn't see an error message or any other indication that the rest of the images I had selected through the Media Image upload form were not successfully added.

I could choose 21, 40, or 500 images, but only 20 were ever added to an album at any time.

There were no apparent warnings on the screen, so I just assumed there was some random bug in the Media Image Entity or Media module suite that limited uploads to 20 files at a time.

But due to an unrelated error, I glanced at the PHP logs one day, and noticed the following error message:

[Fri Dec 23 22:05:53.403709 2016] [:error] [pid 29341] [client ip.address.here:41316] PHP Warning:  Maximum number of allowable file uploads has been exceeded in Unknown on line 0, referer: https://example.com/entity-browser/modal/image_browser?uuid=b6e0c064758fc25f517d276e265585959f18361a&original_path=/node/add/gallery

Quite enlightening!

So looking at the PHP docs for file uploads, it seems the default limit is 20 files. That's a bit low for a photo sharing site, so I decided to lift that limit to 250 for the benefit of family members who are a bit trigger-happy when taking pictures!

I edited php.ini and set the following directive:

max_file_uploads = 250

And now I can upload to my heart's content, without manually batching uploads in groups of 20!

Jan 13 2017
Jan 13

One project I'm working on needed a Behat test added to test whether a particular redirection works properly. Basically, I wanted to test for the following:

  1. An anonymous user accesses a file at a URL like http://www.example.com/pictures/test.jpg
  2. The anonymous user is redirected to the path http://www.example.com/sites/default/files/pictures/test.jpg

Since the site uses Apache, I added the actual redirect to the site's .htaccess file in the docroot, using the following Rewrite configuration:

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Rewrite requests for /profile_images to public files directory location.
  RewriteRule ^ pictures/(.*)$ /sites/default/files/pictures/$1 [L,NC,R=301]
</IfModule>

Testing with curl --head, I could see that the proper headers were set—Location was set to the correct redirected URL, and the response gave a 301. So now I had to add the Behat test.

First, I created a .feature file to contain Redirection tests for my project (since it uses Acquia's BLT, I placed the file in tests/behat/features so it would automatically get picked up when running blt tests:behat):

Feature: Redirects
  In order to verify that redirects are working
  As a user
  I should be able to load a URL
  And I should be redirected to another URL

  # Note: Can't use "When I visit 'path'" because it expects a 200.
  @mink:goutte
  Scenario: Test a Picture redirect.
    Given I am not logged in
    When I do not follow redirects
      And I am on "/pictures/xyz.jpg"
    Then I am redirected to "/sites/default/files/pictures/xyz.jpg"

There are a couple unique aspects of this feature file which are worth highlighting:

  1. Testing redirects requires the GoutteDriver, so I've tagged the scenario (@mink:goutte) to indicate this requirement—see notes on the page behat: intercepting the redirection with behat and mink.
  2. I had to add the line When I do not follow redirects to make sure I can intercept the browser and tell it to not follow redirects automatically. By default, it will follow redirects on the And I am on [path] line, and that would make my ability to test the actual redirection impossible.
  3. The Then I am redirected to [path] line is where the magic happens. I had to write a custom Behat step definition to teach Behat how to test the redirection.

If I ran the test at this point, it would fail on the When I do not follow redirects line, because Behat doesn't yet know how to not follow redirects. So I need to teach it by adding two step definitions to my FeatureContext class. Here's the class in it's entirety:

<?php
<?phpnamespace Drupal;

use

Drupal\DrupalExtension\Context\RawDrupalContext;
use
Behat\Behat\Context\SnippetAcceptingContext;// Needed for assert* functions from PHPUnit.
require_once '../../../../vendor/phpunit/phpunit/src/Framework/Assert/Functions.php';/**
* FeatureContext class defines custom step definitions for Behat.
*/
class FeatureContext extends RawDrupalContext implements SnippetAcceptingContext {/**
* @When /^I do not follow redirects$/
*/
public function iDoNotFollowRedirects() {
$this->getSession()->getDriver()->getClient()->followRedirects(false);
}
/**
* @Then /^I (?:am|should be) redirected to "([^"]*)"$/
*/
public function iAmRedirectedTo($actualPath) {
$headers = $this->getSession()->getResponseHeaders();
assertTrue(isset($headers['Location'][0]));$redirectComponents = parse_url($headers['Location'][0]);
assertEquals($redirectComponents['path'], $actualPath);
}

}

?>

Notes for the customized FeatureContext:

  1. I had to require PHPUnit's functions to be able to assert certain things in the redirect test step definition, so I've manually loaded that file with require_once.
  2. iDoNotFollowRedirects() allows me to disable Goutte's automatic redirect handling.
  3. iAmRedirectedTo() is where the magic happens:
    1. The response headers for the original request are retrieved.
    2. The Location URL is extracted from those headers.
    3. The path (sans protocol/domain/port) is extracted.
    4. I assert that the path from the Location header matches the path that is being tested.

Now, when I run the test on my local environment (which runs Apache, so the .htaccess redirect is used), I get the following result:

1 scenario (1 passed)
4 steps (4 passed)
0m11.24s (31.16Mb)

Unfortunately, I then realized that the tests in our CI environment are currently using Drush's embedded PHP webserver, which doesn't support/use Apache .htaccess files. Therefore I'll either have to set up the CI environment using Apache instead of Drush, or use some other means of testing for the proper redirection (e.g. using PHPUnit to verify the right syntax appears inside the .htaccess file directly).

Jan 11 2017
Jan 11

Yesterday I presented Drupal VM Tips & Tricks at the DrupalDC meetup, remotely. I didn't have a lot of time to prepare anything for the presentation, but I thought it would be valuable to walk through some of the neat features of Drupal VM people might not know about.

Here's the video from the presentation:

[embedded content]

Some relevant links mentioned during the presentation:

Dec 30 2016
Dec 30

Note: Extra special thanks to Doug Vann for providing motivation to finally post this blog post!

[embedded content]

Early in 2016, when the Search API and Solr-related modules for Drupal 8 were in early alpha status, I wrote the blog post Set up a faceted Apache Solr search page on Drupal 8 with Search API Solr and Facets.

That post was helpful during the painful months when Solr search in Drupal 8 was still in a very rough state, but a lot has changed since then, and Solr-based search in Drupal 8 is in a much more stable (and easy-to-configure) place today, so I thought I'd finally write a new post to show how simple it is to build faceted Solr search in Drupal 8, almost a year later.

Build a local development environment with Apache Solr

These days I always build and maintain Drupal sites locally using Drupal VM; doing this allows me to set up a development environment exactly how I like it, and doing things like adding Apache Solr is trivial. So for this walkthrough, I'll start from square one, and show you how to start with absolutely nothing, and build faceted search on a new Drupal 8 site, using Drupal VM and a Composer file:

Download Drupal VM and follow the Quick Start Guide, then add the following config.yml inside the Drupal VM folder to ensure Apache Solr is installed:

---
# Make sure Apache Solr is installed inside the VM.
installed_extras:
  - drush
  - mailhog
  - solr

# Configure Solr for the Search API module.
post_provision_scripts:
  - "../examples/scripts/configure-solr.sh"

# Use custom drupal.composer.json to build and install site.
build_composer: true
build_composer_project: false
drupal_core_path: "{{ drupal_composer_install_dir }}/docroot"
drupal_composer_dependencies:
  - "drupal/devel:1.x-dev"
  - "drupal/search_api:^1.0"
  - "drupal/search_api_solr:^1.0"
  - "drupal/facets:^1.0"

We're going to use Drupal VM's Composer integration to generate a Drupal site that has a Composer-based Drupal install in the synced folder path (by default, in a drupal subdirectory inside the Drupal VM folder).

The drupal_composer_dependencies variable tells Drupal VM to composer require the modules necessary to get Search API Solr and Facets. The post_provision_scripts script is included with Drupal VM, and will configure the version of Apache Solr installed with Drupal VM appropriately for use with the Search API Solr module.

Copy the example.drupal.composer.json to drupal.composer.json, then run vagrant up to build the local development environment and download all the Drupal code required to get started with search.

Note: If you are setting search up on an existing site or don't want to use Drupal VM, download and install the Search API, Search API Solr, and Facets modules manually, and make sure you have Apache Solr running and a search core configured with the latest Search API Solr module version's configuration.

Install the modules

If you want to install the modules via Drupal's UI:

  1. Go to http://drupalvm.dev/, then log in as the administrator (default username and password is admin/admin).
  2. Go to the Extend page (/admin/modules), and enable "Search API", "Facets", "Solr search", and "Solr Search Defaults".

Enable Search API Solr and Facet modules in Drupal 8 UI

If you want to install the modules via Drush:

  1. Run drush @drupalvm.drupalvm.dev en -y facets search_api search_api_so lr search_api_solr_defaults

You should also uninstall the core 'Search' module if it's installed—it is not required for Search API or Facets, and will continue to store extra junk in your site's database if it is installed.

Configure the Solr server

Visit the Search API configuration page, and edit the default Solr Server, making the following changes:

  • Change 'Solr core' to collection1 (default is d8).

Search API Solr search core configuration

At this point, on the server's status page (/admin/config/search/search-api/server/default_solr_server), you should see a message "The Solr server could be reached" and "The Solr core could be accessed":

Apache Solr server connection details in Search API configuration

Once this is done, uninstall the Solr Search Defaults module (drush @drupalvm.drupalvm.dev pmu -y search_api_solr_defaults); this module is no longer required after initial install, as the configuration for your Solr server is now part of the site's active configuration store and won't be deleted.

Configure the Solr index

The Solr Search Defaults module creates a default content index containing all published nodes on the website. In our case, that means all Basic pages and Articles would be included.

You don't need to change anything in the index configuration, but if you want to have a poke around and see how it's set up and what options you have in terms of the data source, fields, or processors, visit /admin/config/search/search-api/index/default_solr_index/edit.

Add default content (if you don't have any)

Assuming you built this site using Drupal VM, it's likely the site is barren, with no content whatsoever to be indexed. To fix that, you can use the Devel module's handy companion, Devel generate:

  1. Enable Devel generate: drush @drupalvm.drupalvm.dev en -y devel_generate
  2. Generate dummy content: drush @drupalvm.drupalvm.dev generate-content 100
    • Note: At the time of this writing, the Drush command didn't result in generated content. Use the UI at /admin/config/development/generate/content if the Drush command isn't generating content.

Now you have a bunch of nodes you can index and search!

Confirm Search indexing is working

It's best to let your production Solr servers wait a couple minutes before freshly-indexed content are made available to search; this way searches are a little more performant as Solr can batch its update operations. But for local development it's nice to have the index be up-to-date as quickly as possible for testing purposes, so Drupal VM's configuration tells Solr to update it's search index immediately after Drupal sends any content.

So, if you generated content with Devel generate, then visit the Index status page for the default search index (/admin/config/search/search-api/index/default_solr_index), you should see all the content on the site indexed:

100 percent of content indexed in Search API

If you're working on an existing site, or if all the content isn't yet indexed for some reason, you can manually index all content by clicking the 'Index now' button and waiting for the operation to complete.

Note that indexing speed can vary depending on the complexity of your site. If you have a site with many complex node types and hundreds of thousands or millions of nodes, you'll need to use more efficient methods for indexing, or else you'll be waiting months for all your content to be searchable!

Make a Faceted Solr Search View

The Solr Search Defaults module creates an example Views-based search page, which you can access at /solr-search/content. It should already be functional, since your content is indexed in Solr (try it out!):

Function Search Content page with Search API Solr in Drupal 8

For many sites, this kind of general keyword-based site search is all that you'd ever need. But we'll spruce it up a bit and make it more functional by changing the path and adding a Content Type Facet.

First, modify the view by visiting /admin/structure/views/view/solr_search_content:

  1. Change the Title to 'Search' (instead of 'Search Content').
  2. Change the Path to '/search' (instead of '/solr-search/content').
  3. Click 'Save'.

Second, create a Content Type facet by visiting /admin/config/search/facets:

  1. Click 'Add facet'.
  2. Choose the 'View Solr search content, display Page' Facet source (this is the view you just edited).
  3. Select 'Content type (type)' for the Facet's Field.
  4. Name the facet 'Search Facet - Content type' (this will help with placing a block later).
  5. Click 'Save'.
  6. On the Facet edit page:
    1. Check the box to 'Show the amount of results'.
    2. Check the 'List item label' checkbox (this will make the facet show 'Basic page' instead of 'page'—the label instead of the machine name for each item).
    3. Click 'Save'.

The facet is now ready to be placed in your theme so it will appear when the Search view is rendered. Visit the Block layout configuration page (/admin/structure/block), and click 'Place block' in the region where you want the facet to appear. In my theme, I chose the 'Sidebar first' region.

Find 'Search Facet - Content type' (the Facet you just created) and click 'Place block'. Then set the block title to something like 'Filter by Type', and click 'Save block'. You don't need to set specific visibility constraints for the block because the Facet is set to not display at all if there aren't search results on the page that require it to be shown.

Click 'Save blocks' on the Block layout page, and then visit your sitewide search page at /search:

Solr Search page with a basic preconfigured Facet in Drupal 8

If you perform a search, then you'll notice the facet's result counts will adjust accordingly:

Facets with result count for Drupal 8 Search API keyword search page

At this point, you should have a fully operational Faceted Solr search on your Drupal 8 site. From here, you can customize the search page further, work on outputting different results (maybe a content teaser instead of the full rendered content?), and add more facets (date, author, taxonomy term, etc.) to make the search work exactly as you'd like!

Next steps

If your hosting provider doesn't provide an Apache Solr search core for your site to use, you might want to consider using Hosted Apache Solr to host your site's Solr search core; it uses a similar setup to what's used in Drupal VM, and I can vouch for it, since I run the service :)

Note that the Search API modules are still in beta as of this blog post; minor details may result in differences from the screenshots and instructions above.

Dec 20 2016
Dec 20

Limiting the amount of surprises you get when developing a large-scale Drupal project is always a good thing. And to that end, Acquia's BLT (Build and Launch Tools) wisely chooses to leave Drupal VM alone when updating BLT itself. Updates to Drupal VM can and should be done independently of install profile and development and deployment tooling.

composer require geerlingguy/drupal-vm:~4.0

But this creates a conundrum: how do you upgrade Drupal VM within a project that uses BLT and has Drupal VM as one of it's composer dependencies? It's actually quite simple—and since I just did it for one of my projects, I thought I'd document the process here for future reference:

  1. In your project's root, require the newer version of Drupal VM: composer require --dev geerlingguy/drupal-vm:~4.0 (in my case, I was updating from the latest 3.x release to 4.x).
  2. Edit your box/config.yml file—it's best to either use BLT's current config.yml template as a guide for updating yours, or read through the Drupal VM release notes to find out what config variables need to be added, changed, or removed.
  3. Commit the updates to your code repository.
  4. (If updating major versions) Instruct all developers to run vagrant destroy -f, then vagrant up to rebuild their local environments fresh, on the new version. (If updating minor versions) Instruct all developers to run vagrant provision to update their environments.

There are a lot of great new features in Drupal VM 4, like the ability to switch PHP versions in the VM on-the-fly. This is great for those testing migrations from PHP 5.6 to 7.0 or even 7.1! There's never been an easier and quicker way to update your projects to the latest VM version.

Dec 15 2016
Dec 15
For a recent project, I needed to use the popular Bootstrap theme, using the Sass starterkit for easy CSS management.
Dec 10 2016
Dec 10

Drupal VM 4.0.0 Release Tag - We've Got Company on GitHub

Seven months after Drupal VM 3 introduced PHP 7.0 and Ubuntu 16.04 as the default, as well as more stable team-based development environment tooling, Drupal VM 4 is here!

Thanks especially to the efforts of Oskar Schöldström and Thom Toogood, who helped push through some of the more tricky fixes for this release!

If you're not familiar with Drupal VM, it's a tool built with Ansible and Vagrant that helps build Drupal development environments. The fourth release brings with it even more flexibility than before. Not only can you choose between Ubuntu and CentOS, Apache and Nginx, MySQL and PostgreSQL, Memcached and Redis... you can now seamlessly switch among PHP 5.6, 7.0, and 7.1—without having to recreate your entire development environment!

See the 4.0.0 release notes for all the details—here are the highlights:

  • Drush is now optional (you can use the version included with your project, or not use it at all!)
  • PHP 5.6, 7.0 and 7.1 are supported—and switching between them is easier than ever. Just update php_version and run vagrant provision to switch!

Download Drupal VM and try out one of the most popular Vagrant-based development environments.

Oct 04 2016
Oct 04

tl;dr: If you want to skip the 'how-to' part and explanation, check out the pix_migrate example Drupal 8 migration module on GitHub.

For a couple years, I wanted to work on my first personal site migration into Drupal 8, for the last Drupal 6 site I had running on my servers. I've run a family photo/audio/video sharing website since 2009, and through the years it has accumulated hundreds of galleries, and over 20,000 media items.

Family Photos and Events website display - desktop and mobile
The home page of the Drupal 8 photo sharing website.

I did a lot of work to get the Drupal 6 site as optimized and efficient as possible, but after official support for Drupal 6 was over, the writing was on the wall. I was fortunate to have a few days of time when I could rebuild the site in Drupal 8, work on a migration, and build a new theme for the site (fully responsive, with retina/responsive images, modern HTML5 media embeds, etc.!). I already wrote a couple other posts detailing parts of the site build and launch:

And this will be the third post with some takeaways/lessons from the site build. I mostly write these as retrospectives for myself, since I'll likely be building another five or ten migrations for personal Drupal projects, and it's easier to refer back to a detailed blog post than to some old source code. Hopefully it helps some other people as well!

Why I Didn't Use Core's 'Migrate Drupal'

Migrate Drupal is a built-in migration path for Drupal 8 that allows some Drupal 6 or 7 sites to be migrated/upgraded to Drupal 8 entirely via the UI. You just tell the migration where your old database is located, click 'Next' a few times, then wait for the migration to complete. Easy, right?

As it turns out, there are still a lot of things that aren't upgraded, like most file attachments, Views, and of course any module data for modules that haven't been ported to Drupal 8 yet (even for those that are, many don't have migration paths yet).

The module is still in 'experimental' status, and for good reason—for now, unless you have a very simple blog or brochure-style Drupal 6 or 7 site, it's a good idea to spin up a local Drupal 8 site (might I suggest Drupal VM?) and test your migration to see how things go before you commit fully to the Drupal 8 upgrade process.

Building Custom Migrations

Since Migrate Drupal was out, I decided to build my own individual migrations, migrating all the core entities out of Drupal 6 and into Drupal 8. Since this was an SQL/database-based migration, almost everything I needed was baked into Drupal core. I also added a few contributed modules to assist with migrations:

  • Migrate Plus - Adds a few conveniences to the migration pipeline, and also includes the best (and most up-to-date) migration example modules.
  • Migrate Tools - Provides all the drush commands to make working with Migrate through a CLI possible.

Other than that, before working on any migration, I had to rebuild the structure of the website in Drupal 8 anew. For me, this is actually a fun process, as I can rebuild the content architecture from scratch, and ditch or tweak a lot of the little things (like extra fields or features) that I found were not needed from the Drupal 6 site.

One of my features of Drupal 8 is the fact that almost everything you need for any given content type is baked into core. Besides adding a number of media-related modules (see the official Drupal 8 Media Guide for more) to support image, video, and audio entities (which are related to nodes), I only added a few contrib modules to add user registration spam prevention (Honeypot, a nicer Admin Toolbar, and a helpful bundle of Twig extensions via Twig Tweak that helped make theming (especially embedding views) easier.

Here's a high-level overview of the content architecture that drives the site:

Drupal image gallery website content architecture diagram

Or in text form (with a chain of dependencies):

  • Users (since all content belongs to a particular User).
  • Files (in Drupal, all files should be imported first, so they can be attached to entities later).
  • Names (this is a taxonomy, with a bunch of names that can be attached to photos (either via keyword match or manually tagging images).
  • Images (a Media Entity, which can have one User reference, one File reference, and one or more Name references).
  • Galleries (a Content Type, which can have one User reference, and one Image reference).

Once the site's structure was built out, and the basic administrative and end-user UX was working (this is important to make it easier to evaluate how the migration's working), I started building the custom migrations to get all the Drupal 6 data into Drupal 8.

Getting Things in Order

Because Images depend on Users, Files, and Names, I had to set up the migrations so they would always run in the right order. It's hard to import an Image when you don't have the File that goes along with it, or the User that created the Image! Also, Galleries depend on Images, so I had to make sure they were imported last.

Knowing the content structure of the new site, and having the database of the old site already set up (I added both databases in the site's settings.php file like so:

<?php
/**
* Database settings.
*/
$databases['default']['default'] = array (
'database' => 'pix_site',
'username' => 'pix_site',
'password' => 'supersecurepassword',
'prefix' => '',
'host' => '127.0.0.1',
'port' => '3306',
'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
'driver' => 'mysql',
);
$databases['migrate']['default'] = array(
'database' => 'pix_site_old',
'username' => 'pix_site_old',
'password' => 'supersecurepassword',
'prefix' => '',
'host' => '127.0.0.1',
'port' => '3306',
'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
'driver' => 'mysql',
);
?>

I decided to name my custom migration module pix_migrate, so I created a directory with that name, with an info file pix_migrate.info.yml that describes the module and its dependencies:

name: Pix Migrate
type: module
description: 'Migrations for media gallery site Drupal 6 to Drupal 8 upgrade.'
package: custom
version: '8.0'
core: '8.x'
dependencies:
  - migrate_plus

Once I had that structure set up, I could go to the Extend page or use Drush to enable the module: drush en -y pix_migrate; this would also enable all the other required Migrate modules, and at this point, I was ready to start developing the individual migrations!

Aside: Developing and Debugging Migrations

During the course of building a migration, it's important to be able to quickly test mappings, custom import plugins, performance etc. To that end, here are a few of the techniques you can use to make migration development easier:

  • First things first: for every module that generates configuration, the module should also clean up after itself. For a Migration-related module, that means adding a hook_install() implementation in the module's .install file. See this hook_uninstall() example for details.
  • When you test something that relies on configuration in your module's 'install' directory (as we'll see Migrations do), you'll find it necessary to reload that configuration frequently. While you could uninstall and reinstall your module to reload the configuration (drush pmu -y pix_migrate && drush en -y pix_migrate), this is a little inefficient. Instead, you can:
  • Use one of the following modules to allow the quick refresh/reload/re-sync of configuration from your module's install directory (note that there are currently a number of solutions to this generic problem of 'how do I update config from my modules?'; it will be some time before the community gravitates towards one particular solution):
  • When running migrations, there are some really helpful Drush commands provided by the Migrate Tools module, along with some flags that can help make the process go more smoothly:
    • When running a migration with drush migrate-import (drush mi), use --limit=X (where X is an integer) to just import X items (instead of the full set). (Note that there's currently a bug that make multiple runs with --limit not work like you'd expect.)
    • When running a migration with drush migrate-import, use --feedback=X (where X is an integer) to show a status message after every X items has been imported.
    • If a migration errors out or seems to get stuck, use drush migrate-stop, then drush migrate-status to see if the migration kicks back to 'Idle'. If it gets stuck on 'Stopping', and you're sure the Migration's not doing anything anymore (check CPU status), then you can use drush migrate-reset-status to reset the status to 'Idle'.
  • If a migration seems to be very slow, you can use XHProf to profile Migrations via Drush, and hopefully figure out what's causing the slowness.

Migration module file structure

As stated earlier, we have a folder with an .info.yml file, but here's the entire structure we'll be setting up in the course of adding migrations via configuration:

pix_migrate/
  config/
    install/
      [migrations go here, e.g. migrate_plus.migration.[migration-name].yml]
    src/
      Plugin/
        migrate/
          source/
            [Source plugins go here]
  pix_migrate.info.yml
  pix_migrate.install

Our first migration: Users

Note: ALL the code mentioned in this post is located in the GitHub repository TODO. This is the module that I used to do all my imports, using Drupal 8.1.x.

Most of the time, the first migration you'll need to work on is the User migration—getting all the users who authored content into the new system. Since most other migrations depend on the authors being imported already, I usually choose to build this migration first. It also tends to be one of the easiest migrations, since Users usually don't have to be related to other kinds of content.

I'll name all the migrations using the pattern pix_[migration_name] for the machine name, and 'Pix [Migration Name]' for the label. Starting with the User migration, here's how I defined the migration itself, in the file pix_migrate/config/install/migrate_plus.migration.pix_user.yml:

id: pix_user
migration_group: pix
migration_tags: {}
label: 'Pix User'

source:
  plugin: pix_user

destination:
  plugin: 'entity:user'

process:
  name: name
  pass: pass
  mail: mail
  init: init
  status: status
  created: created
  access: access
  login: login
  timezone: timezone_name
  roles:
    plugin: default_value
    default_value: 1

migration_dependencies: {}

dependencies:
  module:
    - pix_migrate

Note that all migration configurations follow the basic pattern:

# Meta info, like id, group, label, tags.

source:
  # Source definition or plugin.

destination:
  # Destination definition or plugin.

process:
  # Field mappings.

# migration_dependencies and dependencies.

For more documentation on the structure of the migration itself, please read through the example modules included with the Migrate Plus module.

For the User migration, the configuration is telling Drupal, basically:

  1. Use the pix_user Migration source plugin (we'll create that later).
  2. Save user entities (using the entity:user Migration destination plugin).
  3. Map fields to each other (e.g. name to name, mail to mail, etc.).
  4. This migration requires the pix_migrate module, but doesn't require any other migrations to be run before it.

We could've used a different source plugin (e.g. the csv or xml plugin) if we were importing from different data sources, but since we are importing from an SQL database, we need to do define our own plugin (don't worry, it's just like Drupal 7 Migrations, but even simpler!), so we'll create that inside pix_migrate/src/Plugin/migrate/source/PixUser.php:

<?php
namespace Drupal\pix_migrate\Plugin\migrate\source;

use

Drupal\migrate\Plugin\migrate\source\SqlBase;
use
Drupal\migrate\Row;/**
* Source plugin for Pix site user accounts.
*
* @MigrateSource(
* id = "pix_user"
* )
*/
class PixUser extends SqlBase {/**
* {@inheritdoc}
*/
public function query() {
return
$this->select('users')
->
fields('users', array_keys($this->fields()))
->
condition('uid', 0, '>')
->
condition('uid', 1, '<>');
}
/**
* {@inheritdoc}
*/
public function fields() {
$fields = [
'uid' => $this->t('User ID'),
'name' => $this->t('Username'),
'pass' => $this->t('Password'),
'mail' => $this->t('Email address'),
'created' => $this->t('Account created date UNIX timestamp'),
'access' => $this->t('Last access UNIX timestamp'),
'login' => $this->t('Last login UNIX timestamp'),
'status' => $this->t('Blocked/Allowed'),
'timezone' => $this->t('Timeone offset'),
'init' => $this->t('Initial email address used at registration'),
'timezone_name' => $this->t('Timezone name'),
];

return

$fields;
}
/**
* {@inheritdoc}
*/
public function getIds() {
return [
'uid' => [
'type' => 'integer',
],
];
}

}

?>

In the plugin, we extend the SqlBase class, and only have to implement three methods:

  1. query(): The query that is used to get rows for this migration.
  2. fields(): The available fields from the source data (you should explicitly list every field from which you need to get data).
  3. getIds(): Which source field (or fields) are the unique identifiers per row of source data.

There are other methods you can add to do extra work, or to preprocess data (e.g. prepareRow()), but you only have to implement these three for a database-based migration.

The User migration is pretty simple, and I'm not going to go into details here—check out the examples in the Migrate Plus module for that—but I will step through a couple other bits of the other migrations that warrant further explanation.

Second migration: Files

The File migration needs to migrate managed files from Drupal 6 to core file entities in Drupal 8; these file entities can then be referenced by 'Image' Media Entities later. The file migration is defined thusly, in pix_migrate/config/install/migrate_plus.migration.pix_file.yml:

id: pix_file
[...]
source:
  plugin: pix_file

destination:
  plugin: 'entity:file'
  source_base_path: https://www.example.com/
  source_path_property: filepath
  urlencode: true
  destination_path_property: uri

process:
  fid: fid
  filename: filename
  uri: filepath
  uid:
    -
      plugin: migration
      migration: pix_user
      source: uid
      no_stub: true
    -
      plugin: default_value
      default_value: 1
[...]

The important and unique parts are in the destination and process section, and I'll go through them here:

First, for the entity:file source plugin, you need to define a few more properties to make sure the migration succeeds. source_base_path allows you to set a base path for the files—in my case, I needed to add the URL to the site so Migrate would fetch the files over HTTP. You could also set a local filesystem path or any other base path that's accessible to the server running the migration. I used urlencode: true to make sure special characters and spaces were encoded (otherwise some file paths would fail). Then I told the plugin to use the filepath from the source to migrate files into the uri in the destination (this is a change from Drupal 6 to Drupal 8 in the way Drupal refers to file locations).

Then, for the process, some of the fields were easy/straight mappings (File ID, name, and path—which is morphed using the rules I set in the destination settings mentioned previously). But for the uid, I had to do a little special formatting. Instead of just a straight mapping, I defined two processors—one the migration plugin, which allows me to define a migration from which a mapping should be used (the pix_user migration from earlier), and the other the default_value plugin.

In my case, I didn't import the user with uid of 1 during the pix_user migration, so I have to tell Migrate first, "don't stub out a user for missing users", then "for any files that don't have a user mapped from the old site, set a default_value of uid 1.

For the PixFile.php plugin definition (which defines the pix_file source plugin), I needed to do a tiny bit of extra work to get the filepath working with references in Drupal 8:

<?php
/**
* {@inheritdoc}
*/
public function prepareRow(Row $row) {
// Update filepath to remove public:// directory portion.
$original_path = $row->getSourceProperty('filepath');
$new_path = str_replace('sites/default/files/', 'public://', $original_path);
$row->setSourceProperty('filepath', $new_path);

return

parent::prepareRow($row);
}
?>

If I didn't do this, the files would show up and get referenced properly for Image media entities, but image thumbnails and other image styles wouldn't be generated (they'd show a 404 error). Note that if your old site's files directory is in a site-specific folder (e.g. sites/example.com/files/), you would need to replace the path using that pattern instead of sites/default/files.

Third migration: Names

This is perhaps the simplest of all the migrations—please see the code in the pix_migrate repository on GitHub. It's self-explanatory, and doesn't depend on any other migrations.

Fourth migration: Image entities referencing files and names

The Image media entity migration is where the rubber meets the road; it has to migrate all the image node content from Drupal 6 into Drupal 8, while maintaining a file reference to the correct file, an author reference to the correct user, and name references to all relevant terms in the Names taxonomy.

First, let's look at the migration definition:

id: pix_image
[...]
source:
  plugin: pix_image

destination:
  plugin: 'entity:media'

process:
  bundle:
    plugin: default_value
    default_value: image
  name: title
  uid:
    -
      plugin: migration
      migration: pix_user
      source: uid
      no_stub: true
    -
      plugin: default_value
      default_value: 1
  'field_description/value': body
  'field_description/summary': teaser
  'field_description/format':
    plugin: default_value
    default_value: basic_html
  field_names:
    plugin: migration
    migration: pix_name
    source: names
  status: status
  created: created
  changed: changed
  'field_image/target_id':
    plugin: migration
    migration: pix_file
    source: field_gallery_image_fid

migration_dependencies:
  required:
    - pix_user
    - pix_file
    - pix_name
[...]

There's a bit to unpack here:

  • We're going to use a pix_image migrate source plugin to tell Migrate about source data, which we'll define later in PixImage.php.
  • We're importing Images as media entities, so we set the destination plugin to entity:media.
  • We want to save Image media entities, so we have to define the bundle (in process) with a default_value of image
  • The uid needs to be set up just like with the pix_file migration, referring to the earlier pix_user migration mapping, but defaulting to uid 1 if there's no user migrated.
  • field_description is a complex field, with multiple possible values to map; so we can map each value independently (e.g. field_description/value gets mapped to the body text in Drupal 6, and field_description/format gets a default value of basic_html.
  • field_names, like uid, needs to refer to the pix_name migration for term ID mappings.
  • field_image/target_id needs to refer to the pix_file migration for file ID mappings.
  • This migration can't be run until Users, Names, and Files have been migrated, so we can explicitly define that dependency in the migration_dependencies section. Set this way, Migrate won't allow this Image migration to be run until all the dependent migrations are complete.

There is also a little extra work that's necessary in the pix_image Migrate source plugin to get the body, summary, and Names term IDs from the source (I chose to do it this way instead of trying to get the ::query() method to do all the necessary joins, just because it was a little easier with the weird database structure in Drupal 6):

<?php
/**
* {@inheritdoc}
*/
public function prepareRow(Row $row) {
// Get Node revision body and teaser/summary value.
$revision_data = $this->select('node_revisions')
->
fields('node_revisions', ['body', 'teaser'])
->
condition('nid', $row->getSourceProperty('nid'), '=')
->
condition('vid', $row->getSourceProperty('vid'), '=')
->
execute()
->
fetchAll();
$row->setSourceProperty('body', $revision_data[0]['body']);
$row->setSourceProperty('teaser', $revision_data[0]['teaser']);// Get names for this row.
$name_tids = $this->select('term_node')
->
fields('term_node', ['tid'])
->
condition('nid', $row->getSourceProperty('nid'), '=')
->
condition('vid', $row->getSourceProperty('vid'), '=')
->
execute()
->
fetchCol();
$row->setSourceProperty('names', $name_tids);

return

parent::prepareRow($row);
}
?>

Final migration: Galleries referencing images

The fifth and final migration puts everything together. Since we changed a little of the content architecture from Drupal 6 to Drupal 8, there's a tiny bit of extra work that goes into getting each Gallery's images related to it correctly. In Drupal 6, the images each referenced a gallery node. In Drupal 8, each Gallery has a field_images field that holds the references to Image media entities.

So we can still map the images the same way as other fields are mapped in the migration configuration:

[...]
process:
  [...]
  field_images:
    plugin: migration
    migration: pix_image
    source: images
[...]

But to get the images field definition correct, we need to populate that field with an array of Image IDs from the Drupal 6 site in the pix_gallery source plugin:

<?php
/**
* {@inheritdoc}
*/
public function prepareRow(Row $row) {
[...]
// Get a list of all the image nids that referenced this gallery.
$image_nids = $this->select('content_type_photo', 'photo')
->
fields('photo', ['nid'])
->
condition('field_gallery_nid', $row->getSourceProperty('nid'), '=')
->
execute()
->
fetchCol();
$row->setSourceProperty('images', $image_nids);

return

parent::prepareRow($row);
}
?>

This query basically grabs all the old 'photo' node IDs from Drupal 6 that had references to the currently-being-imported gallery node ID, then spits that out as an array of node IDs. Migrate then uses that array (stored in the images field) to map old image nodes to new image media entities in Drupal 8.

Conclusion

I often think migrations are full of magic... and sometimes they do seem that way, especially when they work on the first try and migrate a few thousand items at once! But when you dig into them, you find that beneath one simple line of abstraction (e.g. title: title for a field mapping), there is a lot of grunt work that Migrate module does to get the source data reliably and repeatably into your fancy new Drupal 8 site.

This blog post is a little more rough than what I normally would write, and less 'tutorial-y', but I figure that most developers are like me—we learn by doing, but need to see real-world, working examples before the light bulb goes off sometimes. Hopefully I've helped with something through the course of writing this post.

Note that the migrations themselves took me a couple days to set up and debug, and I probably read through 10 other earlier blog posts, every line of certain classes' code, and all the documentation on Drupal.org pertaining to migrations in Drupal 8. Hopefully as time goes on and more examples are published, that aspect of migration development becomes less necessary :)

Oct 01 2016
Oct 01

PostgreSQL elephant transparent PNG
The PostgreSQL logo. Same family as PHP's mascot!

For the past few years, I've been intending to kick the tires of PostgreSQL, an open source RDBMS (Relational DataBase Management System) that's often used in place of MySQL, MariaDB, Oracle, MS SQL, or other SQL-compliant servers. Drupal 7 worked with PostgreSQL, but official support was a bit lacking. For Drupal 8, daily automated test builds are finally being run on MySQL, SQLite, and PostgreSQL, so many of the more annoying bugs that caused non-MySQL database engines to fail have finally been fixed!

With Drupal VM, one of my goals is to be able to replicate almost any kind of server environment locally, supporting all of the most popular software. Developers have already been able to choose Apache or Nginx, Memcached, or Redis, Varnish, Solr or Elasticsearch, and many other options depending on their needs. Today I finally had the time to nail down PostgreSQL support, so now developers can choose which database engine they'd like—MySQL, MariaDB, PostgreSQL, or even SQLite!

As of Drupal VM 3.3.0, all four are supported out of the box, though for MariaDB or PostgreSQL, you need to adjust a couple settings in your config.yml before provisioning.

If you want to build a VM running Drupal 8 on PostgreSQL, the process is pretty simple:

  1. Download Drupal VM and follow the Quick Start Guide.
  2. Before running vagrant up, create a config.yml file with the contents:

---
drupalvm_database: pgsql

  1. Run vagrant up.

After a few minutes, you should have a new Drupal 8 site running on top of PostgreSQL!

PostgreSQL database engine Drupal 8 status report page

A few caveats

You should note that, just like with support for Apache vs. Nginx1, there are far fewer Drupal sites running on PostgreSQL than on MySQL (or MariaDB), so if you choose to use PostgreSQL, you'll likely encounter a bump in the road at some point. For example, to get PostgreSQL working at all with Drupal, the database has to use an older PostgreSQL default output method that uses ASCII instead of hex (the default since PostgreSQL 9.0) for transmission.

If you're planning on digging deeper in to PostgreSQL with Drupal (especially if you need to support things like spatial and geographic objects, or full-text search, and don't want to add on Apache Solr or something like it), you should read through this meta issue for Drupal 8: [meta] Remaining Drupal 8 PostgreSQL issues.

Learn More

1Historically, Apache was used by the vast majority of Drupal sites, so many Drupal features, documentation, and hosting providers assume Apache and either don't consider Nginx configuration, or give it 'second-class' status. That's not to say it's _not_supported... just that you often need to do some of your own due diligence to get everything working smoothly and securely when not using the default option!

Sep 22 2016
Sep 22

Recovering from surgery finally gave me time to update my last D6 site—a 7 year old private photo and media sharing site with nearly 10,000 nodes and 20+ GB of content—to Drupal 8. Drupal 8 has become a lot more mature lately, to the point where I'm comfortable building a site and not having the foundation rot out from beneath as large ecosystem shifts have mostly settled down.

One thing that I thought would have the simplest implementation actually took a little while to figure out. I needed to have users' full name display instead of their usernames throughout the site. For D6 (and for similar D7 use cases), the easiest way to do this was to enable the Realname module, configure it a tiny bit, and be done with it.

In Drupal 8, however, Realname doesn't yet have a full release (see this issue for progress), and the way usernames are generated has changed slightly (see change record hook_username_alter() changed to hook_user_format_name_alter()).

So it took a few minutes' fiddling around before I came up with the following hook implementation that reformats the user's display name using a 'Name' field (machine name field_name) added to the user entity (you can add the field at /admin/config/people/accounts/fields), but only if there's a value for that user:

<?php
/**
* Implements hook_user_format_name_alter().
*/
function custom_user_format_name_alter(&$name, $account) {
// Load the full user account.
$account = \Drupal\user\Entity\User::load($account->id());
// Get the full name from field_name.
$full_name = $account->get('field_name')->value;
// If there's a value, set it as the new $name.
if (!empty($full_name)) {
$name = $full_name;
}
}
?>

Note that there's ongoing discussion in the Drupal core issue queue about whether to remove hook_user_format_name_alter(), since there are some caveats with its usage, and edge cases where it doesn't behave as expected or isn't used at all. I'm hoping this situation will be a little better soon—or at least that there will be a Realname release so people who don't like getting their hands dirty with code don't have to :)

Aug 18 2016
Aug 18

During some migration operations on a Drupal 8 site, I needed to make an HTTP request that took > 30 seconds to return all the data... and when I ran the migration, I'd end up with exceptions like:

Migration failed with source plugin exception: Error message: cURL error 28: Operation timed out after 29992 milliseconds with 2031262 out of 2262702 bytes received (see http://curl.haxx.se/libcurl/c/libcurl-errors.html).

The solution, it turns out, is pretty simple! Drupal's \Drupal\Core\Http\ClientFactory is the default way that plugins like Migrate's HTTP fetching plugin get a Guzzle client to make HTTP requests (though you could swap things out if you want via services.yml), and in the code for that factory, there's a line after the defaults (where the 'timeout' => 30 is defined) like:

<?php
$config
= NestedArray::mergeDeep($default_config, Settings::get('http_client_config', []), $config);
?>

Seeing that, I know at a glance that Drupal is pulling any http_client_config configuration overrides from settings.php and applying them to the Guzzle Clients that this factory creates. Therefore, I can add the following to my site's settings.php to set the default timeout to 60 seconds instead of the default 30:

<?php
/**
 * HTTP Client config.
 */
$settings['http_client_config']['timeout'] = 60;
?>

Pretty simple! You can override any of the other settings this way too, like the proxy settings (there's an example in the default settings.php file), headers, and whether to verify certificates for https requests.

Aug 11 2016
Aug 11

DrupalCamp St. Louis 2016 Logo

The time is here! The rest of the DrupalCamp St. Louis 2016 organizers and I were working feverishly this week to get all our ducks in a row, and we now have online registration opened up for DrupalCamp St. Louis 2016! Here are the relevant details:

You'll get a snazzy T-Shirt, a catered lunch, and the fuzzy warm feeling of being part of the great Drupal open source community! Plus I'll be there!

You can still submit session proposals until August 17—so get your proposal in soon; we'll announce the full set of selected sessions on August 20th!

Aug 04 2016
Aug 04

File this one under the 'it's obvious, but only after you've done it' category—I needed to attach a CSS library to a view in Drupal 8 via a custom module so that, wherever the view displayed on the site, the custom CSS file from my module was attached. The process for CSS and JS libraries is pretty much identical, but here's how I added a CSS file as a library, and made sure it was attached to my view:

Add the CSS file as a library

In Drupal 8, drupal_add_css(), drupal_add_js(), and drupal_add_library() were removed (for various reasons), and now, to attach CSS or JS assets to views, nodes, etc., you need to use Drupal's #attached functionality to 'attach' assets (like CSS and JS) to rendered elements on the page.

In my custom module (custom.module), I added the CSS file css/custom_view.css:

.some-class {
  color: #000;
}

Then, to tell Drupal about the CSS file, I added it as a library inside custom.libraries.yml (alongside the .module file):

custom_view:
  css:
    component:
      css/custom_view.css: {}

In this case, the library's name is the top-level element (custom-view), so when I later want to attach this new library to a page, node, view, etc., I can refer to it as custom/custom_view (basically, [module_name]/[library_name]).

Attach the library to your view

Thank goodness for tests! I was looking through the Drupal core issue queue for issues mentioning views using #attached, and eventually found a patch that referred to the test views_test_data_views_pre_render(), which tests the ability to alter a view using the pre-render hook, and thankfully includes an example for attaching a library.

In my case, learning from that test, I want to attach the library to my view by the view ID (in my case super_awesome_view), so I added the following hook in my .module file:

/**
* Implements hook_views_pre_render().
*/
function custom_views_pre_render(ViewExecutable $view) {
  if (isset($view) && ($view->storage->id() == 'super_awesome_view')) {
    $view->element['#attached']['library'][] = 'custom/custom_view';
  }
}

You may be wondering, "What was wrong in the old days with the simplicity of drupal_add_css()? Well, the main reason why much of Drupal 8's awesome caching abilities are possible is due to the fact that all rendered markup can have cacheability metadata attached, and attaching CSS and Javascript like this allows that caching system to work automatically. In Drupal 7, it was just too messy when any code anywhere could toss in a random stylesheet or JS file outside of Drupal's more rigid Render API.

Aug 01 2016
Aug 01

In Drupal 8, many small things have changed, but my willingness to quickly hack something out in a few lines of code/config instead of installing a relatively large module to do the same thing hasn't :-)

I needed to add a checkbox to control whether the page title should be visible in the rendered page for a certain content type on a Drupal 8 site, and there are a few different ways you can do this (please suggest alternatives—especially if they're more elegant!), but I chose to do the following:

  1. Add a 'Display Title' boolean field (checkbox, using the field label as the title, and setting off to 0 and on to 1 in the field settings) to the content type (page in this example).

    Drupal 8 Basic Page 'Display Title' checkbox

  2. Make sure this field is not set to be displayed in the content type's display settings.
  3. In my theme's hook_preprocess_page (inside themename.theme), add the following:
<?php
/**
* Implements hook_preprocess_page().
*/
function themename_preprocess_page(&$variables) {
// Hide title on basic page if configured.
if ($node = \Drupal::routeMatch()->getParameter('node')) {
if (
$node->getType() == 'page') {
if (!
$node->field_display_title->value) {
unset(
$variables['page']['content']['mysite_page_title']);
}
}
}
}
?>

mysite_page_title is the machine name of the block that you have placed on the block layout page (/admin/structure/block) with the page title in it.

After doing this and clearing caches, the page title for Basic Page content was easy to show and hide based on that simple checkbox. Or you can use the Exclude Node Title module, if you don't want to get your hands dirty!

Jul 18 2016
Jul 18

Today I needed to migrate a URL/Link into a Drupal 8 site, and I was scratching my head over how to migrate it so there were distinct values for the URL (the actual link) and the Label (the 'title' that displays to end users and is clickable). Drupal 8's Link field type allows you to set a URL in addition to an optional (or required) label, but by default, if you just migrate the URL, the label will be blank.

I first set up the migration config like so:

...
process:
  field_url: source_url

And source_url was defined in the migration's source.fields configuration.

In my case, the source data didn't have a label, but I wanted to set a default label so the Drupal 8 site could display that as the clickable link (instead of an ugly long URL). To do that, it's similar to migrating a formatted text field, where you can migrate individual components of the field using the syntax [field_name]/[component]. In a Link field's case, it looks like:

...
process:
  'field_url/uri': source_url
  'field_url/title':
    plugin: default_value
    default_value: 'Click here!'

A lot easier than I was expecting—I didn't have to do anything in PrepareRow or even write my own plugin! I found that the parameters were uri and title by digging into the Link module's field propertyDefinitions, which lists a uri, title, and options (the definition is in Drupal\link\Plugin\Field\FieldType.

Special thanks to Richard Allen for cluing me into this after I was looking for documentation to no avail (he pointed out that Link fields are probably just like the core Body field, which is migrated like so, with body/value, body/format, etc.). He also mentioned that pinging people in the #drupal-migrate IRC channel is usually a helpful way to get help at this point in the game!

Jul 13 2016
Jul 13

On a recent project, there was a Migration run that took a very long time, and I couldn't pinpoint why; there were multiple migrations, and none of the others took very long at all (usually processing at least hundreds if not thousands of nodes per minute). In Drupal 7, if you enabled the XHProf module, then you'd get a checkbox on the configuration page that would turn on profiling for all page requests and Drush commands.

In Drupal 8, the XHProf module was completely rewritten, and as a side effect, the Drush/CLI profiling functionality is not yet present (see: Profile drush/CLI with XHProf in Drupal 8).

Since I don't have the time right now to help figure out how to get things working through the official XHProf module, I decided to use a 'poor man's profiling' method to profile a Migration run:

  1. Find the Migrate module's drush command file (migrate_tools.drush.inc).
  2. Inject the appropriate xhprof functions in the right places to set up and save a profiling run.
  3. Run the drush mi [migration-name] command.
  4. Profit!

In my case, since I was using Drupal VM and it's default XHProf configuration, I had to add the following PHP:

<?php
/**
* @param string $migration_names
*/
function drush_migrate_tools_migrate_import($migration_names = '') {
// Enable XHProf profiling for CPU and Memory usage.
xhprof_enable(XHPROF_FLAGS_CPU + XHPROF_FLAGS_MEMORY);

...

migrate drush command code here ...// Disable XHProf, save the run into the configured directory.
$xhprof_data = xhprof_disable();
$XHPROF_ROOT = "/usr/share/php";
include_once
$XHPROF_ROOT . "/xhprof_lib/utils/xhprof_lib.php";
include_once
$XHPROF_ROOT . "/xhprof_lib/utils/xhprof_runs.php";
$xhprof_runs = new XHProfRuns_Default();
$run_id = $xhprof_runs->save_run($xhprof_data, "xhprof_testing");
}
?>

The $XHPROF_ROOT should point to the directory where you have XHProf installed.

After doing this, and running drush mi [migration-name], I looked at the Drupal VM XHProf runs page (configured by default at http://xhprof.[drupal-vm-name]/), and noticed the new run (the one at the top—the second one in this screenshot was from a run I did while viewing the site in the browser):

XHProf Drupal VM Dashboard page screenshot

See more on the PHP.net XHProf documentation pages, most notably the XHProf example with optional GUI example.

Jun 30 2016
Jun 30
Many developers who work on Drupal (or other web/PHP) projects have error reporting disabled in their local or shared dev environments. They do this for a variety of reasons: some don't know how to enable it, some are annoyed by the frequency of notices, warnings, and errors, and some don't like to be reminded of how many errors are logged.But there are a few important reasons you should make sure to show all errors when developing:
Jun 20 2016
Jun 20

DrupalCamp St. Louis logo - Fleur de Lis

DrupalCamp St. Louis 2016 will be held on September 10-11 in St. Louis, MO, on the campus of the University of Missouri, St. Louis, and we're excited to announce that session submissions are open!

We'd love to hear people speak about Drupal business, case studies, coding, community, DevOps, front end, PHP, project management, security, or any other Drupal topic. If you're interested in speaking, please submit a session for consideration, and we'll announce the selected sessions before August 1st.

Need some ideas for inspiration, or some guidelines for what makes a great DrupalCamp session? Please read through the Session FAQs created by the amazing organizers of MidCamp—the same general things apply to DrupalCamp St. Louis.

Haven't spoken at a DrupalCamp before? Now's your chance to shine!

We're also accepting camp sponsorships. Please see our Camp Sponsors page for information on how you can help support one of the midwest's best Drupal events!

Jun 12 2016
Jun 12

Drupal 8 plus Hosted Apache Solr

After a few months of testing, I'm happy to announce Hosted Apache Solr now supports Search API Solr with Drupal 8! Both Search API and Search API Solr have been getting closer to stable releases, and more people have been requesting Drupal 8 search cores, so I decided to finish testing and updating support guides this weekend.

Note that getting Solr and search pages configured in Drupal 8 is a bit different than in Drupal 7 (even if you were using the Search API module instead of the older Apache Solr Search module)—I posted a detailed setup guide earlier this year: Set up a faceted Apache Solr search page on Drupal 8 with Search API Solr and Facets.

I still haven't migrated either this site or Hosted Apache Solr to Drupal 8 yet, so I don't have a canonical Drupal 8 reference site using Hosted Apache Solr at this time, but I've been working on a few other Drupal 8 sites, and am happy to report I can't imagine building a site in Drupal 7 again, if I can help it :) There are a few rough edges, but all the great new features more than make up for that!

Jun 07 2016
Jun 07

DrupalCamp St. Louis 2016 - Landing page

I wanted to post this as a 'save the date' to any other midwestern Drupalists—here in St. Louis, we'll be hosting our third annual DrupalCamp on September 10 and 11 (Saturday and Sunday) in St. Louis, MO. We'll have sessions on Saturday, and a community/sprint day Sunday, and just like last year, we'll record all the sessions and post them to the Drupal STL YouTube channel after the Camp.

We're still working on a few details (nailing down the location, getting things set up so we can accept session submissions and registrations, etc.), but if you're interested in coming, please head over to the official DrupalCamp STL 2016 website and sign up to be notified of further information!

We're also seeking sponsors this year—read through the sponsors page for sponsorship levels and benefits, and help make this another great community event!

Jun 06 2016
Jun 06

Drupal VM is one of the most flexible and powerful local development environments for Drupal, but one the main goals of the project is to build a fully-functional Drupal 8 site quickly and easily without doing much setup work. The ideal would be to install Vagrant, clone or download the project, then run vagrant up. A few minutes later, you'd have a Drupal 8 site ready for hacking on!

In the past, you always had to do a couple extra steps in between, configuring a drupal.make.yml file and a config.yml file. Recently, thanks in huge part to Oskar Schöldström's herculean efforts, we achieved that ideal by switching from defaulting to a Drush make-based workflow to a Composer-based workflow (this will come in the 3.1.0 release, very soon!). But it wasn't without trial and tribulation!

Before we switched the default from Drush make to Composer, I wanted to get initial build times down so users didn't have to wait for an excruciatingly long time to download Drupal. At first, using all the defaults, it took twice as long to build a Drupal site from a composer.json file or using drupal-project as it did to build the Drupal site from an equivalent make file. The main reason is that Composer spends a lot more time than Drush in resolving project dependencies (recursively reading all composer.json files and downloading all the required projects into the vendor directory).

I thought I'd share some of the things we learned concerning speeding up Composer installs and updates for Drupal (and other PHP projects) in a blog post, so the tips aren't buried in issues in Drupal VM's issue queue:

Use prestissimo

prestissimo is a Composer plugin that enables parallel installations. All you have to do is composer global require "hirak/prestissimo:^0.3", and all composer install commands will use parallel package downloads, greatly speeding up the initial installation of Drupal.

For Drupal VM, the Drupal download time went from 400 seconds to 166 seconds—more than 2x faster for the Composer installation!

Make sure XDebug is disabled

This one should be rather obvious, but many times, developers leave XDebug enabled on the CLI, and this slows down Composer substantially—sometimes making installs take 2-4x longer! Make sure php_xdebug_cli_enable is 0 in Drupal VM's config.yml if you have xdebug installed in the installed_extras list.

(If using Vagrant) Use vagrant-cachier

Many Vagrant power users already use vagrant-cachier with their VMs to cache apt or yum packages so rebuilds are quicker (you don't have to re-download frequently-installed packages anymore); but to use it with Composer, you need add one extra bit of configuration in your Vagrantfile:

if Vagrant.has_plugin?('vagrant-cachier')
  ... any other cachier configuration ...

  # Cache the composer directory.
  config.cache.enable :generic, :cache_dir => '/home/vagrant/.composer/cache'
end

We can't use vagrant-cachier's built-in Composer bucket, because PHP isn't preinstalled on the base boxes Drupal VM uses. So we use a :generic bucket instead, and manually point it at the Composer cache directory inside the VM.

Things that didn't seem to help

  • Shallow Git clones: Some people suggested using shallow Git clones (e.g. following this Composer PR), but it didn't make a measurable difference.
  • "minimum-stability": "dev": In the past, setting minimum-stability to dev could speed things up a bit while Composer sorts out the dependency tree (see this post). It seems to not have any measurable impact in this case, though.
  • There are still some other areas ripe for improvement, too—for example, the drupal-packagist project may be able to improve it's caching infrastructure to greatly speed up download times.

Please let me know if there are other tips and tricks you may have that can help speed up Composer—we've almost hit the same build times with Composer that we hit with Drush make files, but make files are still slightly faster.

May 27 2016
May 27

Any time there are major new versions of software, some of the tooling surrounding the software requires tweaks before everything works like it used to, or as it's documented. Since Drupal 8 and Drush 8 are both relatively young, I expect some growing pains here and there.

One problem I ran into lately was quite a head-scratcher: On Acquia Cloud, I had a cloud hook set up that was supposed to do the following after code deployments:

# Build a Drush alias (e.g. [subscription].[environment]).
drush_alias=${site}'.'${target_env}

# Run database updates.
drush @${drush_alias} updb -y

# Import configuration from code.
drush @${drush_alias} cim vcs

This code (well, with fra -y instead of cim) works fine for some Drupal 7 sites I work on in Acquia Cloud, but it seems that database updates were detected but never run, and configuration changes were detected but never made... it took a little time to see what was happening, but I eventually figured it out.

The tl;dr fix?

# Add --strict=0 to resolve the error Drush was throwing due to alias formatting.
drush @${drush_alias} updb -y --strict=0

# I forgot a -y, so Drush never actually executed the changes!
drush @${drush_alias} cim -y vcs

For the first issue, Acquia cloud generates its own Drush alias files, and in all the aliases, it includes some options like site and env. It seems that Drush < 8 would just ignore extra options like those... but Drush 8.x throws an error and stops execution for the current task because of those extra variables. Using --strict=0 tells Drush to squelch any erros thrown by those extra options. Eventually, I'm guessing Acquia Cloud's Drush aliases will be made to be fully-compatible with Drush 8.x, but this workaround is fine for now.

For the second issue, it was just my boneheadedness... if you're running any command that requires a prompt non-interactively (e.g. through an automated system like cloud hooks), you have to add the 'assume-yes' option, -y, just like I used to with fra -y in Drupal 7!

Before, I would get the following error message on every Cloud deployment:

The following updates are pending:

custom module :
  8001 -   Add customizations.

Do you wish to run all pending updates? (y/n): y
Unknown options: --site, --env.  See `drush help                         [error]
updatedb-batch-process` for available options. To suppress this
error, add the option --strict=0.
Unknown options: --site, --env.  See `drush help cache-rebuild` for      [error]
available options. To suppress this error, add the option --strict=0.
Finished performing updates.                                                [ok]

Even though it says 'Finished performing updates', they didn't actually get run. Now it runs the updates without any issue:

The following updates are pending:

custom module :
  8001 -   Add customizations.

Do you wish to run all pending updates? (y/n): y
Performing custom_update_8001                                                                              [ok]
Cache rebuild complete.                                                                                      [ok]
Finished performing updates.                                                                                 [ok]

May 26 2016
May 26

Raspberry Pi Dramble Cluster with Mini Raspberry Pi Zero Cluster

Another year, another field trip for the Pi Dramble—my 5-Raspberry-Pi cluster! I presented a session titled Highly available Drupal on a Raspberry Pi Cluster at php[tek] 2016, which just so happens to have moved to my hometown, St. Louis, MO this year!

For this presentation, I remembered to record the audio using a lav mic plugged into my iPhone, as well as iShowU to record what was on my screen. Sadly, I didn't have a secondary camera to capture the Pi Dramble itself, but you can glance at all the other 'Let's build a Pi Cluster' videos if you want to see it in action!

Here's a video recording of the presentation:

[embedded content]

And here are the slides:

If you haven't yet seen the Dramble, check out all the details at http://www.pidramble.com/.

It was a fun and lively session, and I thank the php[tek] organizers for allowing me to share my passion for electronics, Ansible, PHP, and Drupal with another great group of people. I'll be giving another talk tomorrow, on a completely unrelated topic, ProTips for Staying Sane while Working from Home.

May 19 2016
May 19

Drupal VM 3.0.0 "The Light Sailer" was just released, and you can grab it from the Drupal VM website now. We spent a lot of time during DrupalCon New Orleans sprinting on Drupal VM, fixing bugs, and updating ALL THE THINGS to make sure this release solves a lot of pain points for individuals and teams who need a great local development environment.

Drupal VM - Website Homepage

Let's get right into why this is the best release of Drupal VM EVER!

The fastest and most modern environment

Drupal VM now defaults to Ubuntu 16.04 (which was just released in late April), running MySQL 5.7 and PHP 7. This means you're getting the fastest, most reliable, and most modern development environment for your Drupal 8 projects.

But you can still stick with any of the old OSes and versions of PHP just like you always could: Ubuntu 16.04, 14.04, and 12.04, as well as CentOS 7 and 6, and even Debian Jessie or Wheezy are supported out of the box! Technically, you can still run any version of PHP from 5.1 to 7.0 in Drupal VM (depending on OS selection)... but only PHP 5.5+ is supported right now.

Also, for 2.5.1, Blackfire.io support was added, so you can now profile in any PHP version with either Blackfire or XHProf! (There was a great session on Blackfire at DrupalCon NOLA.)

The best team-based development environment

New features in Drupal VM allow teams to do the following:

  • Add Drupal VM as a composer.json dependency (that's right, Drupal VM is on Packagist! - docs)
  • Commit a shared config.yml, and let developers override only the settings they need in a local.config.yml (docs)
  • Use Drupal VM and Vagrant in a subfolder, meaning you don't have to cd into the Drupal VM directory to run commands like vagrant up
  • Use a custom Vagrantfile to add in or modify Drupal VM's default Vagrant configuration (docs).
  • Add custom shell scripts and/or Ansible task files to be run before and/or after provisioning (docs)

This is the best release yet for development teams, because Drupal VM can be configured specifically for a particular Drupal site—and then parts of the configuration can be overridden by individual developers without any hacks!

A stable, reliable upgrade path

During the course of the Drupal VM 2.x series, one of the major pain points was upgrading Drupal VM to newer versions. At the Drupal VM BoF at DrupalCon, many people mentioned that every time they upgraded Drupal VM, they ended up with some random errors that caused friction. Even if not upgrading, certain Ansible roles would cause problems with older versions of Drupal VM!

Drupal VM now specifies versions of Ansible roles that are used to build the VM (as of 2.5.1), so if you download Drupal VM today, and don't upgrade for a long time, everything should keep working. And if you upgrade to a new version, and read through the release notes, you should have a smooth upgrade process.

600+ stars!

When I started working on Drupal VM, I just tossed my own local Vagrant configuration on GitHub and hoped someone else would see some good ideas in it. When the project had 50 stars (then 100, then 200), I was amazed, and wondered when interest in Drupal VM would start waning.

Ansible Galaxy - Explore Role Downloads

Lo and behold, a couple years in, Drupal VM has been starred over 600 times, and all the most downloaded roles on Ansible Galaxy are roles used in Drupal VM! It's also humbling, and quite awesome, to meet a complete stranger at DrupalCon who uses Drupal VM; thank you to all those who have used Drupal VM, have helped with all the open source Ansible Galaxy roles, and also help fight the good fight of automating all the infrastructure things for Drupal!

Special thanks to all the users who have contributed to the last few releases: oxyc, rodrigoeg, thom8, scottrigby, iainhouston, quicksketch, Mogtofu33, stevepurkiss, slimatic, sarahjean, and derimagia (this list is not exhaustive, and I know I met more people at DrupalCon who helped but I forgot to mention here—please let me know if I forgot about you in the comments!).

What the future holds

I'm starting to work on better planning for future releases (e.g. 3.1.0, etc.), and you can always check the Drupal VM issue queue (especially 'Milestones') to see the latest. Somewhere down the line, parts of Drupal VM will likely start using Docker and/or other containerization tech to make builds and rebuilds much faster. There are already some users exploring the use of vagrant-lxc on Linux for speedier builds!

Where will we go for 4.0? I'm not quite sure yet... but I'll keep the guiding principles for my own development environment in mind:

  • Fast and flexible
  • Stable and secure (for local dev, at least)
  • Cross-platform compatible

Please download Drupal VM, try it out, and see what you think!

May 18 2016
May 18

Since a quick Google search didn't bring up how to do this in Drupal 8 (there are dozens of posts on how to do it in Drupal 7), I thought I'd post a quick blog post on how you can modify a user's roles in Drupal 8. Hint: It's a lot easier than you'd think!

In Drupal 7, $user was an object... but it was more like an object that acted like a dumb storage container. You couldn't really do anything with it directly—instead, you had to stick it in functions (like user_multiple_role_edit()) to do things like add or remove roles or modify account information.

In Drupal 8, $user is a real, useful object. Want to modify the account name and save the change?

<?php
$user
->setUsername('new-username');
$user->save();
?>

There are now dozens of simple methods you can call on a user object to get and set information on a $user object, and what used to be a little annoying in Drupal 7 (modifying a user's roles) is now very straightforward. Let's say I want to add the 'administrator' role to a user account:

<?php
$user
->addRole('administrator');
$user->save();
?>

Done!

Want to remove the role?

<?php
$user
->removeRole('administrator');
$user->save();
?>

Added bonus—you no longer need to retrieve a role ID using user_role_load_by_name(), because in Drupal 8 role IDs are now machine readable strings!

I often need to add a drush command that can be run in non-production environments that will make certain users administrators (e.g. developers who need full access in non-prod, but shouldn't have full access on production), and using this new logic, it's extremely easy to build a drush command to do this.

First, create a drush command file (I generally create them in the drush folder in the site's docroot, titled [command].drush.inc). In my case, I created npadmin.drush.inc (for "Non-Prod Admin"), with the following contents:

<?php/**
 * @file
 * Non-production admins drush command.
 */

/**
 * Implements hook_drush_command().
 */

function npadmin_drush_command() {
 
$items = []; $items['non-prod-admins'] = [
   
'description' => "Makes certain users administrators in non-prod environments.",
   
'examples' => [
     
'drush npadmin' => 'Make certain users admins in non-prod environments.',
    ],
   
'aliases' => ['npadmin'],
  ];

  return

$items;
}
/**
 * Implements drush_hook_COMMAND().
 */
function drush_npadmin_non_prod_admins() {
 
$users_changed = 0; // List of users who should be made administrators in non-prod environments.
 
$users_to_make_admin = [
   
'jeff.geerling',
   
'etc...',
  ];

  foreach (

$users_to_make_admin as $name) {
   
$user = user_load_by_name($name);
    if (
$user) {
     
$user->addRole('administrator');
     
$user->save();
     
$users_changed++;
    }
  }
drush_log(dt('Assigned the administrator role to !count users.', ['!count' => $users_changed]), 'ok');
}
?>

After creating the Drush command file, clear Drush's cache (drush cc drush), and then run drush npadmin in whichever environments should have these users become administrators.

Going further, you could turn $users_to_make_admin into a configuration item, so you could change it without changing code (in Drupal 7, I often used a variable for this purpose).

Programmatically managing a user's roles is a great example of OOP code in Drupal 8 making programming more simple and logical. Check out UserInterface for many more methods you can call on a user object!

May 10 2016
May 10

Another year, another Acquia Certification exam...

Acquia Certified Developer - Drupal 8 Exam Badge 2016

I'm at DrupalCon New Orleans, the first North American DrupalCon since the release of Drupal 8. In addition, this is the first DrupalCon where the Acquia Certified Developer - Drupal 8 Exam is being offered, so I decided to swing by the certification center (it's on the 3rd floor of the convention center, in case you want to take any of the certification exams this week!) and take it.

I've taken all the other exams up to this point, including the general developer exam, the front end and back end specialist exams, and the site builder exam, and I've posted short reviews of each of those exams (click the links to read the reviews), so I thought I'd post a review of this exam as well, for the benefit of others who will take it in coming months!

Note: If you'd like a good overview of my perspective on the Acquia Certifications in general, please read my post on the general developer exam, which was written prior to my Acquia employment. Not that I bias my posts here based on my employer, but I do realize there are many who are conflicted over the value of a Drupal certification (or of any software certifications in general), and there are pros and cons to taking the actual exams. I'm of the opinion that the exams are a good thing for the Drupal community, but probably not a great way to judge individual developers (e.g. for hiring purposes), unless taken as one data point in the evaluation process.

A little bit of the old...

For anyone who's taken one of the previous exams, this exam should feel familiar. Probably half of the questions (mostly the 'foundational web development' and 'site building' ones) could be used on any of the Drupal Certification exams. This is because Drupal 8 does carry over a lot of the same content and admin architecture of Drupal 7, even if things underlying the architecture have changed dramatically.

Things like adding fields, managing content types, adding relationships, displaying lists of content, etc. are fundamentally the same as they were in Drupal 7, though all the modules for performing these operations are included in Drupal 8 core.

Additionally, many parts of the theming realm (preprocess functions and the Render API, mostly) are the same.

...A good dose of the new

However, there are maybe 5-10 questions that are a little trickier for me (and would be for anyone working with both Drupal 7 and Drupal 8 sites), because the questions deal with APIs or bits of Drupal that have changed only subtly, or were only halfway re-architected in Drupal 8. For example, there are a few things in Drupal 8 that use Symfony's Event Dispatcher. But wouldn't you know, nodes still use the old-school hooks! As I have only worked on a couple custom modules that tie into node operations, I wasn't sure what the status of conversion to the new system was... but if you're wondering, HookEvent is still not part of Drupal 8 core.

Besides those ambiguous questions (most of which I'm sure I answered incorrectly), there were a number of questions related to basic Twig syntax (e.g. how do you print variables, how do you filter variables, how do you structure a template, how do you add a template suggestion, etc.), and thankfully, there wasn't anything super-deep, like how Twig blocks (not Drupal blocks) work!

You'll want to have at least a basic understanding of creating a route, creating a block plugin, defining a service, adding a CSS or JS file as a library, and using annotations properly in custom modules. You can look at some core modules (like the Book module) for simple implementation examples.

A few errors

Since this is a relatively new exam (I think it was 'officially' launched this week!), there are a few rough edges (grammatical issues, question style issues) that will be ironed out in the next few months; when I had taken the other exams, those little issues had already been worked out, so all my thought could focus on the question at hand, and the potential solutions, and not on style, phrasing, or content.

There was one question that seems to have been partially cut off, as it had a statement, a code block, then some answers... but no actual question! I inferred what the question would be based on context... but I also let the facilitator know, and by the time you take the exam, the erroneous questions will likely be fixed.

These are common issues with the launch of a new exam, since the exam questions are built up from a large number of contributed questions, and the tool (webassessor) doesn't always seem to offer the best interface/input for technical questions. The same caveats apply to this exam as the others—make sure you read through the question a couple times if it's unclear, and whenever there are code samples (especially in answers), make sure you're parsing things correctly!

My Results

I took the exam early in the week, but didn't have a lot of caffeine yet, so I may have been able to score marginally higher. But I'm happy with my overall 81.66%, broken down into the following:

  • Fundamental Web Development Concepts: 100%
  • Site Building: 77%
  • Front end Development: 80%
  • Back end Development: 80%

I'm guessing that after I have a few more Drupal 8 sites under my belt (especially ones that don't use the default Bartik theme, like the Raspberry Pi Dramble site), I can bump up some of the other scores a bit. There are a lot of subtle differences between Drupal 7 and Drupal 8 that can trip a seasoned Drupalist up in these questions!

May 08 2016
May 08

Raspberry Pi Dramble cluster - with a banana for scale

When I originally built the Raspberry Pi Dramble 6-node Pi cluster in 2014 (for testing Ansible with bare metal hardware on the cheap), I compiled all the code, notes, etc. into a GitHub repository. In 2015, I decided to take it a step further, and I started hosting www.pidramble.com on the cluster, in my basement office!

Every time I've shared the cluster with others during presentations or at events like DrupalCon, someone how it's built. While I do have almost all my notes and instructions for building the entire cluster on the Pi Dramble Wiki, a step-by-step visual guide is better. Therefore, I'm posting a series of videos, "Let's Build a Raspberry Pi Cluster", to my YouTube channel.

The first video goes over hardware parts and setup:

The second goes over microSD card setup and Raspbian OS configuration:

These videos will also serve as good background material for those attending my upcoming talks on Drupal 8 and the Raspberry Pi:

I'll post again once I've wrapped up the video series (it will likely be either 5 or 6 parts total). What do you think of the videos? Should I consider making more of this kind of video, or ditch the idea and stick to writing?

May 06 2016
May 06
When module authors decide to port their modules to a new major version of Drupal (e.g. 6 to 7, or 7 to 8), they often take the time to rearchitect the module (and sometimes an entire related ecosystem of modules) to make development more efficient, clean up cruft, and improve current features.
May 03 2016
May 03

In Drupal 8, Search API Solr is the consolidated successor to both the Apache Solr Search and Search API Solr modules in Drupal 7. I thought I'd document the process of setting up the module on a Drupal 8 site, connecting to an Apache Solr search server, and configuring a search index and search results page with search facets, since the process has changed slightly from Drupal 7.

Install the Drupal modules

In Drupal 8, since Composer is now a de-facto standard for including external PHP libraries, the Search API Solr module doesn't actually include the Solarium code in the module's repository. So you can't just download the module off Drupal.org, drag it into your codebase, and enable it. You have to first ensure all the module's dependencies are installed via Composer. There are two ways that I recommend for doing this (both are documented in the module's issue: Keep Solarium managed via composer and improve documentation):

  1. Run the following commands in the root directory of your site's codebase:
    1. composer config repositories.drupal composer https://packagist.drupal-composer.org
    2. composer require "drupal/search_api_solr 8.1.x-dev" (or 8.1.0-alpha3 for the latest stable alpha release)
  2. Install Composer Manager and follow Composer Manager's installation instructions to let it download module dependencies (after having downloaded search_api_solr into your codebase via Drush or manually).

Basically, you need to make sure Search API, Search API Solr, and Facets (formerly Facet API) are in your codebase before you can install them on the Extend page:

Install Facets, Search API, and Solr Search modules in Drupal 8

Install those three modules on your site (either on the 'Extend' page, or via Drush with drush en [module_name]), and if you don't need the redundant core search functionality (you probably don't!), uninstall the Search module.

Connect to your Solr server

Visit the Search API configuration page (/admin/config/search/search-api, or Configuration > Search and metadata > Search API in the menu), and click 'Add Server'. Add a server with the following configuration (assuming you're running this on a local instance of Drupal VM, which easily installs and configures Solr for you):

  • Server name: Local Solr Server (don't use localhost until this issue is resolved!)
  • Backend: Solr (should be the only option at this point)
  • HTTP protocol: http
  • Solr host: localhost
  • Solr port: 8983
  • Solr path: /solr
  • Solr core: collection1 (this Solr search index/core is set up for you if you use Drupal VM and uncomment the example post-provision script for Solr)
  • Basic HTTP authentication: (make sure this is blank)
  • Advanced: These options are up to you. Leave the version override and HTTP method set to their automatic defaults.

Click 'Save', and the server should be saved. Hopefully under Server Connection on the page that you're taken to, you see the message The Solr server could be reached. This means the server is set up correctly, and you can move on to creating a search index on the server.

Create a search index

Back on the Search API configuration page, click 'Add Index'. For the example, we'll create an index with only Article content, with the following configuration:

  • Index name: Articles
  • Data sources: Content
    • What should be indexed?: None except those selected
    • Bundles: Article
    • Languages: English
  • Server: Local Solr Server

Make sure the index is enabled (there's a checkbox for it), and click 'Save'. By default, Search API will index all article content that currently exists immediately (or none, if none exists at this point).

If you don't have any article content yet, create a few articles with a title, body, and some tags. Now that you have content, make sure it's indexed by running cron (either use drush cron or run it via the Reports > Status report page). Check the index page at /admin/config/search/search-api/index/articles to make sure articles are in the index:

Search API Article index status Drupal 8

Add fields to the index

Before you can search for content in the index, you need to make sure all the fields you're interested in searching are available in Solr. You need to go to the 'Fields' tab for the index, and click the 'Add fields' button to add fields from the content type to this index:

Search API Article index add fields to index Drupal 8

Under the 'Content' section, I chose to add the following fields:

  • ID
  • UUID
  • Content Type
  • Title
  • Authored by > User > Name
  • Authored on
  • URL alias > Path alias
  • Body
  • Tags

Note: As of May 2016, the UI for adding fields feels slightly jarring. There's an open issue to improve the field management UI in Search API: AJAXify and generally improve the new Fields UI.

Click 'Done' once finished adding fields, then make sure all the fields you added are present under 'Content' back on the field overview page. (Later, when you're going further into index customization and optimization, you'll spend a bit more time on this page and in this process of refining the fields, their boost, their types, etc.)

Add processors to the index

The last step in setting up the search index is the addition of 'processors', which allow the index to be more flexible and work exactly how we want it for a fulltext 'fuzzy' search as well as for faceted search results. Go to the 'Processors' tab for the Articles index, and check the following checkboxes:

  • Aggregated fields
  • Highlight
  • Ignore case
  • Node status
  • Tokenizer
  • Transliteration

Then, to give ourselves the ability to search with one search box on multiple fields (e.g. title, body, and tags), at the bottom of the page click 'Add new Field' in the 'Aggregated fields' configuration (note this might currently drop you into a different vertical tab once clicked—switch back to the Aggregated fields tab if so), and then call the new field 'Fulltext Article search'. Check Body, Tags, URL alias, Title, and Authored by.

Click 'Save' at the bottom of the form to save all changes. The search index will need to be updated so all the processors can have an effect, but before doing that, go back to the 'Fields' configuration for the Article search index, and switch the new aggregated field from type 'String' to 'Fulltext':

Use type Fulltext for aggregated field in Search API Article index

Click 'Save changes', then go back to the search index View page and click the 'Index now' button (or use cron to reindex everything).

Configure a search page and search facets

Now we come to the final and most important part of this process: creating a search page where users can search through your articles using a full-text, faceted search powered by Solr. Create a new View (visit /admin/structure/views and click 'Add new view'), and name the view 'Article search'. For the view's general settings:

  • Show: Index Articles
  • Page: Create a page (checked)
    • Page title: Article search
    • Path: search/articles
    • Display format: Unformatted list of Rendered entity

Click 'Save and edit' to configure the view. Now we will configure the search index view to our liking:

  1. Click the 'Settings' for the Rendered entity (Format > Show), and switch from the Default view mode to Teaser.
  2. Click the 'Add' button under Filter criteria, and add the "Fulltext Article search" aggregate field we added earlier.
  3. Check the 'Expose this filter to visitors' button, and change the label to 'Search', then Apply the changes.
  4. Save the view, then visit /search/articles.

Test out your new search page, and make sure you can search any part of any of the aggregated text fields.

Note: If you want to be able to search parts of words (e.g. word stems like 'other' match instances of 'another'), then you have to do that in your Solr schema, using Solr's EdgeNGramFilterFactory; see this documentation from Hosted Apache Solr: Enable partial word match keyword searching with EdgeNGramFilterFactory .

Add search facets

Now that we have the general search page set up, let's add a few facets to allow users to more easily drill down into their search results. Go to the Facets admin page (/admin/config/search/facets or Configuration > Search and metadata > Facets), and click 'Add facet' to add two different facets:

  1. A 'Publish date' facet:
    1. Facet name: Publish date
    2. Facet name for URLs: published
    3. Facet source: Article search > Page
    4. Facet field: Authored on
    5. Weight: 0
  2. A 'Tags' facet:
    1. Facet name: Tags
    2. Facet name for URLs: tags
    3. Facet source: Article search > Page
    4. Facet field: Tags
    5. Weight: 0

For the facet display settings, leave the defaults for now (you can customize the facet display and advanced behavior later), and click 'Save' to save the facet. For the 'Tags' facet, edit the display and check the 'Translate taxonomy terms' checkbox so the term name (and not the term ID) is displayed.

Go to the Block layout page (/admin/structure/block) to place the two new facet blocks you just created into the Left sidebar (or wherever in the theme you'd like them to be visible), and save the updated block layout.

Profit!

Now, if you visit the /search/articles page, you should see a faceted fulltext Apache Solr-powered search page, and from here you can customize anything as much or as little as you need!

Search API Solr Drupal 8 faceted fulltext Apache Solr search page

Note that there are a few bits and pieces of the UI and functionality that are still being worked out between Search API, Search API Solr, Facets, and other addon modules that extend these base modules' functionality. Some features that many rely upon in Drupal 7's Solr/search ecosystem, like 'pretty paths' for Facets (e.g. /search/articles/tag1/tag2 instead of /search/articles?taxonomy=tid1&taxonomy_2=tid2) are not yet available in Drupal 8, though there are some early ports (e.g. a Facet pretty paths port) for many of these modules in the search ecosystem.

I was pleasantly surprised how robust and complete the core search functionality already is in Drupal 8; with Drupal 8.1.0 just released, and more and more companies beginning to move to Drupal 8 as their core platform, I think we'll see more of these addon modules make their way to a stable release.

Apr 28 2016
Apr 28

Recently I needed to migrate a small set of content into a Drupal 8 site from a JSON feed, and since documentation for this particular scenario is slightly thin, I decided I'd post the entire process here.

I was given a JSON feed available over the public URL http://www.example.com/api/products.json which looked something like:

{
  "upcs" : [ "11111", "22222" ],
  "products" : [ {
    "upc" : "11111",
    "name" : "Widget",
    "description" : "Helpful for many things.",
    "price" : "14.99"
  }, {
    "upc" : "22222",
    "name" : "Sprocket",
    "description" : "Helpful for things needing sprockets.",
    "price" : "8.99"
  } ]
}

I first created a new 'Product' content type inside Drupal, with the Title field label changed to 'Name', and with additional fields for UPC, Description, and Price. Then I needed to migrate all the data in the JSON feed into Drupal, in the product content type.

Note: at the time of this writing, Drupal 8.1.0 had just been released, and many of the migrate ecosystem of modules (still labeled experimental in Drupal core) require specific or dev versions to work correctly with Drupal 8.1.x's version of the Migrate module.

Required modules

Drupal core includes the base 'Migrate' module, but you'll need to download and enable all the following modules to create JSON migrations:

After enabling those modules, you should be able to use the standard Drush commands provided by Migrate Tools to view information about migrations (migrate-status), run a migration (migrate-import [migration]), rollback a migration (migrate-rollback [migration]), etc.

The next step is creating your own custom migration by adding custom migration configuration via a module:

Create a Custom Migration Module

In Drupal 8, instead of creating a special migration class for each migration, registering the migrations in an info hook, etc., you can just create a migration configuration YAML file inside config/install (or, technically, config/optional if you're including the migration config inside a module that does a bunch of other things and may or may not be used with the Migration module enabled), then when your module is installed, the migration configuration is read into the active configuration store.

The first step in creating a custom migration module in Drupal 8 is to create an folder (in this case, migrate_custom_product), and then create an info file with the module information, named migrate_custom_product.info.yml, with the following contents:

type: module
name: Migrate Custom Product
description: 'Custom product migration.'
package: Migration
core: 8.x
dependencies:
  - migrate_plus
  - migrate_source_json

Next, we need to create a migration configuration file, so inside migrate_custom_product/config/install, add a file titled migrate_plus.migration.product.yml (I'm going to call the migration product to keep things simple). Inside this file, define the entire JSON migration (don't worry, I'll go through each part of this configuration in detail later!):

# Migration configuration for products.
id: product
label: Product
migration_group: Products
migration_dependencies: {}

source:
  plugin: json_source
  path: http://www.example.com/api/products.json
  headers:
    Accept: 'application/json'
  identifier: upc
  identifierDepth: 0
  fields:
    - upc
    - title
    - description
    - price

destination:
  plugin: entity:node

process:
  type:
    plugin: default_value
    default_value: product

  title: name
  field_upc: upc
  field_description: description
  field_price: price

  sticky:
    plugin: default_value
    default_value: 0
  uid:
    plugin: default_value
    default_value: 0

The first section defines the migration machine name (id), human-readable label, group, and dependencies. You don't need to separately define the group outside of the migration_group defined here, though you might want to if you have many related migrations that need the same general configuration (see the migrate_example module included in Migrate Plus for more).

source:
  plugin: json_source
  path: http://www.example.com/api/products.json
  headers:
    Accept: 'application/json'
  identifier: upc
  identifierDepth: 1
  fields:
    - upc
    - title
    - description
    - price

The source section defines the migration source and provides extra data to help the source plugin know what information to retrieve, how it's formatted, etc. In this case, it's a very simple feed, and we don't need to do any special transformation to the data, so we can just give a list of fields to bring across into the Drupal Product content type.

The most important parts here are the path (which tells the JSON source plugin where to go to get the data), the identifier (the unique ID that should be used to match content in Drupal to content in the feed), and the identifierDepth (the level in the feed's hierarchy where the identifier is located).

destination:
  plugin: entity:node

Next we tell Migrate the data should be saved to a node entity (you could also define a destination of entity:taxonomy, entity:user, etc.).

process:
  type:
    plugin: default_value
    default_value: product

  title: name
  field_upc: upc
  field_description: description
  field_price: price

  sticky:
    plugin: default_value
    default_value: 0
  uid:
    plugin: default_value
    default_value: 0

Inside the process configuration, we'll tell Migrate which specific node type to migrate content into (in this case, product), then we'll give a simple field mapping between the Drupal field name (e.g. title) and the name of the field in the JSON feed's individual record (name). For certain properties, like a node's sticky status, or the uid, you can provide a default using the default_value plugin.

Enabling the module, running a migration

Once the module is ready, go to the module page or use Drush to enable it, then use migrate-status to make sure the Product migration configuration was picked up by Migrate:

$ drush migrate-status
Group: Products  Status  Total  Imported  Unprocessed  Last imported
product          Idle    2      0         2

Use migrate-import to kick off the product migration:

$ drush migrate-import product
Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'product'           [status]

You can then check under the content administration page to see if the products were migrated successfully:

Drupal 8 content admin - successful product JSON migration

If the products appear here, you're done! But you'll probably need to do some extra data transformation using a custom JSONReader to transform the data from the JSON feed into your custom content type. That's another topic for another day!

Note: Currently, the Migrate UI at /admin/structure/migrate is broken in Drupal 8.1.x, so using Drush is the only way to inspect and interact with migrations; even with a working UI, it's generally best to use Drush to inspect, run, roll back, and otherwise interact with migrations.

Reinstalling the configuration for testing

Since the configuration you define inside your module's config/install directory is only read into the active configuration store at the time when you enable the module, you will need to re-import this configuration frequently while developing the migration. There are two ways you can do this. You could use some code like the following in your custom product migration's migrate_custom_product.install file:

<?php
/**
 * Implements hook_uninstall().
 */
function migrate_custom_product_uninstall() {
 
db_query("DELETE FROM {config} WHERE name LIKE 'migrate.migration.custom_product%'");
 
drupal_flush_all_caches();
}
?>

...or you can use the Configuration Development module to easily re-import the configuration continuously or on-demand. The latter option is recommended, and is also the most efficient when dealing with more than just a single migration's configuration. I have a feeling config_devel will be a common module in a Drupal 8 developer's tool belt.

Further Reading

Some of the inspiration for this post was found in this more fully-featured example JSON migration module, which was referenced in the issue Include JSON example in the module on Drupal.org. You should also make sure to read through the Migrate API in Drupal 8 documentation.

Apr 06 2016
Apr 06

As Drupal VM has passed 500 stars on GitHub, and is becoming a fairly mature environment for local development environment—especially for teams of Drupal developers who want to maintain consistency and flexibility when developing many sites, I've been working to get more stable releases, better documentation, and a more focused feature set.

Also, in the past few months, as interest has surged, I've even had the opportunity to talk about all things Drupal VM on the DrupalEasy podcast! Check out DrupalEasy Podcast 172 - The Coup (Jeff Geerling - Drupal VM), which was just posted a few days ago.

And to keep the conversation flowing, I'm going to be moderating a BoF on Drupal VM at DrupalCon New Orleans, Drupal VM and local Drupal development for teams.

The BoF will be Wednesday morning, from 10:45-11:45 a.m., in room 289 (Chromatic), and if you want to talk about local development environments for teams, or just the future of Drupal VM, please stop by—I hope it will be a lively and productive conversation!

And who knows, maybe I'll bring my Raspberry Pi Zero cluster and see how Drupal VM performs on it :D

Pi Zero cluster: “go small or go home” #raspberrypi pic.twitter.com/wn4GDQV4AB

— Jeff Geerling (@geerlingguy) March 26, 2016
Mar 31 2016
Mar 31

For the past few days, I've been diving deep into testing Drupal 8's experimental new BigPipe feature, which allows Drupal page requests for authenticated users to be streamed and loaded in stages—cached elements (usually the majority of a page) are loaded almost immediately, meaning the end user can interact with the main elements on the page very quickly, then other uncacheable elements are loaded in as Drupal is able to render them.

Here's a very quick demo of an extreme case, where a particular bit of content takes five seconds to load; BigPipe hugely improves the usability and perceived performance of the page by streaming the majority of the page content from cache immediately, then streaming the harder-to-generate parts as they become available (click to replay):

BigPipe demonstration in an animated gif
Drupal BigPipe demo - click to play again.

BigPipe takes advantage of streaming PHP responses (using flush() to flush the output buffer at various times during a page load), but to ensure the stream is delivered all the way from PHP through to the client, you need to make sure your entire webserver and proxying stack streams the request directly, with no buffering. Since I maintain Drupal VM and support Apache and Nginx as webservers, as well as Varnish as a reverse caching proxy, I experimented with many different configurations to find the optimal way to stream responses through any part of this open source stack.

And because my research dug up a bunch of half-correct, mostly-untested assumptions about output buffering with PHP requests, I figured I'd set things straight in one comprehensive blog post.

Testing output buffering

I've seen a large number of example scripts used to test output_buffering on Stack Overflow and elsewhere, and many of them assume output buffering is disabled completely. Rather than doing that, I decided to make a little more robust script for my testing purposes, and also to document all the different bits for completeness:

<?php
// Set a valid header so browsers pick it up correctly.
header('Content-type: text/html; charset=utf-8');// Emulate the header BigPipe sends so we can test through Varnish.
header('Surrogate-Control: BigPipe/1.0');// Explicitly disable caching so Varnish and other upstreams won't cache.
header("Cache-Control: no-cache, must-revalidate");// Setting this header instructs Nginx to disable fastcgi_buffering and disable
// gzip for this request.
header('X-Accel-Buffering: no');$string_length = 32;
echo
'Begin test with an ' . $string_length . ' character string...<br />' . "\r\n";// For 3 seconds, repeat the string.
for ($i = 0; $i < 3; $i++) {
 
$string = str_repeat('.', $string_length);
  echo
$string . '<br />' . "\r\n";
  echo
$i . '<br />' . "\r\n";
 
flush();
 
sleep(1);
}

echo

'End test.<br />' . "\r\n";
?>

If you place this file into a web-accessible docroot, then load the script in your terminal using PHP's cli, you should see output like (click to replay):

PHP CLI streaming response test
PHP response streaming via PHP's CLI - click to play again.

And if you view it in the browser? By default, you won't see a streamed response. Instead, you'll see nothing until the entire page loads (click to replay):

PHP webserver streaming response test not working
PHP response not streaming via webserver in the browser - click to play again.

That's good, though—we now have a baseline. We know that the script works on PHP's CLI, but either our webserver or PHP is not streaming the response all the way through to the client. If you change the $string_length to 4096, and are using a normal PHP/Apache/Nginx configuration, you should see the following (click to replay):

PHP webserver streaming response test not working
PHP response streaming via webserver in the browser - click to play again.

The rest of this post will go through the steps necessary to ensure the response is streamed through your entire stack.

PHP and output_buffering

Some guides say you have to set output_buffering = Off in your php.ini configuration in order to stream a PHP response. In some circumstances, this is useful, but typically, if you're calling flush() in your PHP code, PHP will flush the output buffer immediately after the buffer is filled (the default value is 4096, which means PHP will flush it's buffer in 4096 byte chunks).

For many applications, 4096 bytes of buffering offers a good tradeoff for better transport performance vs. more lively responses, but you can lower the value if you need to send back much smaller responses (e.g. tiny JSON responses like {setting:1}).

One setting you definitely do need to disable, however, is zlib.output_compression. Set it to zlib.output_compression = Off in php.ini and restart PHP-FPM to make sure gzip compression is disabled.

There are edge cases where the above doesn't hold absolutely true... but in most real-world scenarios, you won't need to disable PHP's output_buffering to enable streaming responses.

Nginx configuration

I recommend using Nginx with PHP-FPM for the most flexible and performant configuration, but still run both Apache and Nginx in production for various reasons. Nginx has a small advantage over Apache for PHP usage in that it doesn't have the cruft of the old mod_php approach where PHP was primarily integrated with the webserver, meaning the proxied request approach (using FastCGI) has always been the default, and is well optimized.

All you have to do to make streaming responses work with Nginx is set the header X-Accel-Buffering: no in your response. Once Nginx recognizes that header, it automatically disables gzip and fastcgi_buffering for only that response.

header('X-Accel-Buffering: no');

You can also manually disable gzip (gzip off) and buffering (fastcgi_buffering off) for an entire server directive, but that's overkill and would harm performance in any case where you don't need to stream the response.

Apache configuration

Because there are many different ways of integrating PHP with Apache, it's best to discuss how streaming works with each technique:

mod_php

Apache's mod_php seems to be able to handle streaming without disabling deflate/gzip for requests out of the box. No configuration changes required.

mod_fastcgi

When configuring mod_fastcgi, you must add the -flush option to your FastCgiExternalServer directive, otherwise if you have mod_deflate/gzip enabled, Apache will buffer the entire response and delay until the end to deliver it to the client:

# If using PHP-FPM on TCP port 9000.
FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -flush -host 127.0.0.1:9000 -pass-header Authorization

mod_fcgi

I've never configured Apache and PHP-FPM using mod_fcgi, and it seems cumbersome to do so; however, according to the Drupal BigPipe environment docs, you can get output buffering disabled for PHP responses by setting:

FcgidOutputBufferSize 0

mod_proxy_fcgi

If you use mod_proxy_fcgi with PHP-FPM, then you have to disable gzip in order to have responses streamed:

SetEnv no-gzip 1

In all the above cases, PHP's own output buffering will take effect up to the default output_buffering setting of 4096 bytes. You can always change this value to something lower if absolutely necessary, but in real-world applications (like Drupal's use of BigPipe), many response payloads will have flushed output chunks greater than 4096 bytes, so you might not need to change the setting.

Varnish configuration

Varnish buffers output by default, and you have to explicitly disable this behavior for streamed responses by setting do_stream on the backend response inside vcl_backend_response. Drupal, following Facebook's lead, uses the header Surrogate-Control: BigPipe/1.0 to flag a response as needing to b streamed. You need to use Varnish 3.0 or later (see the Varnish blog post announcing streaming support in 3.0), and make the following changes:

Inside your Varnish VCL:

sub vcl_backend_response {
    ...
    if (beresp.http.Surrogate-Control ~ "BigPipe/1.0") {
        set beresp.do_stream = true;
        set beresp.ttl = 0s;
    }
}

Then make sure you output the header anywhere you need to stream a response:

header('Surrogate-Control: BigPipe/1.0');

Debugging output streaming

During the course of my testing, I ran into some strange and nasty networking issue with a VMware vagrant box, which was causing HTTP responses delivered through the VM's virtual network to be buffered no matter what, while responses inside the VM itself worked fine. After trying to debug it for an hour or two, I gave up, rebuilt the VM in VirtualBox instead of VMware, couldn't reproduce the issue, then rebuilt again in VMware, couldn't reproduce again... so I just put that there as a warning—your entire stack (including any OS, network and virtualization layers) has to be functioning properly for streaming to work!

To debug PHP itself, and make sure PHP is delivering the stream even when your upstream webserver or proxy is not, you can analyze packet traffic routed through PHP-FPM on port 9000 (it's a lot harder to debug via UNIX sockets, which is one of many reasons I prefer defaulting to TCP for PHP-FPM). I used the following command to sniff port 9000 on localhost while making requests through Apache, Nginx, and Varnish:

tcpdump -nn -i any -A -s 0 port 9000

You can press Ctrl-C to exit once you're finished sniffing packets.

Mar 24 2016
Mar 24

tl;dr: Drupal 8's defaults make most Drupal sites perform faster than equivalent Drupal 7 sites, so be wary of benchmarks which tell you Drupal 7 is faster based solely on installation defaults or raw PHP execution speed. Architectural changes have made Drupal's codebase slightly slower in some ways, but the same changes make the overall experience of using Drupal and browsing a Drupal 8 site much faster.

When some people see reports of Drupal 8 being 'dramatically' slower than Drupal 7, they wonder why, and they also use this performance change as ammunition against some of the major architectural changes that were made during Drupal 8's development cycle.

First, I wanted to give some more concrete data behind why Drupal 8 is slower (specifically, what kinds of things does Drupal 8 do that make it take longer per request than Drupal 7 on an otherwise-identical system), and also why this might or might not make any difference in your choice to upgrade to Drupal 8 sooner rather than later.

Load test benchmarks with a cluster of Raspberry Pis

For a hobby project of mine, the Raspberry Pi Dramble, I like to benchmark every small change I make to the infrastructure—I poke and prod to see how it affects load capacity (how many requests per second can be served without errors), per-page load performance (how many milliseconds before the page is delivered), and availability (how many requests are served correctly and completely).

I've compiled these benchmarks from time to time on the Dramble - Drupal Benchmarks page, and I also did a much more detailed blog post on the matter (especially comparing PHP 5.6 to 7.0 to HHVM): Benchmarking PHP 7 vs HHVM - Drupal and Wordpress.

The most recent result paints a pretty sad picture if you're blindly comparing Drupal 8's standard configuration with Drupal 7's (with anonymous page caching enabled1):

Drupal 8 vs Drupal 7 standard profile performance on home page load - anonymous vs authenticated

These particular benchmarks highlight the maximum load capacity with 100% availability that the cluster of five (incredibly slow, in comparison to most modern servers) Raspberry Pis. Chances are you'll get more capacity just spinning up an instance of Drupal VM on your own laptop! But the fact of the matter is: Drupal 7, both when loading pages for anonymous and authenticated users, in a very bare (no custom modules, no content) scenario, is much faster than Drupal 8. But why?

XHProf page profiling with Drupal 7 and Drupal 8

With Drupal VM, it's very easy to profile code with XHProf, so I spun up one VM for Drupal 8, then shut that one down and spun up an identical environment for Drupal 7 (both using PHP 5.6 and Apache 2.4), and ran an XHProf analysis on the home page, standard profile, anonymous user, with anonymous page cache enabled, on the first page load (e.g. when Drupal stores its anonymous cache copy).

Subsequent page loads use even less of Drupal's critical code path, and it would be helpful to also analyze what's happening there, but for this post I'll focus on the first anonymous page request to the home page.

Compare, first, the zoomed out callgraph image for Drupal 7 (126ms, 13,406 function calls, 3.7 MB memory) vs Drupal 8 (371ms, 41,863 function calls, 11.1 MB memory):

Drupal 7 vs Drupal 8 - Anonymous request to standard profile home page - XHProf call graph comparison
Call graphs for Drupal 7 (left) vs Drupal 8 (right) - click the links to download full size

Callgraphs allow you to visualize the flow of the code from function to function, and to easily identify areas of the code that are 'hotspots', which either take a long time to run or are called many, many times.

Just glancing at the callgraphs, you can see the difference in the way the page is rendered. In Drupal 7, Drupal's homegrown request routing/menu system efficiently chooses the proper menu callback, and most of the time is spent in regular expressions (that 'preg_grep' red box) during theming and rendering the page.

In Drupal 8, there is a bit of extra time spent routing the request to the proper handler, notifying subscribers of the current request and response flow2, with similar amounts of time as Drupal 7 are spent theming and rendering the page. On top of that, since Drupal 8 has been architected in a more OOP way, especially with the splitting out of functionality into discrete PHP files, more time is spent scanning file data on each page load—this can be mitigated in some circumstances by disabling opcache's stat of each file on each page load, but even then, there is a lot of time spent in file_exists, Composer\Autoload\ClassLoader::findFileWithExtension, is_file, and filemtime.

In both cases, one of the most time-consuming tasks is retrieving data from the database; in Drupal 8, the front page took about 29ms grabbing data from MySQL, in Drupal 7, about 26ms—close enough to be practically the same. In most real-world scenarios, database access is a much larger portion of the page load, so the total page render times in real world usage are often a bit closer between Drupal 7 and Drupal 8. But even there, Drupal 8 adds in a tiny bit of extra time for its more flexible (and thus expensive) entity loading.

So Drupal 8's hot/minimal code path is verifiably slower than Drupal 7 in many small ways, due to additional function calls and object instantiation for Symfony integration, notification handling (on top of some remaining Drupal 7-style hooks) and time spent rummaging through the highly individual-file-per-class-heavy codebase. But does this matter for you? Thats can be a difficult question to answer.

You can download the full .xhprof reports below; if you want to view them in XHProf and generate your own callgraphs, you can do so by placing them in your XHProf output directory without the .txt extension:

Drupal 8 changes - more than just the architecture

Most Drupal 7 site builders feel quite at home in Drupal 8, especially considering many of the features that are baked into Drupal 8 core were the most popular components of many Drupal 7 sites—Views, Wysiwyg, entity relationships, etc. Already, just adding those modules (which are used on many if not most Drupal 7 sites) to a standard Drupal 7 site evens the playing field by a large margin, at least for uncached requests:

Drupal 7 vs Drupal 8 with D8 core modules included in Drupal 7
Drupal 7 and Drupal 8 authenticated requests are much more even when including all of D8's core functionality

It's rare to see a Drupal 7 site with less than ten or fifteen contributed modules; many sites have dozens—or even hundreds—of contributed modules that power the various admin and end-user-facing features that make a Drupal 7 site work well. Using real-world sites as examples, rather than clean-room Drupal installs, benchmarks between functionally similar Drupal 7 and Drupal 8 sites are often much closer (like the one above); though Drupal 7 still takes the raw performance crown per-page-request.

For the above D7 + D8 core module test, I ran the following drush command to get (most of) the modules that are in D8 core, enabled via the standard install profile: drush en -y autoupload backbone bean breakpoints ckeditor date date_popup_authored edit email entityreference entity_translation file_entity filter_html_image_secure jquery_update link magic module_filter navbar phone picture resp_img save_draft strongarm transliteration underscore uuid variable views

So, Drupal 8 is slightly slower than a Drupal 7 site with a comparable suite of modules... excluding many of the amazing new features like Twig templating, built-in Wysiwyg and file upload integration, a better responsive design for everything, more accessibility baked in, and huge multilingual improvements—what else in Drupal 8 makes the raw PHP performance tradeoff worth it?

Easier and more robust caching for anonymous users

Varnish cache hit in Drupal 8

What's the best way to speed up any kind of dynamic CMS? To bypass it completely, using something like Varnish, Nginx caching, or a CDN acting as a caching or 'reverse' proxy like Fastly, CloudFlare or Akamai. In Drupal 7, all of these options were available, and could be made to work fairly easily. However, the elephant in the room was always how do you keep content fresh?

The problem was Drupal couldn't pass along any information with pages that were cached to help upstream reverse proxies to intelligently cache the documents. You'd end up with dozens or custom configured rules and a concoction of modules like Expire, Purge, and/or Varnish, and then you'd still have people who publish content on your site asking why their changes aren't visible on page XYZ.

In Drupal 8, cache tags are built into core and passed along with every page request. Cache tags allow reverse proxies to attach a little extra metadata to every page on the site (this doesn't need to be passed along to the client, since it's only for cacheability purposes), and then Drupal can intelligently say "expire any page where node:118292 appears". Then Varnish could add a ban rule that will mark any view, content listing, block, or other node where node:118292 appears as needing to be refreshed from the backend.

Instead of setting extremely short TTLs (time to live) for content, meaning more requests to Drupal (and thus a slower average response time), you will be free to set TTLs much longer—for some sites, you could even set the cache TTL to days, weeks or longer, so Drupal is only really ever touched when new content is added or specific content is updated.

I wrote a very detailed article on how you can use cache tags with Varnish and the Purge module in Drupal 8; you can also more easily use Drupal 8 with CloudFlare, Fastly, and other CDNs and reverse proxies; for simple cases, you can use Drupal 8 with CloudFlare's free plan, like I did with my Raspberry Pi Dramble. Paid plans allow you to integrate more deeply and use cache tags effectively.

Faster for authenticated users and slow-loading content

If you need to support many logged in users (e.g. a community site/forum, or a site with many content editors), you know how difficult it is to optimize Drupal 6 or 7 for authenticated users; the Authcache module and techniques like Edge-Side Includes have been the most widely-adopted solutions, but if, like me, you've ever had to implement these tools on complex sites, you know that they are hard to configure correctly, and in some cases can cause slower performance while simultaneously making the site's caching layers harder to debug. Authenticated user caching is a tricky thing to get right!

In Drupal 8, because of the comprehensive cacheability metadata available for content and configuration, a new Dynamic Page Cache module is included in core. It works basically the same as the normal anonymous user page cache, but it uses auto-placeholdering to patch in the dynamic and uncacheable parts of the cached page. For many sites, this will make authenticated page requests an order of magnitude faster (and thus more scalable) in Drupal 8 than in Drupal 7, even though the raw Drupal performance is slightly slower.

That's well and good... but the end user still doesn't see the rendered page until Drupal is completely finished rendering the page and placing content inside the placeholders. Right? Well, Drupal 8.1 adds a new and amazing experimental feature modeled after Facebook's "BigPipe" tech:

BigPipe demonstration in an animated gif
BigPipe demo - click the above gif to play it again.

The image above illustrates how BigPipe can help even slow-to-render pages deliver usable content to the browser very quickly. If you have a block in a sidebar that pulls in some data from an external service, or only one tiny user-specific block (like a "Welcome, Jeff!" widget with a profile picture) that takes a half second or longer to render, Drupal can now serve the majority of a page immediately, then send the slower content when it's ready.

To the end user, it's a night-and-day difference; users can start interacting with the page very quickly, and content seamlessly loads into other parts of the page as it is delivered. Read more about BigPipe in Drupal—it's currently labeled as an 'experimental' module in Drupal 8.1, and I'm currently poking and prodding BigPipe with Drupal VM.

Also, in case you're wondering, here's a great overview of the difference between ESI and BigPipe.

There are a few caveats with BigPipe—depending on your infrastructure's configuration, you may need to make some changes so BigPipe can stream the page correctly to the end user. Read BigPipe environment requirements for more information.

Only the beginning of what's possible

Drupal 8's architecture also allows for other innovative ways of increasing overall performance. One trend on the upswing is decoupled Drupal, where all or parts of the front-end of the site (the parts of the website your users see) are actually rendered via javascript either on a server, or on the client (or a mix of both). These decoupled sites have the potential to make more seamless browsing experiences, and can also make some types of sites perform much faster.

Caveat: Before decoupling, have a read through Dries Buytaert's recent (and very insightful) blog post: How should you decouple Drupal?

In Drupal 7, building a fully decoupled site was extremely difficult, as everything would need to work around the fact that Drupal < 8 was built mainly for generating HTML pages. Drupal 8's approach is to generate generic "responses". The default is to generate an HTML page... but it's much easier to generate JSON, XML, or other types of responses. And things like cacheability metadata are also flexible enough to work with any kind of response, so you can have a full-cacheable decoupled Drupal site if you want, without even having to install extra modules or hack around Drupal's rendering system, like you did in Drupal 7.


Click here to download the video if it won't play above.

Another recent development is the RefreshLess module, which uses javascript and the HTML5 history API to make any Drupal 8 site behave more like a one page app—if you click on a link, the page remains in place, but the URL updates, and the parts of the page that are different are swapped out seamlessly, using the cacheability data that's already baked into Drupal 8, powering all the other awesome new caching tools!

On top of all that, we're still very early in Drupal 8's release cycle. Since Drupal is using semantic versioning for releases, new features and improvements can be added to minor releases (e.g. 8.1, 8.2, etc.), meaning as we see more of what's possible with BigPipe, Dynamic page cache, etc., we'll make even more improvements—maybe to the point where even the tiniest Drupal 8 page request is close to Drupal 7 in terms of raw PHP execution speed!

What are your thoughts and experiences with Drupal 8 performance so far?

1 Drupal 7's standard profile doesn't enable the anonymous page cache out of the box. You have to enable it manually on the Performance configuration page. This is one area where Drupal 8's initial out of the box experience is actually faster than Drupal 7. Additionally, Drupal 7's anonymous page cache was much less intelligent than Drupal 8's (any content update or comment posting in Drupal 7 resulted in the entire page cache emptying), meaning content updates and page caching in general are much less painful in Drupal 8.

2 One of the biggest contributors to the slower request routing performance is Drupal 8's use of Symfony components for matching routes, notifying subscribers, etc. chx's comment on the far-reaching nature of this change was prescient; much of Drupal's basic menu handling and access control had to be adapted to the new (less efficient, but more structured) Symfony-based routing system.

Mar 22 2016
Mar 22

Varnish cache hit in Drupal 8

Over the past few months, I've been reading about BigPipe, Cache Tags, Dynamic Page Cache, and all the other amazing-sounding new features for performance in Drupal 8. I'm working on a blog post that more comprehensively compares and contrasts Drupal 8's performance with Drupal 7, but that's a topic for another day. In this post, I'll focus on cache tags in Drupal 8, and particularly their use with Varnish to make cached content expiration much easier than it ever was in Drupal 7.

Purging and Banning

Varnish and Drupal have long had a fortuitous relationship; Drupal is a flexible CMS that takes a good deal of time (relatively speaking) to generate a web page. Varnish is an HTTP reverse proxy that excels at sending a cached web page extremely quickly—and scaling up to thousands or more requests per second even on a relatively slow server. For many Drupal sites, using Varnish to make the site hundreds or thousands of times faster is a no-brainer.

But there's an adage in programming that's always held true:

There are two hard things in computer science: cache invalidation, naming things, and off-by-one errors.

Cache invalidation is rightly positioned as the first of those two (three!) hard things. Anyone who's set up a complex Drupal 7 site with dozens of views, panels pages, panelizer layouts, content types, and configured Cache expiration, Purge, Acquia Purge, Varnish, cron and Drush knows what I'm talking about. There are seemingly always cases where someone edits a piece of content then complains that it's not updating in various places on the site.

The traditional answer has been to reduce the TTL for the caching; some sites I've seen only cache content for 30 seconds, or at most 15 minutes, because it's easier than accounting for every page where a certain type of content or menu will change the rendered output.

In Varnish, PURGE requests have been the de-facto way to deal with this problem for years, but it can be a complex task to purge all the right URLs... and there could be hundreds or thousands of URLs to purge, meaning Drupal (in combination with Purge/Acquia Purge) would need to churn through a massive queue of purge requests to send to Varnish.

Drupal 8 adds in a ton of cacheability metadata to all rendered pages, which is aggregated from all the elements used to build that page. Is there a search block on the page? There will be a config:block.block.bartik_search cache tag added to the page. Is the main menu on the page? There will be a config:system.menu.main cache tag, and so on.

Adding this data to every page allows us to do intelligent cache invalidation. Instead of us having to tell Varnish which particular URLs need to be invalidated, when we update anything in the main menu, we can tell Varnish "invalidate all pages that have the config:system.menu.main cache tag, using a BAN instead of a PURGE. If you're running Varnish 4.x, all you need to do is add some changes to your VCL to support this functionality, then configure the Purge and Generic HTTP Purger modules in Drupal.

Whereas Varnish would process PURGE requests immediately, dropping cached pages matching the PURGE URL, Varnish can more intelligently match BAN requests using regular expressions and other techniques against any cached content. You have to tell Varnish exactly what to do, however, so there are some changes required in your VCL.

Varnish VCL Changes

Borrowing from the well-documented FOSHttpCache VCL example, you need to make the following changes in your Varnish VCL (see the full set of changes that were made to Drupal VM's VCL template):

Inside of vcl_recv, you need to add some logic to handle incoming BAN requests:

sub vcl_recv {
    ...
    # Only allow BAN requests from IP addresses in the 'purge' ACL.
    if (req.method == "BAN") {
        # Same ACL check as above:
        if (!client.ip ~ purge) {
            return (synth(403, "Not allowed."));
        }

        # Logic for the ban, using the Purge-Cache-Tags header. For more info
        # see https://github.com/geerlingguy/drupal-vm/issues/397.
        if (req.http.Purge-Cache-Tags) {
            ban("obj.http.Purge-Cache-Tags ~ " + req.http.Purge-Cache-Tags);
        }
        else {
            return (synth(403, "Purge-Cache-Tags header missing."));
        }

        # Throw a synthetic page so the request won't go to the backend.
        return (synth(200, "Ban added."));
    }
}

The above code basically inspects BAN requests (e.g. curl -X BAN http://127.0.0.1:81/ -H "Purge-Cache-Tags: node:1"), then passes along a new ban() if the request comes from the acl purge list, and if the Purge-Cache-Tags header is present. In this case, the ban is set using a regex search inside stored cached object's obj.http.Purge-Cache-Tags property. Using this property (on obj instead of req) allows Varnish's ban lurker to clean up ban requests more efficiently, so you don't end up with thousands (or millions) of stale ban entries. Read more about Varnish's ban lurker.

Inside of vcl_backend_response, you can add a couple extra headers to help the ban lurker (and, potentially, allow you to make more flexible ban logic should you choose to do so):

sub vcl_backend_response {
    # Set ban-lurker friendly custom headers.
    set beresp.http.X-Url = bereq.url;
    set beresp.http.X-Host = bereq.http.host;
    ...
}

Then, especially for production sites, you should make sure Varnish doesn't pass along all the extra headers needed to make Cache Tags work (unless you want to see them for debugging purposes) inside vcl_deliver:

sub vcl_deliver {
    # Remove ban-lurker friendly custom headers when delivering to client.
    unset resp.http.X-Url;
    unset resp.http.X-Host;
    unset resp.http.Purge-Cache-Tags;
    ...
}

At this point, if you add these changes to your site's VCL and restart Varnish, Varnish will be ready to handle cache tags and expire content more efficiently with Drupal 8.

Drupal Purge configuration

First of all, so that external caches like Varnish know they are safe to cache content, you need to set a value for the 'Page cache maximum age' on the Performance page (admin/config/development/performance). You can configure Varnish or other reverse proxies under your control to cache for as long or short a period of time as you want, but a good rule-of-thumb default is 15 minutes—even with cache tags, clients cache pages based on this value until the user manually refreshes the page:

Set page cache maximum age

Now we need to make sure Drupal does two things:

  1. Send the Purge-Cache-Tags header with every request, containing a space-separated list of all the page's cache tags.
  2. Send a BAN request with the appropriate cache tags whenever content or configuration is updated that should expire pages with the associated cache tags.

Both of these can be achieved quickly and easily by enabling and configuring the Purge and Generic HTTP Purger modules. I used drush en -y purge purge_purger_http to install the modules on my Drupal 8 site running inside Drupal VM.

Purge automatically sets the http.response.debug_cacheability_headers property to true via it's purge.services.yml, so Step 1 above is taken care of. (Note that if your site uses it's own services.yml file, the http.response.debug_cacheability_headers setting defined in that file will override Purge's settings—so make sure it's set to true if you define settings via services.yml on your site!)

Note that you currently (as of March 2016) need to use the -dev release of Purge until 8.x-3.0-beta4 or later, as it sets the Purge-Cache-Tags header properly.

For step 2, you need to add a 'purger' that will send the appropriate BAN requests using purge_purger_http: visit the Purge configuration page, admin/config/development/performance/purge, then follow the steps below:

  1. Add a new purger by clicking the 'Add Purger' button: Add Purger
  2. Choose 'HTTP Purger' and click 'Add': HTTP Purger
  3. Configure the Purger's name ("Varnish Purger"), Type ("Tag"), and Request settings (defaults for Drupal VM are hostname 127.0.0.1, port 81, path /, method BAN, and scheme http): Configure HTTP purger request settings
  4. Configure the Purger's headers (add one header Purge-Cache-Tags with the value [invalidation:expression]): Configure HTTP purger header settings

Testing cache tags

Now that you have an updated VCL and a working Purger, you should be able to do the following:

  1. Send a request for a page and refresh a few times to make sure Varnish is caching it:

    $ curl -s --head http://drupalvm.dev:81/about | grep X-Varnish
    X-Varnish: 98316 65632
    X-Varnish-Cache: HIT

  2. Edit that page, and save the edit.

  3. Run drush p-queue-work to process the purger queue:

    $ drush @drupalvm.drupalvm.dev p-queue-work
    Processed 5 objects...

  4. Send another request to the same page and verify that Varnish has a cache MISS:

    $ curl -s --head http://drupalvm.dev:81/about | grep X-Varnish
    X-Varnish: 47
    X-Varnish-Cache: MISS

  5. After the next request, you should start getting a HIT again:

    $ curl -s --head http://drupalvm.dev:81/about | grep X-Varnish
    X-Varnish: 50 48
    X-Varnish-Cache: HIT

You can also use Varnish's built in tools like varnishadm and varnishlog to verify what's happening. Run these commands from the Varnish server itself:

# Watch the detailed log of all Varnish requests.
$ varnishlog
[wall of text]

# Check the current list of Varnish bans.
$ varnishadm
varnish> ban.list
200       
Present bans:
1458593353.734311     6    obj.http.Purge-Cache-Tags ~ block_view

# Check the current parameters.
varnish> param.show
...
ban_dups                   on [bool] (default)
ban_lurker_age             60.000 [seconds] (default)
ban_lurker_batch           1000 (default)
ban_lurker_sleep           0.010 [seconds] (default)
...

If you're interested in going a little deeper into general Varnish debugging, read my earlier post, Debugging Varnish VCL configuration files.

Other notes and further reading

I spent a few days exploring cache tags, and how they work with Varnish, Fastly, CloudFlare, and other services with Drupal 8, as part of adding cache tag support to Drupal VM. Here are some other notes and links to further reading so you can go as deep as you want into cache tags in Drupal 8:

  • If you're building custom Drupal modules or renderable arrays, make sure you add cacheability metadata so all the cache tag magic just works on your site! See the official documentation for Cacheability of render arrays.
  • The Varnish module is actively being ported to Drupal 8, and could offer an alternative option for using cache tags with Drupal 8 and Varnish.
  • Read the official Varnish documentation on Cache Invalidation, especially regarding the effectiveness and performance of using Bans vs Purges vs Hashtwo vs. Cache misses.
  • There's an ongoing meta issue to profile and rationalize cache tags in Drupal 8, and the conversation there has a lot of good information about cache tag usage in the wild, caveats with header payload size and hashing, etc.
  • As mentioned earlier, if you have a services.yml file for your site, make sure you set http.response.debug_cacheability_headers: true inside (see note here).
  • Read more about Varnish bans
  • Read more about Drupal 8 cache tags
  • Read a case study of cache tags (with Fastly) dramatically speeding up a large Drupal 8 site.
  • Be careful with your ban logic in the VCL; you need to avoid using regexes on req to allow the ban lurker to efficiently process bans (see Why do my bans pile up?).
  • If you find Drupal 8's cache_tags database table is growing very large, please check out the issue Garbage collection for cache tag invalidations. For now, you can safely truncate that table from time to time if needed.
Mar 14 2016
Mar 14

I think today was my most Pi-full ? day, ever! Let's see:

Early in the morning, I finished upgrading all the Ansible playbooks used by the Raspberry Pi Dramble so my cluster of five Raspberry Pis would run faster and better on the latest version of official Raspberry Pi OS, Raspbian Jessie.

Later, opensource.com published an article I wrote about using Raspberry Pis placed throughout my house to help my kids sleep better:

Raspberry Pi project to regulate room temperature and sleep better https://t.co/ikwRbS5wns by @geerlingguy pic.twitter.com/rXA1eWodIm

— Open Source Way (@opensourceway) March 14, 2016

Meanwhile, my wife brought home some tasty Dutch Apple pie, of which I had a slice at lunch:

#PiDay celebration continues… pic.twitter.com/W4HhAmigrT

— Jeff Geerling (@geerlingguy) March 14, 2016

Later, at 3:14 p.m. on 3/14, while I was in the 314 area code, I tweeted:

It’s 3:14 on 3/14 in the 314 #PiDay #stlouis

— Jeff Geerling (@geerlingguy) March 14, 2016

...and I was walking into our Micro Center to pick up a Raspberry Pi 3, which I'm going to benchmark like crazy for my Raspberry Pi Dramble Drupal 8 cluster, and for my Drupal Pi project:

Time to run some Drupal benchmarks on the Pi 3 I picked up from @microcenter on #PiDay #RaspberryPi pic.twitter.com/BwiOsSvLJR

— Jeff Geerling (@geerlingguy) March 14, 2016
Mar 14 2016
Mar 14

I've been supporting Drupal VM (a local Drupal CMS development environment) for Windows, Mac, and Linux for the past couple years, and have been using Vagrant and virtual machines for almost all my development (mostly PHP, but also some Python and Node.js at the moment) for the past four years. One theme that comes up quite frequently when dealing with VMs, open source software stacks (especially Drupal/LAMP), and development, is how much extra effort there is to make things work well on Windows.

Problem: tool-builders use Linux or Mac OS X

The big problem, I see, is that almost all the tool-builders for OSS web software run either Mac OS X or a flavor of Linux, and many don't even have access to a Windows PC (outside of maybe an odd VM for testing sites in Internet Explorer or Edge, if they're a designer/front-end developer). My evidence is anecdotal, but go to any OSS conference/meetup and you'll likely see the same.

When tool-builders don't use Windows natively, and in many cases don't even have access to a real Windows environment, you can't expect the tools they build to always play nice in that kind of environment. Which is why virtualization is almost an essential component of anyone's Windows development workflow.

However, that's all a bit of an aside leading up to the substance of this post: common issues with Windows-based development using virtual machines (e.g. VirtualBox, Hyper-V, VMware, etc.), and some solutions or tips for minimizing the pain.

As an aside, I bought a Lenovo T420 and stuck 2 SSDs and an eMMC card in it, then I purchased and installed copies of Windows 7, 8, and 10 on them so I could make sure the tools I build work at least decently well on Windows in multiple different environments. Many open source project maintainers aren't willing to fork over a $500+ outlay just to test in a Windows environment, especially since the Windows bugs often take 2-4x more time and cause many more grey hairs than similar bugs on Linux/Mac OS X.

Tips for more efficiency on Windows

First: if there's any way you can use Linux or Mac instead of Windows, you'll be in much less pain. This is not a philosophical statement, nor is it an anti-Microsoft screed. Almost all the modern web development toolkits are supported primarily (and sometimes only) for Linux/POSIX-like environments. Getting everything in a modern stack working together in Windows natively is almost never easy; getting things working within a VM is a viable but sometimes annoying alternative.

Second: if you can do all development within the VM, you'll be in somewhat less pain. If you can check out your codebase inside the VM's filesystem (and not use a synced folder), then edit files in the VM, or edit files in Windows through a mounted share (instead of using built-in shared folders), some of the pain will be mitigated.

Here are some of the things I've found that make development on Windows possible; and sometimes even enjoyable:

Working with symbolic links (symlinks)

You should not use Git Bash, Git Shell, or Powershell when managing the Vagrant environment (there is a way, but it's excruciating). It's highly recommended to use either Cygwin (with openssh) or Cmder instead. There are additional caveats when you require symlink support:

  • You need to run Cygwin or Cmder as administrator (otherwise you can't create symlinks).
  • You have to do everything inside the VM (you can do git stuff outside, and edit code outside, but anything you need to do dealing with creating/changing/deleting symlinks should be done inside the VM).
  • If you touch the symlinks outside the VM, bad things will happen.
  • Symlinks only work with SMB or native shares (possibly rsync, too, but that's a bit harder to work with in my experience.
  • If you switch from native to SMB, or vice-versa, you have to rebuild all symlinks (symlinks between the two synced folder types are incompatible).
  • If you use SMB, you have to set custom mount options for Vagrant in the synced folder configuration, e.g.: mount_options: ["mfsymlinks,dir_mode=0755,file_mode=0755"]

VirtualBox Guest Additions

Probably half the problems I've seen are due to outdated (or not-installed) VirtualBox Guest Additions. Many Vagrant box maintainers don't update their boxes regularly, and if this is the case, and you have a newer version of VirtualBox, all kinds of strange issues (ssh errors, synced folder errors, etc.) ensue.

I highly recommend (for Mac, Linux, and Windows) you install the vagrant-vbguest plugin: vagrant plugin install vagrant-vbguest

Delete a deep folder hierarchy (nested directories)

For many of the projects I've worked on, folder hierarchies can get quite deep, e.g. C:/Users/jgeerling/Documents/GitHub/myproject/docroot/sites/all/themes/mytheme/node_modules/dist/modulename/etc. Windows hates deep folder hierarchy, and if you try deleting a folder with such a structure either in Explorer or using rmdir in PowerShell, you'll likely end up with an error message (e.g. File name too long...).

To delete a folder with a deep hierarchy (many nested directories), you need to install robocopy (part of the Windows Server set of tools), then follow these directions to delete the directory.

Node.js and npm problems

There are myriad issues running Node.js, NPM, and the ecosystem of associated build tools. It's hard enough keeping things straight with multiple Node versions and nvm on Mac/Linux... but toss in a Windows environment and most corporate networks/group policies, and you will also need to deal with:

  • If you have certain flavors of antivirus running, you might have trouble with Node.js and NPM.
  • If you are behind a corporate proxy, you will need to run a few extra commands to make sure bolt can do what it needs to do.

If you attempt to use Node/NPM within Windows, you should run Cygwin or Cmder as administrator, and possibly disable AntiVirus software. If working behind a proxy, you will also need to configure NPM to work with the proxy (in addition to configuring the VM/Linux in general to work behind the proxy):

$ npm config set proxy http://username:[email protected]:port/
$ npm config set https-proxy http://username:[email protected]:port/

Intel VT-x virtualization

Many PC laptops (especially those from Lenovo, HP, and Dell) have Intel's VT-x virtualization turned off by default, which can cause issues with many Vagrant boxes. Check your computer manufacturer's knowledge base for instructions for enabling VT-x in your system BIOS/UEFI settings.

I have a Lenovo T420, and had to follow these instructions for enabling virtualization from Lenovo's support site.

Other Notes

I've also compiled a list of Windows tips and tricks in the Drupal VM project documentation: Drupal VM Docs - Windows Notes.

Summary

Developing for Drupal with Vagrant and VMs on Windows is possible—I've used Drupal VM and related projects with Windows 7, 8, and 10, with and without proxies, on a variety of hardware—but not optimal for all cases. If you keep running into issues like those listed above (or others), you might want to investigate switching your development environment to Linux or Mac OS X instead.

Feb 29 2016
Feb 29

I'm going to bring the Raspberry Pi Dramble with me to php[tek] on May 25 in St. Louis this year, and I'm hoping to also bring it with me to DrupalCon New Orleans in early May (I submitted the session Highly-available Drupal 8 on a Raspberry Pi Cluster, hopefully it's approved!).

Raspberry Pi model 3 B from Raspberry Pi Foundation

After this morning's official announcement of the Raspberry Pi model 3 B, I placed two orders with separate vendors as quickly as possible; I'm hoping I can get at least one or two to run some benchmarks and see where the Pi Dramble can get the most benefit from the upgraded ARMv8 processor (it's a 64 bit processor with a higher base clock speed than the model 2); I'm also going to see if any of the other small improvements in internal SoC architecture make an impact on real-world Drupal and networking benchmarks.

I also now have three Raspberry Pi Zeros that I'm working with to build a creative, battery-powered cluster for educational purposes, but without the quad core processor of the Pi 2/3, speed is a huge limitation in what this smaller cluster (it's tiny!) can do.

At a minimum, I'll have a slightly faster single Pi for running Drupal 8 / www.pidramble.com from home while the cluster is on the road, using the Drupal Pi project!

Feb 29 2016
Feb 29

With the rampant speculation there will be a new Raspberry Pi model released next week, I was wondering if the official Raspberry Pi blog might reveal anything of interest; they just posted a Four Years of Pi blog post on the 26th, which highlighted the past four years, and mentioned the excitement surrounding 4th anniversary of Pi sales, coming up on February 29th, 2016.

Glancing at the blog's source, I noticed it looks like a Wordpress blog (using httpie on the cli):

$ http https://www.raspberrypi.org/blog/four-years-of-pi/ | grep generator
<meta name="generator" content="WordPress 4.4.2" />

Having set up a few WP sites in the past, I knew there was a simple way to load content by its ID, using a URL in the form:

https://www.example.org/?p=[post-id-here]

Trying this on the RPi blog, I put in the post ID of the latest blog post, which is set as a class on the <body> tag: postid-20167: https://www.raspberrypi.org/?p=20167

This URL redirects to the pretty URL (yay for good SEO, at least :), so this means if I can put in other IDs, and get back valid pages (or just valid redirects), I can start enumerating the post IDs and seeing what I can find. Checking a few other IDs gets some interesting images, a few 404s with no redirects... and eventually, a 404 after a redirect, with a fairly large spoiler (well, not so large if you're following Raspberry Pi news almost anywhere this weekend!).

$ http head https://www.raspberrypi.org/?p=[redacted]
HTTP/1.1 301 Moved Permanently
Location: https://www.raspberrypi.org/[redacted]

If I load the 'Location' URL, I get:

$ http head https://www.raspberrypi.org/[redacted]
HTTP/1.1 404 Not Found

So... for SEO purposes, it's best to either drop the ?p=[id] format, or redirect it to the pretty URL. However, for information security, this redirect should only happen if the content is published, because it can lead (like we see here) to information disclosure.

Other CMSes like Drupal and Joomla (or most any other CMS/custom CMS I've seen) can also suffer from the same enumeration problem, and I know at least for Drupal, there are tools like Username Enumeration Prevention for usernames and m4032404 for other content. Another way to work around this particular problem is to stage content in a non-production environment, and only have the content exist in production at all once it's published.

Note that enumeration by ID (posts, users, etc.) is not necessarily considered a security vulnerability (and it's really not... it's not like someone can hack your site using this attack). But it can lead to unwanted information leakage.

Feb 25 2016
Feb 25

After months of having this on my todo list, I've finally had the time to record a quick introduction video for Drupal VM. Watch the video below, then a transcript below the video:

Drupal VM is a local development environment for Drupal that's built with Vagrant and Ansible. It helps you build and maintain Drupal sites using best practices and the best tools. In this quick overview, I'll show you where you can learn more about Drupal VM, then show you a simple Drupal VM setup.

The Drupal VM website gives a general overview of the project and links to:

I'm going to build Drupal VM on my Mac using the Quick Start Guide.

First, download Vagrant using the link in the Quick Start Guide and install it on your computer. Vagrant will install the only other required application, VirtualBox, the first time you run it. (If you're on a Mac or Linux PC, you should also install Ansible for the best experience.)

Next, go back to the Drupal VM website, then download Drupal VM using the download link.

  • Copy the example.drupal.make.yml file to drupal.make.yml.
  • Then copy example.config.yml to config.yml, and make changes to suit your environment.

I removed some of the tools inside the installed_extras section since I don't need them for this demonstration.

Open your Terminal, and change directories into the Drupal VM directory using the cd command.

cd ~/Downloads/drupal-vm-master

Type in vagrant up, and after a few minutes, Drupal VM will be set up and ready for you to use.

Once the VM is built, visit http://dashboard.drupalvm.dev/ to see an overview of all the sites and software on your VM. Visit http://drupalvm.dev/ to see the Drupal 8 site that was automatically created.

At this point, after I'm finished working on my project, I can shut down the VM using vagrant halt, restart it with vagrant reload, or delete it and start over from scratch with vagrant destroy.

I'll be posting other videos demonstrating Drupal VM on Windows, Drupal VM with PHP 7, and how to use Drupal VM with existing Drupal sites, or multisite Drupal installs!

For more information about Drupal VM, visit the Drupal VM website at http://www.drupalvm.com/.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web