Jul 04 2018
Jul 04

When you're building a site, it's standard to set up "pretty URLs". For example, we'd rather see /catalog/accessories than /taxonomy/term/18 and we'd rather see /about-us than /node/19. With Drupal, we use the Pathauto module to configure these types of standard content paths. But in some cases, there's no option to create a nice URL pattern, so we end up seeing those URLs.

This happens with Views that are filtered by a taxonomy term in the URL. The result is that you end up seeing taxonomy term IDs in the URL (e.g. catalog/19/search) rather than a pretty URL (e.g. catalog/accessories/search).

In this tutorial, I'll walk you through how to create a field that is used to make generate URLs for Views pages that take taxonomy terms as contextual arguments. Using this approach, a site editor will be able to configure what the URL path will look like.

Assumptions

It has been assumed that you know:

  • The basic concepts of Drupal 8.
  • How to configure fields.
  • How to configure a view with a contextual filter.
  • How to create a custom module in Drupal 8.
  • How to create a custom plugin in Drupal 8.

The solution

We're going to use the core taxonomy argument plugin (Drupal\taxonomy\Plugin\views\argument\Taxonomy) to help us fix this problem. The plugin takes a term ID from the URL and passes it to Views. We'll override the plugin so that it takes a string from the URL (slug), and then looks up the associated term ID. All the rest of the functionality we'll leave as is.

Step 1: Content and field configuration

To make the example work, we need the following configuration to be in place (most of this you get out-of-the-box with Drupal, you'll just need to add the Slug field):

  • A taxonomy vocabulary named tags.
  • Tags should have the following field:
    • Field name: Slug
    • Machine name: field_slug
    • Type: Text (Plain)
    • Size: 32 characters
  • A content type named article.
  • Article should have the following field:
    • Field name: Tags
    • Machine name: field_tags
    • Type: Entity reference (to taxonomy terms from Tags)
    • Number of values: At least one

Configuring field slug on taxonomy termConfiguring field slug on taxonomy term

Step 2: Create a custom module

Create a custom module called custom_views_argument. Declare a dependency on the views module in the .info.yml file.

Step 3: Implement hook_views_data_alter()

Reference: custom_views_argument.module

The hook_views_data_alter() hook tells Views about the various database tables, fields and the relevant plugins associated to them. We implement this hook to tell Drupal to include our custom argument plugin which we will create in the next step.

Step 4: Create a custom views argument plugin

Reference: CustomTaxonomySlug.php

Next we implement the CustomTaxonomySlug class with a proper annotation @ViewsArgument("custom_taxonomy_slug"). This tells the Views module that the class is a special class which implements a custom Views argument plugin. We extend the Drupal\taxonomy\Plugin\views\argument\Taxonomy class and override one important method CustomTaxonomySlug::setArgument().

public function setArgument($arg) {
  // If we are not dealing with the exception argument, example "all".
  if ($this->isException($arg)) {
    return parent::setArgument($arg);
  }
  // Convert slug to taxonomy term ID.
  $tid = is_numeric($arg)
    ? $arg : $this->convertSlugToTid($arg);
  $this->argument = (int) $tid;
  return $this->validateArgument($tid);
}

All we do here is catch the argument from the URL and if it is a slug, we use a convertSlugToTid() method to retrieve the underlying taxonomy term ID. That is it! The rest of the things are handled by the taxonomy plugin.

Step 5: Create Demo Content

Now that everything is in place, we'll put our solution to the test. Start by creating some demo content. Create 2-3 articles and assign them some tags. The tags are created, however, they don't have a slug.

Once done, go to the Admin > Structure > Taxonomy > Tags page and edit the tags and give them nice URL slugs containing only English alphabet letters, numbers and dashes. For real projects, you might need to use a custom or contrib module to automatically generate slugs (see the machine_name module).

Step 6: Configure a View

Now we're all set! The last step is to create and configure a View which will put everything together.

  • Create a View of Content. You can name it Blog.
  • Create a page display and set it's URL to /blog/%.
  • Add a relationship to taxonomy terms referenced from field_tags.
    • We do this to be able to use the Slug field in a filter. 

Configure a relationship with taxonomy termsConfigure a relationship with taxonomy terms

  • Now, define a contextual filter for the Slug using the custom argument plugin which we created.
    • Click on the Add button for Contextual filters
    • Choose the Slug filter which we created. It should have the name we had defined in our plugin, i.e. Custom: Has taxonomy term with slug.
    • Optionally, specify a validation criteria for Taxonomy Term and specify the Tags vocabulary.

Configuring the custom views argument pluginConfiguring the custom views argument plugin for contextual filters

  • Save the view for the new configuration to take effect.

And we're done! If you visit the /blog/SLUG, you should see all the articles which have the taxonomy term associated to SLUG. Here, SLUG refers to the value you put in the Slug field for the tag. E.g. if you have a tag named Accessories and you wrote accessories in the Slug field, you should go the the URL /blog/accessories.

Next steps

If you're looking for hands-on training about how to develop Drupal modules, we offer professional Drupal training online and in-person. See the full Drupal 8 Module Development training curriculum for details.

Jun 14 2018
Jun 14

Drupal's field system is awesome and it is one of the reasons why I started using Drupal in the first place. However, there are some small limitations in it which surface from time to time. Say, you have a Text (Plain) field named field_one_liner which is 64 characters long. You created around 30 nodes and then you realized that the field size should have been 255. Now, if you try to do this from Drupal's field management UI, you will get a message saying:

There is data for this field in the database. The field settings can no longer be changed.

So, the only way you can resize it is after deleting the existing field! This doesn't make much sense because it's indeed possible to increase a field's size using SQL without saying goodbye to the data.

In this tutorial, we'll see how to increase the size of an existing Text (Plain) field in Drupal 8 without losing data using a hook_update_N().

Assumptions

  • You have intermediate / advanced knowledge of Drupal.
  • You know how to develop modules for Drupal.
  • You have basic knowledge of SQL.

Prerequisites

If you're going to try out the code provided in this example, make sure you have the following field on any node type:

  • Name: One-liner
  • Machine name: field_one_liner
  • Type: Text (Plain)
  • Length: 64

After you configure the field, create some nodes with some data on the One-liner field.

Note: Reducing the length of a field might result in data loss / truncation.

Implementing hook_update_N()

Reference: Custom Field Resize module on GitHub

hook_update_N() lets you run commands to update the database schema. You can create, update and delete database tables and columns using this hook after your module has been installed. To implement this hook, you need to have a custom module. For this example, I've implemented this hook in a custom module which I've named <a>custom_field_resize</a>. I usually name all my custom modules custom_ to namespace them. In the custom module, we implement the hook in a MODULE.install file, where MODULE is the machine-name of your module.

/**
 * Increase the length of "field_one_liner" to 255 characters.
 */
function custom_field_resize_update_8001() {}

To change the field size, there are four things we will do inside this hook.

Resize the Columns

We'll run a set of queries to update the relevant database columns.

$database = \Drupal::database();
$database->query("ALTER TABLE node__field_one_liner MODIFY field_one_liner_value VARCHAR(255)");
$database->query("ALTER TABLE node_revision__field_one_liner MODIFY field_one_liner_value VARCHAR(255)");

If revisions are disabled then the node_revision__field_one_liner table won't exist. So, you can remove the second query if your entity doesn't allow revisions.

Update Storage Schema

Resizing the columns with a query is not sufficient. Drupal maintains a record of what database schema is currently installed. If we don't do this then Drupal will think that the database schema needs to be updated because the column lengths in the database will not match the configuration storage.

$storage_key = 'node.field_schema_data.field_one_liner';
$storage_schema = \Drupal::keyValue('entity.storage_schema.sql');
$field_schema = $storage_schema->get($storage_key);
$field_schema['node__field_one_liner']['fields']['field_one_liner_value']['length'] = 255;
$field_schema['node_revision__field_one_liner']['fields']['field_one_liner_value']['length'] = 255;
$storage_schema->set($storage_key, $field_schema);

The above code will update the key_value table to store the updated length of the field_one_liner in its configuration.

Update Field Configuration

We took care of the database schema data. However, there are other places where Drupal stores the configuration. Now, we will need to tell the Drupal config management system that the field length is 255.

// Update field configuration.
$config = \Drupal::configFactory()
  ->getEditable('field.storage.node.field_one_liner');
$config->set('settings.max_length', 255);
$config->save(TRUE);

Finally, Drupal also stores info about the actively installed configuration and schema. To refresh this, we will need to re-save the field storage configuration to make Drupal detect all our changes.

// Update field storage configuration.
FieldStorageConfig::loadByName($entity_type, $field_name)->save();

After this, running drush updb or running update.php from the admin interface should detect your hook_update_N() and it should update your field size. If you're committing your configuration to git, you'll need to run drush config-export after running the database updates to update the config in the filesystem and then commit it.

Conclusion

Though we've talked about resizing a Text (Plain) or varchar field in this tutorial, we can resize any field type which can be safely resized using SQL. In certain rare scenarios, it might be necessary to create a temporary table with the new data-structure, copy the existing data into that table with queries and once all the data has been copied successfully, replace the existing table with the temporary table. For example, if you want to convert a Text (Plain) field to a Text (Long) field or some other type.

Maybe someday we'll have a resizing feature in Drupal where Drupal will intelligently allow us to increase a field's size from it's field UI and only deny reduction of field size where there is a possibility of data loss. But, in the meanwhile, we can use this handy trick to resize our fields. Thanks for reading! Please leave your comments / questions in the comments below and I'll get back to them as soon as I have time.

Mar 19 2018
Mar 19

One of the most interesting features added in Drupal 8.5 is the new layout builder module. The layout builder lets you change the way your content is presented. You can add sections to display content using different layouts, and build out according to your design requirements. The exciting part is that you can combine these sections together to create truly customized pages. The user interface, though still a work-in-progress, is similar to page builders in systems like Square Space, WordPress, and Wix. Combine this UI with Drupal's content management features, and the layout builder is a really powerful site building tool.

The layout builder can be used in two ways. You can use it to create a layout for each content type on your site and you can also use it to create a layout for each individual piece of content. This second use case makes the layout builder a landing-page-building tool that content editors and marketers can use to create flexible pages within a Drupal site.

One might think of this module as a Drupal core version of the Display Suite or Panels modules.

The layout builder module is currently experimental, meaning that its API might change and it's not recommended to use it in production sites yet. That's because there's a risk that your layouts will stop working when you do an update because of changes to the module.

Configuring the Layout Builder Module for Content Types

Let's say that we want to display an article in two columns. One column for the image and another for our text fields:

Screenshot of a two column layout created with the layout builderA two-column layout created with the layout builder.

To be able to use the layout builder, enable the module named Layout Builder from the Extend page (admin/modules) in the admin section.

Layout builder module on the 'Extend' page

Having enabled the module, the next step is to configure the display of our article content type. For this example, we will modify the Default display of the Article content type. Simply go to Admin > Structure > Content types > Article > Manage Display (admin/structure/types/manage/article/display) and click on the Manage Layout button.

Manage layout for the article default display

Clicking on Manage Layout should take you to a page where you can modify the layout for articles. The layout is made up of sections and each section can display blocks and fields.

  • Sections: Each section can contain content arranged in a certain layout. Example: 2 columns, 3 columns, etc.
  • Inside each section, you can display:
    • Fields from the content being displayed. Example: title, body, tags, etc.
    • Blocks which appear on the Structure > Block Layout page. Example: Page title, tabs, blocks from the custom block library, etc.

For this example, we configure a 2-column layout with the image in the left column and some other fields like author name, body and tags in the right column.

Configure sections and blocks for the contentChoose the layout and arrange your blocks​​​​.

Once you are done configuring, click on the Save Layout link towards the top of the page. That's it! Now, when you visit the article view page, you should be able to see your layout in action.

Configuring the Layout Builder Module for Specific Nodes

On the Manage Display tab, you can also select a checkbox to 'Allow each content item to have a customized layout'. This means that each piece of content has its own 'Layout' tab where you can add sections and change the layout and content for each individual article.

The layout mechanism works the same way, and you can place sections on the page and pick the layout and content for each one.

Video Tutorial

Confused? Check out this video created by my colleague Suzanne Dergacheva explaining how the layout builder works. For more in-depth training on creating landing pages with Drupal using this and other techniques, see our Landing Page Architecture and Theming course. We're offering this course at DrupalCon Nashville in April.

[embedded content]

Feb 26 2018
Feb 26

While working on a project with Acquia Dev Desktop (ADD), we needed to run a specific version of PHP which is not included with ADD by default. We started hacking ADD and came up with our own solution. For those in a hurry, you can go directly to the solution.

The Problem

While working for one of our clients, we had to work with some tools which were a part of their workflow. Their websites were hosted on Acquia Cloud so they were using Acquia Dev Desktop (ADD) - Acquia's development stack and cloud client. This tool allows developers to "run and develop Drupal sites locally and to optionally sync them with Acquia Cloud".

At the time of writing this post ADD is was version 2.1 and it supported only up to PHP 7.0.14. At first, we thought it was good enough. But soon we discovered that the cloud servers were running more cutting-edge versions such as PHP 7.1.8. 

The first issue came when we had to update our Composer packages. Running composer from different environments running different versions of PHP may result in each environment downloading a different version of the same package. A workaround is to always have the same environment run the composer update. However, we needed to ensure that the environment with the oldest version of PHP was able to run all the updated libraries. For example, the latest doctrine/common libraries require PHP ~7.1, so using ADD 2 with PHP 7.0.14 we couldn't run it.

Another issue arose when we decided to update to Drupal 8.4.x. According to an issue on drupal.org, Drupal 8.4.x requires Drush 8.1.12+, but ADD 2 ships with Drush 8.1.10.

Facing such issues, we asked Google, "how to add a version of PHP to Acquia Dev Desktop?" No good results came up so we decided to start hacking ADD and ultimately found a solution.

Adding a specific version of PHP to Acquia Dev Desktop

Our solution worked on a Mac, but we believe it should work somewhat similarly on Windows.

Step 1: Install the required version of PHP

Install the desired PHP version with Brew, in our situation it was PHP 7.1.8_20:

$ brew install homebrew/php/php71

Step 2: Copy PHP files into Acquia Dev Desktop

Stop ADD if it's running and copy the PHP files into the Acquia Dev Desktop directories:

$ cp -r /usr/local/Cellar/php71/7.1.8_20/ /Applications/DevDesktop​/php7_1
$ cp /usr/local/etc/php/7.1/php.ini /Applications/DevDesktop​​/php7_1/bin/php.ini

Note: We tried to create a symlink to prevent file duplication but it didn't work, so we ended simply copying the files into ADD, respecting its directories nomenclature and structure.

Step 3: Make Acquia Dev Desktop detect the new version of PHP

Edit /Applications/DevDesktop/Acquia Dev Desktop.app/Contents/MacOS/static.ini and after the last contiguous line under [php/4] add the following lines:

[php/5]
id=php7_1
executablePath=php7_1/bin/php
cgiExecutablePath=php7_1/bin/php-cgi
configPath=/Applications/DevDesktop/php7_1/bin/php.ini

Note: If you are using Finder to navigate to the file, you may have to right click on the Acquia Dev Desktop application icon to see its content.

Step 4: Configure Acquia Dev Desktop

Open the ADD Preferences and select the Config tab. Now, in the Default PHP Version, select the one we just added. Make sure you select PHP mode Fast CGI and have PHP use the php.ini we previously copied into ADD in Step 2. Back in the ADD Preferences main window, in the left panel select the site you are working on, and then in Default PHP Version select the desired version. Then, click Ok and restart Apache.

Step 5: Configure Drush to use the same PHP version

To configure ADD's Drush to use the same PHP version, edit /Applications/DevDesktop/tools/drush and replace the following:

# Before
[ -z "$PHP_ID" ] && PHP_ID=php5_5    
# After
export PHP_ID="php7_1"

Step 6: Update Acquia Dev Desktop's version of Drush

To udpate Acquia Dev Desktop's Drush, edit /Applications/DevDesktop/tools/composer.json and make the following change:

// Before
"drush/drush": "8.1.10"
// After
"drush/drush": "^8.1.12"

Note: Drupal 8.4.x requires Drush 8.1.12+.

Now, in the terminal, go to the same directory as composer.json and run this command to finish:

composer update --optimize-autoloader

Et voilà! We have just installed a custom version of PHP on Acquia Dev Desktop. Feel free to share your experiences in the comments below to help other fellow readers.

Feb 26 2018
Feb 26

While working on a project with Acquia Dev Desktop (ADD), we needed to run a specific version of PHP which is not included with ADD by default. We started hacking ADD and came up with our own solution. For those in a hurry, you can go directly to the solution.

This article would not have been possible without significant contributions from my colleague Benoit Borrel.

The Problem

While working for one of our clients, we had to work with some tools which were a part of their workflow. Their websites were hosted on Acquia Cloud so they were using Acquia Dev Desktop (ADD) - Acquia's development stack and cloud client. This tool allows developers to "run and develop Drupal sites locally and to optionally sync them with Acquia Cloud".

At the time of writing this post ADD is was version 2.1 and it supported only up to PHP 7.0.14. At first, we thought it was good enough. But soon we discovered that the cloud servers were running more cutting-edge versions such as PHP 7.1.8. 

The first issue came when we had to update our Composer packages. Running composer from different environments running different versions of PHP may result in each environment downloading a different version of the same package. A workaround is to always have the same environment run the composer update. However, we needed to ensure that the environment with the oldest version of PHP was able to run all the updated libraries. For example, the latest doctrine/common libraries require PHP ~7.1, so using ADD 2 with PHP 7.0.14 we couldn't run it.

Another issue arose when we decided to update to Drupal 8.4.x. According to an issue on drupal.org, Drupal 8.4.x requires Drush 8.1.12+, but ADD 2 ships with Drush 8.1.10.

Facing such issues, we asked Google, "how to add a version of PHP to Acquia Dev Desktop?" No good results came up so we decided to start hacking ADD and ultimately found a solution.

Adding a specific version of PHP to Acquia Dev Desktop

Our solution worked on a Mac, but we believe it should work somewhat similarly on Windows.

Step 1: Install the required version of PHP

Install the desired PHP version with Brew, in our situation it was PHP 7.1.8_20:

$ brew install homebrew/php/php71

Step 2: Copy PHP files into Acquia Dev Desktop

Stop ADD if it's running and copy the PHP files into the Acquia Dev Desktop directories:

$ cp -r /usr/local/Cellar/php71/7.1.8_20/ /Applications/DevDesktop​/php7_1
$ cp /usr/local/etc/php/7.1/php.ini /Applications/DevDesktop​​/php7_1/bin/php.ini

Note: We tried to create a symlink to prevent file duplication but it didn't work, so we ended simply copying the files into ADD, respecting its directories nomenclature and structure.

Step 3: Make Acquia Dev Desktop detect the new version of PHP

Edit /Applications/DevDesktop/Acquia Dev Desktop.app/Contents/MacOS/static.ini and after the last contiguous line under [php/4] add the following lines:

[php/5]
id=php7_1
executablePath=php7_1/bin/php
cgiExecutablePath=php7_1/bin/php-cgi
configPath=/Applications/DevDesktop/php7_1/bin/php.ini

Note: If you are using Finder to navigate to the file, you may have to right click on the Acquia Dev Desktop application icon to see its content.

Step 4: Configure Acquia Dev Desktop

Open the ADD Preferences and select the Config tab. Now, in the Default PHP Version, select the one we just added. Make sure you select PHP mode Fast CGI and have PHP use the php.ini we previously copied into ADD in Step 2. Back in the ADD Preferences main window, in the left panel select the site you are working on, and then in Default PHP Version select the desired version. Then, click Ok and restart Apache.

Step 5: Configure Drush to use the same PHP version

To configure ADD's Drush to use the same PHP version, edit /Applications/DevDesktop/tools/drush and replace the following:

# Before
[ -z "$PHP_ID" ] && PHP_ID=php5_5    
# After
export PHP_ID="php7_1"

Step 6: Update Acquia Dev Desktop's version of Drush

To udpate Acquia Dev Desktop's Drush, edit /Applications/DevDesktop/tools/composer.json and make the following change:

// Before
"drush/drush": "8.1.10"
// After
"drush/drush": "^8.1.12"

Note: Drupal 8.4.x requires Drush 8.1.12+.

Now, in the terminal, go to the same directory as composer.json and run this command to finish:

composer update --optimize-autoloader

Et voilà! We have just installed a custom version of PHP on Acquia Dev Desktop. Feel free to share your experiences in the comments below to help other fellow readers.

Jan 04 2018
Jan 04

In the last few projects I've worked on at Evolving Web, we've come across a common requirement: having a collapsible section of the site on mobile devices containing the site's logo, a menu and other Drupal blocks. The Responsive Menus module is quite popular, but it only works with menu blocks - no logo, no custom text. Since we couldn't find any contrib module to solve the problem, we wrote some custom JavaScript to integrate a JavaScript plugin called Sidr, which inspired me to write a Sidr integration module for Drupal. In this article, we will discuss how the module works and how you can get it working in your next project.

Here's a screenshot from a quick demo site I prepared. You click on the hamburger menu, and the black sidr region appears on the left. You click it again and the region slides back out.

Screenshot of a collapsible Sidr in DrupalA quick demo with a Sidr panel in Drupal with the Dark theme

Installing the Sidr module and libraries

To install the module, we must install the module files and then the Sidr libraries.

  • Download the Sidr module into the modules or modules/contrib directory in your Drupal project and install the module
  • Install the Sidr libraries
    • Download the version of Sidr recommended in the module's README file, which at the time of writing this article is Sidr 2.2.1
    • Once downloaded, copy the dist directory in the Sidr project to the libraries directory in your Drupal project and rename it to sidr
    • At this point, there should be valid JavaScript file at DRUPAL/libraries/sidr/jquery.sidr.js and the Drupal status report page should show the Sidr libraries as Installed

Sidr library statusSidr library status

  • Configure the Sidr theme from the admin/config/media/sidr page. Here's some quick info about the themes:
    • The dark (default) and the light themes are provided by Sidr.
    • Typically, for a project with a custom look and feel, you'll use the bare theme. This provides minimal CSS, allowing you to style the .sidr element and its children in your theme. Yeah!

Sidr global settingsSidr global settings

Congratulations! You now have the module installed. All that's left now is a bit of configuration.

Configure collapsible content

Depending on the project requirements, you might have one or more sidr instances. For this article, let's say you want a sliding panel on the left with the following contents:

  • Site logo (the site branding block).
  • The main menu (the main menu block).

We can achieve this using two different approaches:

With a custom region (recommended)

Create a custom region named, say, Drawer (Left) in your theme where you can place whatever blocks you want to show in your Sidr. We will then configure the Sidr plugin to use the contents of this region to populate the collapsible panel (discussed below).

Note: Make sure you hide this region using CSS because Sidr will copy the contents of the region in to a div.sidr element during its initialization.

Without a custom region

If all your blocks are already present on your page, you can use multiple jQuery selectors in the Source configuration for the Sidr trigger block (discussed below) and the Sidr plugin will copy the contents of those elements and put them in the Sidr. Sidr will not copy the elements for the jQuery selector, but all of their children. This is the reason why I prefer to use a custom region. Using the above-mentioned custom region approach, you can preserve wrapper elements which give you nice CSS selectors for theming your Sidr.

Configure a trigger button

Now that we have set up the contents for the collapsible region, we are ready to create a Sidr trigger block. This trigger block will provide a button to open and close the Sidr panel. To do this,

  • Go to the Block management admin/structure/block page.
  • Click the Place block button for the region where you want to place the trigger button (usually somewhere in the header).
  • Choose the Sidr trigger block, configure it and save it. Some of the configuration options have been discussed below.

Sidr trigger settingsSidr trigger settings

Trigger text and icon

The trigger text is the text which is displayed on the Sidr trigger button. You can also enter some custom HTML in the Advanced settings > Trigger icon field to configure an icon, say, a hamburger icon or an <i class="fa fa-bars"></i>.

Note: It is compulsory to have either a trigger text or a trigger icon. You can also have both if you want.

Source

The source is from where the Sidr panel will be populated. It can be one of the following:

  • A jQuery selector: If you provide a jQuery selector, the inner HTML of the selected elements will be copied into the relevant Sidr panel. You might be interested in the renaming option which makes Sidr rename the ID attributes of the copied elements to avoid getting repeated DOM IDs. Here are some examples:
    • Using a custom region, source should look like .region-drawer-left or .region-drawer-right or whatever you name your custom region.
    • Without a custom region, you will have to refer to various blocks like #block-site-branding, #block-main-menu, #block-copyright-notice.
  • A URL: If source is a URL, the content on the URL will be fetched via AJAX and displayed in the Sidr panel.
  • A callback: You can even provide a JavaScript function as the source, in which case, the contents returned by the callable will be used to populate the Sidr panel. If you do not like the idea of copied elements provided by the jQuery selector option above, a callback might give you more flexibility.

Most of the Advanced settings are optional and you can learn about them by reading the Sidr documentation. Once you have configured everything, all you'll be missing is some custom styling to bring your Sidr to life.

Feel free to leave comments about problems you face and any suggestions you might have. You might want to read the Sidr module's project page and issue queue for the latest updates.

Dec 18 2017
Dec 18

Usually, Drupal migrations get run at the beginning of a project to get all your content into Drupal. But sometimes, you need to create a migration that runs on a regular basis. For example, you have an external database of courses and programs maintained by a university that needs to be displayed on a Drupal site. Another example: you have a database of book data that needs to be pulled into Drupal nightly.

When working with migrations where the source files are updated every day, it can get really tedious to download the updated source files manually each time the migration runs.

In this tutorial, we'll write a source plugin based on the CSV source plugin which will allow us to automatically download CSV files from a remote server via SFTP before running migrations. This article was co-authored by my colleague David Valdez - gracias David for your contribution.

The Problem

In a project we worked on recently, we had the following situation:

  • CSV files are updated by a PowerShell script every night on the client's server.
  • These CSV files are accessible via SFTP.

Our task is to download the CSV source files over SFTP and to use them as our migration source.

Before We Start

  • This articles assumes that you can write custom modules. If you have never written a custom module, you can try reading this article on creating custom modules.
  • It is assumed that you have working knowledge of migrations in Drupal 8. If you are new to Drupal 8 migrations, I recommend you to start by reading these articles first:

The Plan

The goal is to avoid downloading the file manually every time we run our migrations. So we need a way to doing this automatically everytime we execute a migration. To achieve this, we create a custom source plugin extending the CSV plugin provided by the Migrate Source CSV module, which will download CSV files from a remote server and pass it to the CSV plugin to process them.

The Source Migrate Plugin

To start, let's create a custom module and call it migrate_example_source and implement a custom migrate source plugin by creating a PHP class inside it at /src/Plugin/migrate/source/MigrateExampleSourceRemoteCSV.php

We start implementing the class by simply extending the CSV plugin provided by the migrate_source_csv module:
namespace Drupal\migrate_source_csv\Plugin\migrate\source;

use Drupal\migrate_source_csv\Plugin\migrate\source\CSV as SourceCSV;
use phpseclib\Net\SFTP

/**
 * @MigrateSource(
 *   id = "migrate_example_source_remote_csv"
 * )
 */
class MigrateExampleSourceRemoteCSV extends SourceCSV {}

If you are building a source plugin from scratch, you will need to extend the SourcePluginBase class instead of the CSV class given in this example. Adding the annotation @MigrateSource is very important because that is what will make the migrate module detect our source plugin. In our plugin, we use the phpseclib/phpseclib libraries to make SFTP connections. Hence, we need to include the libraries in our project by running the following command in the Drupal root:

composer require phpseclib/phpseclib

Our plugin will download the source CSV file and will simply pass it to the CSV plugin to do the rest. We do the download when the plugin is being instantiated like this:

/**
 * {@inheritdoc}
 */
public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration) {
  // If SFTP connection parameters are present.
  if (!empty($configuration['sftp'])) {
    // A settings key must be specified.
    // We use the settings key to get SFTP configuration from $settings.
    if (!isset($configuration['sftp']['settings'])) {
      throw new MigrateException('Parameter "sftp/settings" not defined for Remote CSV source plugin.');
    }
    // Merge plugin settings with global settings.
    $configuration['sftp'] += Settings::get('sftp', []);
    // We simply download the remote CSV file to a temporary path and set
    // the temporary path to the parent CSV plugin.
    $configuration['path'] = $this->downloadFile($configuration['sftp']);
  }
  // Having downloaded the remote CSV, we simply pass the call to the parent plugin.
  parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);
}

In the constructor we are using global SFTP credentials with Settings::get(). We need to define the credentials in settings.php like this:

$settings['sftp'] = array(
  'default' => [
    'server' => 'ftp.example.com',
    'username' => 'username',
    'password' => 'password',
    'port' => '22',
  ],
);

Once we have the credentials of the FTP server we use a downloadFile() method to download the remote CSV file. Here's an extract of the relevant code:

protected function downloadFile(array $conn_config) {
  ...
  // Prepare to download file to a temporary directory.
  $path_remote = $conn_config['path'];
  $basename = basename($path_remote);
  $path_local = file_directory_temp() . '/' . $basename;
  ...
  // Download file by SFTP and place it in temporary directory.
  $sftp = static::getSFTPConnection($conn_config);
  if (!$sftp->get($path_remote, $path_local)) {
    throw new MigrateException('Cannot download remote file ' . $basename . ' by SFTP.');
  }
  ...
  // Return path to the local of the file.
  // This will in turn be passed to the parent CSV plugin.
  return $path_local;
}

Note: The code block above has been simplified a bit. If you see the actual source plugin, there are some lines of code which make things more compatible with the migration_lookup plugin.

This method creates an SFTP connection, downloads the file to a temporary location and returns the path to the downloaded file. The temporary file path is then passed to the Migrate Source CSV and that's it! Finally, to use the plugin in our migration we just set our plugin as the source/plugin:

id: migrate_example_content
label: 'Example content'
...
source:
  plugin: migrate_example_source_remote_csv
  # Settings for our custom Remote CSV plugin.
  sftp:
    settings: sftp
    path: "/path/to/file/example_content.csv"
  # Settings for the contrib CSV plugin.
  header_row_count: 1
  keys:
    - id
...

The code for this plugin and the example module is available at migrate_example_source. Great!

Nov 08 2017
Nov 08

A few weeks ago, us at Evolving Web finished migrating the Princeton University Press website to Drupal 8. The project was over 70% migrations. In this article, we will see how Blackfire helped us optimize our migrations by changing around two lines of code.

Before we start

  • This article is mainly for PHP / Drupal 8 back-end developers.
  • It is assumed that you know about the Drupal 8 Migrate API.
  • Code performance is analyzed with a tool named Blackfire.
  • Front-end performance analysis is not in the scope of this article.

The Problem

Here are some of the project requirements related to the problem. This would help you get a better picture of what's going on:

  • A PowerShell script exports a bunch of data into CSV files on the client's server.
  • A custom migration plugin PUPCSV uses the CSV files via SFTP.
  • Using hook_cron() in Drupal 8, we check hashes for each CSV.
  • If a file's MD5 hash changes, the migration is queued for import using the Drupal 8 Queue API.
  • The CSV files usually have 2 types of changes:
    • Certain records are updated here and there.
    • Certain records are added to the end of the file.
  • When a migration is executed, migrate API goes line-by-line, doing the following things for every record:
    • Read a record from the data source.
    • Merge data related to the record from other CSV files (kind of an inner join between CSVs).
    • Compute hash of the record and compare it with the hash stored in the database.
    • If a hash is not found in the database, the record is created.
    • If a hash is found and it has changed, the record is updated.
    • If a hash is unchanged, no action is taken.

While running migrations, we figured out that it was taking too much time for migrations to go through the CSV files, simply checking for changes in row hashes. So, for big migrations with over 40,000 records, migrate was taking several minutes to reach the end of file even on a high-end server. Since we were running migrate during cron (with Queue Workers), we had to ensure that any individual migration could be processed below the 3 minute PHP maximum execution time limit available on the server.

Analyzing migrations with Blackfire

At Evolving Web, we usually analyze performance with Blackfire before any major site is launch. Usually, we run Blackfire with the Blackfire Companion which is currently available for Google Chrome and Firefox. However, since migrations are executed using drush, which is a command line tool, we had to use the Blackfire CLI Tool, like this:

$ blackfire run /opt/vendor/bin/drush.launcher migrate-import pup_subjects
Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'pup_subjects'

Blackfire Run completed

Upon analyzing the Blackfire reports, we found some 50 unexpected SQL queries being triggered from somewhere within a PUPCSV::fetchNextRow() method. Quite surprising! PUPCSV refers to a migrate source plugin we wrote for fetching CSV files over FTP / SFTP. This plugin also tracks a hash of the CSV files and thereby allows us to skip a migration completely if the source files have not changed. If the source hash changes, the migration updates all rows and when the last row has been migrated, we store the file's hash in the database from PUPCSV::fetchNextRow(). As a matter of fact, we are preparing another article about creating custom migrate source plugin, so stay tuned.

We found one database query per row even though no record was being created or updated. Didn't seem to be very harmful until we saw the Blackfire report.

Screenshot of Blackfire report highlighting the problem

Code before Blackfire

Taking a closer look at the RemoteCSV::fetchNextRow() method, a call to MigrateSourceBase::count() was found. It was found that the count() method was taking 40% of processing time! This is because it was being called for every row in the CSV. Since the source/cache_counts parameter was not set to TRUE in the migration YAML files, the count() method was iterating over all items to get a fresh count for each call! Thus, for a migration with 40,000 records, we were going through 40,000 x 40,000 records and the PHP maximum execution time was being reached even before migrate could get to the last row! Here's a look at the code.

protected function fetchNextRow() {
  // If the migration is being imported...
  if (MigrationInterface::STATUS_IMPORTING === $this->migration->getStatus()) {
    // If we are at the last row in the CSV...
    if ($this->getIterator()->key() === $this->count()) {
      // Store source hash to remember the file as "imported".
      $this->saveCachedFileHash();
    }
  }
  return parent::fetchNextRow();
}

Code after Blackfire

We could have added the cache_counts parameter in our migration YAML files, but any change in the source configuration of the migrations would have made migrate API update all records in all migrations. This is because a row's hash is computed as something like hash($row + $source). We did not want migrate to update all records because we had certain migrations which sometimes took around 7 hours to complete. Hence, we decided to statically cache the total record count to get things back in track:

protected function fetchNextRow() {
  // If the migration is being imported...
  if (MigrationInterface::STATUS_IMPORTING === $this->migration->getStatus()) {
    // Get total source record count and cache it statically.
    static $count;
    if (is_null($count)) {
      $count = $this->doCount();
    }
    // If we are at the last row in the CSV...
    if ($this->getIterator()->key() === $count) {
      // Store source hash to remember the file as "imported".
      $this->saveCachedFileHash();
    }
  }
  return parent::fetchNextRow();
}

Problem Solved. Merci Blackfire!

After the changes, we ran Blackfire again and found things to be 52% faster for a small migration with 50 records.

Blackfire before-after comparison report

For a bigger migration with 4,359 records the migration import time reduced from 1m 47s to only 12s which means a 98% improvement. Asking why we didn't include the screenshot for the bigger migration? We did not (or rather could not) generate a report for the big migration because of two reasons:

  • While working, Blackfire stores function call and other information to memory. Running a huge migration with Blackfire might be a bit slow. Besides, our objective was to find the problem and we could do that more easily while looking at smaller figures.
  • When running a migration with thousands of rows, the migration functions are called over thousands of times! Blackfire collects data for each of these function calls, hence, the collected data sometimes becomes too heavy and Blackfire rejects the huge data payload with an error message like this:
The Blackfire API answered with a 413 HTTP error ()
Error detected during upload: The Blackfire API rejected your payload because it's too big.

Which makes a lot of sense. As a matter of fact, for the other case study given below, we used the --limit=1 parameter to profile code performance for a single row.

A quick brag about another 50% Improvement?

Apart from this jackpot, we also found room for another 50% improvement (from 7h to 3h 32m) for one of our migrations which was using the Touki FTP library. This migration was doing the following:

  • Going through around 11,000 records in a CSV file.
  • Downloading the files over FTP when required.

A Blackfire analysis of this migration revealed something strange. For every row, the following was happening behind the scenes:

  • If a file download was required, we were doing FTP::findFileByName($name).
  • To get the file, Touki was:
    • Getting a list of all files in the directory;
    • Creating File objects for every file;
    • For every file object, various permission, owner and other objects were created.
    • Passing all the files through a callback to see if it's name was $name.
    • If the name was matching, the file was returned and all other File objects were discarded.

Hence, for downloading every file, Touki FTP was creating 11,000 File objects of which it was only using one! To resolve this, we decided to use a lower-level FTP::get($source, $destination) method which helped us bypass all those 50,000 or more objects which were being created per record (approximately, 11,000 * 50,000 or more for all records). This almost halved the import time for that migration when working with all 11,000 records! Here's a screenshot of Blackfire's report for a single row.

Blackfire before-after comparison report

So the next time you think something fishy is going on with code you wrote, don't forget to use use Blackfire! And don't forget to leave your feedback, questions and even article suggestions in the comments section below.

More about Blackfire

Blackfire is a code profiling tool for PHP which gives you nice-looking reports about your code's performance. With the help of these reports, you can analyze the memory, time and other resources consumed by various functions and optimize your code where necessary. If you are new to Blackfire, you can try these links:

Apart from all this, the paid version of Blackfire lets you set up automated tests and gives you various recommendations for not only Drupal but various other PHP frameworks.

Next Steps

  • Try Blackfire for free on a sample project of your choice to see what you can find.
  • Watch video tutorials on Blackfire's YouTube channel.
  • Read the tutorial on creating custom migration source plugins written by my colleague (coming soon).
Aug 24 2017
Aug 24

When content URLs change during migrations, it is always a good idea to do something to handle the old URLs to prevent them from suddenly starting to throw 404s which are bad for SEO. In this article, we'll discuss how to migrate URL aliases provided by the path module (part of D8 core) and URL redirects provided by the redirect module.

The Problem

Say we have two CSV files (given to us by the client):

The project requirement is to:

  • Migrate the contents of article.csv as article nodes.
  • Migrate the contents of category.csv as terms of a category terms.
  • Make the articles accessible at the path blog/{{ category-slug }}/{{ article-slug }}.
  • Make blog/{{ slug }}.php redirect to article/{{ article-slug }}.

Here, the term slug refers to a unique URL-friendly and SEO-friendly string.

Before We Start

Migrate Node and Category Data

This part consists of two simple migrations:

The article data migration depends on the category data migration to associate each node to a specific category like:

# Migration processes
process:
  ...
  field_category:
    plugin: 'migration_lookup'
    source: 'category'
    migration: 'example_category_data'
    no_stub: true
  ...

So, if we execute this migration, we will have all categories created as category terms and 50 squeaky new nodes belonging to those categories. Here's how it should look if we run the migrations using drush:

$ drush migrate-import example_article_data,example_category_data
Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'example_category_data'
Processed 50 items (50 created, 0 updated, 0 failed, 0 ignored) - done with 'example_article_data'

Additionally, we will be able to access a list of articles in each category at the URL blog/{{ category-slug }}. This is because of the path parameter we set in the category data migration. The path parameter is processed by the path module to create URL aliases during certain migrations. We can also use the path parameter while creating nodes to generate URL aliases for those nodes. However, in this example, we will generate the URL aliases in a stand-alone migration.

Generate URL Aliases with Migrations

The next task will be to make the articles available at URLs like /blog/{{ category-slug }}/{{ article-slug }}. We use the example_article_alias migration to generate these additional URL aliases. Important sections of the migration are discussed below.

Source

source:
  plugin: 'csv'
  path: 'article.csv'
  ...
  constants:
    slash: '/'
    source_prefix: '/node/'
    alias_prefix: '/blog/'
    und: 'und'

We use the article.csv file as our source data to iterate over articles. Also, we use source/constants to define certain data which we want to use in the migration, but we do not have in the CSV document.

Destination

destination:
  plugin: 'url_alias'

Since we want to create URL aliases, we need to use the destination plugin url_alias provided by the path module. Reading documentation or taking a quick look at the plugin source at Drupal\path\Plugin\migrate\destination\UrlAlias::fields(), we can figure out the fields and configuration supported by this plugin.

Process

...
temp_nid:
  plugin: 'migration_lookup'
  source: 'slug'
  migration: 'example_article_data'
...
temp_category_slug:
    # First, retrieve the ID of the taxonomy term created during the "category_data" migration.
    -
      plugin: 'migration_lookup'
      source: 'category'
      migration: 'example_category_data'
    # Use a custom callback to get the category name.
    -
      plugin: 'callback'
      callable: '_migrate_example_paths_load_taxonomy_term_name'
    # Prepare a url-friendly version for the category.
    -
      plugin: 'machine_name'

Since we need to point the URL aliases to the nodes we created during the article data migration, we use use the migration_lookup plugin (formerly migration) to read the ID of the relevant node created during the article data migration. We store the node id in temp_nid. I added the prefix temp_ to the property name because we just need it temporarily for calculating another property and not for using it directly.

Similarly, we need to prepare a slug for the category to which the node belongs. We will use this slug to generate the alias property.

source:
  plugin: 'concat'
  source:
    - 'constants/source_prefix'
    - '@temp_nid'

Next, we generate the source, which is the path to which the alias will point. We do that by simply concatenating '/nid/' and '@temp_nid' using the concat plugin.

alias:
  plugin: 'concat'
  source:
    - 'constants/alias_prefix'
    - '@temp_category_slug'
    - 'constants/slash'
    - 'slug'

And finally, we generate the entire alias by concatenating '/article/', '@temp_category_slug', a '/' and the article's '@slug'. After running this migration like drush migrate-import example_article_alias, all the nodes should be accessible at /article/{{ category-slug }}/{{ article-slug }}.

Generate URL Redirects with Migrations

For the last requirement, we need to generate redirects, which takes us to the redirect module. So, we create another migration named example_article_redirect to generate redirects from /blog/{{ slug }}.php to the relevant nodes. Now, let's discuss some important lines of this migration.

Source

constants:
  # The source path is not supposed to start with a "/".
  source_prefix: 'blog/'
  source_suffix: '.php'
  redirect_prefix: 'internal:/node/'
  uid_admin: 1
  status_code: 301

We use source/constants to define certain data which we want to use in the migration, but we do not have in the CSV document.

Destination

destination:
  plugin: 'entity:redirect'

In Drupal 8, every redirect rule is an entity. Hence, we use the entity plugin for the destination.

Process

redirect_source:
  plugin: 'concat'
  source:
    - 'constants/source_prefix'
    - 'slug'
    - 'constants/source_suffix'

First, we determine the path to be redirected. This will be the path as in the old website, example, blog/{{ slug }}.php without a / in the front.

redirect_redirect:
  plugin: 'concat'
  source:
    - 'constants/redirect_prefix'
    - '@temp_nid'

Just like we did for generating aliases, we read node IDs from the article data migration and use them to generate URIs to which the user should be redirected when they visit one of the /blog/{{ slug }}.php paths. These destination URIs should be in the form internal:/node/{{ nid }}. The redirect module will intelligently use these URIs to determine the URL alias for those paths and redirect the user to the path /article/{{ slug }} instead of sending them to /node/{{ nid }}. This way, the redirects will not break even if we change the URL alias for a particular node after running the migrations.

# We want to generate 301 permanent redirects as opposed to 302 temporary redirects.
status_code: 'constants/status_code'

We also specify a status_code and set it to 301. This will create 301 permanent redirects as opposed to 302 temporary redirects. Having done so and having run this third migration as well, we are all set!

Migration dependencies

migration_dependencies:
  required:
    - 'example_article_data'

Since the migration of aliases and the migration of redirects both require access to the ID of the node which was generated during the article data migration, we need to add the above lines to define a migration_dependency. It will ensure that the example_article_data migration is executed before the alias and the redirect migrations. So if we run all the migrations of this example, we should see them executing in the correct order like:

$ drush mi --tag=example_article
Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'example_category_data'
Processed 50 items (50 created, 0 updated, 0 failed, 0 ignored) - done with 'example_article_data'
Processed 50 items (50 created, 0 updated, 0 failed, 0 ignored) - done with 'example_article_alias'
Processed 50 items (50 created, 0 updated, 0 failed, 0 ignored) - done with 'example_article_redirect'

Next steps

Jun 17 2017
Jun 17

Since the release of Drupal 8 with a standardized way of managing translations, many sites running Drupal 7 are making a switch to Drupal 8. In Drupal 7 there are two ways to translate content:

  1. Using the content_translation module. The D7 core way of translating content, where every translation is a separate node.
  2. Using the entity_translation module. Maintains one node with a unique nid, while translations take place at the field level.

In this article we will discuss how to migrate content translations created with the content_translation module from Drupal 7 to Drupal 8. You can find our tutorial about migrating translations that use Entity Translation here.

This article would not have been possible without the help of my colleague Dave. ¡Gracias Dave!

The problem

We have a Drupal 7 database containing article nodes, which might have translations in English, Spanish and French. Some of these nodes are language-neutral, i.e. non-translatable. Our target is to migrate the Drupal 7 nodes into a Drupal 8 website, preserving the translations.

Before we start

  • Since this is an advanced migration topic, it is assumed you already know the basics of migration. If are new to migrations in Drupal 8, I recommend that you read about migrating basic data to Drupal 8 first.
  • If you'd like to run the migrations in this example yourself, see the quick-start documentation in our drupal migration i18n example repository.
  • The source website used in this example is Drupal 7.54.
  • The destination website used in this example is Drupal 8.3.x. However, an alternative solution for earlier versions is included towards the end of the article.

The module

To write the migrations, we create a module - in our case, migrate_example_i18n. There's nothing special about the module declaration, except for the dependencies:

  • migrate_plus and migrate_tools provide various features for defining and executing migrations.
  • migrate_source_csv: Will be used for demonstrating migration of translated content from non-Drupal sources in an upcoming article.
  • migrate_drupal: This module provides tools for migrating data from older versions of Drupal. It comes with Drupal 8.x core. Since this migration uses a Drupal 7 site as a source for its data, we need the migrate_drupal module.

How do translations work?

Before jumping into writing these migrations, it is important to mention that Drupal 7 and Drupal 8 translations work very differently. Here's the difference in a nutshell:

  • Drupal 7: When we translate a node, a new node is created with a different ID. This translated node has a property named tnid, which stores the ID of the original node, linking the two nodes together. For language-neutral or untranslated content, the tnid is set to 0.
  • Drupal 8: When we translate a node, no new node is created! The translation is saved in the fields of the original node, but with a different language code.

So just like we do when migrating translated content from Drupal 6 to Drupal 8, we create two migrations:

  • The example_dog_base migration will migrate the original content of each node, untranslated.
  • The example_dog_i18n migration will migrate only translations and associate them with original content created by example_dog_base.

We group the two migrations using the example_dog migration group to keep things clean and organized. Then we can execute both migrations with drush migrate-import --group=example_dog --update.

Step 1: Base migration

We start with example_dog_base to migrate all base data or non-translations. Described below are some noteworthy parameters:

Source

source:
  plugin: d7_node
  node_type: article
  key: drupal_7_content
  constants:
    uid_root: 1
    node_article: 'article'
  • plugin: Since we want to import data from a Drupal installation, we need to set the source plugin to d7_node. The d7_node source plugin is introduced by the migrate_drupal, module and it helps us read nodes from a Drupal 7 database without having to write queries manually. Since Drupal 8.3.x, this plugin supports translations created with the content_translation module. If you are using an older version of Drupal 8, then check the alternative solution provided towards the end of this article.
  • node_type: This tells the source plugin that we are interested in just one particular Drupal 7 node type, namely article.
  • key: Our Drupal 7 data doesn't come from our main Drupal 8 database - instead it comes from a secondary database connection. We choose a key to identify each such connection and we need to tell the source which such key to use. The keys themselves are defined in the $databases variable in our settings.php or settings.local.php. See the example settings.local.php file to see how it's done.
  • constants: We define some hard-coded values under this parameter.
  • translations: Notice there is no translations parameter here. The default value (false) tells the source plugin that we're only interested in migrating non-translations, i.e. content in the base language and language-neutral content.

Destination

destination:
  plugin: 'entity:node'
  • plugin: Since we want to create node entities in Drupal 8, we specify this as entity:node. That's it.
  • translations: Again we do not define the translations parameter while migrating base data. Omitting the parameter tells the destination plugin that we are interested in creating fresh nodes for each record, not translations of existing nodes.

Process

type: constants/node_article
langcode:
  plugin: default_value
  source: language
  default_value: und
uid: constants/uid_root
title: title
body: body
field_one_liner: field_one_liner
sticky: sticky
status: status
promote: promote

This is where we map the old node properties to the new node properties. Most of the properties have been assigned as is, without alteration, however, some noteworthy properties have been discussed below:

  • nid: There is no nid parameter here, because we don't care what nid each new node has in Drupal 8. Drupal can just assign a new nid to each node in the normal way.
  • type: We specify that we want to create article nodes.
  • langcode: The langcode parameter was formerly language in Drupal 7, so we rename it here. Also, if a Drupal 7 node is language-neutral, the language property will have no value. In that case,  we default to und.

This takes care of the base data. If we run this migration with drush migrate-import example_hybrid_base --update, all Drupal 7 nodes which are in base language or are language-neutral will be migrated into Drupal 8.

Step 2: Translation migration

We are halfway through now! All that's missing is migrating translations of the nodes we migrated above. To do this, we create another migration with the ID example_dog_i18n:

source:
  plugin: d7_node
  node_type: article
  translations: true
  # ...
destination:
  plugin: 'entity:node'
  translations: true
process:
  nid:
    plugin: migration
    source: tnid
    migration: example_dog_base
  langcode: language
  # ...
migration_dependencies:
  required:
    - example_dog_base
  • source:
    • translations: We set this to true to make the source plugin read only translations.
  • destination:
    • translations: We set this to true to make the destination plugin create translations for existing nodes instead of creating fresh new nodes.
  • process:
    • nid: In this case, we do care what the Drupal 8 nid is for each node. It has to match the nid for the untranslated version of this content, so that Drupal can add a translation to the correct node. This section uses the migration (migration_lookup) process plugin to figure out the right nid. It tells Drupal to check the previously-executed example_hybrid_base migration for a D6 node that has the same tnid as this D6 node. It will then then reuse the resulting nid here.
    • langcode: We define the language in which the translation should be created.
  • migration_dependencies: Since we cannot add translations to nodes that do not yet exist, we tell Drupal that this migration depends on the base migration example_dog_base. That way, the base migration will run before this migration.

That's it! We can run our translation migration with drush migrate-import example_dog_i18n --update and the translations will be imported into Drupal 8. Alternatively, we can use the migration group we defined to run both these migrations at once - the base migration will automatically be executed first and then the i18n migration. Here's how the output should look:

$ drush migrate-import --group=example_dog --update
Processed 7 items (7 created, 0 updated, 0 failed, 0 ignored) - done with 'example_dog_base'
Processed 7 items (7 created, 0 updated, 0 failed, 0 ignored) - done with 'example_dog_i18n'

You can check if everything went alright by clicking the Translate option for any translated node in Drupal 8. If everything went correctly, you should see that the node exists in the original language and has one or more translations.

Article migrated from Drupal 7 to Drupal 8Article migrated from Drupal 7 to Drupal 8

Alternate Solution for Drupal 8.2.x and Older

The example code for this article works out of the box with Drupal 8.3 or higher. However, it will not work with earlier versions of Drupal 8. For Drupal 8.2 or older, we need to use a custom source plugin (inspired by the d6_node plugin). All we have to do is use the D7NodeContnentTranslation source plugin included in the code for this example, like source: d7_node_content_translation. This custom source plugin adds support for the translations parameter, which in turn makes the migration of content translations work correctly.

Next Steps

Jun 17 2017
Jun 17

Since the release of Drupal 8 with a standardized way of managing translations, many sites running Drupal 7 are making a switch to Drupal 8. In Drupal 7 there are two ways to translate content:

  1. Using the content_translation module. The D7 core way of translating content, where every translation is a separate node.
  2. Using the entity_translation module. Maintains one node with a unique nid, while translations take place at the field level.

In this article we will discuss how to migrate content translations created with the entity_translation module from Drupal 7 to Drupal 8. You can find our tutorial about migrating translations that use Content Translation here.

This article would not have been possible without the help of my colleague Dave. Merci Dave!

The problem

We have a Drupal 7 database containing article nodes, which might have translations in English, Spanish and French. Some of these nodes are language-neutral, i.e. non-translatable. Our target is to migrate the D7 nodes into a D8 website, preserving the translations.

Before we start

  • Since this is an advanced migration topic, it is assumed you already know the basics of migration. If you are new to migrations in Drupal 8, I recommend that you read about migrating basic data to Drupal 8 first.
  • This article assumes that you have read our previous article on how to migrate content translations from Drupal 7 to Drupal 8 or have the relevant knowledge.
  • To execute the migrations in this example, you can download the drupal migration i18n example repository from GitHub. The module should work without any trouble for a standard Drupal 8 install. See quick-start for more information.
  • To see the example migrations in action, you need:
    • A Drupal 8 site.
    • The relevant D7 database, since we are migrating data from a Drupal 6 site.
    • Drush will be required to execute migration commands.

The module

To write the migrations, we create a module - in our case, it has been named migrate_example_i18n. Just like migrating content translations from D7 to D8, we create 2 YML files to define:

  • The example_creature_base migration will migrate all base data or non-translations.
    • The source/translations parameter is omitted or set to false.
    • The destination/translations parameter is omitted or set to false.
  • The example_creature_i18n migration will migrate all translations.
    • The process/nid is configured to use the migration plugin to lookup the node in the base language.
    • The source/translations parameter is set to true.
    • The destination/translations parameter is to true.
    • The migration_dependencies parameter declares example_creature_base as a dependency.

We group the two migrations using the example_creature migration group to keep things clean and organized. Then we can execute both migrations with drush migrate-import --group=example_creature --update.

How to migrate Entity Translations?

Entity translations! Drupal 7 content translations are supported since Drupal 8.3. At the point of writing this, there is no standard method for migrating entity translations to Drupal 8. In this example, we will migrate D7 nodes translated with the entity_translation module, however, the procedure should be similar for other entity types as well. Before we start, here are some notes about what's so different about entity translations:

  • All translations have the same entity_id. So, for a translated node, the entity_translation module will result in only one entry in the node table.
  • Translation information, certain metadata and revision information for entities is stored in the entity_translation table.

So if an English node with ID 19 has translations in Spanish and French, the entity_translations table has the following records:

An extract from the entity_translation table. entity_type entity_id revision_id language source uid status translate created changed node 19 1 en   1 1 0 1485800973 1487198982 node 19 1 es en 1 1 0 1485802336 1487199003 node 19 1 fr en 1 1 0 1487185898 1487198969

The above data structure is significantly different from the content translation structure. In fact, Drupal 8 handles translations much like the entity translation module! Hence, to handle entity-translations, we must take the entity_translation table into consideration, which the core d7_node source plugin does not do at the time of writing this article. Hence, we override the d7_node source with a custom source plugin named d7_node_entity_translation.

This is where we jump into code! We override certain methods of d7_node source to add support for the entity_translation table.

class D7NodeEntityTranslation extends D7Node {
  // Determines if the node-type being translated supports entity_translation.
  protected function isEntityTranslatable() {}
  // Depending on the "source/translations" parameter, this method alters
  // the migration query to return only translations or non-translations.
  protected function handleTranslations(SelectInterface $query) {}
  // This method has been overridden to ensure that every node's fields are
  // are loaded in the correct language.
  public function prepareRow(Row $row) {}
  // This method is called by the prepareRow() method to load field values
  // for source nodes. We override this method to add support for $language.
  protected function getFieldValues($entity_type, $field, $entity_id, $revision_id = NULL, $language = NULL) {}
  // Since all source nodes have the same "nid", we need to use a
  // combination of "nid:language" to distinguish each source translation.
  public function getIds() {}
}

Here's a quick look at the changes we need to make:

  • function getIds() tells the migrate API to use one or more source properties which should be used to uniquely identify source records. When working with entity translations, all translations have the same entity_id, but they have a different language. We override this method to tell Drupal to consider both the entity_id and the language properties to uniquely identify source records. So, the source records are uniquely identified something like 19:en, 19:es, 19:fr instead of using just 19.
  • function handleTranslations() is the method which adds support for the translations parameter we use in the source plugin. The translations parameter tells Drupal whether to migrate entities in their base language or to migrate translations. We override this method to:
    • See if the node type being migrated supports entity translations.
    • If the node type supports entity translations, then we INNER JOIN entity_translation and read translation data and some entity metadata, like date of creation, date of updation, etc from that table.
  • function prepareRow() as the name suggests, prepares a row of source data before it is passed to the process plugins. At this stage, field data is also attached to the source data. However, it does not load field data in the language specified in the source row. To overcome this problem, we override the getFieldValues() method and make sure it loads the field data in the same language as specified in the source row.

That's it! You should now be able to run the migration with drush migrate-import --group=example_creature --update. The output should look something like this:

$ drush mi --group=example_creature --update
Processed 9 items (9 created, 0 updated, 0 failed, 0 ignored) - done with 'example_creature_base'
Processed 9 items (9 created, 0 updated, 0 failed, 0 ignored) - done with 'example_creature_i18n'

Note: Keep an eye out for Drupal core updates. If the drupal_migrate module adds support for entity translations, migrating entity translations might become much easier.

Next Steps

May 05 2017
May 05

If you have ever worked with sites that deal with events, you've probably been asked to create some type of calendar display. In this article, we'll discuss how to set up a basic events calendar using the Calendar (8.x-1.x-dev) for Drupal 8.

Configure the event content type

In our example, to handle events, we create a new content type called Event. You can put any content type in a calendar as long as it has a Date field. Though we might need date ranges to handle multi-day events, at the time of writing this article, there is no support for using the date range field with the calendar module, so we will use a simple date field for this example.

Event node formThe Event node form. Right now, events only have a title, description and a date field labelled Schedule.

Configuring the "events" view

With the node type in place, the next step will be to display the nodes using a view, using calendar display settings. To create a view with the calendar settings in place, you can go to Structure > Views > Add view from template page (admin/structure/views/template/list). Here, we choose the template which allows us to create a calendar for our date field.

Screenshot of the "add view from template" pageWe click the Add button corresponding to the date field (which we called Schedule).

The template provides you some options to configure certain aspects of the calendar to be generated, namely:

  • View name
  • Description
  • Base view path: The base path to use for the calendar pages and tabs. In our example, we choose events as the base so the calendar will generate paths like:
    • events/day: For a day-wise view
    • events/week: For a week-wise view
    • events/month: For a month-wise view
    • events/year: For a year-wise view

From the views configuration page, we can also configure the path for our calendar page(s) and create blocks with mini-calendars.

Calendar view generation optionsSome settings for the calendar to be generated.

Once done fine-tuning, we save the view and visit the relevant front-end page, which in this example is events/month. Here's how the calendar looks out of the box with the Bartik theme.

How the calendar looks in front-end

Conclusion

The calendar module will be a great contrib module while working with calendars. However, it may or may not serve your needs depending on your project requirements. Here are certain points (at the time of writing this article) which might affect the usability of this module:

  • No support for date range: Lack of support for date ranges makes it hard to work with multi-day event scenarios.
  • No support for start & end date: Though we can setup two separate date fields for start and end date, multi-day events are not visible as multi-column rows in the calendar.
Apr 20 2017
Apr 20

In my last post, I showed you how to migrate translated content from Drupal 6 to Drupal 8. But clients often don't start with their data in Drupal 6. Instead there's some other source of data that may include translations, like a CSV spreadsheet. In this article, I'll show you how to migrate multilingual content from such sources to Drupal 8.

This article would not have been possible without the help of my colleague Dave. Gracias Dave!

The problem

We have two CSV files containing some data about chemical elements in two languages. One file contains data in English and the other file, in Spanish. Our goal is to migrate these records into a Drupal 8 website, preserving the translations.

Before we start

  • Since this is an advanced migration topic, it is assumed you already know the basics of migration.
  • To execute the migrations in this example, you can download the migrate example i18n. The module should work without any trouble for a standard Drupal 8 install. See quick-start for more information.

Migrating JSON, XML and other formats

Though this example shows how to work with a CSV data source, one can easily work with other data sources. Here are some quick pointers:

  • Find and install the relevant migrate source module. If you do not have a standard source module available, you can:
    • try converting your data to a supported format first.
    • write your own migration source plugin, if you're feeling adventurous.
  • Modify the migration definitions to include custom parameters for the data source.
  • Some useful source formats are supported by these projects:

The module

To write the migrations, we create a module—in our case, it is named migrate_example_i18n. There's nothing special about the module declaration except for the dependencies:

How to migrate translations

Before we start writing migrations, it is important to mention how Drupal 8 translations work. In a nutshell:

  • First, we create content in its base language, say in English. For example, we could create a brand new node for the element Hydrogen, which might have a unique node ID 4.
  • Now that the base node is in place, we can translate the node, say to Spanish. Unlike some previous versions of Drupal, this won't become a new node with its own node ID. Instead, the translation is saved against the same node generated above, and so will have the same node ID—just a different language setting.

Hence, the migration definition for this example includes the following:

  • We migrate the base data in English using in example_element_en migration.
  • We migrate the Spanish translations using the example_element_es migration, and link each translation to the original English version.
  • We group the two migrations in the example_element migration group to keep things clean and organized.

Thus, we can execute the migrations of this example with the command drush migrate-import --group=example_element.

Warning

Note that this plan only works because every single node we are importing has at least an English translation! If some nodes only existed in Spanish, we would not be able to link them to the (non-existent) original English version. If you encounter data like this, you'll need to handle it in a different way.

Step 1: Element base migration (English)

To migrate the English translations, we define the example_element_en migration. Here is a quick look at some important parameters used in the migration definition.

Source

source:
  plugin: csv
  path: 'element.data.en.csv'
  header_row_count: 1
  keys:
    - Symbol
  fields:
    Name: 'Name'
    Symbol: 'Symbol'
    'Atomic Number': 'Atomic number'
    'Discovered By': 'Name of people who discovered the element'
  constants:
    lang_en: en
    node_element: 'element'
  • plugin: Since we want to import data from a CSV file, we need to use the csv plugin provided by the migrate_source_csv module.
  • path: Path to the CSV data source so that the source plugin can read the file. Our source files for this example actually live within our module, so we modify this path at runtime using hook_migration_plugins_alter() in migrate_example_i18n.module.
  • header_row_count: Number of initial rows in the CSV file which do not contain actual data. This helps ignore column headings.
  • keys: The column(s) in the CSV file which uniquely identify each record. In our example, the chemical symbol in the column Symbol is unique to each row, so we can use that as the key.
  • fields: A description for every column present in the CSV file. This is used for displaying source details in the UI.
  • constants: Some static values for use during the migration.

Destination

destination:
  plugin: 'entity:node'
  • plugin: Nothing fancy here. We aim to create node entities, so we set the plugin as entity:node.
  • translations: Since we are importing the content in base language, we do not specify the translations parameter. This will make Drupal create new nodes for every record.

Process

process:
  type: constants/node_element
  title: Name
  langcode: constants/lang_en
  field_element_symbol: Symbol
  field_element_discoverer:
    plugin: explode
    delimiter: ', '
    source: Discovered By

This is where we map the columns of the CSV file to properties of our target nodes. Here are some mappings which require a special mention and explication:

  • type: We hard-code the content type for the nodes we wish to create, to type element.
  • langcode: Since all source records are in English, we tell Drupal to save the destination nodes in English as well. We do this by explicitly specifying langcode as en.
  • field_element_discoverer: This field is a bit tricky. Looking at the source data, we realize that every element has one or more discoverers. Multiple discoverer names are separated by commas. Thus, we use plugin: explode and delimiter: ', ' to split multiple records into arrays. With the values split into arrays, Drupal understands and saves the data in this column as multiple values.

When we run this migration like drush migrate-import example_element_en, we import all the nodes in the base language (English).

Step 2: Element translation migration (Spanish)

With the base nodes in place, we define a migration similar to the previous one with the ID example_element_es.

source:
  plugin: csv
  path: 'element.data.es.csv'
  header_row_count: 1
  keys:
    - 'Simbolo'
  constants:
    lang_en: en
  # ...
destination:
  plugin: 'entity:node'
  translations: true
process:
  nid:
    plugin: migration
    source: Simbolo
    migration: example_element_en
  langcode: constants/lang_es
  content_translation_source: constants/lang_en
  # ...
migration_dependencies:
  required:
    - example_element_en

Let us look at some major differences between the example_element_es migration and the example_element_en migration:

  • source:
    • path: Since the Spanish node data is in another file, we change the path accordingly.
    • keys: The Spanish word for Symbol is Símbolo, and it is the column containing the unique ID of each record. Hence, we define it as the source data key. Unfortunately, Drupal migrate support keys with non-ASCII characters such as í (with its accent). So, as a workaround, I had to remove all such accented characters from the column headings and write the key parameter as Simbolo, without the special í.
    • fields: The field definitions had to be changed to match the Spanish column names used in the CSV.
  • destination:
    • translations: Since we want Drupal to create translations for English language nodes created during the example_element_en migration, we specify translations: true.
  • process:
    • nid: We use the plugin: migration to make Drupal lookup nodes which were created during the English element migration and use their ID as the nid. This results in the Spanish translations being attached to the original nodes created in English.
    • langcode: Since all records in element.data.es.csv are in Spanish, we hard-code the langcode to es for each record of this migration. This tells Drupal that these are Spanish translations.
    • content_translation_source: Each translation of a Drupal node comes from a previous translation—for example, you might take the Spanish translation, and translate it into French. In this case, we'd say that Spanish was the source language of the French translation. By adding this process step, we tell Drupal that all our Spanish translations are coming from English.
  • migration_dependencies: This ensures that the base data is migrated before the translations. So to run this migration, one must run the example_element_en migration first.

Voilà! Run the Spanish migration (drush migrate-import example_element_es) and you have the Spanish translations for the elements! We can run both the English and Spanish migration at once using the migration group we created. Here's how the output should look in the command-line:

$ drush migrate-import --group=example_element
Processed 111 items (111 created, 0 updated, 0 failed, 0 ignored) - done with 'example_element_en'
Processed 105 items (105 created, 0 updated, 0 failed, 0 ignored) - done with 'example_element_es'

If we had another file containing French translations, we would create another migration like we did for Spanish, and import the French data in a similar way. I couldn't find a CSV file with element data in French, so I didn't include it in this example—but go try it out on your own, and leave a comment to tell me how it went!

Next steps

Apr 12 2017
Apr 12

Now that Drupal 6 has reached end-of-life, many sites are moving to Drupal 8. If you had multilingual content in Drupal 6, this upgrade used to be very difficult—but since Drupal 8.2 there is support for migrating all your translations! In this article, we will discuss how to migrate translated content from Drupal 6 to Drupal 8.

This article would not have been possible without the help of my colleague Dave. Gracias Dave!

The problem

We have a Drupal 6 database containing story nodes about animal hybrids. Some nodes have translations in English, Spanish and French; some are untranslated; and others are language-neutral (non-translatable). Our goal is to migrate the D6 nodes into a D8 website, preserving the translations.

Before we start

The module

To write the migrations, we create a module - in our case, migrate_example_i18n. There's nothing special about the module declaration, except for the dependencies:

  • migrate_plus and migrate_tools provide various features for defining and executing migrations.
  • migrate_source_csv: Will be used for demonstrating migration of translated content from non-Drupal sources in an upcoming article.
  • migrate_drupal: This module provides tools for migrating data from older versions of Drupal. It comes with Drupal 8.x core. Since this migration uses a Drupal 6 site as a source for its data, we need the migrate_drupal module.

How do translations work?

Before jumping into writing these migrations, it is important to mention that Drupal 6 and Drupal 8 translations work very differently. Here's the difference in a nutshell:

  • Drupal 6: When we translate a node, a new node is created with a different ID. This translated node has a property named tnid, which stores the ID of the original node, linking the two nodes together. For language-neutral or untranslated content, the tnid is set to 0.
  • Drupal 8: When we translate a node, no new node is created! The translation is saved in the fields of the original node, but with a different language code.

To map between the D6 and D8 translation models, we'll use two migrations:

  • The example_hybrid_base migration will migrate the original content of each node, untranslated.
  • The example_hybrid_i18n migration will migrate in all the translations, and connect each one to the original node from example_hybrid_base..

We group the two migrations using the example_hybrid migration group to keep things clean and organized. Then we can execute both migrations with drush migrate-import --group=example_hybrid --update.

Step 1: Base migration

Let's start with the example_hybrid_base, to migrate all the base data (non-translations) in this migration. Described below are some noteworthy parameters:

Source

source:
  plugin: d6_node
  node_type: story
  key: drupal_6
  constants:
    node_article: article
    body_format: full_html
  • plugin: Since we want to import data from a Drupal installation, we need to set the source plugin to d6_node. The d6_node source plugin is introduced by the migrate_drupal, module and it helps us read nodes from a Drupal 6 database without having to write queries manually.
  • node_type: This tells the source plugin that we are interested in just one particular Drupal 6 node type, namely story.
  • key: Our Drupal 6 data doesn't come from our main Drupal 8 database—instead it comes from a secondary database connection. We choose a key to identify each such connection, and we need to tell the source which such key to use. The keys themselves are defined in the $databases variable in our settings.php or settings.local.php. See the example settings.local.php file to see how it's done.
  • constants: We define some hard-coded values under this parameter.
  • translations: Notice there is no translations parameter here. The default value (false) tells the source plugin that we're only interested in migrating non-translations, i.e. content in the base language and language-neutral content.

Destination

destination:
  plugin: 'entity:node'
  • plugin: Since we want to create node entities in Drupal 8, we specify this as entity:node. That's it.
  • translations: Again we do not define the translations parameter while migrating base data. Omitting the parameter tells the destination plugin that we are interested in creating fresh nodes for each record, not translations of existing nodes.

Process

process:
  type: constants/node_article
  langcode:
    plugin: default_value
    source: language
    default_value: und
  'body/value': body
  'body/format': constants/body_format
  title: title
  field_one_liner: field_one_liner
  sticky: sticky
  status: status
  promote: promote

This is where we map the old node properties to the new node properties. Most of the properties have been assigned as is, without alteration, however, some noteworthy properties have been discussed below:

  • nid: There is no nid parameter here, because we don't care what nid each new node has in Drupal 8. Drupal can just assign a new nid to each node in the normal way.
  • type: We specify that we want to create article nodes.
  • langcode: The langcode parameter was formerly language in Drupal 6, so we rename it here. Also, if a Drupal 6 node is language-neutral, it will have no value at all here. In that case,  we default to und.
  • body: We can assign this property directly to the body property. However, the Drupal 6 data is treated as plain text in Drupal 8 in that case. So migrating with body: body, the imported nodes in Drupal 8 would show visible HTML markup on your site. To resolve this, we explicitly assign the old body to body/value and specify that the text is in HTML by assigning full_html to body/format. That tells Drupal to treat the body as Full HTML.

This takes care of the base data. If you run this migration with drush migrate-import example_hybrid_base --update, all Drupal 6 nodes which are in base language or are language-neutral will be migrated into Drupal 8.

Step 2: Translation migration

We are halfway through now! All that's missing is migrating translations of the nodes we migrated above. To do this, we create another migration with the ID example_hybrid_i18n:

source:
  plugin: d6_node
  node_type: story
  translations: true
  # ...
destination:
  plugin: 'entity:node'
  translations: true
process:
  nid:
    plugin: migration
    source: tnid
    migration: example_hybrid_base
  langcode: language
  # ...
migration_dependencies:
  required:
    - example_hybrid_base

The migration definition remains mostly the same but has the following important differences as compared the base migration:

  • source:
    • translations: We set this to true to make the source plugin read only translations.
  • destination:
    • translations: We set this to true to make the destination plugin create translations for existing nodes instead of creating fresh new nodes for each source record.
  • process:
    • nid: In this case, we do care what the Drupal 8 nid is for each node. It has to match the nid for the untranslated version of this content, so that Drupal can add a translation to the correct node. This section uses the migration process plugin to figure out the right nid. It tells Drupal to check the previously-executed example_hybrid_base migration for a D6 node that has the same tnid as this D6 node. It will then then reuse the resulting nid here.
    • langcode: We define the language in which the translation should be created.
  • migration_dependencies: Since we cannot add translations to nodes that do not yet exist, we tell Drupal that this migration depends on the base migration example_hybrid_base. That way, the base migration will run before this migration.

That's it! We can run our translation migration with drush migrate-import example_hybrid_i18n --update and the translations will be imported into Drupal 8. Alternatively, we can use the migration group we defined to run both these migrations at once - the base migration will automatically be executed first and then the i18n migration. Here's how the output should look:

$ drush migrate-import --group=example_hybrid --update
Processed 8 items (8 created, 0 updated, 0 failed, 0 ignored) - done with 'example_hybrid_base'
Processed 9 items (9 created, 0 updated, 0 failed, 0 ignored) - done with 'example_hybrid_i18n'

You can check if everything went alright by clicking the Translate option for any translated node in Drupal 8. If everything went correctly, you should see that the node exists in the original language and has one or more translations.

Next steps

Jan 30 2017
Jan 30

Having completed the migration of academic program nodes as mentioned in Drupal 8 Migration: Migrating Basic Data (Part 1) and the migration of taxonomy terms as mentioned in Drupal 8 Migration: Migrating Taxonomy Term References (Part 2), this article would focus on the third requirement. We have images for each academic program. The base name of the images are mentioned in the CSV data-source for academic programs. To make things easy, we have only one image per program. This article assumes:

  • You have read the first part of this article series on migrating basic data.
  • You are able to write basic entity migrations.
  • You understand how to write multiple process plugins in migrations.

Though the problem might sound complex, the solution is as simple as following two steps.

Step 1: Importing images as "file" entities

First we need to create file entities for each file. This is because Drupal treats files as file entities which have their own ID. Then Drupal treats node-file associations as entity references, referring to the file entities with their IDs.

We create the file entities in the migrate_plus.migration.program_image.yml file, but this time, using some other process plugins. We re-use the program.data.csv file to import the files, so the source definition again uses the CSV plugin. We specify the key parameter in source as the column containing file names, ie, Image file. This way, we would be refer to these files in other migrations using their names, eg, engineering.png.

keys:
  - Image file

Apart from that, we use some constants to refer to source and destination paths for the images.

constants:
  file_source_uri: public://import/program
  file_dest_uri: 'public://program/image'

file_source_uri is used to refer to the path from which files are to be read during the import, and file_dest_uri is used to refer to the destination path where files should be copied to. The newly created file entities would refer to files stored in this directory. The public:// URI refers to the files directory inside the site in question. This is where all public files related to the site are stored.

file_source:
  -
    plugin: concat
    delimiter: /
    source:
      - constants/file_source_uri
      - Image file
  -
    plugin: urlencode
file_dest:
  -
    plugin: concat
    delimiter: /
    source:
      - constants/file_dest_uri
      - Image file
  -
    plugin: urlencode

Where do we use these constants? In the process element, we prepare two paths - the file source path (file_source) and the file destination path (file_dest).

  • file_source is obtained by concatenating the file_source_uri with the Image file column which stores the file's basename. Using delimiter: / we tell the migrate module to join the two strings with a / (slash) in between to ensure we have a valid file name. In short, we do file_source_uri . '/' . basename using the concat plugin.
  • file_dest, in a similar way, is file_dest_uri . '/' . basename. This is where we utilize the constants we defined in the source element.

Now, we use the file_source and file_dest paths generated above with plugin: file_copy. The file_copy plugin simply copies the files from the file_source path to the file_dest path. All the steps we did above were just for being able to refer to complete file source and destination paths during the process of copying files. The file gets copied and the uri property gets populated with the destination file path.

uri:
  plugin: file_copy
  source:
    - [email protected]_source'
    - [email protected]_dest'

We also use the existing file names as names of the newly created files. We do this using a direct assignment of the Image file column to the filename property as follows:

filename: Image file

Finally, since the destination of the migration is entity:file, the migrate module would use the filename and uri properties to generate a file entity, thereby generating a unique file ID.

Step 2: Associating files to academic programs

Once the heavy-lifting is done and we have our file entities, we need to put the files to use by associating them to academic programs. To do this, we add processing instructions for file_image in migration_plus.migration.program_data.yml. Just like we did for taxonomy terms, we tell the migrate module that the Image file column contains a unique file name, which refers to a file entity created during the program_image migration. We assign these file references to the field_image/target_id property as in Drupal 8, file associations are also treated as entity references.

'field_image/target_id':
  plugin: migration
  migration: program_image
  source: Image file

However, in the data-source for academic program data, we see a column named Image alt as well. Can we migrate these as well? We can! With an additional line of YAML.

'field_image/alt': Image alt

And we are done! If you update the configuration introduced by the c11n_migrate module and run the migration with the command drush config-import --partial --source=sites/sandbox.com/modules/c11n_migrate/config/install -y && drush migrate-import --group=c11n --update -y, you should see the following output::

$ drush mi --group=c11n --update -y
Processed 8 items (0 created, 8 updated, 0 failed, 0 ignored) - done with 'program_tags'
Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'program_image'
Processed 4 items (0 created, 4 updated, 0 failed, 0 ignored) - done with 'program_data'

To make sure that tag data is imported and available during the academic program migration, we specify the program_image migration in the migration_dependencies for the program_data migration. Now, when you run these migrations, the image files get associated to the academic program nodes.

Migrated image visible in UIMigrated image visible in UI.

Next steps

This is part three in a series of three articles on migrating data in Drupal 8. If you missed it, go back and read part one: Migration basics or part two: Migrating taxonomy terms and term references.

Jan 25 2017
Jan 25

If you've ever edited code for a website, you know that a seemingly simple change to just one page can unexpectedly cause other pages to change. Similarly, if you've used content management systems like Drupal, WordPress or Joomla, you've likely seen platform updates cause unexpected problems. Especially for a website with thousands of pages, it is nearly impossible to visit every page to ensure that nothing broke after an update.

SiteDiff to the rescue! SiteDiff is a command-line tool which helps you compare two versions of your site—for example, a known-good version versus an updated version. SiteDiff makes it easy to see how a website changes. It is useful for performing QA on re-deployments, site upgrades, and more!

Process

To demonstrate the usage of SiteDiff, I'll show you what I did when upgrading the website of Evolving Web from Drupal 8.1.x to Drupal 8.2.x. I wanted to check if the upgrade caused anything to break on our site, so I needed to classify all changes to the site's HTML as either harmless or undesirable. I used the following process:

  1. Install and setup sitediff.
  2. Run sitediff diff to find differences between the old version of the site and the updated version.
  3. If differences are found, SiteDiff produces a report listing them:
    • Pick a difference, eg: "every menu link has a new class in 8.2.x".
    • Determine whether the difference is harmless, or causes problems.
      • If the difference causes a problem, fix the site so that it functions the way it used to.
      • If the difference is harmless, configure SiteDiff so that it ignores the difference when you run it again.
    • Go back to step 2 to find remaining differences.
  4. If there are no differences remaining, that means I've successfully classified all changes. I'll know my updated site is in good shape!

Installing SiteDiff

To install SiteDiff, you can follow the SiteDiff installation instructions in the SiteDiff GitHub repository. It involves just three basic steps:

  • Installing Ruby, if it's not already installed. Refer to the SiteDiff's README for Ruby version requirements.
  • Installing other dependencies.
  • Installing SiteDiff itself, with sudo gem install sitediff.

Setup

To get started, we need to tell SiteDiff which website we wish to work with. We'll use the command sitediff init before-url after-url. To get two different versions of the site running, you can:

  • Use the live version of your site as the before version, and a development version of the modified site as the after version. The disadvantage of using the live site is that SiteDiff will crawl all your site's pages, which might lead to heavy load on your live site. This only has to happen once, though—SiteDiff can cache the results of its crawl.
  • Set up two development environments - one with a copy of the live version of the site and the other with the changed version of the site, which in my case was the Drupal 8.2 version of the site.

I used Docker to set up 2 containers - one running the original site at http://localhost:10380/ and the other running the upgraded Drupal 8.2 version of the site at http://localhost:10480/. Once these two sites were up and running, I ran sitediff init http://localhost:10380/ http://localhost:10480/. This crawled my whole site, and automatically created the configuration file sitediff/sitediff.yaml. We'll edit this sitediff.yaml file as we go along. You can refer to the example sitediff.yaml file for more information.

$ sitediff init http://localhost:10380 http://localhost:10480
[sitediff] Visited http://localhost:10480, cached
[sitediff] Visited http://localhost:10380, cached
[sitediff] Visited http://localhost:10480/about-evolvingweb, cached
[sitediff] Visited http://localhost:10480/blog, cached
[sitediff] Visited http://localhost:10480/feed, cached
...
[sitediff] Created /path/to/project/sitediff/sitediff.yaml
[sitediff] You can now run 'sitediff diff'

To see a list of all available parameters for this command, you can use sitediff help init.

Finding differences

This is where things get interesting. You can now issue the command sitediff diff -q --cached=all and you'll see a report of paths which have changed like this:

$ sitediff diff -q --cached=all
[sitediff] Reading config file: /path/to/project/sitediff/sitediff.yaml
[sitediff] Using sites from cache: after, before
[sitediff] FAILURE /
[sitediff] SUCCESS /about-evolvingweb
[sitediff] SUCCESS /blog
[sitediff] FAILURE /contact
[sitediff] SUCCESS /feed
...

Here, I used two optional parameters to modify SiteDiff's output:

  • -q: Without this optional parameter, SiteDiff shows a diff of each page as the output of the diff command. I chose to run SiteDiff in quiet mode as I wanted to view a detailed report of these changes using sitediff serve (explained in the next step).
  • --cached=all: With this command, I tell SiteDiff to use the cached version of both the before and after versions of the site to make the diff work faster. Without this parameter only the before site would be read from cache and the after site would be read at run-time.

To see a list of all available parameters for this command, you can use sitediff help diff. To see a detailed report of the exact changes which were found per-page, we can use the command sitediff serve, and SiteDiff renders a nice HTML page with a list of all changes found. This gives you the option to view the before and after versions side-by-side, or view the textual diff of any particular page.

SiteDiff HTML reportsitediff --serve shows a list of all pages with changed pages highlighted in red.

Handling acceptable differences

When you run sitediff diff, some pages are highlighted in red—these are the pages which have differences. Clicking on the DIFF link for a particular page, we see the full HTML source of the page, along with the changes SiteDiff found. Initially, all of your pages might be highlighted in red! But that's nothing to be worried about. These differences are often harmless or even expected.

In these cases, we just want to ignore the change, which we'll do using rules in our sitediff.yaml file.

Normalizing output: Sanitization rules

In my case, one site ran on http://localhost:10380 and the other on http://localhost:10480, which caused differences in CSS/JS link tags:

SiteDiff report showing domain differences.

We know these differences don't reflect a real change in the site's structure or markup, and can safely be ignored. To handle cases like this, we can define rules in the sitediff.yaml file. These rules are evaluated by SiteDiff during the diff operation, so that these unimportant differences do not appear in the report. To handle the difference above, we use the following sanitization rule in the configuration file:

sanitization:
- title: Strip domain names from absolute URLs
  pattern: http:\/\/localhost:10[34]80
  substitute: http://localhost

This rule asks SiteDiff to look for the regular expression defined in the pattern element and replace it with the text given in the substitute element. The title is for us to know as to what the rule is intended to do. We can add such a rule under the global sanitization key to apply to both sites we're comparing; or put a sanitization key in the before or after sections of the file, to limit its application to the before or after version of the site respectively.

Some other cases where we can use sanitization rules are:

  • Removing randomized content added to parts of the page. For example, Drupal forms are expected to have random form IDs.
  • Handling changes that we like! For example, an update to a Drupal module might correctly add type="email" to email fields. After checking that this looks and behaves ok, we approve the change and write a rule to exclude it from the report.

Be careful writing your regex patterns. Patterns including .+ or .* are greedy, and might eat up more characters than you intend. It's better to use more restricted patterns like [^"]+.

Normalizing output: DOM transformation rules

Similarly, there are cases when we might wish to remove certain DOM elements or unwrap them to normalize certain differences between the two versions of the site. For this we can use DOM transformations, such as:

  • unwrap: This transformation removes a given HTML element, but keeps its contents without the wrapping element. For example, if one version of the site wraps articles inside an article element, while the other version does not have that additional article element, we can use the unwrap transformation to remove the article tag while keeping its contents.
  • remove: As the name suggests, we can use this to remove a given HTML element. For example, if we have a block containing random articles or the current time, we can remove the block to normalize the two versions of the site.

You can see a full list in the SiteDiff docs. Here's what the syntax for a transformation in your sitediff.yaml looks like:

dom_transform:
- type: unwrap
  selector: article

It's usually best to use DOM transformation rules instead of regex sanitization rules when possible, since they're harder to get wrong, and easier to read and understand later.

Handling undesirable differences

Not all differences are trivial or harmless, like those above! In my case, I found an unexpected change - a div element acting as a wrapper for a group of checkboxes was changed to a fieldset element due to an update in a module.

SiteDiff report showing a div replaced by a fieldset

The HTML still looks reasonable—but since the change was unexpected, I needed to verify that it hadn't caused any problems. I opened the changed page in my browser and found that it looked wrong! Due to different HTML structure, our existing CSS rules turned a label from black to red!

Form item label looks different after update.The form item's label is all in red after the update.

SiteDiff helped me discover this bug in my site—thanks SiteDiff! I probably wouldn't have found this if I had just updated my site and manually checked a few pages.

For each confirmed bug, we have to fix the problem in the site's code or CSS. Sometimes we'll reproduce the exact HTML the site used to have. Other times, we'll adapt to the HTML change, and then add a sanitization rule or DOM transformation to ignore the change on future runs, since we've taken care of it.

Moving along

Now that you've classified a couple of differences, you might think you're done. But probably not yet! It often takes a few passes to classify all the differences, so you should now run sitediff diff again. Here's a tip to make things go quicker: As you're testing new rules or changes, don't run sitediff diff on your entire site. Instead, you can make SiteDiff look at only the paths you know are relevant. You can do this with the --paths parameter, for example, sitediff diff --paths /path/one /path/two.

Eventually, you'll have handled every last difference, and SiteDiff's report will contain nothing in red. Congratulations! Now you can deploy the updated site knowing you haven't broken anything.

Next steps

  • Read SiteDiff documentation to learn more about it.
  • See sitediff.yaml.example file for example rules.
  • Try out SiteDiff the next time you make updates and/or upgrades to a website.
  • Read YAML documentation to understand the sitediff.yaml file better.
Jan 23 2017
Jan 23

Having completed the migration of academic program nodes as mentioned in Drupal 8 Migration: Migrating Basic Data (Part 1), this article will focus on the second requirement - importing tags (taxonomy terms) related to the academic programs. This article assumes:

  • You have read the first part of this article series on migrating basic data.
  • You are able to write basic entity migrations.
  • You are aware of taxonomy vocabularies / terms and their usage.

Importing tags as taxonomy terms

As a general rule for migrating relations between two entities, first we need to write a migration for the target entities. In this case, since academic programs has a field named field_tags and we wish to store the IDs of certain taxonomy terms of the vocabulary tags in the program nodes, we need to write a migration to import tags first. Thus, while running the migrations for academic programs, the tags would already exist on the Drupal 8 site and migrate API would be able to refer to these tags using their IDs.

For achieving this, we write a simple migration for the tags as in the migrate_plus.migration.program_tags.yml file. One noteworthy thing about this migration is that we use the tags themselves as the unique ID for the tags. This is because:

  • The tags data-source, program.tags.csv, does not provide any unique key for the tags.
  • The academic programs data-source, program.data.csv, refers to the tags using the tag text (instead of unique IDs).

Once the tag data is imported, all we have to do is add some simple lines of YAML in the migration definition for academic programs to tell Drupal how to migrate the field_tags property of academic programs. As we did for program_level in the previous article, we will be specifying multiple plugins for this property:

field_tags:
  -
    plugin: explode
    delimiter: ', '
    source: Tags
  -
    plugin: migration
    migration: program_tags

Here is an explanation of what we are actually doing with the multiple plugins:

  • explode: Taking a look at the data-source, we notice that academic programs have multiple tags separated by commas. So, as a first step, we use the explode plugin which would split / explode the tags by the delimiter , (comma), thereby creating an array of tags. We do this using plugin: explode and delimiter: ', '. We have a space after the comma because the data source has a space after it's commas and adding that space in the delimiter, we would be able to exclude the spaces from the imported tags.
  • migration: Now that we have an array of tags, each tag identifying itself using it's unique tag text, we tell the migrate module that these tags are the same ones we imported in migrate_plus.migration.program_tags.yml and that the tags generated during that migration are to be used here in order to associate them to the academic programs. We do this using plugin: migration and migration: program_tags. You can read more about the migration plugin on Drupal.org.
migration_dependencies:
  optional:
    - program_tags
#     - program_image

To make sure that tag data is imported and available during the academic program migration, we specify the program_tags migration in the migration_dependencies for the program_data migration. Now, when you re-run these migrations, the taxonomy terms get associated to the academic program nodes. At this stage, you can keep the program_image dependency still commented, because we haven't written it yet.

Migrated tags visible in UIMigrated tags visible in UI.

As simple as it may sound, this is all that is needed to associate the tags to the academic programs! All that is left is re-installing the c11n_migrate module and executing the migrations using the following drush command: drush mi --group=c11n --update. You should see the following output:

$ drush mi --group=c11n --update
Processed 8 items (8 created, 0 updated, 0 failed, 0 ignored) - done with 'program_tags'
Processed 4 items (0 created, 4 updated, 0 failed, 0 ignored) - done with 'program_data'

Because of the migration_dependencies we specified, the program_tags migration was run before the program_data migration.

Importing terms without a separate data-source

For the sake of demonstration, I also included an alternative approach for the migration of the field_program_type property. For program type, I used the entity_generate plugin which comes with the migrate_plus module. This is how the plugin works:

  • Looks up for an entity of a particular type (in this case, taxonomy_term) and bundle (in this case, program_types) based on a particular property (in this case, name).
  • If no matching entity is found, an entity is created on the fly.
  • The ID of the existing / created entity is returned for use in the migration.
field_program_type:
  plugin: entity_generate
  source: Type
  entity_type: taxonomy_term
  bundle_key: vid
  bundle: program_types
  value_key: name

So, in the process instructions for field_program_type, I use plugin: entity_generate. So, during migration, for every program type, the entity_generate plugin is called and a particular taxonomy term is associated to the academic programs. The disadvantage of using the entity_generate method is when we rollback the migration, these taxonomy terms created during the migration would not be deleted.

Next steps

This is part two in a series of three articles on migrating data in Drupal 8. Check this space next week for part three: Migrating files and images. If you missed it, go back and read part one: Migration basics.

Jan 16 2017
Jan 16

Usually when a huge site makes the decision to migrate to Drupal, one of the biggest concerns of the site owners is migrating the old site's data into the new Drupal site. The old site might or might not be a Drupal site, but given that the new site is on Drupal, we can make use of the cool migrate module to import data from a variety of data sources including but not limited to XML, JSON, CSV and SQL databases.

This article revolves around an example module named c11n_migrate showing how to go about importing basic data from a CSV data source, though things would work pretty similarly for other types of data sources.

The problem

As per project requirements, we wish to import certain data for an educational and cultural institution.

  • Academic programs: We have a CSV file containing details related to academic programs. We are required to create nodes of type program with the data. This is what we discuss in this article.
  • Tags: We have a CSV file containing details related to tags for these academic programs. We are required to import these as terms of the vocabulary named tags. This will be discussed in a future article.
  • Images: We have images for each academic program. The base name of the images are mentioned in the CSV file for academic programs. To make things easy, we have only one image per program. This will be discussed in a future article.

Executing migrations

Before we start with actual migrations, a few things to note about running your migrations:

  • Though the basic migration framework is a part of the D8 core as the migrate module, to be able to execute migrations, you must install the migrate_tools module. You can use the command drush migrate-import --all to execute all migrations. In this tutorial, we also install some other modules like migrate_plus, migrate_source_csv.
  • Migration definitions in Drupal 8 are in YAML files, which is great. But the fact that they are located in the config/install directory implies that these YAML files are imported when the module is installed. Hence, any subsequent changes to the YAML files would not be detected until the module is re-installed. We solve this problem by re-importing the relevant configurations manually like drush config-import --partial --source=path/to/module/config/install.
  • While writing a migration, we usually update the migration over and over and re-run them to see how things go. To do this quickly, you can re-import config for the module containing your custom migrations (in this case the c11n_migrate module) and execute the relevant migrations in a single command like drush config-import --partial --source=sites/sandbox.com/modules/c11n_migrate/config/install -y && drush migrate-import --group=c11n --update -y.
  • To execute the migrations in this example, you can download the c11n_migrate module sources and rename the downloaded directory to c11n_migrate. The module should work without any trouble for a standard Drupal 8 install.

The module

Though a matter of personal preference, I usually prefer to name project-specific custom modules with a prefix of c11n_ (being the numeronym for the word customization). That way, I have a naming convention for custom modules and I can copy any custom module to another site without worrying about having to change prefixes. Very small customizations, can be put into a general module named c11n_base.

To continue, there is nothing fancy about the module definition as such. The c11n_migrate.info.yml file includes basic project definition with certain dependencies on other modules. Though the migrate module is in Drupal 8 core, we need most of these dependencies to enable / enhance migrations on the site:

  • migrate: Without the migrate module, we cannot migrate!
  • migrate_plus: Improves the core migrate module by adding certain functionality like migration groups and usage of YML files to define migrations. Apart from that, this module includes an example module which I referred to on various occasions while writing my example module.
  • migrate_tools: General-purpose drush commands and basic UI for managing migrations.
  • migrate_source_csv: The core migrate module provides a basic framework for migrations, which does not include support for specific data sources. This module makes the migrate module work with CSV data sources.

Apart from that, we have a c11n_migrate.install file to re-position the migration source files in the site's public:// directory. Most of the migration magic takes place in config/install/migrate_plus.* files.

Migration group

Like we used to implement hook_migrate_api() in Drupal 7 to declare the API version, migration groups, individual migrations and more, in Drupal 8, we do something similar. Instead of implementing a hook, we create a migration group declaration inside the config/install directory of our module. The file must be named something like migrate_plus.migration_group.NAME.yml where NAME is the machine name for the migration group, in this case, migrate_plus.migration_group.c11n.yml.

id: c11n
label: Custom migrations
description: Custom data migrations.
source_type: CSV files
dependencies:
  enforced:
    module:
      - c11n_migrate

We create this group to act as a container for all related migrations. As we see in the extract above, the migration group definition defines the following:

  • id: A unique ID for the migration. This is usually the NAME part of the migration group declaration file name as discussed above.
  • label: A human-friendly name of the migration group as it would appear in the UI.
  • description: A brief description about the migration group.
  • source_type: This would appear in the UI to provide a general hint as to where the data for this migration comes from.
  • dependencies: Though this might sound a bit strange to Drupal 7 users, this segment is used to define modules on which the migration depends. When one of these required modules are missing / removed, the migration group is also automatically removed.

Once done, if you install/re-install the c11n_migrate module and visit the admin/structure/migrate page, you should see the migration group we created above!

Migration group visible in UI

Migration definition: Metadata

Now that we have a module to put our migration scripts in and a migration group for grouping them together, it's time we write a basic migration! To get started, we import basic data about academic programs, ignoring complex stuff such as tags, files, etc. In Drupal 7 we used to do this in a file containing a PHP class which used to extend the Migration class provided by the migrate module. In Drupal 8, like many other things, we do this in a YML file, in this case, the migrate_plus.migration.program_data.yml file.

id: program_data
label: Academic programs and associated data.
migration_group: c11n
migration_tags:
  - academic program
  - node
# migration_dependencies:
#   optional:
#     - program_tags
#     - program_image
dependencies:
  enforced:
    module:
      - c11n_migrate

In the above extract, we declare the following metadata about the migration:

  • id: A unique identifier for the migration. In this example, I allocated the ID program_data, hence, the migration declaration file has been named migrate_plus.migration.program_data.yml. We can specifically execute this with the command drush migrate-import ID.
  • label: A human-friendly name of the migration as it would appear in the UI.
  • migration_group: This puts the migration into the migration group c11n we created above. We can execute all migrations in a given group with the command drush migrate-import --group=GROUP.
  • migration_tags: Here we provide multiple tags for the migration and just like groups, we can execute all migrations with the same tag using the command drush migrate-import --tag=TAG
  • dependencies: Just like in case of migration groups, this segment is used to define modules on which the migration depends. When one of these required modules are missing / removed, the migration is automatically removed.
  • migration_dependencies: This element is used to mention IDs of other migrations which must be run before this migration. For example, if we are importing articles and their authors, we need to import author data first so that we can refer to the author's ID while importing the articles. Note that we can leave this undefined / commented for now as we do not have any other migrations defined. I defined this section only after I finished writing the migrations for tags, files, etc.

Migration definition: Source

source:
  plugin: csv
  path: 'public://import/program/program.data.csv'
  header_row_count: 1
  keys:
    - ID
  fields:
    ID: Unique identifier for the program as in the data source.
    Title: Name of the program.
    Body: A description for the program.
    Level: Whether the program is for undergraduates or graduates.
    Type: Whether it is a full-time or a part-time program.
    Image file: Name of the image file associated with the program.
    Image alt: Alternate text for the image for accessibilty.
    Tags: Comma-separated strings to use as tags.
    Fees: We will ignore this field as per requirement.

Once done with the meta-data, we define the source of the migration data with the source element in the YAML.

  • plugin: The plugin responsible for reading the source data. In our case we use the migrate_source_csv module which provides the source plugin csv. There are other modules available for other data sources like JSON, XML, etc.
  • path: Path to the data source file - in this case, the program.data.csv file.
  • header_row_count: This is a plugin-specific parameter which allows us to skip a number of rows from the top of the CSV. I found this parameter reading the plugin class file modules/contrib/migrate_source_csv/src/Plugin/migrate/source/CSV.php, but it is also mentioned in the docs for the migrate_source_csv module.
  • keys: This parameter defines a number of columns in the source data which form a unique key in the source data. Luckily in our case, the program.data.csv provides a unique ID column so things get easy for us in this migration. This unique key will be used by the migrate module to relate records from the source with the records created in our Drupal site. With this relation, the migrate module can interpret changes in the source data and update the relevant data on the site. To execute an update, we use the parameter --update with our drush migrate-import command, for example drush migrate-import --all --update.
  • fields: This parameter provides a description for the various columns available in the CSV data source. These descriptions just appear in the UI and explain purpose behind each column of the CSV.
  • constants: We define certain values which we would be hard-coding into certain properties which do not have relevant columns in the data-source.

Once done, the effect of the source parameter should be visible on the admin/structure/migrate/manage/c11n/migrations/program_data/source page as follows:

Migration source visible in UI

Migration definition: Destination

destination:
  plugin: 'entity:node'
  default_bundle: program

In comparison to the source definition, the destination definition is much simpler. Here, we need to tell the migrate module how we want it to use the source data. We do this by specifying the following parameters:

  • plugin: Just like source data is handled by separate plugins, we have destination plugins to handle the output of the migrations. In this case, we want Drupal to create node entities with the academic program data, so we use the entity:node plugin.
  • default_bundle: Here, we define the type of nodes we wish to obtain using the migration. Though we can override the bundle for individual item, this parameter provides a default bundle for entities created by this migration. We will be creating only program nodes, so we mention that here.

Fields definitions for program nodes

Provided above is a quick look at the program node fields.

Migration definition: Mapping and processing

If you ever wrote a migration in an earlier version of Drupal, you might already know that migrations are usually not as simple as copying data from one column of a CSV file to a given property of the relevant entity. We need to process certain columns and eliminate certain columns and much more. In Drupal 8, we define these processes using a process element in the migration declaration. This is where we put our YAML skills to real use.

process:
  title: Title
  sticky: constants/bool_0
  promote: constants/bool_1
  uid: constants/uid_root
  'body/value': Body
  'body/format': constants/restricted_html
  field_program_level:
    -
      plugin: callback
      callable: strtolower
      source: Level
    -
      plugin: default_value
      default_value: graduate
    -
      plugin: static_map
      map:
        graduate: gr
        undergraduate: ug

Here is a quick look at the parameters we just defined:

  • title: An easy property to start with, we just assign the Title column of the CSV as the title property of the node. Though we do not explicitly mention any plugin for this, in the background, Drupal uses the get plugin to handle this property.
  • sticky: Though Drupal can apply the default value for this property if we skip it (like we have skipped the status property), I wanted to demonstrate how to specify a hard-coded value for a property. We use the constant constants/bool_0 to make the imported nodes non-sticky with sticky = 0.
  • promote: Similarly, we ensure that the imported nodes are promoted to the front page by assigning constants/bool_1 for the promote property.
  • uid: Similarly, we specify default owner for the article as the administrative user with uid = 1.
  • body: The body for a node is a filtered long text field and has various sub-properties we can set. So, we copy the Body column from the CSV file to the body/value property (instead of assigning it to just body). In the next line, we specify the body/format property as restricted_html. Similarly, one can also add a custom summary for the nodes using the body/summary property. However, we should keep in mind that while defining these sub-properties, we need to wrap the property name in quotes because we have a / in the property name.
  • field_program_level: With this property I intend to demonstrate a number of things - multiple plugins, the static_map plugin, the callback plugin and the default_value plugin.
    • Here, we have the plugin specifications as usual, but we have small dashes with which we are actually defining an array of plugins or a plugin pipeline. The plugins would be called one by one to transform the source value to a destination value. We specify a source parameter only for the first plugin. For the following plugins, the output of the previous plugin would be used as the input.
    • The source data uses the values graduate/undergraduate with variations in case as Undergraduate or UnderGraduate. With the first plugin, we call the function strtolower (with callback: strtolower) on the Level property (with source: Level) to standardize the source values. After this plugin is done, all Level values would be in lower-case.
    • Now that the values are in lower-case, we face another problem. The Math & Economics row, no Level value is specified. If no value exists for this property, the row would be ignored during migration. As per client's instructions, we can use the default value graduate when a Level is not specified. So, we use the default_value plugin (with plugin: default_value) and assign the value graduate (using default_value: graduate) for rows which do not have a Level. Once this plugin is done, all rows would technically have a value for Level.
    • We notice that the source has the values graduate/undergraduate, whereas the destination field only accepts gr/ug. In Drupal 7, we would have written a few lines of code in a ProgramDataMigration::prepareRow() method, but in Drupal 8, we just write some more YAML. To tackle this, we pass the value through a static_map (with plugin: static_map) and define a map of new values which should be used instead of old values (with the map element). And we are done! Values would automatically be translated to gr or ug and assigned to our program nodes.

With the parameters above, we can write basic migrations with basic data-manipulation. If you wish to see another basic migration, you can take a look at migrate_plus.migration.program_tags.yml. Here is how the migration summary looks once the migration has been executed.

$ drush migrate-import program_data --update
Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'program_data'

Once done correctly, the nodes created during the migration should also appear in the content administration page just as expected.

Nodes created during migration

Next steps

This is part one in a series of three articles on migrating data in Drupal 8. Check this space next week for part two: Migrating taxonomy terms and term references, or the week after for part three: Migrating files and images.

Nov 02 2016
Nov 02

When we build Drupal websites, we often have to create a custom data structure which can hold multiple values, each value having its own set of fields. For example, a phone number might have a country code, area code, the actual number, and then a type (work, home, cell). And you might have to create a field that accepts multiple phone numbers.

You might be tempted to build out these types of fields using a module like field collections or paragraphs, but if you're a Drupal developer, you can easily create custom reusable fields to accomplish the same thing. The Drupal 8 Field API gives us a pretty straight-forward way to do this in a custom module, and the result is a field that we can easily re-use across content types.

In this blog post, I'll show you how to create a custom compound field for Drupal 8, using burrito ingredients as the example. Creating our field will involve three main steps:

  • Defining a custom field type.
  • Defining a custom field widget.
  • Defining a custom field formatter.

As I do for any new Drupal adventure, I started with the Drupal 8 examples module - in this case, the field_example module. I also referred to the poutine_maker project for Drupal 7 and tried to create something similar. Reading this post should give you a basic idea as to how to do the three tasks listed above and adapt the example provided to build your own shiny new custom field!

The Challenge

Having spent 6 months in Bogotá DC, the capital of Colombia, I came across a yummy food item - the Burrito (originally a Mexican recipe) - a big flour tortilla, filled with a variety of vegetables, meat (optional), sauces and cheese, wrapped in a foil - worth every dollar (or should I say peso) you pay for it! Makes you wanna exclaim "¡Es muy rico!"

So I thought, what if we had a site where visitors could write articles and with each article, they could include the set of ingredients they prefer in their burritos? Every user would be able to create a piece of content (node) specifying multiple burrito recipes, each recipe having:

  • A string name for the burrito;
  • Boolean flags for every ingredient / topping;

Thus, as per requirements, I decided to make the burrito_maker module.

Setting up the Module

There's nothing fancy about the burrito_maker module, except for the fact that it defines a custom field. It includes the following important files:

  • A standard info.yml file (successor of the info file) with inc files for utility functions.
  • A Field Type plugin implementation.
  • A Field Widget plugin implementation.
  • A Field Formatter plugin implementation.

Unlike Drupal 7, field types, field widgets and field formatters are defined and detected using Plugin definitions with annotations like @FieldType, @FieldWidget and @FieldFormatter respectively. Hence, we have no hook_field_* implementations in the .module file. Instead, there are separate files to define our plugins.

All the plugin declarations in the burrito_maker include namespace declarations and use statements at the beginning of the class files. These are used for organizing code and for autoloading classes in Drupal 8.

Step 1: Define field type with @FieldType annotation

The first thing to do is create a FieldType plugin implementation. This is done using a class which represents a single item of the Burrito field type. By convention, it has been named the BurritoItem. Notice the @FieldType part of the class documentation - this replaces the Drupal 7 hook_field_info() implementation.

/**
 * Contains field type "burrito_maker_burrito".
 *
 * @FieldType(
 *   id = "burrito_maker_burrito",
 *   label = @Translation("Burrito"),
 *   description = @Translation("Custom burrito field."),
 *   category = @Translation("Food"),
 *   default_widget = "burrito_default",
 *   default_formatter = "burrito_default",
 * )
 */
class BurritoItem extends FieldItemBase implements FieldItemInterface {
  public static function schema(FieldStorageDefinitionInterface $field_definition) {...}
  public static function propertyDefinitions(FieldStorageDefinitionInterface $field_definition) {...}
  public function isEmpty() {...}
}

The field type machine-name is burrito_maker_burrito. At times, it may seem tempting to name your field only burrito instead of burrito_maker_burrito. However, it is good practice to define stuff in the namespace of your own module. Here's a quick look at what we defined:

  • id: The unique machine-name.
  • label: The human-readable name.
  • description: A brief description.
  • category: The category under which the field appears in the Field UI. Example: In the Add a field page in the Field type selection widget.
  • default_widget: The default input widget to use for the field type. I added this line after I created the burrito_default field widget plugin.
  • default_formatter: The default output formatter to use for the field type. I added this line after I created the burrito_default field formatter plugin.

The class also contains some required methods (which previously used to be hook_field_* functions) given below:

BurritoItem::schema() - Tell Drupal how to store your values

Here, we define storage columns to be created in the database table for storing each value of the given field type. The method returns an array using the same format as hook_schema() implementations.

BurritoItem::propertyDefinitions() - Provide meta data for field properties

Here we define additional infomation about sub-properties of the field.

public static function propertyDefinitions(FieldStorageDefinitionInterface $field_definition) {

  module_load_include('inc', 'burrito_maker');

  $properties['name'] = DataDefinition::create('string')
    ->setLabel(t('Name'))
    ->setRequired(FALSE);

  $topping_coll = burrito_maker_get_toppings();
  foreach ($topping_coll as $topping_key => $topping_name) {
    $properties[$topping_key] = DataDefinition::create('boolean')
      ->setLabel($topping_name);
  }

  return $properties;

}

BurritoItem::isEmpty() - Tell Drupal when a value should be considered empty

After a user enters values into your custom field and hits submit, Drupal checks to see if the fields are empty. If they are empty, it doesn't try to validate or save anything. Hence, the sole purpose of this isEmpty() method is to help Drupal understand when a field item should be considered empty (and hence ignored). If this method returns FALSE, Drupal knows that the field has some value which needs to be validated and saved.

In the example below, I check if the user has entered anything in the text boxes or checked any of the checkboxes - if yes, then we treat the BurritoItem as non-empty by returning FALSE.

public function isEmpty() {

  $item = $this->getValue();

  $has_stuff = FALSE;

  // See if any of the topping checkboxes have been checked off.
  foreach (burrito_maker_get_toppings() as $topping_key => $topping_name) {
    if (isset($item[$topping_key]) && $item[$topping_key] == 1) {
      $has_stuff = TRUE;
      break;
    }
  }

  // Has the user entered a name for the Burrito?
  if (isset($item['name']) && !empty($item['name'])) {
    $has_stuff = TRUE;
  }

  return !$has_stuff;

}

Note: If you ever find that your values are not being saved to the database, there is high probability that this function is telling Drupal that the values are empty.

It's alive! My field is in the list!

After you define the field type, you can enable the Burrito Maker module and you'll be able to see your field type on the Add field screen. You can try it out from Admin > Structure > Content types > Basic page > Manage fields for example.

Custom field type visible in field type selectionCustom field type visible in field type selection.

Note: If you try to create a field instance at this moment, you'll probably receive an error message (even though your field instance might get created). This is because we have not defined the field widget and field formatters yet. Though we could have defined all the three classes first and then enabled the module, I personally like to see visible evidence of progress (like clients) when I code. Seeing my custom field in the list of field types gets me motivated me to write the rest of the classes.

Step 2: Define field widget with @FieldWidget annotation

So, the database tables are ready and your field type appears in the UI. But how would the user enter data for their custom burritos? Drupal doesn't know anything about them burritos! So, we need to implement a FieldWidget plugin to tell Drupal exactly how it should build forms for accepting burrito data.

Now, I add the default_widget definition in the FieldType annotation data - the machine name of our field widget being burrito_default. To define the structure of the widget, we create the following class with the @FieldWidget annotation:

/**
 * Contains field widget "burrito_default".
 *
 * @FieldWidget(
 *   id = "burrito_default",
 *   label = @Translation("Burrito default"),
 *   field_types = {
 *     "burrito_maker_burrito",
 *   }
 * )
 */
class BurritoDefaultWidget extends WidgetBase implements WidgetInterface {
  public function formElement(FieldItemListInterface $items, $delta, array $element, array &$form, FormStateInterface $form_state) {...}
}

The data we specify for the field widget is as follows:

  • id: Unique identifier for the widget.
  • label: Human-readable name.
  • field_types: An array of field types (referenced by field type id) supported by the widget.

This class must implement the required method formElement(). However, in our example, we have two other utility methods as well.

BurritoDefaultWidget::formElement()

This method is pretty straight-forward and is responsible for defining form items which map to various schema columns provided by the field type. In our case, this method looks a bit complicated (though it is quite simple) because we have various checkboxes for the toppings/ingredients of the burrito, placed inside two fieldsets - Meat and Toppings - the meat fieldset being hidden if the field settings do not allow meat.

Here is a screenshot of the type of form we intend to build:

Burrito field widget visible in form.

You'll notice that in the form above, we have the Add another item button. This is because the Number of values option in the field settings is set to Unlimited. You can add as many burrito definitions as you want, just like any other multi-value field. Below are some details about how the form elements are defined. For more background information about defining form elements, you can refer to the Drupal 8 Form API.

First, I prepare the field item for which we are building the form element. $items contains all field items added to the entity - hence, it is a FieldItemListInterface. We are interested in the $delta index of the $items array - the field item for which we are building the form.

// $item is where the current saved values are stored.
$item =& $items[$delta];

Since our form widget has many fields, I thought it would be better to wrap it in a fieldset to make the fields appear grouped in the UI.

// In this example, $element is a fieldset, but it could be any element
// type (textfield, checkbox, etc.)
$element += array(
  '#type' => 'fieldset',
);

Inside this fieldset, we define a self-explanatory textfield where the user would type the Name of the burrito.

$element['name'] = array(
  '#title' => t('Name'),
  '#type' => 'textfield',
  // Use #default_value to pre-populate the element
  // with the current saved value.
  '#default_value' => isset($item->name) ? $item->name : '',
);

Now, we show a meat fieldset only if the field settings do not disallow meat.

// Show meat options only if allowed by field settings.
if ($this->getFieldSetting('allow_meat')) {

  // Have a separate fieldset for meat.
  $element['meat'] = array(
    '#title' => t('Meat'),
    '#type' => 'fieldset',
    '#process' => array(__CLASS__ . '::processToppingsFieldset'),
  );

  // Create a checkbox item for each meat on the menu.
  foreach (burrito_maker_get_toppings('meat') as $topping_key => $topping_name) {
    $element['meat'][$topping_key] = array(
      '#title' => t($topping_name),
      '#type' => 'checkbox',
      '#default_value' => isset($item->$topping_key) ? $item->$topping_key : '',
    );
  }

}

We iterate over all possible meat toppings in a foreach loop and define a checkbox for each ingredient. We do the same for regular vegetarian toppings as well - the only difference is we put the veggie toppings in a separate fieldset. Once done, we simply return the form element we built.

Note: You might notice a #process attribute defined in the toppings and meat fieldsets - we use the BurritoDefaultWidget::processToppingsFieldset() method to flatten the fieldset values to the format expected by the Field API. Also, you might notice the use of settings in the code - these settings will be described in the Plugin settings section in this article.

Step 3: Define field formatter with @FieldFormatter annotation

Now for presentation of the burrito data, we define the BurritoDefaultFormatter class. Its main purpose is to take a list of BurritoItem objects and display them as per the field's display settings. This is done by the only required method in the formatter plugin implementation, BurritoFormatter::viewElements(). After defining the formatter, I added the default_formatter declaration in the field type implementation.

/**
 * Contains field widget "burrito_default".
 *
 * @FieldFormatter(
 *   id = "burrito_default",
 *   label = @Translation("Burrito default"),
 *   field_types = {
 *     "burrito_maker_burrito",
 *   }
 * )
 */
class BurritoDefaultFormatter extends FormatterBase {
  public function viewElements(FieldItemListInterface $items, $langcode) {...}
}

Note the annotation @FieldFormatter which defines the following:

  • id: Unique ID of the field formatter.
  • label: Human readable name for the formatter.
  • field_types: The field types supported by this formatter (referenced by field type id).

The viewElements() method receives a list of BurritoItem objects and prepares a huge render array containing render data for all of the field values.

Note: You might notice the use of settings in the code. These settings are described in the Plugin settings section of this article.

It's alive! I can view burrito data!

Formatted output for a burrito field.Formatted output for a burrito field.

Plugin settings - Could it be any better?

More often than not, we have settings associated to various plugins. For example, the burrito formatter could provide the administrator an option to choose how the field items are displayed - whether as comma-separated values or as an unordered list. Here's how this setting would look in the UI:

Field formatter settings form in action.Field formatter settings form in action.

Though it might sound difficult, it is actually very easy to do this in Drupal 8. All you have to define is the two methods defaultSettings() and settingsForm() to enable settings. The good news is that this works the same way for pretty much every plugin implementing the PluginSettingsInterface:

PluginClass::defaultSettings()

This method returns an associative array of all options provided by the plugin. Here is the BurritoDefaultFormatter::defaultSettings() method:

public static function defaultSettings() {
  return array(
    'toppings' => 'csv',
  ) + parent::defaultSettings();
}

Note: For the plugin settings to auto-save, a default value for every parameter must be declared.

PluginClass::settingsForm(array $form, FormStateInterface $form_state)

This method simply returns an array of form elements for accepting the settings from the UI. Here is the BurritoDefaultFormatter::settingsForm(array $form, FormStateInterface $form_state) method:

public function settingsForm(array $form, FormStateInterface $form_state) {

  $output['toppings'] = array(
    '#title' => t('Toppings'),
    '#type' => 'select',
    '#options' => array(
      'csv' => t('Comma separated values'),
      'list' => t('Unordered list'),
    ),
    '#default_value' => $this->getSetting('toppings'),
  );

  return $output;

}

PluginClass - Other settings related methods

Apart from the defaultSettings() and settingsForm() method, you can also use the getSettings(), getSetting(), setSettings() and setSetting() methods.

Next steps

Now that you have an idea of how to build a custom multi-part field, you can try this out on your next project. Here are some resources to help you out:

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web