Mar 09 2016
Mar 09

Migrate is one of the most established modules in the Drupal ecosystem. So much so that with Drupal 8, a decision has been made to get some of its functionality ported and added to Drupal core. An important reason was that the traditional upgrade between major releases was replaced with a migration of Drupal 6 or 7 content and configuration to Drupal 8.

Drupal 8 logo

Not all the functionality of the Migrate module has been moved into core though. Rather, we have the basic framework within the core migrate module and the functionality serving the upgrade path from Drupal 6 and 7 within the core migrate_drupal module. What’s left can be found in a host of contrib modules. The most important is Migrate Tools which contains, among other things, drush commands and the UI for running migrations. Additionally, the Migrate Source CSV, Migrate Source XML and Migrate Source JSON modules provide plugins for the most used types of migration sources.

In this article we are going to look at how migration works in Drupal 8 by migrating some content into node entities. For simplicity, the data we play with resides in tables in the same database as our Drupal installation. If you want to see the final code, you can check it out in this repository.

Drupal 8 Migration Theory

The structure of a migration in Drupal 8 is made up of three main parts: the source, the process and the destination, all three being built using the new Drupal 8 plugin system. These three parts form a pipeline. The source plugin represents the raw data we are migrating and is responsible for delivering individual rows of it. This data is passed to one or more process plugins that perform any needed manipulation on each row field. Finally, once the process plugins finish preparing the data, the destination plugins save it into Drupal entities (either content or configuration).

Creating a migration involves using such plugins (by either extending or directly using core classes). All of these are then brought together into a special migration configuration entity. This is shipped as module config that gets imported when the module is first enabled or can be constructed using a template (a case we won’t be exploring today). The migration configuration entity provides references to the main plugins used + additional metadata about the migration.

Movie Migration

Let us now get down to it and write a couple of migrations. Our data model is the following: we have two tables in our database: movies and movies_genres. The former has an ID, name and description while the latter has an ID, name and movie ID that maps to a movie from the first table. For the sake of simplicity, this mapping is one on one to avoid the complication of a third table. Here is the MySQL script that you can use to create these tables and populate with a few test records (if you want to follow along):

CREATE TABLE `movies` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `description` text,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

CREATE TABLE `movies_genres` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `movie_id` int(11) DEFAULT NULL,
  `name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;

INSERT INTO `movies` (`id`, `name`, `description`)
VALUES
        (1, 'Big Lebowsky', 'My favorite movie, hands down.'),
        (2, 'Pulp fiction', 'Or this is my favorite movie?');

INSERT INTO `movies_genres` (`id`, `movie_id`, `name`)
VALUES
        (1, 1, 'Comedy'),
        (2, 1, 'Noir'),
        (3, 2, 'Crime');
        

What we want to achieve is migrate the movies into basic Drupal Article nodes and the genres into taxonomy terms of the Tags vocabulary (which the Article nodes reference via a field). Naturally, we also want the migration to mirror the association between the movies and the genres.

Genres

Let us first take care of the genre migration because in Drupal, the movies will depend on them (they will reference them).

Inside the config/install folder of our module, we need the following configuration entity file called migrate.migration.genres.yml:

id: genres
label: Genres
migration_group: demo
source:
  plugin: genres
  key: default
destination:
  plugin: entity:taxonomy_term
process:
  vid:
    plugin: default_value
    default_value: tags
  name: name

The first three elements of this configuration are the migration ID, label and group it belongs to. The following three keys are responsible for the three main parts of the migration pipeline mentioned earlier. Let’s talk about the latter three separately.

Source

Under the source key we specify which plugin the migration should use to represent and deliver the source data (in our case the plugin with the ID of genres). The key key is used to specify which database should our source query use (that is where our data is).

So in the Plugin/migrate/source folder of our module, let’s create our SQL based source plugin for our genres:

Genres.php

namespace Drupal\demo\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SqlBase;


class Genres extends SqlBase {

  
  public function query() {
    $query = $this->select('movies_genres', 'g')
      ->fields('g', ['id', 'movie_id', 'name']);
    return $query;
  }

  
  public function fields() {
    $fields = [
      'id' => $this->t('Genre ID'),
      'movie_id' => $this->t('Movie ID'),
      'name' => $this->t('Genre name'),
    ];

    return $fields;
  }

  
  public function getIds() {
    return [
      'id' => [
        'type' => 'integer',
        'alias' => 'g',
      ],
    ];
  }
}

Since we are using a SQL source, we need to have our own source plugin class to provide some information as to how the data needs to be read. It’s not enough to use an existing one from core. The query() method creates the query for the genre data, the fields() method defines each individual row field and the getIds() method specifies the source row field that acts as the unique ID. Nothing complicated happening here.

We are extending from SqlBase which, among other things, looks for the plugin configuration element named key to learn which database it should run the query on. In our case, the default one, as we detailed in the migration configuration entity.

And this is all we need for our simple source plugin.

Destination

For the migration destination, we use the default core taxonomy term destination plugin which handles everything for us. No need to specify anything more.

Process

Under the process key of the migration, we list each destination field we want populated and one or more process plugins that transform the source row fields into data for the destination fields. Since we want the genres to be all terms of the Tags vocabulary, for the vid field we use the default_value plugin which accepts a default_value key that indicates the value each record will have. Since all will be in the same vocabulary, this works well for us.

Lastly, for the term name field we can simply specify the source row field name without an explicit plugin name. This will, under the hood, use the get plugin that simply takes the data from the source and copies it over unaltered to the destination.

For more information on how you can chain multiple process plugins in the pipeline or what other such plugins you have available from core, I recommend you check out the documentation.

Movies

Now that our genres are importable, let’s take a look at the movies migration configuration that resides in the same folder as the previous (config/install):

migrate.migration.movies.yml

id: movies
label: Movies
migration_group: demo
source:
  plugin: movies
  key: default
destination:
  plugin: entity:node
process:
  type:
    plugin: default_value
    default_value: article
  title: name
  body: description
  field_tags:
    plugin: migration
    migration: genres
    source: genres
migration_dependencies:
  required:
    - genres

We notice the same metadata as before, the three main parts of the migration (source, process and destination) but also the explicit dependency which needs to be met before this migration can be successfully run.

Source

Like before, let’s take a look at the movies source plugin, located in the same place as the genres source plugin (Plugin/migrate/source ):

Movies.php:

namespace Drupal\demo\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SqlBase;
use Drupal\migrate\Row;


class Movies extends SqlBase {

  
  public function query() {
    $query = $this->select('movies', 'd')
      ->fields('d', ['id', 'name', 'description']);
    return $query;
  }

  
  public function fields() {
    $fields = [
      'id' => $this->t('Movie ID'),
      'name' => $this->t('Movie Name'),
      'description' => $this->t('Movie Description'),
      'genres' => $this->t('Movie Genres'),
    ];

    return $fields;
  }

  
  public function getIds() {
    return [
      'id' => [
        'type' => 'integer',
        'alias' => 'd',
      ],
    ];
  }

  
  public function prepareRow(Row $row) {
    $genres = $this->select('movies_genres', 'g')
      ->fields('g', ['id'])
      ->condition('movie_id', $row->getSourceProperty('id'))
      ->execute()
      ->fetchCol();
    $row->setSourceProperty('genres', $genres);
    return parent::prepareRow($row);
  }
}

We have the same three required methods as before, that do the same thing: query for and define the data. However, here we also use the prepareRow() method in order to alter the row data and available fields. The purpose is to select the ID of the movie genre that matches the current row (movie). That value is populated into a new source field called genres, which we will see in a minute how it’s used to save the Tags taxonomy term reference.

Destination

In this case, we use the node entity destination plugin and we need nothing more.

Process

There are four fields on the Article node we want populated with movie data. First, for the node type we use the same technique as before for the taxonomy vocabulary and set article to be the default value. Second and third, for the title and body fields we map the movie name and description source fields unaltered.

Lastly, for the tags field we use the migration process plugin that allows us to translate the ID of the genre (that we added earlier to the genres source row field) into the ID of its corresponding taxonomy term. This plugin does this for us by checking the migration mapping of the genres and reading these IDs. And this is why the genres migration is also marked as a dependency for the movies import.

Activating and Running the Migration

Now that we have our two migration configuration entities and all the relevant plugins, it’s time to enable our module for the first time and have the configuration imported by Drupal. If your module was already enabled, uninstall it and then enable it again. This will make the config import happen.

Additionally, in order to run the migrations via Drush (which is the recommended way of doing it), install the Migrate Tools module. Then all that’s left to do is to use the commands to migrate or rollback the movies and genres.

To see the available migrations and their status:

drush migrate-status

To import all migrations:

drush migrate-import --all

To roll all migrations back:

drush migrate-rollback --all

Conclusion

And there we have it – a simple migration to illustrate how we can now import, track and roll back migrations in Drupal 8. We’ve seen how the plugin system is used to represent all these different components of functionality, and how the migration definition has been turned into configuration that brings these elements together.

There is much more to learn, though. You can use different source plugins, such as for data in CSV or JSON, complex process plugins (or write your own), or even custom destination plugins for whatever data structure you may have. Good luck!

Daniel Sipos

Meet the author

Daniel Sipos is a Drupal developer who lives in Brussels, Belgium. He works professionally with Drupal but likes to use other PHP frameworks and technologies as well. He runs webomelette.com, a Drupal blog where he writes articles and tutorials about Drupal development, theming and site building.
Jun 13 2014
Jun 13

How to Build a Drupal 8 Module

Drupal 8 brings about a lot of changes that seek to enroll it in the same club other modern PHP frameworks belong to. This means the old PHP 4 style procedural programming is heavily replaced with an object oriented architecture. To achieve this, under the initiative of Proudly Found Elsewhere, Drupal 8 includes code not developed specifically for Drupal.

One of the most important additions to Drupal are Symfony components, with 2 major implications for Drupal developers. First, it has the potential to greatly increase the number of devs that will now want to develop for Drupal. And second, it gives quite a scare to some of the current Drupal 7 developers who do not have much experience with modern PHP practices. But that’s ok, we all learn, and lessons taken from frameworks like Symfony (and hopefully Drupal 8), will be easily extensible and applicable to other PHP frameworks out there.

In the meantime, Drupal 8 is in a late stage of its release cycle, the current version at the time of writing being alpha11. We will use this version to show some of the basic changes to module development Drupal 7 devs will first encounter and should get familiar with. I set up a Git repo where you can find the code I write in this series so you can follow along like that if you want.

How do I create a module?

The first thing we are going to look at is defining the necessary files and folder structure to tell Drupal 8 about our new module. In Drupal 7 we had to create at least 2 files (.info and .module), but in Drupal 8, the YAML version of the former is enough. And yes, .info files are now replaced with .info.yml files and contain similar data but structured differently.

Another major change is that custom and contrib module folders now go straight into the root modules/ folder. This is because all of the core code has been moved into a separate core/ folder of its own. Of course, within the modules/ directory, you are encouraged to separate modules between custom and contrib like in Drupal 7.

Let’s go ahead and create a module called demo (very original) and place it in the modules/custom/ directory. And as I mentioned, inside of this newly created demo/ folder, all we need to begin with is a demo.info.yml file with the following required content:

name: Drupal 8 Demo module
description: 'Demo module for Drupal 8 alpha11'
type: module
core: 8.x

Three out of four you should be familiar with (name, description and core). The type is now also a requirement as you can have yml files for themes as well. Another important thing to keep in mind is that white spaces in yml files mean something and proper indentation is used to organize data in array-like structures.

You can check out this documentation page for other key|value pairs that can go into a module .info.yml file and the change notice that announced the switch to this format.

And that’s it, one file. You can now navigate to the Extend page, find the Demo module and enable it.

As I mentioned, we are no longer required to create a .module file before we can enable the module. And architecturally speaking, the .module files will be significantly reduced in size due to most of the business logic moving to service classes, controllers and plugins, but we’ll see some of that later.

What is ‘routing’ and what happened to hook_menu() and its callbacks?

In Drupal 7, hook_menu() was probably the most implemented hook because it was used to define paths to Drupal and connect these paths with callback functions. It was also responsible for creating menu links and a bunch of other stuff.

In Drupal 8 we won’t need hook_menu() anymore as we make heavy use of the Symfony2 components to handle the routing. This involves defining the routes as configuration and handling the callback in a controller (the method of a Controller class). Let’s see how that works by creating a simple page that outputs the classic Hello world!.

First, we need to create a routing file for our module called demo.routing.yml. This file goes in the module root folder (next to demo.info.yml). Inside this file, we can have the following (simple) route definition:

demo.demo:
  path: '/demo'
  defaults:
    _content: '\Drupal\demo\Controller\DemoController::demo'
    _title: 'Demo'
  requirements:
    _permission: 'access content'

The first line marks the beginning of a new route called demo for the module demo (the first is the module name and the second the route name). Under path, we specify the path we want this route to register. Under defaults, we have two things: the default page title (_title) and the _content which references a method on the DemoController class. Under requirements, we specify the permission the accessing user needs to have to be able to view the page. You should consult this documentation page for more options you can have for this routing file.

Now, let’s create our first controller called DemoController that will have a method named demo() getting called when a user requests this page.

Inside the module directory, create a folder called src/ and one called Controller/ inside of it. This will be the place to store the controller classes. Go ahead and create the first one: DemoController.php.

The placement of the Controllers and, as we will see, other classes, into the src/ folder is part of the adoption of the PSR-4 standard. Initially, there was a bigger folder structure we had to create (PSR-0 standard) but now there is a transition phase in which both will work. So if you still see code placed in a folder called lib/, that’s PSR-0.

Inside of our DemoController.php file, we can now declare our class:

<?php
/**
 * @file
 * Contains \Drupal\demo\Controller\DemoController.
 */

namespace Drupal\demo\Controller;

/**
 * DemoController.
 */
class DemoController {
  /**
   * Generates an example page.
   */
  public function demo() {
    return array(
      '#markup' => t('Hello World!'),
    );
  }      
}

This is the simplest and minimum we need to do in order to get something to display on the page. At the top, we specify the class namespace and below we declare the class.

Inside the DemoController class, we only have the demo() method that returns a Drupal 7-like renderable array. Nothing big. All we have to do now is clear the caches and navigate to http://example.com/demo and we should see a Drupal page with Hello World printed on it.

In Drupal 7, when we implement hook_menu(), we can also add the registered paths to menus in order to have menu links showing up on the site. This is again no longer handled with this hook but we use a yml file to declare the menu links as configuration.

Let’s see how we can create a menu link that shows up under the Structure menu of the administration. First, we need to create a file called demo.menu_links.yml in the root of our module. Inside this yml file we will define menu links and their position in existing menus on the site. To achieve what we set out to do, we need the following:

demo.demo:
  title: Demo Link
  description: 'This is a demo link'
  parent: system.admin_structure
  route_name: demo.demo

Again we have a yml structure based on indentation in which we first define the machine name of the menu link (demo) for the module demo (like we did with the routing). Next, we have the link title and description followed by the parent of this link (where it should be placed) and what route it should use.

The value of parent is the parent menu link (appended by its module) and to find it you need to do a bit of digging in *.menu_links.yml files. I know that the Structure link is defined in the core System module so by looking into the system.menu_links.yml file I could determine the name of this link.

The route_name is the machine name of the route we want to use for this link. We defined ours earlier. And with this in place, you can clear the cache and navigate to http://example.com/admin/structure where you should now see a brand new menu link with the right title and description and that links to the demo/ path. Not bad.

Conclusion

In this article we began exploring module development in Drupal 8. At this stage (alpha11 release), it is time to start learning how to work with the new APIs and port contrib modules. To this end, I am putting in writing my exploration of this new and exiting framework that will be Drupal 8 so that we can all learn the changes and hit the ground running when release day comes.

For starters, we looked at some basics: how you start a Drupal 8 module (files, folder structure etc), all compared with Drupal 7. We’ve also seen how to define routes and a Controller class with a method to be called by this route. And finally, we’ve seen how to create a menu link that uses the route we defined.

In the next tutorial, we will continue building this module and look at some other cool new things Drupal 8 works with. We will see how we can create blocks and how to work with forms and the configuration system. See you then.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web