The Big Picture: Drupal 8 Migrate in Core

Migrating from major version to major version of Drupal core has always been a significantly large task for all but the simplest sites. The upgrade path that has traditionally been part of Drupal core has always been limited in what it can do, so most sites were forced to use alternative methods to migrate configuration and content. Sometimes these migrations were manual, sometimes automated, and most often a combination of the two.

Drupal 8 aims to greatly reduce the friction of migrating sites from Drupal 6 and Drupal 7 by adopting a proven and extensible approach to site migrations. The Migrate module has been the go-to tool for migrating a large number of sites to Drupal 7 from earlier versions of Drupal as well as from other content management systems (including custom ones.)

This blog post aims to provide an overview of how the migration system in Drupal 8 works, our current progress, and how new contributors can get involved. The Migrate in Core initiative began in earnest about a year ago at DrupalCon Prague, when it was decided to use some code and concepts from the Migrate and Drupal-to-Drupal Data Migration modules as a starting point for a new and improved upgrade path.

At the current time, the Drupal 6 to Drupal 8 migration is almost complete, while the Drupal 7 to Drupal 8 migration is just getting started. There are a few blocking issues that we're trying to get past in the next couple of weeks (including files migration and link field migration). We feel that we'll be able to leverage much of the work we've done on the Drupal 6 to Drupal 8 migration for the Drupal 7 to Drupal 8 migration. In fact, we have a great issue for a new contributor to help us kick of the Drupal 7 work just waiting for someone to tackle.

The Drupal 7 Migrate and Drupal-to-Drupal Data Migration modules have inspired the Drupal 8 "Migrate" and "Migrate Drupal" core modules. Before we get into some of the details of how things work, it is important to note that these two modules differ from their Drupal 7 counterparts in many ways, but perhaps none more importantly than the fact that they are designed to migrate both content and configuration. This significantly increased the complexity of these modules, but was a requirement to provide functionality above and beyond previous major Drupal version upgrade paths.

Similar to previous upgrade paths, the Drupal 8 "Migrate Drupal" core module is designed to migrate only core content and configuration to Drupal 8. Contributed module maintainers are responsible for providing migration classes for data that they manage. For example, if you have Panels on your Drupal 7 site, these won't be migrated by the Migrate Drupal module, rather the Panels maintainers will have to add migration classes that include the logic for migrating Panels-related data to Drupal 8.

Of course, there are caveats for this "core only" migration approach - in the Drupal 6 to Drupal 8 migration, we've also included migrations for "core CCK" and the Link modules. Furthermore, there's a rumor that work is ongoing on migration classes for Views data.

If you're familiar with the Drupal 7 version of the Migrate module, many of the concepts of the Drupal 8 core Migrate and Migrate Drupal will be familiar, but most of them have been implemented in vastly different ways. This is most evident in the use of Drupal 8 configuration (.yml) files. These per-migration configuration files specify the various plugins that are used for each aspect of the migration, including defining the source data, the "process pipeline", and the destination.

For example, here's the configuration for the a taxonomy vocabulary migration (core/modules/migrate_drupal/config/install/migrate.migration.d6_taxonomy_vocabulary.yml):

id: d6_taxonomy_vocabulary
label: Drupal 6 taxonomy vocabularies
migration_groups:
  - Drupal 6
source:
  plugin: d6_taxonomy_vocabulary
process:
  vid:
    -
      plugin: machine_name
      source: name
    -
      plugin: dedupe_entity
      entity_type: taxonomy_vocabulary
      field: vid
      length: 32
  label: name
  name: name
  description: description
  hierarchy: hierarchy
  weight: weight
destination:
  plugin: entity:taxonomy_vocabulary

Parsing this configuration section-by-section:

id: d6_taxonomy_vocabulary
label: Drupal 6 taxonomy vocabularies

The "id" is the unique identifier for the migration. This is used in an annotation on the source plugin as well as an identifier in dependent migrations. The "label" is simply a friendly name for the migration.

migration_groups:
  - Drupal 6

Similar to the Drupal 7 Migrate module, migrations can be placed into groups for organization and querying. For example, “run all the ‘Drupal 6’ migrations.”.

source:
  plugin: d6_taxonomy_vocabulary

The "source" identifies the Drupal 8 migration plugin that is responsible for gathering the data from the source database. In this case, it is referring to the "Vocabulary" migration class (core/modules/migrate_drupal/src/Plugin/migrate/source/d6/Vocabulary.php), with the @MigrateSource annotation in that file linking the configuration file with the class. Like the Drupal 7 Migrate module, the source plugin passes data one row at a time to the process pipeline.

process:
  vid:
    -
      plugin: machine_name
      source: name
    -
      plugin: dedupe_entity
      entity_type: taxonomy_vocabulary
      field: vid
      length: 32
  label: name
  name: name
  description: description
  hierarchy: hierarchy
  weight: weight

The "process" section is where much of the real work happens. In this section, fields are mapped from the source to the destination, and various "process plugins" can modify the data as it moves from source to destination. There are various process plugins available for use, and custom plugins can also be added as necessary. Similar to field mapping classes in the Drupal 7 version of the Migrate module, the format of this section is {destination field name}: {source field name}. The "get" process plugin is used by default to get the field data from the source class, so in the case of "label: name" in the example above, this is actually using the "get" plugin to grab the "name" field from the source and assigning it to the "label" field on the destination.

As data needs to be manipulated between source and destination, additional process plugins can be utilized. In this example, "vid" field is set by first using the implicit "get" plug to grab the value of the "name" field from the source, then is it processed by the machine_name plugin (which ensures it is suitable to be a machine name), followed by the dedupe entity plugin which ensures it isn't a duplicate of another vid.

destination:
  plugin: entity:taxonomy_vocabulary

Finally, the destination plugin is specified; all processed data from the source is passed to the destination plugin that ultimately saves it to the Drupal 8 database.

If that much makes sense, then you're well on your way to understanding how Migrate in Core works. There are currently 85 different migrations defined for Drupal 6, each one works in a similar manner to the example above.

In future blog posts, we'll dive down a bit further into the actual mechanics of how a migration is run, the relationship between the Drupal 8 Migrate and Migrate Drupal modules, and how to create your own migration classes (pay attention module maintainers!)

If you can't wait, there's already some helpful documentation available for how the Drupal 8 migrate system works, we'll be updating it as the initiative progresses. If you have any questions, pop into the #drupal-migrate IRC channel and ask away.

Finally, if you're interested in helping out with the Migrate in Core initiative, we're always looking for help testing, documenting, organizing the issue queue, helping out in #drupal-migrate, and (of course) coding.