Feeds

Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jul 27 2020
Jul 27

Following the previous post, in this article I wanted to share again some guidelines for debugging Drupal Migrations. While in the last article I wrote down some questions about modules, plugins and configuration object management, in this case I felt like stopping at something more “code-oriented”, more specific. I have chosen for this article some guidelines to configure Xdebug in an IDE (in this case I have used PHPStorm) for migrations running in a locally deployed Drupal using DDEV. With the aim of debugging a Migration process -although actually the process is similar for any other development, since it is something certainly transversal-, here I share with you my fifth article about migrations at Drupal. I hope you find it useful.

Picture from Unsplash, user Krzysztof Niewolny, @epan5

Table of Contents

1- Introduction to Xdebug
2- Xdebug for DDEV
3- Steps for seeting up the IDE
4- Errors launching Drush in containers
5- :wq!

This article is part of a series of posts about Drupal Migrations:

1- Drupal Migrations (I): Basic Resources

2- Drupal Migrations (II): Examples

3- Drupal Migrations (III): Migrating from Google Spreadsheet

4- Drupal Migrations (IV): Debugging Migrations First Part

5- Drupal Migrations (V): Debugging Migrations-II

1- Introduction to Xdebug

In this section we’re going to share some debugging tactics for our migrations that can complement the actions discussed in the previous article: we already know how to check the configuration files, how to use certain processing plugins to extract information from the process and at this moment we’re going to add the third piece to the construction: how to debug the operation of a migration from the point of view of the related code, that is, through the PHP classes involved in all the mechanics.

Xdebug is a classic PHP extension that in its simplest dimension, allows us to track variables from the code to see their values through the execution flows of your software. In this post we won’t stop to explain how to install, configure and manage Xdebug in general terms, but we can be interested in a particular case: in order to run your local Drupal installations and test your migrations, you can use DDEV, a tool for creating local development environments based on Docker containers that already has a pre-installation of Xdebug ready to use.

DDEV avoids having to implement classic LAMP environments by compartmentalizing networks of Docker containers already pre-configured with all the development tools already included, including Xdebug. Here you can find more information, articles and resources about the tool:
https://github.com/drud/awesome-ddev.

2- Xdebug for DDEV

Accepting the premise that we are in a environment build with DDEV, the activation of Xdebug can easily be done in two ways:

Either by editing the config.yaml file in the .ddev resource folder, path: your_project/.ddev/config.yaml, where you can activate xdebug on line 9 of the file:

DDEV enabling Xdebug

Or by command line, using the DDEV-related commands outside the container:

ddev exec enable_xdebug
ddev exec disable_xdebug

Or also:

ddev xdebug
ddev xdebug on
ddev xdebug off

The above commands will serve to activate and deactivate the resource (for performance reasons, the initial state of Xdebug in a DDEV installation is deactivated by default). Since Xdebug is a server-side tool, it is already in the container acting as a web server and requires no further configuration. It’s just a matter of preparing our usual IDE.

In essence, any IDE works the same way with Xdebug: it listens on a port and reacts as soon as it receives an input. Xdebug usually uses port 9000, and for each IDE there are some configuration examples that you can follow here:

And the basic technique was already exposed at the time by the PHPStorm team (Now I’m using PHPStorm for this example): https://blog.jetbrains.com/phpstorm/2012/03/new-in-4-0-easier-debugging-of-remote-php-command-line-scripts/.

3- Steps for setting up the IDE

Let’s look at some basic steps of setting up Xdebug in the PHPStorm IDE, although the philosophy behind it in others like VSCode is just the same.
The first idea is that by activating Xdebug, we’ll have it available for any PHP script running from the command line. This makes it especially valuable for debugging Drush or Drupal Console (Old and abandoned Drupal Console, BTW) . In this case, we also add the enormous facilitation of automated activation already in Xdebug in local DDEV-based deployments, so we add a lot of facilities to debug our migration.

Let’s take the next steps:

DDEV enabling Xdebug Server

1- Server for the project

Open the Server configuration for your project: go to File-> Settings -> Languages & Frameworks -> PHP -> Servers and create a new Server in the side menu. Give it a name (it will be easier if you use the name of your DDEV deployment for the project). Add port 80 and activate Xdebug as debugger.

2- Initial setup

When using PHPStorm non-locally, we need to set up an “external” server and map the project files. In the case of not being in a classic LAMP environment (that would be local environment), we will need to make this configuration (Docker or DDEV suppose a non-local way of deployment in some way). DDEV serves the Drupal installation files from within its web container, starting with /var/www/html/ but the files are located locally within a typical /home/user/folder/projects/my_project path, so we will load this file mapping.

3- Listening Debug connections

Activate the active listening button for debugging.

4- Load environment variable in the container

For this we have two options. We can either load it for each debugging session, entering it into the DDEV container using the ddev ssh instruction and then loading the environment variable from inside the web container with the next instruction, just execute:
export PHP_IDE_CONFIG="serverName=your.project.name.ddev.site"

You can also permanently load the environment variable by creating a companion file in the .ddev/ folder of the project, a docker-compose.env.yaml file that is located next to the other docker-compose related files from the DDEV configuration, with the content:

version: '3.6'
services:
 web:
   environment:
     - PHP_IDE_CONFIG=serverName=your.project.name.ddev.site

This file will be merged with the other files when you restarting the project container network (ddev start / ddev stop) and the variable will be loaded more persistently and automatically.

5- Breakpoints

We can then try to place various breakpoints along the code involved in a migration. For example, for processing that does not require transformation treatment (only the transfer from a source to a destination), we already know that a Process Plugin provided by default by the Migrate API is used: the Get.php class, present in /core/modules/migrate/src/Plugin/migrate/process/Get.php and in this case we can insert some breakpoints in what will be a safe place for certain migration examples: for example in the transform() method of the class (line 106):

DDEV enabling Xdebug with breakpoints

4- Errors launching Drush in containers

When we have the IDE in active listening for debugging, some breakpoints strategically located and we launch some Drush command in the context of DDEV, it is possible that when executing Drush commands, from the IDE we receive an error about the mapping of files associated to Drush. Without meaning to, it is executed on the global Drush launcher of the container, installed in the path /usr/local/bin/drush (of the operating system of the web container).

DDEV enabling Xdebug Drush mapping in containers

In that case, we have two options:

Ỳou can run drush directly on the /vendor address of the project (which is mapped in the previous configuration of our remote server), using the forms:

  • From outside the container: ddev exec /vendor/bin/drush migrate-status
  • From inside the container (ddev ssh): /vendor/bin/drush migrate-status

Another option is recommended by Randy Fay, @randyfay (one of the creators of DDEV) in the commentary to this Github issue:
github.com/ddev/issues/2341#issuecomment-650618158.

That is, take advantage of the Drush command example as usually described in the drush.example file in the path /your-name-project/.ddev/commands/web/ and modify the Drush use case it contains by directly placing the previous path to the project vendor, going from :
drush [email protected] to: /var/www/html/vendor/bin/drush [email protected]

This will allow us to run debugging with Xdebug for our migrations in DDEV environments. Here you can find more information about it:

Now you can run your migration with all the safety and comfort!

5- :wq!

[embedded content]

Jun 29 2020
Jun 29

The Drupal migrations, despite their linearity in terms of definitions, contain a lot of inherited complexity. The reason is very intuitive: although the Migrate API is a supersystem that offers a very simple “interface” of interactions for the user-developer who wants to build migration processes, in reality several subsystems work by interacting with each other throughout a migration process: Entities, Database, Plugins…There are a lot of classes involved in even the simplest migration process. If we add the irrefutable fact that a migration will tend to generate errors in many cases until it has been refined, it’s clear then that one of our first needs will be to learn…how to debug migrations.

Picture from Unsplash, user Krzysztof Niewolny, @epan5

Table of Contents

1- Introduction
2- Basic Debugging: Keep an eye on your file
3- Average Debugging with Migrate Devel
4- :wq!

This article is part of a series of posts about Drupal Migrations:

1- Drupal Migrations (I): Basic Resources

2- Drupal Migrations (II): Examples

3- Drupal Migrations (III): Migrating from Google Spreadsheet

4- Drupal Migrations (IV): Debugging Migrations First Part

5- Drupal Migrations (V): Debugging Migrations-II

1- Introduction

In the wake of the latest articles, I wanted to continue expanding information about migration in Drupal. I was thinking about writing a sub-series of debugging migrations (inside the main series about Drupal Migrations), and I want to publish now the first part, just a set of basic steps in order to get all the available information from a migration process. All the examples in this post were taken of the migration_google_sheet example, from my Gitlab account.

2- Basic Debugging (Keep an eye on your files)

First, we will start with a very primary approach to error detection during a migration. To begin with, it is essential to keep the focus on reducing the range of error possibilities as much as possible by approaching the migration in an iterative and incremental manner. In other words: we will go step by step and expand our migrated data.

2.1- Reviewing your Migration description file

First of all we are going to comment on the most intuitive step of all we will take, since sometimes there are errors that occur at first sight and not because they are recurrent but end up being more obvious.

The first steps in our process of debugging a migration will be a review of two fundamental issues that usually appear in many migrations. So before anything else, we’ll do a quick review of:

Whitespaces: Any extra whitespace may be causing us problems at the level of the migration description file: we review all lines of the file in a quick scan in order to detect extra whitespace.

Errors in the indentation: The migration description file has a format based on YAML, a language for data serialization based on a key scheme: a value where it is structured by parent - child levels and an indentation of two spaces to the right at each level down in the hierarchy. It is very frequent that some indentation is not the right one and this ends up producing an error in the processing of the file. As a measure, as in the previous case, we will review all the cases of indentations registered in the file.

You can rely on a YAML syntax review service such as www.yamllint.com, but you will have to monitor the result as well.

2.2- Reviewing registers in database

If you’re in a basic Drupal installation (standard profile) we have seventy-three tables, after the activation of the basic modules related to migrations: migrate, migrate_plus, migrate_tools and in this case the custom migration_google_sheet_wrong the number of tables in the database is seventy-five. Two more tables have been generated:

cache_migrate
cache_discovery_migration

But also, later, after executing the migration with ID taxonomy_google_sheet_wrong contained in our custom module, we see in the database that two new tables have been generated related to the executed migration:

  • migrate_map_taxonomy_google_sheet
    This table contains the information related to the movements of a row of data (migrations are operated ‘row’ to ‘row’). Migrate API is in charge of storing in this table the source ID, the destination ID and a hash related to the ‘row’ in this data mapping table. Combinations between the source ID and the hash of the row operation then make it easier to track changes, progressively update information, and cross dependencies when performing a batch migration (see below for how they are articulated).
    The lookup processes for migrations are supported by this data: for example, to load a taxonomy term you must first lookup its “parent” term to maintain the hierarchy of terms. If we go to our database and we do not see recorded results after launching a migration, no data was stored and the migration requires debugging.

  • migrate_message_taxonomy_google_sheet
    In this table, messages associated to the executed migration will be stored, structured in the same way as the previous table (based on the processing of a ‘row’ of the migration), each message with its own identifier and an association to the id_hash of the ‘row’ of origin of the data:

Drupal Migration columns from table Messages

This information can be obtained through Drush, since the content of this table is what is shown on the screen when we execute the instruction:

drush migrate:messages id-migration

And this can be a useful way to start getting information about the errors that occurred during our migration.

2.4- Reloading Configuration objects

Another issue we’ll need to address while debugging our migration is how to make it easier to update the changes made to the configuration object created from the migration description file included in the config/install path.

As we mentioned earlier, each time the module is installed a configuration object is generated that is available in our Drupal installation. In the middle of debugging, we’ll need to modify the file and reload it to check that our changes have been executed. How can we make this easier? Let’s take a look at some guidelines.

On the one hand, we must associate the life cycle of our migration-type configuration object with the installation of our module. For it, as we noted in the section 2.3.2- Migration factors as configuration, we will declare as forced dependency our own custom module of the migration:

dependencies:
   enforced:
      module:
          - migration_google_sheet

We can use both Drush and Drupal Console to perform specific imports of configuration files, taking advantage of the single import options of both tools:

Using Drupal Console

drupal config:import:single --directory="/modules/custom/migration_google_sheet/config/install" --file="migrate_plus.migration.taxonomy_google_sheet.yml"


Using Drush

drush cim --partial --source=/folder/

Similarly, we can also remove active configuration objects using either Drush or Drupal Console:

drush config-delete "migrate_plus.migration.taxonomy_google_sheet"
drupal config:delete active "migrate_plus.migration.taxonomy_google_sheet"

If we prefer to use the Drupal user interface, there are options such as the contributed Config Delete module drupal.org/config_delete , which activates extra options to the internal configuration synchronization menu to allow the deletion of configuration items from our Drupal installation. It’s enough to download it through Composer and enable it through Drush or Drupal Console:

composer require drupal/config_delete
drush en config_delete -y

Drupal Config Delete actions

This way we can re-import configuration objects without colliding with existing versions in the database. If you choose to update and compare versions of your configuration, then maybe the Configuration Update Manager contributed module can be a good option https://www.drupal.org/project/config_update.

3- Average Debugging with Migrate Devel

Well, we have looked closely at the data as we saw in the previous section and yet our migration of taxonomy terms from a Google Spreadsheet seems not to work.

We have to resort to intermediate techniques in order to obtain more information about the process. In this phase our allies will be some modules and plugins that can help us to better visualize the migration process.

3.1- Migrate Devel

Migrate Devel https://www.drupal.org/project/migrate_devel is a contributed module that brings some extra functionality to the migration processes from new options for drush. This module works with migrate_tools and migrate_run.

UPDATE (03/07/2020): Version 8.x-2.0-alpha2

Just as I published this article, Andrew Macpherson (new maintainer of the Migrate Devel module and one of the accessibility maintainers for Drupal Core), left a comment that you can see at the bottom of this post with some important news. Well, since I started the first draft of this article, a new version had been published, released on June 28th and it’s already compatible with Drush 9 (and I didn’t know…) So now you know there’s a new version available to download compatible with Drush 9 and which avoids having to install the patch exposed below.

To install and enable the module, we proceed to download it through composer and activate it with drush: Migrate Devel 8.x-2.0-alpha2.

composer require drupal/migrate_devel
# To install the 2.0 branch of the module:
composer require drupal/migrate_devel:^2.0
drush en migrate_devel -y

Follow for versions prior to 8.x-2.0-alpha2:

If you’re working with versions prior to 8.x-2.0-alpha2, then you have to know some particularities: The first point is that it’s was optimized for a previous version of Drush (8) and it does not seem to have closed its portability to Drush 9 and higher.

There’s a tag 8.x.1.4 from two weeks ago in the 8.x-1.x branch: migrate_devel/tree/8.x-1.4

There is a necessary patch in its Issues section to be able to use it in versions of Drush > 9 and if we make use of this module this patch https://www.drupal.org/node/2938677 will be almost mandatory. The patch does not seem to be in its final version either, but at least it allows a controlled and efficient execution of some features of the module. Here will see some of its contributions.

And to apply the patch we can download it with wget and apply it with git apply:

cd /web/modules/contrib/migrate_devel/
wget https://www.drupal.org/files/issues/2018-10-08/migrate_devel-drush9-2938677-6.patch 
git apply migrate_devel-drush9-2938677-6.patch

Or place it directly in the patch area of our composer.json file if we have the patch management extension enabled: https://github.com/cweagans/composer-patches.

Using:

composer require cweagans/composer-patches 

And place the new patch inside the “extra” section of our composer.json file:

Drupal Debugging adding the patch

How it works:

The launch of a migration process with the parameters provided by Migrate Devel will generate an output of values per console that we can easily check, for example using –migrate-debug:

Drupal Devel Output first part
Drupal Devel Output second part

This is a partial view of the processing of a single row of migrated data, showing the data source, the values associated with this row and the final destination ID, which is the identifier stored in the migration mapping table for process tracking:

Drupal Devel Output in database

Now we can see in the record that for the value 1 in origin (first array of values), the identifier 117 was assigned for the load in destination. This identifier will also be the internal id of the new entity (in this case taxonomy term) created within Drupal as a result of the migration. This way you can relate the id of the migration with the new entity created and stored.

What about event subscribers?, Migrate Devel creates an event subscriber, a class that implements EventSubscriberInterface and keeps listening to events generated from the event system of the Drupal’s Migrate API, present in the migrate module of Drupal’s core:

Called from +56 
/var/www/html/web/modules/contrib/migrate_devel/src/EventSubscriber/MigrationEventSubscriber.php

The call is made from the class where events are heard and actions from the module’s Event classes are read. Many events are defined there modules/migrate/src/Event, but in particular, two that are listened to by Migrate Devel:

  1. MigratePostRowSaveEvent.php
  2. MigratePreRowSaveEvent.php

What are the two Drush options offered by Migrate Devel, and in both cases results in a call to the Kint library dump() function provided by the Devel module to print messages. In fact the call to Kint has changed in the last version 8.x-2.0-alpha2, where Kint is replaced by a series of calls to the Dump method of ths Symfony VarDumper. Where we used to do:

  /**
   * Pre Row Save Function for --migrate-debug-pre.
   *
   * @param \Drupal\migrate\Event\MigratePreRowSaveEvent $event
   *    Pre-Row-Save Migrate Event.
   */
  public function debugRowPreSave(MigratePreRowSaveEvent $event) {
    $row = $event->getRow();

    $using_drush = function_exists('drush_get_option');
    if ($using_drush && drush_get_option('migrate-debug-pre')) {
      // Start with capital letter for variables since this is actually a label.
      $Source = $row->getSource();
      $Destination = $row->getDestination();

      // We use kint directly here since we want to support variable naming.
      kint_require();
      \Kint::dump($Source, $Destination);
    }
  }

Now we’re doing:

  /**
   * Pre Row Save Function for --migrate-debug-pre.
   *
   * @param \Drupal\migrate\Event\MigratePreRowSaveEvent $event
   *    Pre-Row-Save Migrate Event.
   */
  public function debugRowPreSave(MigratePreRowSaveEvent $event) {
    if (PHP_SAPI !== 'cli') {
      return;
    }

    $row = $event->getRow();

    if (in_array('migrate-debug-pre', \Drush\Drush::config()->get('runtime.options'))) {
      // Start with capital letter for variables since this is actually a label.
      $Source = $row->getSource();
      $Destination = $row->getDestination();

      // Uses Symfony VarDumper.
      // @todo Explore advanced usage of CLI dumper class for nicer output.
      // https://www.drupal.org/project/migrate_devel/issues/3151276
      dump(
        '---------------------------------------------------------------------',
        '|                             $Source                               |',
        '---------------------------------------------------------------------',
        $Source,
        '---------------------------------------------------------------------',
        '|                           $Destination                            |',
        '---------------------------------------------------------------------',
        $Destination
      );
    }
  }

You can see the update and changes in migrate_devel/8.x-2.0-alpha2/src/EventSubscriber/MigrationEventSubscriber.php.

And you can get more information about creating events and event subscribers in Drupal here in The Russian Lullaby: Building Symfony events for Drupal.

3.2- Debug Process Plugin

The contributed module Migrate Devel also brings a new processing plugin called “debug” and defined in the Debug.php class. This PHP class can be found in the module path: /web/modules/contrib/migrate_devel/src/Plugin/migrate/process/Debug.php and we can check its responsibility by reading its annotation section in the class header:

/**
 * Debug the process pipeline.
 *
 * Prints the input value, assuming that you are running the migration from the
 * command line, and sends it to the next step in the pipeline unaltered.
 *
 * Available configuration keys:
 * - label: (optional) a string to print before the debug output. Include any
 *   trailing punctuation or space characters.
 * - multiple: (optional) set to TRUE to ask the next step in the process
 *   pipeline to process array values individually, like the multiple_values
 *   plugin from the Migrate Plus module.

And it consists directly with the transform() method - inherited from the ProcessPluginBase abstract class, where instead of applying transformation actions during processing, it simply uses PHP’s print_r function to display information by console and will print both scalar values and value arrays.

This plugin can be used autonomously, being included as part of the migration pipeline, so that it prints results throughout the processing of all value rows. In this case, we are going to modify the pipeline of the processing section of our taxonomy terms migration, with the idea of reviewing the values being migrated.

To begin with, we are going to modify the structure. We already know (from previous chapters) that this is actually the way:

process:
 name: name
 description: description
 path: url
 status: published

It’s just an implicit way of using the Get.php Plugin which is equivalent to:

process:
 name:
   plugin: get
   source: name
 description:
   plugin: get
   source: description
 path:
   plugin: get
   source: url
 status:
   plugin: get
   source: published

Now we add to the pipeline the Debug plugin with an information label for processing:

process:
 name:
   plugin: debug
   label: 'Processing name field value: '
   plugin: get
   source: name
 description:
   plugin: debug
   label: 'Processing description field value: '
   plugin: get
   source: description
 path:
   plugin: debug
   label: 'Processing path field value: '
   plugin: get
   source: url
 status:
   plugin: debug
   label: 'Processing status field value:  '
   plugin: get
   source: published

After this change we reload the migration configuration object by uninstalling and installing our module (as it is marked as a dependency, when uninstalled the migration configuration will be removed):

drush pmu migration_google_sheet && drush en migration_google_sheet -y 

So when we run the migration now we will get on screen information about the values:

Drupal Migrate Devel feedback

This way we get more elaborated feedback on the information to be migrated. If we want to complete this information and thinking about more advanced scenarios, we can combine various arguments and options to gather as much information as possible. Let’s think about reviewing the information related to only one element of the migration. We can run something like:

 drush migrate-import --migrate-debug taxonomy_google_sheet --limit="1 items"

Which will combine the output after storage (unlike its –migrate-debug-pre option), showing in a combined way the output of the Plugin, the values via Kint and the final storage ID of the only processed entity.

In this case, we only see basic values and with little processing complexity (we only extract from Source and load in Destiny) but in successive migrations we will be doing more complex processing treatments and it will be an information of much more value. Interesting? think about processing treatment for data values that must be adapted (concatenated, cut, added, etc)…if at each step we integrate a feedback, we can better observe the transformation sequence.

Here you can check the Plugin code: migrate_devel/src/Plugin/migrate/process/Debug.php.

Here you can review the Drupal.org Issue where the idea of implementing this processing Plugin originated: https://www.drupal.org/node/3021648.

Well, with this approach to Migrations debugging we will start the series on debugging…soon more experiences!

4- :wq!

[embedded content]

May 27 2020
May 27

The systems and subsystems related to Drupal’s migration API are certainly exciting. In the previous articles in this series, I wanted to draw as complete a map as possible (part one) of the vast amount of resources, possibilities and referenced experts. In the second part I wanted to expose some basic mechanics of the migration processes in Drupal and knowing that this opens the door to thousands of options, possibilities and techniques….I didn’t want to let a third article go by without sharing some experiences migrating data from a common format as a Google Spreadsheet, just an usual way in wich sometimes the data are sent.

Picture from Unsplash, user Pan Xiaozhen, @zhenhappy

Table of Contents

1- Introduction and remember ETL processes
2- Exposing data through Google Spreadsheet
3- Special Properties from the JSON transformation
4- Characteristics of the Migration
5- Custom Module for Migration
6- :wq!

This article is part of a series of posts about Drupal Migrations

1- Drupal Migrations (I): Basic Resources

2- Drupal Migrations (II): Examples

3- Drupal Migrations (III): Migrating from Google Spreadsheet

4- Drupal Migrations (IV): Debugging Migrations First Part

5- Drupal Migrations (V): Debugging Migrations-II

1- Introduction (and remember ETL processes)

Well, I would like to start by posing a scenario: imagine that we must migrate a list of taxonomy terms to our Drupal Installation. The source of the data is important, since it defines what type of Plugins we will have to use (source Plugins for the extract, but it can also have influence for the processing and for the final load in destination).

The imagined scenario of this article will be the following: we have “inherited” a migration task already initiated by a previous partner. This migration consists of populating an existing vocabulary in our Drupal installation with taxonomy terms.
To practice with other Plugins and other migration models, I thought we can take a Google spreadsheet as a source of data. Let’s not forget the frame we’re in:

ETL Scheme and Drupal Migrate API overview

Today our goal will be to fill the fields of a taxonomy term using the data contained in a external Google Spreadsheet. So we have to complete these fields:

Taxonomy term basic fields

And we’re going to do this describing a Migrate Process and executing it as Configuration. Do you know the differences between migrations as code and as configuration? You can learn some keys about this topic here, in the previous article about Migrations.

2- Exposing data through Google Spreadsheet

So now, to perform source data extraction, we’ll need a spreadsheet with exposed values. I’ve created a Google spreadsheet here at this address. This Google Spreadsheet contains some columns with values related with fields of a taxonomy term, ready for migration.

In order to processing the data source we’ll use the Migrate Google Sheets contrib module from Drupal.org, so you can run your Composer to download the resource. Just launch:

composer require drupal/migrate_plus #If applicable
composer require drupal/migrate_google_sheets
drush en migrate_plus migrate_google_sheet -y

This contrib module will treat the resource as a JSON file, though for that we have to do some tasks previously. For example, we have to expose the Google Spreadsheet like a JSON datasource, using the tools provided by Google.

  • First, we need extracting the workbook-id of our Spreadsheet: docs.google.com/spreadsheets/d/1bKGbPbgeuXaBfcKetaDqoDimmYcerQY_hT1rqzw4TbM/ -> workbook-id: 1bKGbPbgeuXaBfcKetaDqoDimmYcerQY_hT1rqzw4TbM.

  • Second, we need the worksheet-index too. This is only the index of the tab with data from the Spreadsheet. In this case -> worksheet-index: 1.

  • Third, Building the JSON exposed URL using the pattern: spreadsheets.google.com/feeds/list/[workbook-id]/[worksheet-index]/public/values?alt=json, for us: http://spreadsheets.google.com/feeds/list/1bKGbPbgeuXaBfcKetaDqoDimmYcerQY_hT1rqzw4TbM/1/public/values?alt=json.

That’s all! now we got an exposed Google Spreadsheet as a JSON file and now we can get the values from the selected Source Plugins of the Drupal Migrate API.

Google Spreadsheet JSON dataformat

3- Special Properties from the JSON transformation

Now we’re seeing our Json datasource from our browser (maybe better you use some kind of extension for your browser in order to get a well-formed view of the Json data, like Firefox is doing by default):

      {
        "gsx$id": {
          "$t": "1"
        },
        "gsx$name": {
          "$t": "Term 1"
        },
        "gsx$description": {
          "$t": "Descrip 1"
        },
        "gsx$url": {
          "$t": "/t1"
        }
        "gsx$published": {
          "$t": "1"
        }
      }

As we can see, some items has been changed from the original format. Look there:

Comparing data formats

An important observation about this transformation is that the move to JSON format includes some changes over the original spreadsheet. For example, the field identification is modified by Google by adding certain prefixes to the name:

  • gsx$ for field names.
  • $t for values stored in fields.

And the headers has been changed:

  1. ‘Id’ is now ‘gsx$id’
  2. ‘Name’ is now ‘gsx$name’
  3. ‘Description’ is now ‘gsx$description’
  4. ‘Url’ is now ‘gsx$url’
  5. ‘Published’ is now ‘gsx$published’

But since these changes are stable, the Plugin takes care of their processing:

// Class GoogleSheets.php
// Module migrate_google_sheets
// @see https://git.drupalcode.org/project/migrate_google_sheets/

// Actual values are stored in gsx$<field>['$t'].
$this->currentItem[$field_name] = $current['gsx$' . $selector]['$t'];

This allows us use simply the names of the data in their exposed ‘flat’ transformations: all will be in lowercase and with spaces or special characters removed. In addition, the values are stored in an XPath feed/entry path, which the module will also take over:

// For Google Sheets, the actual row data lives under feed->entry.
if (isset($array['feed']) && isset($array['feed']['entry'])) {
 $array = $array['feed']['entry'];
}

4- Characteristics of the Migration

The first observation we will make is that the Google Spreadsheet Plugin is actually an extension of the Json Processing Plugin, as we can see in the class:

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\migrate\MigrateException;
use Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json;
use GuzzleHttp\Exception\RequestException;

/**
 * Obtain Google Sheet data for migration.
 *
 * @DataParser(
 *   id = "google_sheets",
 *   title = @Translation("Google Sheets")
 * )
 */
class GoogleSheets extends Json implements ContainerFactoryPluginInterface {
...
}

Also we can see that the GoogleSheet Plugins is marked as a Data Parser in its block annotations, sowe’ll need some more resources: a base source Plugin and a Data Fetcher (then the Google Spreadsheet Plugin will be the third part in the process).

Ok, What Source Plugins are available in my Drupal installation? Let’s see. Launching:

drupal debug:plugin migrate.source

We’ll get by prompt:

[email protected]:/var/www/html$ drupal debug:plugin migrate.source  
 --------------- --------------------------------------------------------- 
  Plugin ID       Plugin class                                             
 --------------- --------------------------------------------------------- 
  embedded_data   Drupal\migrate\Plugin\migrate\source\EmbeddedDataSource  
  empty           Drupal\migrate\Plugin\migrate\source\EmptySource         
  url             Drupal\migrate_plus\Plugin\migrate\source\Url            
 --------------- --------------------------------------------------------- 

[email protected]:/var/www/html$ 

Ok, as a Source Plugin we can use the class Url.php (our file is exposed by URL). In the Url.php class, we see that we need some king of data parser (The Google Spread Sheet class).

  /**
   * The data parser plugin.
   *
   * @var \Drupal\migrate_plus\DataParserPluginInterface
   */
  protected $dataParserPlugin;

And looking for a fetcher / handler, we can find out a data fetcher for http processing, the class Http.php ready to work with a Url Plugin as source:

/**
 * Retrieve data over an HTTP connection for migration.
 *
 * Example:
 *
 * @code
 * source:
 *   plugin: url
 *   data_fetcher_plugin: http
 *   headers:
 *     Accept: application/json
 *     User-Agent: Internet Explorer 6
 *     Authorization-Key: secret
 *     Arbitrary-Header: foobarbaz
 * @endcode
 *
 * @DataFetcher(
 *   id = "http",
 *   title = @Translation("HTTP")
 * )
 */
class Http extends DataFetcherPluginBase implements ContainerFactoryPluginInterface {

From another side, seems that the Url.php class requires url directions from its constructor method. We can use single URL directions or a set:

  /**
   * {@inheritdoc}
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration) {
    if (!is_array($configuration['urls'])) {
      $configuration['urls'] = [$configuration['urls']];
    }
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);

    $this->sourceUrls = $configuration['urls'];
  }

So by putting together the different parts we’re looking at, it looks like we’ll have to give a structured form to the elements, according to their order and position. In short, our first section for the Source would look like this:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: google_sheets
  urls: 'http://spreadsheets.google.com/feeds/list/1bKGbPbgeuXaBfcKetaDqoDimmYcerQY_hT1rqzw4TbM/1/public/values?alt=json'

5- Custom Module for Migration

We’ll create a new custom module called migration_google_sheet and with structure:

/project/web/modules/custom/  
                     \__migration_google_sheet/  
                         \__migration_google_sheet.info.yml
                           \__config/
                                \__install/
                                     \__migrate_plus.migration.migration.taxonomy_google_sheet.yml

Migration Description File: migrate_plus.migration.taxonomy_google_sheet.yml

langcode: en
status: true
dependencies:
  enforced:
    module:
      - migration_google_sheet
id: taxonomy_google_sheet
label: 'Migrating Taxonomy'
source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: google_sheets
  urls: 'http://spreadsheets.google.com/feeds/list/1bKGbPbgeuXaBfcKetaDqoDimmYcerQY_hT1rqzw4TbM/1/public/values?alt=json'
  fields:
    - name: id
      label: 'Id'
      selector: id
    - name: name
      label: 'Name'
      selector: name
    - name: description
      label: 'Description'
      selector: description
    - name: url
      label: 'Url'
      selector: url
    - name: published
      label: 'Published'
      selector: published
  ids:
    id:
      type: integer
process:
  name: name
  description: description
  path: url
  status: published
destination:
  plugin: 'entity:taxonomy_term'
  default_bundle: tags

So now, executing the migration from prompt:

drush en migration_google_sheet -y 
drush migrate:import taxonomy_google_sheet

[notice] Processed 30 items (30 created, 0 updated, 0 failed, 0 ignored) - done with 'taxonomy_google_sheet'

Well done! We just migrated thirty taxonomy terms from a Google spreadsheet:

Taxonomy Terms Just migrated

You can download or clone the custom Migration module from my gitlab repository.

6- :wq!

[embedded content]

Apr 10 2020
Apr 10

Everything around Test-driven development (TDD) is a very interesting and very motivating world and besides, these are topics with a certain antiquity. So it is very easy to find related contents. But the relationship between this area and Drupal is even more interesting: there are multiple options to implement testing of different types and different orientation (Unit Test, Kernel, JavaScript, functional &mldr;) so it can be complex to introduce in this world. Today I want to take advantage of the experience of porting to Drupal 8 a small contributed module to share examples about browser-focused Functional Testing in Drupal 8 (or Drupal 9). We will learn together with simple use cases. It may be a small slice (Functional Testing of a very small features) of a big cake (Testing in Drupal), but it will be a nice entry point. Follow me.

Picture from Unsplash, user National Cancer Institute, @nci

Table of Contents

1- Introduction
2- Arrangements
3- The BrowserTestBase class
4- Basic Scaffolding
5- Your tests
6- Running the test
7- Read More
8- :wq!

1- Introduction

Well, as I said in the obligatory introductory paragraph, everything related to Testing is extensive, very broad and combines an ungraspable conjunction of philosophical-theoretical elements with practical-technical issues&mldr;so I guess that’s why I was so happy when in the context of the migration to Drupal 8|9 of the contributed module humans.txt, Pedro Cambra as its maintainer proposed in an issue to provide the module with a certain type of test.
It was a great opportunity to use it as a simple, didactic and intuitive approach.

About the testing we are going to see in this article, it is important to situate it and give it context: we will make some types of browser tests, which are part of the test types that come from PHPUnit classes and resources. How are they related? Let’s see this introduction made by James G. Robertson in the Atlantic BT website:

PHPUnit can handle different types of tests in Drupal core: unit tests, kernel tests, and functional tests. With these three tests, you can confirm the quality and reaction of code on edge cases in different layers. Unit tests test functions, methods, and classes without requiring a database connection to run. On the other hand, kernel tests test the integration of module APIs and require a database connection to be configured to run. Functional tests test the entire system and require more setup than the others. Within the functional tests, there are both Browser and JavaScript tests. In addition to these PHP-based tests, you may also run core JavaScript tests using the Nightwatch framework.

Right, so what we’re going to do in this case has to do with testing actions to be performed in the web interface of our Drupal installation but running through code using classes that already provide methods for replicating “manual actions”. Intuitively&mldr; What can we think that we will need? Let’s advance a possible outline of our possible actions:

  1. Maybe, loading a URL.
  2. Perhaps, pressing buttons on a form.
  3. We may need to load values into this former form.
  4. Or also check users, roles and permissions.

So we’ll need classes and resources that allow us to reproduce these actions through code (so that they can be automated). This is just a sketch of our possible needs, as we must first be clear about which features we want to test. So the first guideline shared in this introduction will be: You must know well your needs to test.

Ok, and what is the features we’ll have to test? the Humans.txt contrib module was created years ago to offers a way for building the humans.txt file from within Drupal. For several months we have been working on its portability to Drupal 8 and its features are summarized in two main tasks:

  1. Generating an object as a humans.txt file with values loaded from a configuration form.
  2. Offering to create a link to the object/file in the <head> section of pages.

Mainly, these will be the features that we will have to test.

This article is intended as a theoretical - practical guide to browser-based functional testing for Drupal, based on a certain testing issue history of the Humans.txt module. It includes bugs, more bugs, errors, misses, mismatched versions&mldr;uploads and more uploads, testing bots&mldr;and above all&mldr; a conversational sample of pair between a maintainer and a contributor when working in parallel. It may not be very agile, but it is very real. And I think that here is the most interesting part of the article: exposing the work cycle combined with all the successes and failures.

2- Arrangements

Well, in this section I’ve included the fun little story about how I discovered that I didn’t have the necessary resources installed in the test environment I chose. Pay attention.

It all started when I switched environments and realized that I didn’t have phpunit installed&mldr;I always use DDEV (- Get more info about DDEV - ) as a tool for creating local development environments and in this case I switched to one that didn’t have a Drupal installation with the –dev resources. So if this is your case, take advantage and review the following steps.

Environment

  • First, stop your local apache, if exists: sudo /etc/init.d/apache2 stop

  • Second, up with your DDEV project: ddev start. Make sure your DDEV project is up, running and functioning normally. You’re going to execute PHPUnit from within the DDEV main web container and you’ll use the db container too.

  • Third, go to the web container and install with composer the next resources:

    ddev ssh
    composer require phpunit/phpunit
    
    

Note: In my first iteration, I launched one request like this (opening the phpunit’s version to the latest available) and it caused me a lot of problems when trying to use PHPUnit:

Could not use "\Drupal\Tests\Listeners\HtmlOutputPrinter" as printer: class does not exist
PHPUnit 9.1.1 

What’s the problem? Drupal 8.x is not compatible with PHPUnit in that version (In fact there’are some issues proposing an update to PHPUnit 8 for Drupal 9, like this), so it’s better to uninstall it and request a slightly more limited version, which is not the last one, something around PHPUnit 7 for example.

For the rest of the dependencies, I recognize that I was playing to solve them just following the successive error messages trying to launch several tests again, like Class "Symfony\Bridge\PhpUnit\SymfonyTestsListener" does not exist and many others. In summary I have installed all the following resources in the approximate versions indicated:

composer remove phpunit/phpunit
composer require phpunit/phpunit:^7 # In my case, this installed phpunit 7.5.20
composer require symfony/phpunit-bridge:^3.4.3
composer require behat/behat:^3.4
composer require behat/mink:^1.8
composer require behat/mink-goutte-driver:^1.2

PHPUnit in Drupal Core

The next step was adjust the use of PHPUnit from the file provided by Drupal, present in /project/web/core/phpunit.xml.dist. It’s necessary to make a copy of the file and rename it as phpunit.xml simply, leaving this copy in the same location /core.

Now we need modify two environment variables within the copied file. Beware:

<!-- Example SIMPLETEST_BASE_URL value: http://localhost -->
<env name="SIMPLETEST_BASE_URL" value="http://localhost"/>
<!-- Example SIMPLETEST_DB value: mysql://username:[email protected]/databasename#table_prefix -->
<env name="SIMPLETEST_DB" value="mysql://db:[email protected]/db"/>

As you can see, now we need an internal URL and all the connection data to your database. Ok, in my case as I’m using DDEV I can extract the information from my prompt, just launching ddev describe. Remember that we’re looking for a string formatted as: mysql://username:[email protected]/databasename. And in the DDEV context, by default, all these data are the same: db. Let’s see:

Showing values from a DDEV installation for Drupal

Like an extra, I’ve configured a folder to save results from tests in a HTML format using the variables:

<env name="BROWSERTEST_OUTPUT_DIRECTORY" value="/var/www/html/web/sites/default/simpletest/browser_output/"/>
<env name="BROWSERTEST_OUTPUT_BASE_URL" value="http://migrations.ddev.site"/>

Where BROWSERTEST_OUTPUT_DIRECTORY is used as the directory where the output data will be saved by PHPUnit and needs to be an absolute local path, in my case the result HTML from test goes to /var/www/html/web/sites/default/simpletest/browser_output/ and while BROWSERTEST_OUTPUT_BASE_URL help to register the future links to the saved output, in my case is: migrations.ddev.site and is just the name of the DDEV project that I’m using. This will write all the output from the executed test in the directory using HTML format, and so you can see the pages that the emulated browser visited during the test. Theses HTML pages are saved as files and linked under the URL.

When you launch the test, Drupal 8 will use a simulated browser to execute actions and check assertions, is like an emulator offered by Mink, just a pure headless browser from the installed dependency behat/mink.
Now remember that you must ensure the write permissions in the destiny folder for the user that will launch the test. You can see my whole configuration for PHPUnit using DDEV here in the next gist:

So the second guideline shared in this section will be: tune up your environment well.

When you have available all the libraries and dependencies along with the configuration of the phpunit.xml file, you can try to run the tests that bring many modules through a console execution instruction.
The format of the instruction we will use will be related to the position for ourselves within the path /project/web/ and launch the call to phpunit in a format like this:

../vendor/bin/phpunit -c core modules/contrib/config_inspector 

Where the param -c points to the localization of the phpunit.xm file and it will run all the test located in the marked direction (in this example will be executed tests from the Config Inspector Contrib Module).

Executing test from the Config Inspector Contrib Module

UPDATE (15/04/2020) Well, After a change in the renaming of the /test folder to /tests, the Drupal.org Testing bot has started working and has detected&mldr;that two methods I used in the Test Class were not available, they didn’t exist. At that moment I thought about the version of phpunit I was using (and that I chose and installed myself) and as recommended for Drupal 8 it should be PHPUnit 6.5 and not 7.5 as I am using. So downgrading and adapt the methods to others available&mldr;Which are?

These methods:

$this->assertStringNotContainsString($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is not shown in the -head- section.', $this->fileLink));
$this->assertStringContainsString($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is shown in the -head- section.', $this->fileLink));

Should be replaced by these others, compatible with PHPUnit 6.5:

$this->assertNotContains($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is not shown in the -head- section.', $this->fileLink));

$this->assertContains($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is shown in the -head- section.', $this->fileLink));

From the Assert.php Class available in vendor/phpunit/phpunit/src/Franework/Assert.php

So note to my future me: Drupal 8 + PHPUnit 6.5, man.

3- The BrowserTestBase class

Now we are going to deal with a fundamental PHP class for this test case we are going to make, the BrowserTestBase.php class of the Drupal core test context, path: /core/tests/Drupal/Tests/BrowserTestBase.php (Don’t confuse it with an already deprecated version from the context of the simpletest core module called so BrowserTestBase).

This abstract class extends the TestCase class from PHPUnit (one of the main reasons why we need the use of PHPUnit here) and uses a lot of PHP traits providing many methods from different origins, finally grouped by this base class that we’ll have to extend for our goals. In general terms, we know that abstract classes are classes that are not instantiated and can only be inherited, thus transferring an obligatory functioning to the daughter classes. In this example, using BrowserTestBase we’ll have nearly forty methods available within the class (there are dozens of possible assertions) to perform specific checks on actions to be performed through the browser.

The info from the class is very explicit: You must use the class for functional Drupal test (ok), You have to put your new classes extending BrowserTestBase as a base class in path: /modules/custom/your_custom_module/test/src/Functional and avoid using t() function for translate unless you’re testing translation functionality.

/**
 * Provides a test case for functional Drupal tests.
 *
 * Tests extending BrowserTestBase must exist in the
 * Drupal\Tests\yourmodule\Functional namespace and live in the
 * modules/yourmodule/tests/src/Functional directory.
 *
 * Tests extending this base class should only translate text when testing
 * translation functionality. For example, avoid wrapping test text with t()
 * or TranslatableMarkup().
 *
 * @ingroup testing
 */  

Remember what we said before about knowing well what kind of actions you want to test? Think that this class will provide us with functions that will align possible actions with concrete methods to reproduce mechanics through code.
For example:

As you see, the possibilities are many and only require that we have some knowledge of the capabilities that the BrowserTestBase class offers to automate our tests.

4- Basic scaffolding

I will now share some actions that I need to do before I can start testing functions. As my goal is to prepare test for the contrib module “Humans.txt” the first thing I will do is download the module, install it and make sure I’m in the last commit of the branch I’m interested in testing (the one in version 8.x).

As I’m in a DDEV based context the first thing I’ll do is to access my web container and from there perform the rest of initial commands:

ddev ssh
composer require drupal/humanstxt
drupal moi humanstxt
cd modules/contrib/humanstxt
git checkout 8.x-1.x

Now I’m in the initial point for my job. Another aspect that I must evaluate is if it is convenient (or not) to create a submodule inside Humanstxt as a “test”, with its own installation and testing paths that respond to the same controllers as the main module, or if on the contrary it’s excessive (for now) for the testing of this module. At this moment I think I will create the tests directly, testing against the resources of the main module.

5- Your tests

In our current context, using a new class HumansTxtBasicText which extends BrowserTestBase, every method inside our class will be a unique test by itself, but there will a lot of assertions more to check, cause of in one of our used functions, we’re calling some internal assertions. For example, if we’re using the function drupalCreateUser(['permission name']) from the UserCreationTrait and originally named as createUser , with our direct assertions, we’re going to check other internals just like:

$valid_user = $account->id() !== NULL;
$this->assertTrue($valid_user, new FormattableMarkup('User created with name %name and pass %pass', ['%name' => $edit['name'], '%pass' => $edit['pass']]), 'User login');

Due to this, you’ll see more assertions than yours (or the explicit yours), like this feedback when I’m only using four explicit assertions:

Testing modules/contrib/humanstxt
..                                     2 / 2 (100%)

Time: 1.12 minutes, Memory: 4.00 MB

OK (2 tests, 16 assertions)

Ok, don’t fear. Let’s move on.

The next important point to know is that all your methods should start with the word ‘test’ in lowercase. So any method using this as prefix with public visibility will be automatically detected by PHPUnit and will be launched too.

And there’s one more important thing: any stuff you make within a method/test, won’t live outside it. So, for example if in a test you’re creating a new user, then in the next test you’ll have to create it again. Ok? That’s because every time you run a method/test, each test function will have a full new Drupal instance to execute tests.

Thinking about how to do it in an organized way, I thought about doing it in three functional blocks:

  1. First, check the access control to the configuration form and its fields, that is, check the behavior for different roles (administrators, users with basic and anonymous permissions).

  2. Then, check complementary questions with the information collected in the headers of the answers to requests or the cache tags stored.

  3. Finally, test if the file is created, kept accessible and its link is located in the right way. Everything for different user profiles: administrators, basic and anonymous permissions.

So, I’ll create an new folder within the module with path: humanstxt/test/src/Functional/, using a new class called HumansTxtBasicTest.php with namespace: Drupal\Tests\humanstxt\Functional.

Phase one: Checking access control

We’re going to test if three different users, one with admin permissions, other as a basic user without specific permissions and a last anonymous user can reach the configuration page for the Humans.txt module. Theoretically, only the admin user can access and the basic user or the anonymous cannot reach the config page. Let’s see.

First I’m gonna test the access for admins:

  /**
   * Checks if an admin user can access to the configuration page.
   */
  public function testHumansTxtAdminAccess() {
    // Build initial paths.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->adminUser = $this->drupalCreateUser(['administer humans.txt']);

    // Login for the former admin user.
    $this->drupalLogin($this->adminUser);

    // Access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // Check the response returned by Drupal.
    $this->assertResponse(200);
  }

I’m using the Url class cause I don’t like work with explicit paths in testing. So, if a path changes in a routing.yml file, the test continues being valid and it will be run normally.

And then, I’ll check the access for the pair of users no-admin, using the same test:

  /**
   * Checks if a non-administrative user cannot access to the config page.
   */
  public function testHumansTxtUserNoAccess() {
    // Build initial path.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->normalUser = $this->drupalCreateUser(['access content']);

    // Login for the former basic user.
    $this->drupalLogin($this->normalUser);

    // Try access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // Check the response returned by Drupal.
    $this->assertResponse(403);

    // Logout as normal user and repeat the former cycle as anonymous user.
    $this->drupalLogout();
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(403);
  }

Now It’s time to check the access to the fields of the configuration form.

  /**
   * Checks if an administrator can see the fields.
   */
  public function testHumansTxtAdminFields() {
    // Build initial path.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->adminUser = $this->drupalCreateUser(['administer humans.txt']);

    // Login for the the former admin user.
    $this->drupalLogin($this->adminUser);

    // Access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // The textarea for configuring humans.txt is shown.
    $this->assertSession()->fieldExists('humanstxt_content');

    // The checkbox for configuring the creation of the humanstxt link is shown.
    $this->assertSession()->fieldExists('humanstxt_display_link');
  }

  /**
   * Checks if a non-administrative user cannot use the configuration page.
   */
  public function testHumansTxtUserFields() {
    // Build initial path.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->normalUser = $this->drupalCreateUser(['access content']);

    // Login for the former basic user.
    $this->drupalLogin($this->normalUser);

    // Access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // The textarea is not shown for basic users.
    $this->assertNoFieldById('edit-humanstxt-content', NULL);

    // The checkbox is not shown for basic users.
    $this->assertNoFieldById('edit-humanstxt-display-link', NULL);
  }

Phase Two: Checking Complementary Items

In this block I would like to check stuff like the headers of the returned object/file or the cache tags.

  /**
   * Checks if the header is right.
   */
  public function testHumansTxtHeader() {
    // Build initial path.
    $humanstxt_path = Url::fromRoute('humanstxt.content', [], ['absolute' => FALSE])->toString();

    // Access to the path of humans.txt object/file.
    $this->drupalGet($humanstxt_path);

    // Check the returned response.
    $this->assertResponse(200);

    // Check if the file was served as text/plain with charset=UTF-8.
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');
  }

  /**
   * Checks if cache tags exists.
   */
  public function testHumansTxtCacheTags() {
    // Build initial path.
    $humanstxt_path = Url::fromRoute('humanstxt.content', [], ['absolute' => FALSE])->toString();

    // Access to the path of humans.txt object/file.
    $this->drupalGet($humanstxt_path);

    // Check the returned response.
    $this->assertResponse(200);

    // Check the related cache tag.
    $this->assertCacheTag('humanstxt');
  }

Phase Three: Checking the delivered content

Ok, in order to check the content file or the link write in the section, I thought to make a central and unified test that would group everything for the different user profiles to be tested. What kind of things am I interested in checking? Well I would like to test if the access to the configuration form is possible for administrators (although this was already tested in a previous test) and then creating a new configuration of the Humans.txt in order to test if the saved content corresponds with the one visible inside the object/file (testing also the access to the file).
Finally I would like to test if the link associated to the meta tag in the section of the pages is being loaded, choosing one at random.

  /**
   * Checks if humans.txt file is delivered for Different Users as was configured.
   */
  public function testHumansTxtConfigureHumansTxtDifferentUsers() {
    // Build initial paths.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();
    $humanstxt_path = Url::fromRoute('humanstxt.content', [], ['absolute' => TRUE])->toString();
    $humanstxt_link = '<link rel="author" type="text/plain" hreflang="x-default" href="https://www.therussianlullaby.com/blog/functional-testing-for-browser-in-drupal-using-phpunit//' . $humanstxt_path . '">';

    // Create users for testing.
    $this->adminUser = $this->drupalCreateUser(['administer humans.txt']);
    $this->normalUser = $this->drupalCreateUser(['access content']);

    // Login as admin and get the config form page.
    $this->drupalLogin($this->adminUser);
    $this->drupalGet($humanstxt_config);

    // Load a new configuration for Humans.txt file and submit config Form.
    $test_string = "# Testing Humans.txt {$this->randomMachineName()}";
    $this->submitForm(['humanstxt_content' => $test_string, 'humanstxt_display_link' => TRUE], t('Save configuration'));

    // Check the object/file created.
    $this->drupalGet($humanstxt_path);
    $this->assertResponse(200);

    // Test header.
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');

    // Get page content.
    $content = $this->getSession()->getPage()->getContent();

    // Test the assert- if exists the test_string in the content.
    $this->assertTrue($content == $test_string, sprintf('Test string [%s] is shown in the configured humans.txt file [%s].', $test_string, $content));

    // Test if the link to the object/file is in HTML <head> section for Admins.
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(200);
    $tags = $this->getSession()->getPage()->getHtml();
    $this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link [%s] is shown in the HTML -head- section from [%s].', $humanstxt_link, $tags));

    // Logout as admin and login as normal user.
    $this->drupalLogout();
    $this->drupalLogin($this->normalUser);

    // Repeat the previous cycle now as normal user.
    $this->drupalGet($humanstxt_path);
    $this->assertResponse(200);
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');
    $content = $this->getSession()->getPage()->getContent();
    $this->assertTrue($content == $test_string, sprintf('Test string [%s] is shown in the configured humans.txt file [%s].', $test_string, $content));
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(403);
    $tags = $this->getSession()->getPage()->getHtml();
    $this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link [%s] is shown in the HTML -head- section from [%s].', $humanstxt_link, $tags));

    // Logout as normal user.
    $this->drupalLogout();

    // Now a third iteration as anonymous user.
    $this->drupalGet($humanstxt_path);
    $this->assertResponse(200);
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');
    $content = $this->getSession()->getPage()->getContent();
    $this->assertTrue($content == $test_string, sprintf('Test string [%s] is shown in the configured humans.txt file [%s].', $test_string, $content));
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(403);
    $tags = $this->getSession()->getPage()->getHtml();
    $this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link [%s] is shown in the HTML -head- section from [%s].', $humanstxt_link, $tags));
  }

To check the insertion of the tag in I’ve used a double version of the former codeblock, playing with the values of the element from the Configuration Form: 'humanstxt_display_link' => FALSE and changing it for the next codeblock. This checkbox when false doesn’t insert the link to the humans.txt object/file in , and set to TRUE it will put the tag. The occurrence of the mentioned tag in is managed using a special kind of assertion method from PHPUnit called assertStringContainsString, available in my installed version of the testing framework (7.5) and its inverse for negative versions: assertStringNotContainsString().

So my idea is using the getSession() method from BrowserTestBase class which returns a Mink Session Object. This session object from Mink offers a method called getPage() that can return an object DocumentElement and use its method getHtml() that returns al the HTML code formatted as string. All of this is in the line:
$tags = $this->getSession()->getPage()->getHtml(); and in the variable $tags I’ll save all the HTML response ready to search the link, using the variable as haystack. In any case, all the HTML code from a page is too much for processing, and we don’t need to get all the HTML code, so we’ll get a substring cutting the output from the getHtml() method, extracting up to two thousand characters, enough to evaluate the whole section.

// All the HTML code is too much, we just need to inspect the <head> section.
$tags = substr($this->getSession()->getPage()->getHtml(), 0, 2020);
$this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link: [%s] is NOT shown in the head section from [%s] and this shouldn\'t happen.', $humanstxt_link, $tags));

Finally you can see this first version of the TestClass as Gist in Github. After uploading the patch to the issue there will surely be revisions and changes to the patch, so I promise to link the final version of the Test class that will be committed to the 8.x-1.x branch.

First version of the Testing Class in Gist.

UPDATE (14/04/2020) After the first feedback from the maintainer, I have taken all his indications and prepared a new refactored version of the Test class. I’ve reduced, adapted and simplified several things following the feedback from the maintainer. Now the class is using the setUp() method, the numbers of methods was reduced from eight to five, the assertions from hundred-six to sixty-four and the codelines from almost three hundred lines to hundred and forty lines of code.

I’m also using simple paths, except in the path to the humans.txt file to be inserted in the section as link, since originally the absolute path to the file is being saved: see this Issue and the last uploaded patch.

This second reduced version of the class seems to pass the tests well and gives positive results in all tests and assertions.

Executing the second version of the test for Humans.txt module

Many thanks to Pedro Cambra from Cambrico for the feedback and revisions, I am very grateful.

Here is the second version of the test class after refactoring:

Second version of the Testing Class in Gist.

UPDATE (15/04/2020) After starting the Drupal.org testing bot and checking that my version of PHPUnit was not the one needed for the Drupal core, and has been necessary to make changes in two methods. The semantics of the messages in those methods were set for testing by me and were confusing, were also wrong. They have been changed and now the tests are going well.

This will be the final version of the class of Tests that will be committed to the repository:

By the way, with this last commit, the migration to Drupal 8 of the contributed module Humans.txt has been completed. The module has been finally ported to Drupal 8 | Drupal 9. Humans.txt Releases. Mission accomplished.

6- Running the test

Well, and now with our test stored and the phpunit configuration initially resolved, it’s time to run our test and observe the results. To do this, in my case I’m located in /project/web/ and with phpunit.xml placed in /project/web/core/, I launch the instruction:

../vendor/bin/phpunit -c core modules/contrib/humanstxt

Which throws all the test located inside the humanstxt contrib module&mldr;

Executing test from the Humans.txt Contrib Module

As we can see, we’re executing eight tests with hundred and six assertions. Due to our phpunit.xml configuration, these test are writing over sixty four HTML files based in results using the browser emulator from Mink, thanks to which you will be able to reproduce certain actions by opening the files in your web browser.

In the next caption we can see the HTML output nº64 generated from one of last assertions in the code, just when an anonymous user try to get the Humanstxt configuration form and receive and error message with code HTTP 403 ‘Forbidden’:

Executing test from the Humans.txt Contrib Module

As you can see, you can move through the files using the Previous | Next Links in header, connecting all the secuence of HTML results from tests.

7- Read More

8- :wq!

[embedded content]

Apr 01 2020
Apr 01

Recently I was preparing some events to intercept requests in Symfony and sharing some approaches with my colleagues. Then I discovered that in my environment the topic of Event Management in Drupal (dispatching events, subscribing events) was not very known, so I prepared some snippets to share and from there I thought to write some introductory article. Undoubtedly, in the latest versions of Drupal, certain components of Symfony have not only made their appearance, but also have been gaining importance and surely this will extend further in time, given its elasticity and fully integrated OOP approach. One of these components is the HttpKernel, a very valuable subsystem for building request-response relationships in a technology (Silex, Symfony, Drupal).

Picture from Unsplash, user Noiseporn, @noiseporn

Table of Contents

1- Introduction
2- Concepts of Event and Event Subscriber
3- Building the Event Dispatcher
4- Building the Event Subscriber
5- Read More
6- :wq!

1- Introduction

As we said, HttpKernel very flexible that allows us to structure relationships requests - response. Let’s see what it says about itself in its Symfony documentation:

The HttpKernel component provides a structured process for converting a Request into a Response by making use of the EventDispatcher component. It’s flexible enough to create a full-stack framework (Symfony), a micro-framework (Silex) or an advanced CMS system (Drupal).

In this context where I wanted to practice a little bit with the concepts of “Event” and “Event Subscription”. Let’s see.

2- Event and Event Subscriber

Concept of Event: Event systems are ways to allow extensions in many applications. Generally there are some similar topics and concepts, wich usually include common elements:

  1. Event Registry: Just a place where event subscribers will be grouped.
  2. Event Subscribers / Event Listeners: Callable functions or methods that react to an event.
  3. Event Dispatcher: The way the event is triggered.
  4. Event Context: Some information shared for the subscribers of an event.

Getting information about services and events: Services? Events? but&mldr;how do I know how many I have available at my Drupal installation? Well you can use Drupal Console with a pair of commands to get the info:

drupal debug:container #Get list of available services in a Drupal site.
drupal debug:event #Get a list of available events in your Drupal site.

Events in Drupal: From Drupal 8, the Symfony Event Dispatcher is one the most important components in a Drupal installation. This component is responsible for launching notifications from your applications. You can be listening these notifications and returning some responses executing your custom code.
In Drupal 8, events are defined in a class and declared in a YML file, then dispatched using a function call on the core Drupal event_dispatcher service.

What is an Event Subscriber: A very common task in the Drupal development it’s intercept some request from the browser to your Drupal, getting something interesting on it, and then redirect the user to another location in the site.

Well, in Drupal 7 we were using a combination of hook_init() with a weird function called drupal_goto(), but now in Drupal >= 8 we’re doing this using with the symfony model for subscribing us to the kernel.request event.

We can launch a redirect from a Controller, but in this case we want to test the topic of Event-Subscribers in Drupal.

3- Building the Event Dispatcher

Note: The example requires that you have previously created a new custom module with its custom_module.info.yml in your Drupal installation.

Note II: This example is from Drupal 8 Module Development (second edition) by Daniel Sipos, the Holy Bible of the Drupal Development.
Pag 65: “We will create an event to be dispatched whenever our HelloWorldSalutation::getSalutation() method is called. The purpose is to inform other modules that this happened and potentially allow them to alter the message."

3.1- Creating an event class

File: SalutationEvent.php
Location: /custom_module/

<?php
namespace Drupal\custom_module;

use Symfony\Component\EventDispatcher\Event;

/**
 *Event class to be dispatched from the HelloWorldSalutation service.
 */
class SalutationEvent extends Event {

const EVENT = 'hello_world.salutation_event';

/**
 * The salutation message.
 *
 * @var string
 */
protected $message;

/**
 * @return mixed
 */
public function getValue() {
    return $this->message;
}

/**
 * @param mixed $message 
 */
public function setValue($message) {
    $this->message = $message;
 } 
}


3.2- Injecting the Event Dispatcher service

services:
  hello_world.salutation:
    class: 'Drupal\hello_world\HelloWorldSalutation'
    arguments: ['@config.factory', '@event_dispatcher']
    tags:
      - { name: salutation }

4- Building the Event Subscriber

Given an internal path in Drupal, we’re going to define a new event subscriber to catch the request, review certain conditions of the current user of Drupal and if there’s a matching, then we’ll set a new redirect to another route in your system bypassing the process.

We know that the system allows subscribers to change data before the business logic uses it for something. So in order to register the new event subscriber, you have to create a new service related with the former class (that implements the interface) and tagged with the event_subscriber key.

In this example, when a user request our custom route, we’ll check if the current user has assigned a “non_grata” role and if the result is positive, then we’ll redirect him to the home page of the web portal, without allowing him to access our route. In order to execute this, we’ll need two injected services: the AccountProxyInterface for get the user related data and CurrentRouteMatch to test the requested route.

Required parts:

  1. A declared internal route (file custom_module.routing.yml in custom_module/)
  2. A new declared service (file custom_module.services.yml in custom_module/)
  3. A new class CustomModuleRedirectSubscriber implementing the EventsSubscriberInterface (file in custom_module/src/EventSubscriber/)

Note: The example requires that you have previously created a new custom module with its custom_module.info.yml to register in your Drupal.

4.1- The route

custom_module.hello:
  path: '/hello'
  defaults:
    _controller: '\Drupal\custom_module\Controller\CustomModuleController::helloWorld'
    _title: 'Our Testing Route'
  requirements:
    _permission: 'access content'

4.2- The Service

 custom_module.redirect_subscriber:
    class: '\Drupal\custom_module\EventSubscriber\CustomModuleRedirectSubscriber'
    arguments: ['@current_user', '@current_route_match']
    tags:
      - { name: event_subscriber }

4.3- The Class CustomModuleRedirectSubscriber.php

<?php


namespace Drupal\custom_module\EventSubscriber;


use Drupal\Core\Routing\CurrentRouteMatch;
use Drupal\Core\Routing\LocalRedirectResponse;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Drupal\Core\Session\AccountProxyInterface;
use Symfony\Component\HttpKernel\Event\GetResponseEvent;
use Symfony\Component\HttpKernel\KernelEvents;

/**
 * Class CustomModuleRedirectSubscriber Subscribes to the Kernel Request events.
 */
class CustomModuleRedirectSubscriber implements EventSubscriberInterface {

  /**
   * @var \Drupal\Core\Session\AccountProxyInterface
   */
  protected $currentUser;

  /**
   * @var \Drupal\Core\Routing\CurrentRouteMatch
   */
  protected $currentRouteMatch;

  /**
   * HelloWorldRedirectSubscriber constructor.
   *
   * @param \Drupal\Core\Session\AccountProxyInterface $currentUser
   * @param CurrentRouteMatch $currentRouteMatch
   */
  public function __construct(AccountProxyInterface $currentUser, 
                              CurrentRouteMatch $currentRouteMatch) {
    $this->currentUser = $currentUser;
    $this->currentRouteMatch = $currentRouteMatch;
  }



  /**
   * {@inheritdoc}
   */
  public static function getSubscribedEvents() {
    $events[KernelEvents::REQUEST][] = ['onRequest', 0];
    return $events;
    }

  /**
   * Handler for the kernel request event.
   *
   * @param \Symfony\Component\HttpKernel\Event\GetResponseEvent $event
   *
   */
  public function onRequest(GetResponseEvent $event){
    $route_name = $this->currentRouteMatch->getRouteName();
    if ($route_name !== 'custom_module.hello') {
      return;
    }

    $roles = $this->currentUser->getRoles();
    if(in_array('non_grata', $roles)) {
      $url = Url::fromUri('internal:/');
      $event->setResponse(new LocalRedirectResponse($url->toString()));
   }
  }
}

5- Read More

Here you can find much more information about the concepts previously exposed and consult a multitude of use cases and examples:

6- :wq!

[embedded content]

Mar 17 2020
Mar 17

As I said in the previous post, during these months I will be playing with migrations, preparing some cases for a future (I hope) book. Well, during these days of confinement, I intend to continue with small articles around here to show experiences related to migrations.

In the former post, I was writing about migrations in Drupal from a point of view based in the look for a tool-box, just a set of basic resources in order to focusing a migration.

There’s a lot of information to process about it and some more concepts, technics and tactics to resolving a migration, you can be sure. So this month I want to write something that allows me play with migrations, maybe more practical than theorical.

This article was originally published in https://davidjguru.github.io
Picture from Unsplash, user Émile Séguin, @emileseguin

Table of Contents

1- Introduction
2- Arrangements
3- Approach
4- Migrations
5- Key Concepts
6- Resources
7- :wq!

This article is part of a series of posts about Drupal Migrations

1- Thinking about Drupal Migrations (I): Resources
2- Thinking about Drupal Migrations (II): Examples

1- Introduction

The Drupal Migration API can be one of the most interesting, but also one of the most complex, since its activities are often related to classes and methods of other Drupal APIs (so it’s especially particular when debugging). In any case, and as the amount of concepts can be overwhelming, I think we could practice migration mechanics through a couple of exercises.

Well, for this article I had proposed to model two different migration processes, under a point of view that could be summarized as “primum vivere, deinde philosophari” (first you experiment, then you theorize). This is why I have decided to organize it in a particular way:

  • The first thing to say is that the two processes are divided into sections that are common to both and instead of finishing one and starting the next one, both go in parallel (you choose your own adventure).

  • Then, Only at the end of this post you will find some key concepts used in this article. First we gonna to play with the structures, then we’ll understand them.

So, in the next steps, we’ll working around two certain experiencies:

  1. Migrating Data from a embedded format (maybe the most simple example of Drupal migrations).

  2. Migration Data from a classical CSV file format (just a little more complex than the previous example).

Both of the cases are perhaps the most basic scenarios for a migration, so I recommend reading this article for those who want to get started on its own mechanics, as a practical complement to get into Drupal migrations.

2- Arrangements

First case: Migrating embedded data

For our first case we will need, on the one hand, to enable the Migrate module of the Drupal core, and on the other hand, to download and install a contributed module to be able to manage migrations.

From the different options we have, we are going to choose migrate_run, which we have already mentioned in the previous post and could be interpreted as a light version of migrate_tools (although it’s actually a fork of the project): both of wich provide drush commands to run migrations, so if you have migrate_tools installed you must uninstall it in order to avoid collide with migrate_run.

As a curious note, the first lesson here is that for running Drupal migrations, neither migrate_plus nor migrate_tools are “hard” dependencies, that is, we can implement migrations without having these modules enabled in our Drupal installation.

By the way I have to say that it’s important to know that migrate_run is optimized for Drush 9 and later. If you use Drush 8 you will have to use an adapted version, like the Alpha 4, which was still prepared for Drush 8.

Using Composer and Drush:

composer require drupal/migrate_run
drush pmu migrate_tools # If you need
drush en migrate migrate_run -y
drush cr

Using Drupal Console:

composer require drupal/migrate_run
drupal mou migrate_tools # If you need
drupal moi migrate migrate_run

And you will see in the path /admin/modules:

Enabling Migrate and Migrate Run modules

Building the resources

Now, we’re going to create a new custom module for our first Migration:

cd project/web/modules/custom
mkdir migration_basic_module

Then, the migration_basic_module.info.yml file with content:

name: 'Migration Basic Module'
type: module
description: 'Just a basic example of basic migration process.'
package: 'Migrations Examples 2000'
core: 8.x
dependencies:
  - drupal:migrate

Create the new migration definition file with path: /migration_basic_module/migrations/basic_migration_one.yml.

In our new declarative file basic_migration_one.yml, which describes the migration as a list of parameters and values in a static YAML-type file, we will include the embedded data of two nodes for the content type “basic page” to be migrated, loading only two values:

  1. A title (a text string).
  2. A body (A text based on the ChiquitoIpsum generator*, http://www.chiquitoipsum.com).

*Chiquito de La Calzada was a national figure in the Spanish state, a legendary comedian.

basic_migration_one.yml

id: basic_migration_one
label: 'Custom Basic Migration 2000'
source:
  plugin: embedded_data
  data_rows:
    -
      unique_id: 1
      page_title: 'Title for migrated node - One'
      page_content: 'Lorem fistrum mamaar se calle ustée tiene musho pelo.'
    -
      unique_id: 2
      page_title: 'Title for migrated node - Two'
      page_content: 'Se calle ustée caballo blanco caballo negroorl.'
  ids:
    unique_id:
      type: integer
process:
  title: article_title
  body: article_content
destination:
  plugin: 'entity:node'
  default_bundle: page

And this will be the structure of the new custom module for basic migration example:

/project/web/modules/custom/  
                     \__migration_basic_module/  
                         \__migration_basic_module.info.yml  
                             \__migrations/  
                                 \__basic_migration_one.yml  

Enabling all the required modules using Drush:

drush pm:enable -y migrate migrate_run migration_basic_module
drush cr

Or using Drupal Console:

drupal moi migrate migrate_run migration_basic_module

Second Case: Migrating from csv files

For this second case we are going to deactivate migrate_run (if applicable) and activate the superset of modules: migrate, migrate_plus and migrate_tools. Besides, for the treatment of CSV files we are going to use a Source Plugin stored in a contrib module called Migrate Source CSV migrate_source_csv. This contrib module in its version 3.x is using league/csv for processing CSV files. Ok, let’s go. So using Composer + Drush:

composer require drupal/migrate_plus drupal/migrate_tools drupal/migrate_source_csv
drush pmu migrate_run # If you need 
drush en migrate migrate_plus migrate_tools migrate_source_csv -y
drush cr

So, now in the path /admin/modules/:

Enabling Migrate and Migrate Plus Migrate Tools

Building the resources

We’re going to create another new custom module for our second Migration:

cd project/web/modules/custom
mkdir migration_csv_module

With a new migration_csv_module.info.yml file:

name: 'Migration CSV Module'
type: module
description: 'Just a basic example of basic migration process with a CSV source.'
package: 'Migrations Examples 2000'
core: 8.x
dependencies:
  - drupal:migrate
  - drupal:migrate_tools
  - drupal:migrate_plus

In this example we’re going to require a declarative file of the migration too (as in the previous case) but with the exception that we’re going to locate it in a different place. This will be placed in the /migration_csv_module/config/install/ path.

The structure will look like this just now:

/project/web/modules/custom/  
                     \__migration_csv_module/  
                         \__migration_csv_module.info.yml
                          \__csv/
                               \_migration_csv_articles.csv
                           \__config/
                                \__install/
                                     \__migrate_plus.migration.article_csv_import.yml

So we need a csv with original data to migrate. It’s easy to solve this using web tools like Mockaroo, a pretty good random data generator. I’ve created a CSV file with some fields like: id, title, body, tags, image. Download it from here. This file will be our datasource for the Migration process. Ok, by now create the directories for the module and put the new custom CSV in the /csv path:

CSV Migrate module structure

And now, our migrate_plus.migration.article_csv_import.yml file (In later sections we will explain its construction and sections):

uuid: 1bcec3e7-0a49-4473-87a2-6dca09b91aba
langcode: en
status: true
dependencies: {  }
id: article_csv_import
label: 'Migrating articles'
source:
  plugin: csv
  path: modules/custom/migration_csv_module/csv/migration_csv_articles.csv
  delimiter: ','
  enclosure: '"'
  header_offset: 0
  ids:
    - id
  fields:
    -
      name: id
      label: 'Unique Id'
    -
      name: title
      label: Title
    -
      name: body
      label: 'Post Body'
    -
      name: tags
      label: 'Taxonomy Tag'
    -
      name: image
      label: 'Image Field'
process:
  title: title
  body: body
  tags: field_tags
  image: field_image
  type:
    plugin: default_value
    default_value: article
destination:
  plugin: 'entity:node'

Okay, we now have all the resources we need to create our new migration. Now let’s see how we approach the process.

3- Approaches

We’re going to describe the different approaches that we will apply to our example cases, in order to understand them better.

First case: Migrating embedded data

In this first case, we considered making the lightest possible case of migration in Drupal: Only two nodes with two basic fields each under an embedded format: the lightest possible.

Also, in this example we are going to use for the three ETL phases of the migration (Extract, Transformation and Loading) processing plugins already provided by Drupal (we will not develop any custom plugin). If you don’t know anything about the concept of Migration Plugins, please stop by for a moment and back here to read a little introduction to the topic.

To make things lighter, we will keep the “lite” version of Migration Tools, Migrate Run. Besides, we will only use the basic commands without any other options or complementary parameters, only with the basic argument of the migration file identifier.

Second Case: Migrating from csv files

For this execution, I would like to play with something pretty interesting&mldr;due to we’ll running this second migration example as configuration, I was thinking that will be funny do the inverse road&mldr;Yes, I propose not to install (activate, drush enable) the new custom module for CSV and leave it&mldr;only as storage for the CSV file.

Let’s move and run the migration from somewhere else. Surprise. Visit the path /admin/config/development/configuration/single/import into your Drupal installation and we’ll see there!.

4- Migrations

First case: Migrating embedded data

Getting info about the available migrations

drush migrate:status
drush ms

Output from console:
----------------- -------- ------- ---------- ------------- --------------------- 
  Migration ID      Status   Total   Imported   Unprocessed   Last Imported        
----------------- -------- ------- ---------- ------------- --------------------- 
basic_migration_one   Idle     2       0          2               
----------------- -------- ------- ---------- ------------- --------------------- 

Running migrations

drush migrate:import basic_migration_one
drush mi basic_migration_one  

Output from console:
----------------- -------- ------- ---------- ------------- ------------------- 
  Migration ID      Status   Total   Imported   Unprocessed  Last Imported        
----------------- -------- ------- ---------- ------------- ------------------- 
basic_migration_one   Idle     2       2 (100%)   0            2020-03-17 23:19:36  
----------------- -------- ------- ---------- ------------- ------------------- 

And so, going to the path /admin/content you’ll see the two new nodes:

Drupal Basic Migration Embedded Data

Rollbacking migrations (undoing)

drush migrate:rollback basic_migration_one
drush mr basic_migration_one  

Output from console: 

[notice] Rolled back 2 items - done with 'basic_migration_one'

Drupal Basic Migration Commands

Second Case: Migrating from csv files

Well, now in the path /admin/config/development/configuration/single/import we have to import our new custom migration definition file, Ok?

Loading the migration config data

Just go to Import -> Single Item, select the configuration type as “Migration” and paste the content of the original migration file:

Drupal Migration load File by Config

Click The “Import” button and the new Config object will be created in the Config System.

And now?

Running the migration

With the Migration file under the Config management, you can run the process with the same tools as in the former case. Now, we have available a new migration that we can run from console: drush migrate:status

Drush Migrate Status

Now you can execute the migration with: drush migrate-import article_csv_import And all the new nodes will be created. The limit? well, tags and image not will be migrated, cause tag is an entity reference and image is not a link, is a file, and both types must use some differents Plugins&mldr;but we’ll talk about this in future posts.

Drush cex / Drush cim

With the migration under the config system, now you can edit, import and export the migration using the basic resources from Drush. For example, testing drush cex:

Drush Cex

As you can see, the Config System has directly put the new migration file under the management of Migrate Plus and It has performed some actions, such as: renamed the file by placing migrate_plus.migration as a prefix in the file name or added a new file for group (only a way to group migration processes).

Remember the name of the file? It’s just the same that we were using in the /config/install directory, the so-called migrate_plus.migration.article_csv_import.yml. We’ve done exactly the same process, but from a different direction. Are you impressed? No? Do you find it interesting?

Remember also that with this config file, you can use drush cim and load the migration in any other Drupal (with access to the CSV file as datasource, indeed).

Thus we have migrated some 102 new nodes using two different approaches and different methodologies. Not bad.

5- Key Concepts

Migration Plugins

Ok, It’s very important so we have to repeat one more time the same song&mldr;You must to know the Plugin Format and the diverse world of the existing Migration Plugins.

Every Plugin points to a specific data type, a specific format or a different source. You should know the main ones very well and also investigate those you may need, since in migrations they are used extensively. Because of this, for example, we have not been able to migrate taxonomy terms or images in the second case from the CSV file as datasource.

Let’s see the Plugins involved in these two migrations, watching its descriptive files:

Basic Embedded Migration

source:
  plugin: embedded_data
  data_rows:
         ...
process:
  title: creative_title
  body: engaging_content
destination:
  plugin: 'entity:node'
  default_bundle: page

We’re using for extract data from the source the Embedded Data Plugin, a PHP class available in /web/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php where in its annotations block you can see some configuration keys that you can use in your migrate file:

 *
 * Available configuration keys
 * - data_rows: The source data array.
 * - ids: The unique ID field of the data.
 *

And data_rows and ids are the keys that we’re using in our migration description file. Read more about the EmbeddedDataSource class in Drupal.org API.

Now, watching the process block and looking for&mldr;where’s the Processing Plugin? Well I think this might be interesting&mldr;usually, all the field mappings in a processing block requires a process plugin for each. Then, with some of “sintactic sugar”, the Migrate API offers a way to reduce and simplify this: if no specific treatment is required for each field, then a single Plugin can take care of all the processing. This “default” Plugin may also be implicit, so that in the absence of a declaration, the Drupal Migrate API will always apply the same Processing Plugin by default.

This “implicit” and by-default Plugin is the Get class and is provided as the basic solution in processing fields. You can find the Get class in the path /web/core/modules/migrate/src/Plugin/migrate/process/Get.php. Read more info about the Get.php class in Drupal.org API. So actually, what we are saying in a complementary way is that is the same thing write:

process:
  title: page_title

as this other:

process:
  title:
    plugin: get
    source: page_title

And so life is a little simpler, isn’t it? Remember: in the absence of a processing plugin declaration for a field, Drupal will apply the “Get” plugin by default.

Ok and finally, for destination we’re using the Entity General Plugin with param “node”, in order to create diverse elements with node type and for bundles “page”. This calls to the Destinatio Plugin Entity.php, abstract class in path: web/core/modules/migrate/src/Plugin/migrate/destination/Entity.php and get its own derivative Plugin. Read more about derivative Plugins in Drupal and read about the Entity.php destination Plugin or the derivative migration class.

CSV datasource Migration

I think that the review of the plugins in this case could be easier and more intuitive.

source:
  plugin: csv
 ...
 process:
 ...
   type:
     plugin: default_value
     default_value: article
 destination:
   plugin: 'entity:node'

For the source, the CSV Plugin, from the migrate_source_csv contrib module. For processing, by default is using Get and for type the Default Value Plugin. For destination, the same plugin as the previous migration: new entities.

Migration as code or as configuration

As you could see, we have treated each migration process differently. The first process (Embedded Data) has been treated as part of the “code”, without any further particularities.
But the second process has been treated as a configuration element of the system itself, making it part of the config/install path, which will create a new configuration object from the installation.

In both cases you write the migration definition in a YAML format and then you put the migration file in a place or another. But there are more differences&mldr;Let’s make a little summary of these keys:

  • Migration “as code” is provided out of the box, but the module “Migrate Plus” allows you treating the file as a configuration object.

  • Depending on which approach you use, the location of the files and the workflow will differ:

    • As code, in order to make changes to the migration definition you’ll need access to the file system and manage the migration file as a code file, something developers-oriented.

    • As configuration, you’ll can do changes to the migration definition file using the config sync interface in Drupal, path: /admin/config/development/configuration, in addition to being able to use configuration export/import tools: drush cex, drush cim, cause now you sync the migration (the migration file will be saved in database). This means that you can write, modify, and execute migrations using the user interface. Big surprise.

    • As a configuration object, now your migration file will be create a new configuration registry in your Drupal Config System, and keep it alive also when your migrate module will be disabled. To avoid this and delete the config, put your own custom module as a new dependency of the migration in your migration description file.yml, so the migration will be deleted from Drupal’s Active Config just in this moment:

  dependencies:
    enforced:
      module:
        - my_own_migration_custom_module

  • Another change is that now, in a config-way, your migration file needs a UUID, just a global identifier for the Drupal Config Management System. Add at first line an unique and custom UUID for your file, to facilitate the processing of the configuration. Remember: UUID is a string of 32 hexadecimal digits in blocks of 5 groups using the pattern: 8-4-4-4-12. Make your own!
    uuid: cacafuti-1a23-2b45-3c67-4d567890a1b2.

6- Resources

Download, play and test the different resources using along this post. I uploaded to Github ready to use.

  1. Basic Migration File, basic_migration_one.yml, available in Github as Gist.

  2. CSV Migration File, article_csv import.yml, available in Github as Gist.

  3. CSV Source File with random data, Gist in Github.

  4. Codebase of the two migration modules (basic and csv), Available in Github. This will be a central repository for all the modules of this series of posts about Migrations, so get the direct link to these two examples:

  5. In parallel to this series of articles I’m also publishing a series of snippets in Gitlab under the topic “Migrations”, with a more simplified format, less verbose. Here you can access to the first snippet and get links to the rest of the series. Drupal Migrations Tips (I): Creating a new basic migration structure.

7- :wq!

[embedded content]

Mar 17 2020
Mar 17

As I said in the previous post, during these months I will be playing with migrations, preparing some cases for a future (I hope) book. Well, during these days of confinement, I intend to continue with small articles around here to show experiences related to migrations.

In the former post, I was writing about migrations in Drupal from a point of view based in the look for a tool-box, just a set of basic resources in order to focusing a migration.

There’s a lot of information to process about it and some more concepts, technics and tactics to resolving a migration, you can be sure. So this month I want to write something that allows me play with migrations, maybe more practical than theorical.

This article was originally published in https://davidjguru.github.io
Picture from Unsplash, user Émile Séguin, @emileseguin

Table of Contents

1- Introduction
2- Arrangements
3- Approaches
4- Migrations
5- Key Concepts
6- Resources
7- :wq!

1- Drupal Migrations (I): Basic Resources

2- Drupal Migrations (II): Examples

3- Drupal Migrations (III): Migrating from Google Spreadsheet

4- Drupal Migrations (IV): Debugging Migrations First Part

5- Drupal Migrations (V): Debugging Migrations-II

1- Introduction

The Drupal Migration API can be one of the most interesting, but also one of the most complex, since its activities are often related to classes and methods of other Drupal APIs (so it’s especially particular when debugging). In any case, and as the amount of concepts can be overwhelming, I think we could practice migration mechanics through a couple of exercises.

Well, for this article I had proposed to model two different migration processes, under a point of view that could be summarized as “primum vivere, deinde philosophari” (first you experiment, then you theorize). This is why I have decided to organize it in a particular way:

  • The first thing to say is that the two processes are divided into sections that are common to both and instead of finishing one and starting the next one, both go in parallel (you choose your own adventure).

  • Then, Only at the end of this post you will find some key concepts used in this article. First we gonna to play with the structures, then we’ll understand them.

So, in the next steps, we’ll working around two certain experiencies:

  1. Migrating Data from a embedded format (maybe the most simple example of Drupal migrations).

  2. Migration Data from a classical CSV file format (just a little more complex than the previous example).

Both of the cases are perhaps the most basic scenarios for a migration, so I recommend reading this article for those who want to get started on its own mechanics, as a practical complement to get into Drupal migrations.

2- Arrangements

First case: Migrating embedded data

For our first case we will need, on the one hand, to enable the Migrate module of the Drupal core, and on the other hand, to download and install a contributed module to be able to manage migrations.

From the different options we have, we are going to choose migrate_run, which we have already mentioned in the previous post and could be interpreted as a light version of migrate_tools (although it’s actually a fork of the project): both of wich provide drush commands to run migrations, so if you have migrate_tools installed you must uninstall it in order to avoid collide with migrate_run.

As a curious note, the first lesson here is that for running Drupal migrations, neither migrate_plus nor migrate_tools are “hard” dependencies, that is, we can implement migrations without having these modules enabled in our Drupal installation.

By the way I have to say that it’s important to know that migrate_run is optimized for Drush 9 and later. If you use Drush 8 you will have to use an adapted version, like the Alpha 4, which was still prepared for Drush 8.

Using Composer and Drush:

composer require drupal/migrate_run
drush pmu migrate_tools # If you need
drush en migrate migrate_run -y
drush cr

Using Drupal Console:

composer require drupal/migrate_run
drupal mou migrate_tools # If you need
drupal moi migrate migrate_run

And you will see in the path /admin/modules:

Enabling Migrate and Migrate Run modules

Building the resources

Now, we’re going to create a new custom module for our first Migration:

cd project/web/modules/custom
mkdir migration_basic_module

Then, the migration_basic_module.info.yml file with content:

name: 'Migration Basic Module'
type: module
description: 'Just a basic example of basic migration process.'
package: 'Migrations Examples 2000'
core: 8.x
dependencies:
  - drupal:migrate

Create the new migration definition file with path: /migration_basic_module/migrations/basic_migration_one.yml.

In our new declarative file basic_migration_one.yml, which describes the migration as a list of parameters and values in a static YAML-type file, we will include the embedded data of two nodes for the content type “basic page” to be migrated, loading only two values:

  1. A title (a text string).
  2. A body (A text based on the ChiquitoIpsum generator*, http://www.chiquitoipsum.com).

*Chiquito de La Calzada was a national figure in the Spanish state, a legendary comedian.

basic_migration_one.yml

id: basic_migration_one
label: 'Custom Basic Migration 2000'
source:
  plugin: embedded_data
  data_rows:
    -
      unique_id: 1
      page_title: 'Title for migrated node - One'
      page_content: 'Lorem fistrum mamaar se calle ustée tiene musho pelo.'
    -
      unique_id: 2
      page_title: 'Title for migrated node - Two'
      page_content: 'Se calle ustée caballo blanco caballo negroorl.'
  ids:
    unique_id:
      type: integer
process:
  title: article_title
  body: article_content
destination:
  plugin: 'entity:node'
  default_bundle: page

And this will be the structure of the new custom module for basic migration example:

/project/web/modules/custom/  
                     \__migration_basic_module/  
                         \__migration_basic_module.info.yml  
                             \__migrations/  
                                 \__basic_migration_one.yml  

Enabling all the required modules using Drush:

drush pm:enable -y migrate migrate_run migration_basic_module
drush cr

Or using Drupal Console:

drupal moi migrate migrate_run migration_basic_module

Second Case: Migrating from csv files

For this second case we are going to deactivate migrate_run (if applicable) and activate the superset of modules: migrate, migrate_plus and migrate_tools. Besides, for the treatment of CSV files we are going to use a Source Plugin stored in a contrib module called Migrate Source CSV migrate_source_csv. This contrib module in its version 3.x is using league/csv for processing CSV files. Ok, let’s go. So using Composer + Drush:

composer require drupal/migrate_plus drupal/migrate_tools drupal/migrate_source_csv
drush pmu migrate_run # If you need 
drush en migrate migrate_plus migrate_tools migrate_source_csv -y
drush cr

So, now in the path /admin/modules/:

Enabling Migrate and Migrate Plus Migrate Tools

Building the resources

We’re going to create another new custom module for our second Migration:

cd project/web/modules/custom
mkdir migration_csv_module

With a new migration_csv_module.info.yml file:

name: 'Migration CSV Module'
type: module
description: 'Just a basic example of basic migration process with a CSV source.'
package: 'Migrations Examples 2000'
core: 8.x
dependencies:
  - drupal:migrate
  - drupal:migrate_tools
  - drupal:migrate_plus

In this example we’re going to require a declarative file of the migration too (as in the previous case) but with the exception that we’re going to locate it in a different place. This will be placed in the /migration_csv_module/config/install/ path.

The structure will look like this just now:

/project/web/modules/custom/  
                     \__migration_csv_module/  
                         \__migration_csv_module.info.yml
                          \__csv/
                               \_migration_csv_articles.csv
                           \__config/
                                \__install/
                                     \__migrate_plus.migration.article_csv_import.yml

So we need a csv with original data to migrate. It’s easy to solve this using web tools like Mockaroo, a pretty good random data generator. I’ve created a CSV file with some fields like: id, title, body, tags, image. Download it from here. This file will be our datasource for the Migration process. Ok, by now create the directories for the module and put the new custom CSV in the /csv path:

CSV Migrate module structure

And now, our migrate_plus.migration.article_csv_import.yml file (In later sections we will explain its construction and sections):

uuid: 1bcec3e7-0a49-4473-87a2-6dca09b91aba
langcode: en
status: true
dependencies: {  }
id: article_csv_import
label: 'Migrating articles'
source:
  plugin: csv
  path: modules/custom/migration_csv_module/csv/migration_csv_articles.csv
  delimiter: ','
  enclosure: '"'
  header_offset: 0
  ids:
    - id
  fields:
    -
      name: id
      label: 'Unique Id'
    -
      name: title
      label: Title
    -
      name: body
      label: 'Post Body'
    -
      name: tags
      label: 'Taxonomy Tag'
    -
      name: image
      label: 'Image Field'
process:
  title: title
  body: body
  tags: field_tags
  image: field_image
  type:
    plugin: default_value
    default_value: article
destination:
  plugin: 'entity:node'

Okay, we now have all the resources we need to create our new migration. Now let’s see how we approach the process.

3- Approaches

We’re going to describe the different approaches that we will apply to our example cases, in order to understand them better.

First case: Migrating embedded data

In this first case, we considered making the lightest possible case of migration in Drupal: Only two nodes with two basic fields each under an embedded format: the lightest possible.

Also, in this example we are going to use for the three ETL phases of the migration (Extract, Transformation and Loading) processing plugins already provided by Drupal (we will not develop any custom plugin). If you don’t know anything about the concept of Migration Plugins, please stop by for a moment and back here to read a little introduction to the topic.

To make things lighter, we will keep the “lite” version of Migration Tools, Migrate Run. Besides, we will only use the basic commands without any other options or complementary parameters, only with the basic argument of the migration file identifier.

Second Case: Migrating from csv files

For this execution, I would like to play with something pretty interesting&mldr;due to we’ll running this second migration example as configuration, I was thinking that will be funny do the inverse road&mldr;Yes, I propose not to install (activate, drush enable) the new custom module for CSV and leave it&mldr;only as storage for the CSV file.

Let’s move and run the migration from somewhere else. Surprise. Visit the path /admin/config/development/configuration/single/import into your Drupal installation and we’ll see there!.

4- Migrations

First case: Migrating embedded data

Getting info about the available migrations

drush migrate:status
drush ms

Output from console:
----------------- -------- ------- ---------- ------------- --------------------- 
  Migration ID      Status   Total   Imported   Unprocessed   Last Imported        
----------------- -------- ------- ---------- ------------- --------------------- 
basic_migration_one   Idle     2       0          2               
----------------- -------- ------- ---------- ------------- --------------------- 

Running migrations

drush migrate:import basic_migration_one
drush mi basic_migration_one  

Output from console:
----------------- -------- ------- ---------- ------------- ------------------- 
  Migration ID      Status   Total   Imported   Unprocessed  Last Imported        
----------------- -------- ------- ---------- ------------- ------------------- 
basic_migration_one   Idle     2       2 (100%)   0            2020-03-17 23:19:36  
----------------- -------- ------- ---------- ------------- ------------------- 

And so, going to the path /admin/content you’ll see the two new nodes:

Drupal Basic Migration Embedded Data

Rollbacking migrations (undoing)

drush migrate:rollback basic_migration_one
drush mr basic_migration_one  

Output from console: 

[notice] Rolled back 2 items - done with 'basic_migration_one'

Drupal Basic Migration Commands

Second Case: Migrating from csv files

Well, now in the path /admin/config/development/configuration/single/import we have to import our new custom migration definition file, Ok?

Loading the migration config data

Just go to Import -> Single Item, select the configuration type as “Migration” and paste the content of the original migration file:

Drupal Migration load File by Config

Click The “Import” button and the new Config object will be created in the Config System.

And now?

Running the migration

With the Migration file under the Config management, you can run the process with the same tools as in the former case. Now, we have available a new migration that we can run from console: drush migrate:status

Drush Migrate Status

Now you can execute the migration with: drush migrate-import article_csv_import And all the new nodes will be created. The limit? well, tags and image not will be migrated, cause tag is an entity reference and image is not a link, is a file, and both types must use some differents Plugins&mldr;but we’ll talk about this in future posts.

Drush cex / Drush cim

With the migration under the config system, now you can edit, import and export the migration using the basic resources from Drush. For example, testing drush cex:

Drush Cex

As you can see, the Config System has directly put the new migration file under the management of Migrate Plus and It has performed some actions, such as: renamed the file by placing migrate_plus.migration as a prefix in the file name or added a new file for group (only a way to group migration processes).

Remember the name of the file? It’s just the same that we were using in the /config/install directory, the so-called migrate_plus.migration.article_csv_import.yml. We’ve done exactly the same process, but from a different direction. Are you impressed? No? Do you find it interesting?

Remember also that with this config file, you can use drush cim and load the migration in any other Drupal (with access to the CSV file as datasource, indeed).

Thus we have migrated some 102 new nodes using two different approaches and different methodologies. Not bad.

5- Key Concepts

Migration Plugins

Ok, It’s very important so we have to repeat one more time the same song&mldr;You must to know the Plugin Format and the diverse world of the existing Migration Plugins.

Every Plugin points to a specific data type, a specific format or a different source. You should know the main ones very well and also investigate those you may need, since in migrations they are used extensively. Because of this, for example, we have not been able to migrate taxonomy terms or images in the second case from the CSV file as datasource.

Let’s see the Plugins involved in these two migrations, watching its descriptive files:

Basic Embedded Migration

source:
  plugin: embedded_data
  data_rows:
         ...
process:
  title: creative_title
  body: engaging_content
destination:
  plugin: 'entity:node'
  default_bundle: page

We’re using for extract data from the source the Embedded Data Plugin, a PHP class available in /web/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php where in its annotations block you can see some configuration keys that you can use in your migrate file:

 *
 * Available configuration keys
 * - data_rows: The source data array.
 * - ids: The unique ID field of the data.
 *

And data_rows and ids are the keys that we’re using in our migration description file. Read more about the EmbeddedDataSource class in Drupal.org API.

Now, watching the process block and looking for&mldr;where’s the Processing Plugin? Well I think this might be interesting&mldr;usually, all the field mappings in a processing block requires a process plugin for each. Then, with some of “sintactic sugar”, the Migrate API offers a way to reduce and simplify this: if no specific treatment is required for each field, then a single Plugin can take care of all the processing. This “default” Plugin may also be implicit, so that in the absence of a declaration, the Drupal Migrate API will always apply the same Processing Plugin by default.

This “implicit” and by-default Plugin is the Get class and is provided as the basic solution in processing fields. You can find the Get class in the path /web/core/modules/migrate/src/Plugin/migrate/process/Get.php. Read more info about the Get.php class in Drupal.org API. So actually, what we are saying in a complementary way is that is the same thing write:

process:
  title: page_title

as this other:

process:
  title:
    plugin: get
    source: page_title

And so life is a little simpler, isn’t it? Remember: in the absence of a processing plugin declaration for a field, Drupal will apply the “Get” plugin by default.

Ok and finally, for destination we’re using the Entity General Plugin with param “node”, in order to create diverse elements with node type and for bundles “page”. This calls to the Destinatio Plugin Entity.php, abstract class in path: web/core/modules/migrate/src/Plugin/migrate/destination/Entity.php and get its own derivative Plugin. Read more about derivative Plugins in Drupal and read about the Entity.php destination Plugin or the derivative migration class.

CSV datasource Migration

I think that the review of the plugins in this case could be easier and more intuitive.

source:
  plugin: csv
 ...
 process:
 ...
   type:
     plugin: default_value
     default_value: article
 destination:
   plugin: 'entity:node'

For the source, the CSV Plugin, from the migrate_source_csv contrib module. For processing, by default is using Get and for type the Default Value Plugin. For destination, the same plugin as the previous migration: new entities.

Migration as code or as configuration

As you could see, we have treated each migration process differently. The first process (Embedded Data) has been treated as part of the “code”, without any further particularities.
But the second process has been treated as a configuration element of the system itself, making it part of the config/install path, which will create a new configuration object from the installation.

In both cases you write the migration definition in a YAML format and then you put the migration file in a place or another. But there are more differences&mldr;Let’s make a little summary of these keys:

  • Migration “as code” is provided out of the box, but the module “Migrate Plus” allows you treating the file as a configuration object.

  • Depending on which approach you use, the location of the files and the workflow will differ:

    • As code, in order to make changes to the migration definition you’ll need access to the file system and manage the migration file as a code file, something developers-oriented.

    • As configuration, you’ll can do changes to the migration definition file using the config sync interface in Drupal, path: /admin/config/development/configuration, in addition to being able to use configuration export/import tools: drush cex, drush cim, cause now you sync the migration (the migration file will be saved in database). This means that you can write, modify, and execute migrations using the user interface. Big surprise.

    • As a configuration object, now your migration file will be create a new configuration registry in your Drupal Config System, and keep it alive also when your migrate module will be disabled. To avoid this and delete the config, put your own custom module as a new dependency of the migration in your migration description file.yml, so the migration will be deleted from Drupal’s Active Config just in this moment:

  dependencies:
    enforced:
      module:
        - my_own_migration_custom_module

  • Another change is that now, in a config-way, your migration file needs a UUID, just a global identifier for the Drupal Config Management System. Add at first line an unique and custom UUID for your file, to facilitate the processing of the configuration. Remember: UUID is a string of 32 hexadecimal digits in blocks of 5 groups using the pattern: 8-4-4-4-12. Make your own!
    uuid: cacafuti-1a23-2b45-3c67-4d567890a1b2.

6- Resources

Download, play and test the different resources using along this post. I uploaded to Github ready to use.

  1. Basic Migration File, basic_migration_one.yml, available in Github as Gist.

  2. CSV Migration File, article_csv import.yml, available in Github as Gist.

  3. CSV Source File with random data, Gist in Github.

  4. Codebase of the two migration modules (basic and csv), Available in Github. This will be a central repository for all the modules of this series of posts about Migrations, so get the direct link to these two examples:

  5. In parallel to this series of articles I’m also publishing a series of snippets in Gitlab under the topic “Migrations”, with a more simplified format, less verbose. Here you can access to the first snippet and get links to the rest of the series. Drupal Migrations Tips (I): Creating a new basic migration structure.

7- :wq!

[embedded content]

Feb 25 2020
Feb 25

I am working on notes for a draft that will be a book about migration processes made with Drupal and its Migrate API. It is expected to be released in June 2020 and the work of collecting, experimenting and articulating the content is being quite extensive. As there are still some months left for the launch, in order not to lose the mental sanity and to be able to give partial sense to these tasks, I have thought to publish here some small posts derived from the working notes.

This way I will be able to give something useful to the complementary notes and if the COVID-19 attacks me before seeing the book come out, at least I will have shared something before (I guess).

Well, what do I want to talk about in this post? I would like to make a list of Drupal modules related to migration processes, available as contrib modules and that can be used to provide functionality to a migration. This article will be only a lightweight set of basic resources (I swear).

This article was originally published in https://davidjguru.github.io
Picture from Unsplash, user Nils Nedel, @nilsnedel

Table of Contents

1- Introduction
2- Basic Resources - Core Modules
3- Other Basic Resources - Contrib Modules
4- Extra Resources - Contrib Modules for Plugins
5- Migration Runners - Contrib Modules Drush-Related
6- Authors you should know
7- :wq!

This article is part of a series of posts about Drupal Migrations:

1- Drupal Migrations (I): Basic Resources

2- Drupal Migrations (II): Examples

3- Drupal Migrations (III): Migrating from Google Spreadsheet

4- Drupal Migrations (IV): Debugging Migrations First Part

5- Drupal Migrations (V): Debugging Migrations-II

1- Introduction

It’s not very easy to talk about migrations in general and, of course, it is not easy in the context of Drupal either. To perform migrations it is necessary to have a good knowledge of the technology, data models (in origin and in destination), experience in ETL processes and a certain know-how about how to implement Drupal Plugins (In migrations there is an extensive use of Drupal-Way Plugins).

In any case, since the topic is extensive and my time is now short, I thought of this article as a summary catalogue (for quick consumption) of tools and basic resources for working with migrations.

2- Basic Resources - Core Modules

3- Other Basic Resources - Contrib Modules

  • Migrate Plus: [migrate_plus](https://www.drupal .org/project/migrate_plus). Migrate Plus is an essential contrib module wich extends the features and capabilities of the Migrate core module with a lot of plugins and extensions.

  • Migrate Tools: migrate_tools. Another essential resource: provides a lot of Drush commands for running and managing Migrations.

  • Migrate Status: migrate_status. This little contrib module allows get a feedback about a migration process. Do you need to know if a migration is running? this module gives you a service that you can call in order to check the migration.

  • Migrate Files: migrate_files. It’s such an interesting set of process plugins that you will want to move files and images with it.

  • Migrate Commerce: commerce_migrate. General-purpose framework that extends to the main Migrate module from Drupal Core, for moving data in a Drupal Commerce scenario.

In the Drupal migration processes, we’ll use diverse resources in order to processing the ETL migration plan. One of these basic resources (as I mentioned in the introduction) are the Drupal Plugins, of which you need to have good knowledge and some practice. In a migration scenario, plugins help us processing information from the E:Source (Source Plugins), making T:Processing(ProcessPlugins) in order to save data at L:Destination (Destination Plugins). The assembly of these three parts (usually) results in a correct migration process.

Many modules of the core already bring their own Plugins to facilitate migration processes (as the user module). So let’s review some migration plugins packaged in contributed modules.

Source Plugins Migrate Source Plugin

  • Migrate Source CSV: migrate_source_csv. Contrib Module for migrating data to Drupal 8 from a classical and simple CSV file.

  • Migrate Source SQL: custom_sql_migrate_source_plugin. As a peculiarity of the Plugins used for databases, this module allows to integrate in the .yml file describing a migration, directly SQL queries that will be executed against the source database.

  • Migrate Source YAML: migrate_source_yaml. It’s just a simple tool for migrating content from YAML files.

Processing Plugins Migrate Process Plugins

Destination Plugins: Migrate Destination Plugins & Examples

What kind of Drupal entities will be created in the migrating process? content entities? configuration entities? Take a look.

  • Migrate Destination CSV: d8migrate. It’s a light custom module created by @jonathanfranks.

  • Migrate Destination Config: Class Config.php. Offers a plugin for config migration.

  • Migrate Destination Block: Class EntityBlock.php. Just like and example about the resources that every element can offers in a migration scene, in case of moving Block Entities (are Config Entities) see the PHP classes included in its own module for migrating (Source, Process and Destination).

Drupal 8 Migrate Entity Block

6- Authors you should know

  • Mauricio Dinarte: Mauricio is a developer, consultant, trainer and owner of his own business https://agaric.coop, wrote what is probably the mandatory reading guide for all people who want to learn how to migrate on Drupal: 31 Days of Drupal Migrations, a set of 31 articles published in https://understanddrupal.com/migrations with the most important aspects&mldr;examples, exercises, descriptions&mldr;to understand the whole internal world of migrations inside Drupal.

An essential training material. In addition, his company’s website, under the tag “migrate” also hosts many very good articles about migration topics: https://agaric.coop/tags/migrate.

Some examples from Mauricio Dinarte:

  1. Introduction to paragraphs migrations in Drupal:
  1. Using migration groups to share configuration among Drupal migrations:
  1. What is the difference between migration tags and migration groups in Drupal?

His profile in Drupal.org: https://www.drupal.org/u/dinarcon.

  • Tess Flynn: I heard about Tess Flynn reading articles by Mauricio Dinarte. That’s how I met this expert developer, speaker and communicator of the Drupal community. On her website https://deninet.com I found content of a different nature, but above all, a series of very interesting articles about migrations under the tag “drupal-migration”: https://deninet.com/tag/drupal-migration.

    Along the way I also discovered that it has several contrib modules related to Migrations and Processing Plugins.

Some examples from Tess Flynn:

  1. Migrate Process URL: Provides Process Plugin to migrate link fields.
  1. Migrate Process Vardump: Helping to debugging migrations.
  1. Many Process Plugins:

Her profile in Drupal.org: https://www.drupal.org/u/socketwench.

Some examples from Danny Sipos:

  1. Your first Drupal 8 Migration:
  1. Dynamic migrations using “templates” in Drupal 8:
  1. Quickly generate the headers for the CSV migrate source plugin using Drush:

His profile in Drupal.org: https://www.drupal.org/u/upchuk.

7- :wq!

[embedded content]

Jun 09 2019
Jun 09

As I mentioned in a previous month’s article, I’m still using / playing / practicing with DDEV to build development environments in an agile way. And the truth is that the experience couldn’t be more nicer. We know that for some years, working with Docker Engine has become a MUST. From our ability to define specifications(Dockerfiles), configure autobuilds in the cloud to generate images from our repositories(Dockerhub), run images and containers(Docker), connect containers and relate them(Docker-Composer) or deploy container networks( Docker Swarm) &mldr; on this depends the agility that we can give to our daily work. So I’ve gathered the most frequent commands in a cheatsheet.

This article was originally published in https://davidjguru.github.io
Picture from Unsplash, user Markus Spiske, @markusspiske

Table of Contents

1- Introduction
2- Basics
3- Dockerfile
4- Containers
5- Docker-Compose
6- Others
7- DDEV
8- :wq!

1- Introduction

Due to the importance of knowing (and practicing) with these processes and daily mechanics, I have thought about gathering the most used commands in my day to day work in the context of Docker.

As a Fast-Cheatsheet of work that compiles the most usual instructions of the different parts of the Docker Engine in the day to day, including DDEV, since for me it is already an inseparable part of both: the Docker Universe and the daily work with projects based on Drupal.

I have assembled everything by specific blocks (Basics, Docker, Dockerfile, Docker-Composer, Images, Containers and last but not least, my new love DDEV. And I also took advantage of it to give it a certain didactic approach: some instructions can be summarized or executed more directly, but I think that many colleagues can learn much more when they understand well the logic under which these tools operate.

I would have liked to have included many more things, like DDEV hooks, but this was already on the way to an encyclopedia. I promise to write about it soon.

2- Basics

# Uninstall old versions of Docker
sudo apt-get remove docker docker-ce docker-engine \
docker.io containerd runc

# Installing Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
| sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt install -y docker-ce
sudo chmod 666 /var/run/docker*

# Test if Docker is running or not
systemctl is-active docker

# List all the Docker CLI commands
docker

# Get a whole resume about the Docker installation
docker info

# Get the current version installed of Docker in a fast
# and short response 
docker --version

# Get a extended report about the Docker installation 
# with info about client and server
docker version

3- Dockerfile

# Mapping ports in a Dockerfile
ports:
  - "8080:80"
     (Host:Guest)

# Mapping volumes in a Dockerfile
volumes:
  - "local_directory:remote_directory"
volumes:

        - type: volume
          source: /data/mysql
          target: /var/lib/mysql


# Create a tag for a new Dockerfile
docker build . -t vendor/name-tag 

# Remote login to DockerHub
docker login 

# Push the selected Dockerfile tagged to Dockerhub
docker push vendor/tag

4- Containers

# Run a container but in detached mode and
# back to your prompt
docker run -d vendorexample/appexample

# Run a container with  custom name.
docker run -d --name web-custom-name nginx:1.14-alpine

# Restart a Container
docker restart IDCONTAINER

# Run a container from the centOS image with bash and
# login in prompt
docker run -it centos bash

# Deploy a mysql database using the mysql image and
# name it mysql-db. Set the database password to 
# use db_pass123. Lookup the mysql image on 
# Docker Hub and identify the correct environment
# variable to use for setting the root password.
docker run -d -e MYSQL_ROOT_PASSWORD=db_pass123 --name mysql-db mysql

# Run a container in background with a end of life
docker run -d centos sleep 100

# Run a container, mapping ports and mapping volumes and
# using a user from the container
docker run -p 80:8080 -v /locahost/folder:/container/folder \
-u root jenkins/jenkins

# Get a list of existing containers
docker ps

# List all containers showing it by its ID
docker ps -q

# Kill all containers running selected by ID
docker kill $(docker ps -q)

# Remove only a container
docker rm IDCONTAINER

# Remove all containers with status=exited
docker rm $(docker ps -q -f status=exited)

# Executes a command inside a running container
docker exec idcontainer unixcommand 

# Connect to the Prompt of a Container
docker exec -it IDCONTAINER /bin/bash

# Connect to the Prompt of a Container as root
docker exec -ti -u root IDCONTAINER /bin/bash

# Copying files with Docker
# From Local to Remote Docker Container
docker cp db/dump.sql IDCONTAINER:/tmp/dump.sql
# From Remote Docker Container to local
docker cp IDCONTAINER:/tmp/dump_test.sql ./db 

# Attach local standard output, input and error 
# streams to a running container
docker attach IDCONTAINER 

# Inspect all the info about a Docker Container
docker container inspect IDCONTAINER 

# Show the log of a container
docker logs -f IDCONTAINER

# Tailing logs:
# Searching for 'error' (case - insensitive) in the 
# last 1000 log lines of my jenkins (example) 
# container adding the timestamp at the beginning 
# of each line.
sudo docker logs -t --tail 1000 jenkins 2 >&1 \
| grep -i error

# Describe all the existing Docker Compose Networks
docker network ls

# Get all the info about an specific network
docker network inspect name_network 

# Delete a network in your Docker Compose system
docker network rm name_network

# Remove unusued data and clean the Docker System
docker system prune -f

# Show stats about the running containers.
docker stats

# Same but with a formatted output. 
docker stats --all --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}"

# Fixing results in prompt.
docker ps -q | xargs  docker stats --no-stream


5- Docker-Compose

# Run a multi-container application with Docker Compose
docker-compose up -d
# Turn on the Docker Compose network
# but building images before starting containers
docker-compose up --build

# Stop a multi-container application with Docker Compose
docker-compose stop

# Kill and delete containers based in Docker Compose
docker-compose down

# Strem the container events for every container 
# in a project
docker-compose events --json 

# Connect to a container using its alias (no ID, no IP)
docker-compose exec ALIAS /bin/bash
# Example: executing drush cache rebuild in a Drupal Container
docker-compose exec web ./vendor/bin/drush cr

# Connecting to a Container (called mysql) as root by prompt
docker-compose exec -u root mysql /bin/bash

# Get the Docker container's log
docker-compose logs -t ALIAS
# Get the Docker container's log 
# with direct connection in real time
docker-compose logs -t -f ALIAS

6- Others

# Docker Swarm: Deploy instances of application 
# across docker host
docker stack deploy -c docker-compose.yml

# Remove ALL: stopped containers, all networks 
# not used and all dangling images
docker system prune

7- DDEV

If you need an introduction to DDEV, I recommend you read this article that I wrote recently (the previous month): Development environments for Drupal with DDEV.

# Git Clone Project and launch composer install
git clone https://github.com/randomuser/my-drupal8
cd my-drupal8
ddev composer install

# Initial Project 
mkdir my-drupal8
cd my-drupal8
ddev config --project-type php --php-version 7.3
ddev composer create drupal-composer/drupal-project:8.x-dev \
--stability dev --no-interaction
ddev config --project-type drupal8
ddev restart

# Creating a CMS-specific settings file with 
# DDEV credentials pre-populated
ddev config

# DDEV Commons
ddev start
ddev list
ddev describe [project-name]

# Using SSH in DDEV Containers
ddev ssh 

# Executing Drush in DDEV Containers
ddev exec drush status
ddev exec drush cex
ddev exec drush site-install
ddev exec drush site-install standard \
--site-name='Drupal Site Install Test' \
--account-name=admin --account-pass=admin \
[email protected] -y


# Installing dependencies from a ddev container
ddev composer require drupal/devel

# Install a complete Drupal Site using ddev 
# in a "single" instruction
mkdir NAMEPROJECT && cd NAMEPROJECT \
&& ddev config --project-type php \
--php-version 7.3 \
&& ddev composer create drupal-composer/drupal-project:8.x-dev \
--stability dev --no-interaction && ddev config \
--project-type drupal8 \
&& ddev exec drush site-install standard \
--site-name='NAMEPROJECT' \
--account-name=admin \
--account-pass=admin \
[email protected] -y \
&& ddev start \
&& sensible-browser http://NAMEPROJECT.ddev.local

# Get a list of projects using DDEV
ddev list

# Describe a project
ddev describe project-name

# Importing a database file (launch a prompt to set 
# the location and values of the database dump)
ddev import-db

# Exporting a database
ddev export-db

# Importing files assets
ddev import-files

# Snapshotting your database
ddev snapshot // Same as in ddev stop --remove-data

# Clean the database and avoid the snapshot
ddev stop --remove-data --omit-snapshot


8- :wq!

[embedded content]

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web