Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jul 13 2020
Jul 13

In a previous article we explained the syntax used to write Drupal migration. We also provided references of subfields and content entities' properties including those provided by the Commerce module. This time we are going to list the configuration options of many migrate source plugins. For example, when importing from a JSON file you need to specify which data fetcher and parser to use. In the case of CSV migrations, the source plugin configuration changes depending on the presence of a headers row. Finding out which options are available might require some Drupal development knowledge. To make the process easier, in today’s article we are presenting a reference of available configuration options for migrate source plugins provided by Drupal core and some contributed modules.

List of configuration options for source plugins

For each migrate source plugin we will present: the module that provides it, the class that defines it, the class that the plugin extends, and any inherited options from the class hierarchy. For each plugin configuration option we will list its name, type, a description, and a note if it is optional.

SourcePluginBase (abstract class)

Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Extends: Drupal\Core\Plugin\PluginBase

This abstract class is extended by most migrate source plugins. This means that the provided configuration keys apply to any source plugin extending it.

List of configuration keys:

  1. skip_count: An optional boolean value. If set, do not attempt to count the source. This includes status and import operations.
  2. cache_counts: An optional boolean value. If set, cache the source count. This saves time calculating the number of available records in the source when a count operation is requested.
  3. cache_key: An optional string value. Uniquely named cache key used for cache_counts.
  4. track_changes: An optional boolean value. If set, the Migrate API will keep a hash of the source rows to determine whether the incoming data has changed. If the hash is different the record is re-imported.
  5. high_water_property: An optional array value. If set, only content with a higher value will be imported. The value is usually a timestamp or serial ID indicating what was the last imported record. This key is configured as an associate array. The name key indicates the column in the source that will be used for the comparison. The alias key is optional and if set it serves as a table alias for the column name.
  6. source_module: An optional string value. Identifies the system providing the data the source plugin will read. If not set, the Migrate API tries to read the value from the source plugin annotation. The source plugin itself determines how the value is used. For example, Migrate Drupal's source plugins expect source_module to be the name of a module that must be installed and enabled in the source database.

The high_water_property and track_changes are mutually exclusive. They are both designed to conditionally import new or updated records from the source. Hence, only one can be configured per migration definition file.

SqlBase (abstract class)

Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\source\SqlBase
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This abstract class is extended by migrate source plugins whose data may be fetched via a database connection. This means that the provided configuration keys apply to any source plugin extending it.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

  1. key: An optional string value. The database key name. Defaults to 'migrate'.
  2. target: An optional string value. The database target name. Defaults to 'default'.
  3. database_state_key: An optional string value. Name of the state key which contains an array with database connection information. The Migrate API will consult the States API using the provided key. The returned value should be an associative array with at least two keys: key and target to determine which database connection to use. A third key database can also be included containing database connection information as seen in the snippet below.
  4. batch_size: An optional integer value. Number of records to fetch from the database during each batch. If omitted, all records are fetched in a single query.
  5. ignore_map: An optional boolean value. Source data is joined to the map table by default to improve performance. If set to TRUE, the map table will not be joined. Using expressions in the query may result in column aliases in the JOIN clause which would be invalid SQL. If you run into this, set ignore_map to TRUE.

To explain how these configuration keys are used, consider the following database connections:

<?php $databases['default']['default'] = [ 'database' => 'drupal-8-or-9-database-name', 'username' => 'drupal-8-or-9-database-username', 'password' => 'drupal-8-or-9-database-password', 'host' => 'drupal-8-or-9-database-server', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ]; $databases['migrate']['default'] = [ 'database' => 'drupal-6-or-7-database-name', 'username' => 'drupal-6-or-7-database-username', 'password' => 'drupal-6-or-7-database-password', 'host' => 'drupal-6-or-7-database-server', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ];

This snippet can be added to settings.php or settings.local.php. The $databases array is a nested array of at least three levels. The first level defines the database keys: default and migrate in our example. The second level defines the database targets: default in both cases. The third level is an array with connection details for each key/target combination. This documentation page contains more information about database configuration.

Based on the specified configuration values, this is how the Migrate API determines which database connection to use:

  • If the source plugin configuration contains database_state_key, its value is taken as the name of a States API key that specifies an array with the database configuration.
  • Otherwise, if the source plugin configuration contains key, the database configuration with that name is used.
  • Otherwise, load a fallback state key from the States API. The value that it tries to read is the global state key: migrate.fallback_state_key.
  • Otherwise, the database connection named migrate is used by default.
  • If all of the above steps fail, a RequirementsException is thrown.

Note that all values configuration keys are optional. If none is set, the plugin will default to use the connection specified under $databases['migrate']['default']. At least, set the key configuration even if the value is migrate. This would make it explicit which connection is being used.

DrupalSqlBase (abstract class)

Module: Migrate Drupal (Drupal Core)
Class: Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase
Extends: Drupal\migrate\Plugin\migrate\source\SqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, and ignore_map.

This abstract class provides general purpose helper methods that are commonly needed when writing source plugins that use a Drupal database as a source. For example, check if the given module exists and read Drupal configuration variables. Check the linked class documentation for more available methods.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration key:

  1. constants: An optional array value used to add module dependencies. The value is an associative array with two possible elements. The first uses the entity_type key to add a dependency on the module that provides the specified entity type. The second uses the module key to directly add a dependency on the specified module. In both cases, the DependencyTrait is used for setting the dependencies.

Warning: A plugin extending this abstract class might want to use this configuration key in the source definition to set module dependencies. If so, the expected keys might clash with other source constants used in the process pipeline. Arrays keys in PHP are case sensitive. Using uppercase in custom source constants might avoid this clash, but it is preferred to use a different name to avoid confusion.

This abstract class is extended by dozens of core classes that provide an upgrade path from Drupal 6 and 7. It is also used by the Commerce Migrate module to read product types, product display types, and shipping flat rates from a Commerce 1 database. The same module follows a similar approach to read data from an Ubercart database. The Paragraphs module also extends it to add and implement Configurable Plugin interface so it can import field collection types and paragraphs types from Drupal 7.

Config

Module: Migrate Drupal (Drupal Core). Plugin ID: d8_config
Class: Drupal\migrate_drupal\Plugin\migrate\source\d8\Config
Extends: Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, ignore_map, and constants.

This plugin allows reading configuration values from a Drupal 8 site by reading its config table.

In addition to the keys provided in the parent class chain, this plugin does not define extra configuration keys. And example configuration for this plugin would be:

source: plugin: d8_config key: migrate skip_count: true

In this case we are setting the key property from SqlBase to use the migrate default database connection. The skip_count from SourcePluginBase indicates that there is no need to count how many records exist in the source database before executing migration operations like importing them.

This plugin is presented to show that Drupal core already offers a way to migrate data from Drupal 8. Remember that there are dozens of other plugins extending DrupalSqlBase. It would be impractical to list them all here. See this API page for a list of all of them.

CSV

Module: Migrate Source CSV. Plugin ID: csv
Class: Drupal\migrate_source_csv\Plugin\migrate\source\CSV
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin allows reading data from a CSV file. We used this plugin in the CSV migration example of the 31 days of migration series.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:

  1. path: A string value. It contains the path to the CSV file. Starting with the 8.x-3.x branch, stream wrappers are supported. The original article contains more details on specifying the location of the CSV file.
  2. ids: An array of string values. The column names listed are used to uniquely identify each record.
  3. header_offset: An optional integer value. The index of record to be used as the CSV header and the thereby each record's field name. It defaults to zero (0) because the index is zero-based. For CSV files with no header row the value should be set to null.
  4. fields: An optional associative array value. It contains a nested array of names and labels to use instead of a header row. If set, it will overwrite the column names obtained from header_offset.
  5. delimiter: An optional string value. It contains a one character column delimiter. It defaults to a comma (,). For example, if your file uses tabs as delimiter, you set this configuration to \t.
  6. enclosure: An optional string value. It contains one character used to enclose the column values. Defaults to double quotation marks (").
  7. escape: An optional string value. It contains one character used for character escaping in the column values. It defaults to a backslash (\).

Important: The configuration options changed significantly between the 8.x-3.x and 8.x-2.x branches. Refer to this change record for a reference of how to configure the plugin for the 8.x-2.x.

For reference, below is the source plugin configuration used in the CSV migration example:

source: plugin: csv path: modules/custom/ud_migrations/ud_migrations_csv_source/sources/udm_photos.csv ids: [photo_id] header_offset: null fields: - name: photo_id label: 'Photo ID' - name: photo_url label: 'Photo URL'

Spreadsheet

Module: Migrate Spreadsheet. Plugin ID: spreadsheet
Class: Drupal\migrate_spreadsheet\Plugin\migrate\source\Spreadsheet
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin allows reading data from Microsoft Excel and LibreOffice Calc files. It requires the PhpOffice/PhpSpreadsheet library and many PHP extensions including ext-zip. Check this page for a full list of dependencies. We used this plugin in the spreadsheet migration examples of the 31 days of migration series.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:

  1. file: A string value. It stores the path to the document to process. You can use a relative path from the Drupal root, an absolute path, or stream wrappers.
  2. worksheet: A string value. It contains the name of the one worksheet to process.
  3. header_row: An optional integer value. This number indicates which row contains the headers. Contrary to CSV migrations, the row number is not zero-based. So, set this value to 1 if headers are on the first row, 2 if they are on the second, and so on.
  4. origin: An optional string value. It defaults to A2. It indicates which non-header cell contains the first value you want to import. It assumes a grid layout and you only need to indicate the position of the top-left cell value.
  5. columns: An optional array value. It is the list of columns you want to make available for the migration. In case of files with a header row, use those header values in this list. Otherwise, use the default title for columns: A, B, C, etc. If this setting is missing, the plugin will return all columns. This is not ideal, especially for very large files containing more columns than needed for the migration.
  6. row_index_column: An optional string value. This is a special column that contains the row number for each record. This can be used as a unique identifier for the records in case your dataset does not provide a suitable value. Exposing this special column in the migration is up to you. If so, you can come up with any name as long as it does not conflict with header row names set in the columns configuration. Important: this is an autogenerated column, not any of the columns that comes with your dataset.
  7. keys: An optional associative array value. If not set, it defaults to the value of row_index_column. It contains the "columns" that uniquely identify each record. The keys are column names as defined in the data_rows key. The values is an array with a single member with key 'type' and value a column type such as 'integer'. For files with a header row, you can use the values set in the columns configuration. Otherwise, use default column titles like A, B, C, etc. In both cases, you can use the row_index_column column if it was set.

Note that nowhere in the plugin configuration you specify the file type. The same setup applies for both Microsoft Excel and LibreOffice Calc files. The library will take care of detecting and validating the proper type.

For reference, below is the source plugin configuration used in the LibreOffice Calc migration example:

source: plugin: spreadsheet file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods worksheet: 'UD Example Sheet' header_row: 1 origin: A2 columns: - book_id - book_title - 'Book author' row_index_column: 'Document Row Index' keys: book_id: type: string

SourcePluginExtension (abstract class)

Module: Migrate Plus
Class: Drupal\migrate_plus\Plugin\migrate\source\SourcePluginExtension
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This abstract class provides extra configuration keys. It is extended by the URL plugin (explained later) and by source plugins provided by other modules like Feeds Migrate.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

  1. fields: An associative array value. Each element represents a field that will be made available to the migration. The following options can be set:
  2. name: A string value. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contains spaces, you need to put double quotation marks (") around it when referring to it in the migration.
  3. label: An optional string value. This is a description used when presenting details about the migration. For example, in the user interface provided by the Migrate Tools module. When defined, you do not use the label to refer to the field. Keep using the name.
  4. selector: A string value. This is another XPath-like string to find the field to import. The value must be relative to the location specified by the item_selector configuration. In the example, the fields are direct children of the records to migrate. Therefore, only the property name is specified (e.g., unique_id). If you had nested objects or arrays, you would use a slash (/) character to go deeper in the hierarchy. This will be demonstrated in the image and paragraph migrations.
  5. ids: An associative array value. It contains the "columns" that uniquely identify each record. The keys are column names as defined in the data_rows key. The values is an array with a single member with key 'type' and value a column type such as 'integer'.

See the code snippet for the Url plugin in the next section for an example of how these configuration options are used.

Url (used for JSON, XML, SOAP, and Google Sheets migrations)

Module: Migrate Plus. Plugin ID: url
Class: Drupal\migrate_plus\Plugin\migrate\source\Url
Extends: Drupal\migrate_plus\Plugin\migrate\source\SourcePluginExtension
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, fields, and ids.

This plugin allows reading data from URLs. Using data parser plugins it is possible to fetch data from JSON, XML, SOAP, and Google Sheets. Note that this source plugin uses other plugins provided by Migrate Plus that might require extra configuration keys in addition to the ones explicitly defined in the plugin class. Those will also be listed.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:

  1. urls: An array of string values. It contains the source URLs to retrieve. If the file data fetcher plugin is used, the location can be a relative path from the Drupal root, an absolute path, a fully-qualified URL, or a stream wrapper. See this article for details on location of the JSON file. If the HTTP data fetcher plugin is used, the value can use any protocol supported by curl. See this article for more details on importing remote JSON files.
  2. data_parser_plugin: A string value. It indicates which data parser plugin to use. Possible values provided by the Migrate Plus module are json, xml, simple_xml, and soap. If the Migrate Google Sheets module is installed, it is possible to set the value to google_sheets. Review the relevant articles in the 31 days of migration series to know more about how each of them are used.

The data parser plugins provide the following configuration keys:

  1. item_selector: A string value. It indicates where in the source file lies the array of records to be migrated. Its value is an XPath-like string used to traverse the file hierarchy. Note that a slash (/) is used to separate each level in the hierarchy.
  2. data_fetcher_plugin: A string value. It indicates which data parser plugin to use. Possible values provided by the Migrate Plus module are file and http.

The HTTP data fetcher plugins provide the following configuration keys:

  1. headers: An associative array value. The key/value pairs represent HTTP headers to be sent when the request is made.
  2. authentication: An associative array value. One of the elements in the array should be the plugin key which indicates the authentication plugin to use. Possible values provided by the Migrate Plus module are basic, digest, and oauth2. Other elements in the array will depend on the selected authentication plugin.

The basic and digest authentication plugins provide the following configuration keys:

  1. username: A string value.
  2. password: A string value.

The OAuth2 authentication plugin requires the sainsburys/guzzle-oauth2-plugin composer package to work. It provides the following configuration keys:

  1. base_uri: A string value.
  2. grant_type: A string value. Possible values are authorization_code, client_credentials, urn:ietf:params:oauth:grant-type:jwt-bearer, password, and refresh_token. Each of these might require extra configuration values.

The client credentials grant type requires the following configuration keys:

  1. token_url: A string value.
  2. client_id: A string value.
  3. client_secret: A string value.

For configuration keys required by other grant types, refer to the classes that implement them. Read this article on adding HTTP request headers and authentication parameters for example configurations.

There are many combinations possible to configure this plugin. In the 31 days of migration series there are many example configurations. For reference, below is the source plugin configuration used in the local JSON node migration example:

source: plugin: url data_fetcher_plugin: file data_parser_plugin: json urls: - modules/custom/ud_migrations/ud_migrations_json_source/sources/udm_data.json item_selector: /data/udm_people fields: - name: src_unique_id label: 'Unique ID' selector: unique_id - name: src_name label: 'Name' selector: name - name: src_photo_file label: 'Photo ID' selector: photo_file - name: src_book_ref label: 'Book paragraph ID' selector: book_ref ids: src_unique_id: type: integer

EmbeddedDataSource

Module: Migrate (Drupal Core). Plugin ID: embedded_data
Class: Drupal\migrate\Plugin\migrate\source\EmbeddedDataSource
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin allows the definition of data to be imported right inside the migration definition file. We used this plugin in many of the examples of the 31 days of migration series. It is also used in many core tests for the Migrate API itself.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

  1. data_rows: An array of all the records to be migrated. Each record might contain an arbitrary number of key-value pairs representing "columns" of data to be imported.
  2. ids: An associative array. It contains the "columns" that uniquely identify each record. The keys are column names as defined in the data_rows key. The values is an array with a single member with key 'type' and value a column type such as 'integer'.

Many examples of 31 days of migration series use this plugin. You can get the example modules from this repository. For reference, below is the source plugin configuration used in the first migration example:

source: plugin: embedded_data data_rows: - unique_id: 1 creative_title: 'The versatility of Drupal fields' engaging_content: 'Fields are Drupal''s atomic data storage mechanism...' - unique_id: 2 creative_title: 'What is a view in Drupal? How do they work?' engaging_content: 'In Drupal, a view is a listing of information. It can a list of nodes, users, comments, taxonomy terms, files, etc...' ids: unique_id: type: integer

This plugin can also be used to create default content when the data is known in advance. We often present Drupal site building workshops. To save time, we use this plugin to create nodes which are later used when explaining how to create Views. Check this repository for an example of this. Note that it uses a different directory structure to store the migrations as explained in this blog post.

ContentEntity

Module: Migrate Drupal (Drupal Core). Plugin ID: content_entity
Class: Drupal\migrate_drupal\Plugin\migrate\source\ContentEntity
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin returns content entities from a Drupal 8 or 9 installation. It uses the Entity API to get the data to migrate. If the source entity type has custom field storage fields or computed fields, this class will need to be extended and the new class will need to load/calculate the values for those fields.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration key:

  1. entity_type: A string value. The entity type ID of the entities being migrated. This is calculated dynamically by the deriver so it is only needed if the deriver is not utilized, i.e., a custom source plugin.
  2. bundle: An optionals string value. If set and the entity type is bundleable, only return entities of this bundle.
  3. include_translations: An optional boolean value. If set, entity translations are included in the returned data. It defaults to TRUE.

For reference, this is how this plugin is configured to get all nodes of type article in their default language only:

source: plugin: content_entity:node bundle: article include_translations: false

Note: this plugin was brought into core in this issue copied from the Drupal 8 migration (source) module. The latter can be used if the source database does not use the default connection.

Table

Module: Migrate Plus. Plugin ID: table
Class: Drupal\migrate_plus\Plugin\migrate\source\Table
Extends: Drupal\migrate\Plugin\migrate\source\SqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, and ignore_map.

This plugin allows reading data from a single database table. It uses one of the database connections for the site as defined by the options. See this test for an example on how to use this plugin.

In addition to the keys provided in the parent class chain, this plugin provides the following configuration key:

  1. table_name: An string value. The table to read values from.
  2. id_fields: An associative array value. IDMap compatible array of fields that uniquely identify each record.
  3. fields: An array value. The elements are fields ("columns") present on the source table.

EmptySource (migrate module)

Module: Migrate (Drupal Core). Plugin ID: empty
Class: Drupal\migrate\Plugin\migrate\source\EmptySource
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

This plugin returns an empty row by default. It can be used as a placeholder to defer setting the source plugin to a deriver. An example of this can be seen in the migrations for Drupal 6 and Drupal 7 entity reference translations. In both cases, the source plugin will be determined by the EntityReferenceTranslationDeriver.

In addition to the keys provided in the parent class chain, this plugin does not define extra configuration keys. If the plugin is used with source constants, a single row containing the constant values will be returned. For example:

source: plugin: empty constants: entity_type: node field_name: body

The plugin will return a single row containing 'entity_type' and 'field_name' elements, with values of 'node' and 'body', respectively. This is not very useful. For the most part, the plugin is used to defer the definition to a deriver as mentioned before.

EmptySource (migrate_drupal module)

Module: Migrate Drupal (Drupal Core). Plugin ID: md_empty
Class: Drupal\migrate_drupal\Plugin\migrate\source\EmptySource
Extends: Drupal\migrate\Plugin\migrate\source\EmptySource
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.

By default, this plugin returns an empty row with Drupal specific config dependencies. If the plugin is used with source constants, a single row containing the constant values will be returned. These can be seen in the user_picture_field.yml and d6_upload_field.yml migrations.

In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:

  1. constants: An optional array value used to add module dependencies. The value is an associative array with one possible element. An entity_type key is used to add a dependency on the module that provides the specified entity type.

Available configuration for other migrate source plugins

In Drupal core itself there are more than 100 migrate source plugins, most of which come from the Migrate Drupal module. And many more are made available by contributed modules. It would be impractical to document them all here. To get a list by yourself, load the plugin.manager.migrate.source service and call its getFieldStorageDefinitions() method. This will return all migrate source plugins provided by the modules that are currently enabled on the site. This Drush command would get the list:

# List of migrate source plugin definitions. $ drush php:eval "print_r(\Drupal::service('plugin.manager.migrate.source')->getDefinitions());" # List of migrate source plugin ids. $ drush php:eval "print_r(array_keys(\Drupal::service('plugin.manager.migrate.source')->getDefinitions()));"

To find out which configuration options are available for any source plugin consider the following:

  • Find the class that defines the plugin and find out which configuration values are read. Some plugins even include the list in the docblock of the class. Search for a pattern similar to $this->configuration['option_name'] or $configuration['option_name']. The plugins can be found in the Drupal\module_name\Plugin\migrate\source namespace. The class itself would be in a file under the /src/Plugin/migrate/source/ directory of the module.
  • Look up in the class hierarchy. The source plugin can use any configuration set directly in its definition class and any parent class. There might be multiple layers of inheritance.
  • Check if the plugin offers some extension mechanism that would allow it to use other types of plugins. For example, the data fetcher, data parses, and authentication plugins provided by Migrate Plus. The Url migrate source plugin does this.

What did you learn in today’s article? Did you know that migrate source plugins can inherit configuration keys from their class hierarchy? Were you aware that there are so many source plugins? Other than the ones listed here, which source plugins have you used? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Jul 09 2020
Jul 09

In a previous article we explained the syntax used to write Drupal migrations. When migrating into content entities, these define several properties that can be included in the process section to populate their values. For example, when importing nodes you can specify the title, publication status, creation date, etc. In the case of users, you can set the username, password, timezone, etc. Finding out which properties are available for an entity might require some Drupal development knowledge. To make the process easier, in today’s article we are presenting a reference of properties available in content entities provided by Drupal core and some contributed modules.

Example migrations mapping of content entity properties

For each entity we will present: the module that provides it, the class that defines it, and the available properties. For each property we will list its name, field type, a description, and a note if the field allows unlimited values (i.e. it has an unlimited cardinality). The list of properties available for a content entity depend on many factors. For example, if the entity is revisionable (e.g. revision_default), translatable (e.g. langcode), or both (e.g. revision_translation_affected). The modules that are enabled on the site can also affect the available properties. For instance, if the “Workspaces” module is installed, it will add a workspace property to many content entities. This reference assumes that Drupal was installed using the standard installation profile and all modules that provide content entities are enabled.

It is worth noting that entity properties are divided in two categories: base field definitions and field storage configurations. Base field configurations will always be available for the entity. On the other hand, the presence of field storage configurations will depend on various factors. For one, they can only be added to fieldable entities. Attaching the fields to the entity can be done manually by the user, by a module, or by an installation profile. Again, this reference assumes that Drupal was installed using the standard installation profile. Among other things, it adds a user_picture image field to the user entity and body, comment, field_image, and field_tags fields to the node entity. For entities that can have multiple bundles, not all properties provided by the field storage configurations will be available in all bundles. For example, with the standard installation profile all content types will have a body field associated with it, but only the article content type has the field_image, and field_tags fields. If subfields are available for the field type, you can migrate into them.

Content (Node) entity

Module: Node (Drupal Core)
Class: Drupal\node\Entity\Node
Related article: Writing your first Drupal migration

List of base field definitions:

  1. nid: (integer) The node ID.
  2. uuid: (uuid) The node UUID.
  3. vid: (integer) Revision ID.
  4. langcode: (language) Language code (e.g. en).
  5. type: (entity_reference to node_type) Content type machine name.
  6. revision_timestamp: (created) The time that the current revision was created.
  7. revision_uid: (entity_reference to user) The user ID of the author of the current revision.
  8. revision_log: (string_long) Briefly describe the changes you have made.
  9. status: (boolean) Node published when set to TRUE.
  10. uid: (entity_reference to user) The user ID of the content author.
  11. title: (string) Title.
  12. created: (created) The time that the node was created.
  13. changed: (changed) The time that the node was last edited.
  14. promote: (boolean) Node promoted to front page when set to TRUE.
  15. sticky: (boolean) Node sticky at top of lists when set to TRUE.
  16. default_langcode: (boolean) A flag indicating whether this is the default translation.
  17. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  18. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  19. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

List of field storage configurations:

  1. body: text_with_summary field.
  2. comment: comment field.
  3. field_image: image field.
  4. field_tags: entity_reference field.

User entity

Module: User (Drupal Core)
Class: Drupal\user\Entity\User
Related articles: Migrating users into Drupal - Part 1 and Migrating users into Drupal - Part 2

List of base field definitions:

  1. uid: (integer) The user ID.
  2. uuid: (uuid) The user UUID.
  3. langcode: (language) The user language code.
  4. preferred_langcode: (language) The user's preferred language code for receiving emails and viewing the site.
  5. preferred_admin_langcode: (language) The user's preferred language code for viewing administration pages.
  6. name: (string) The name of this user.
  7. pass: (password) The password of this user (hashed).
  8. mail: (email) The email of this user.
  9. timezone: (string) The timezone of this user.
  10. status: (boolean) Whether the user is active or blocked.
  11. created: (created) The time that the user was created.
  12. changed: (changed) The time that the user was last edited.
  13. access: (timestamp) The time that the user last accessed the site.
  14. login: (timestamp) The time that the user last logged in.
  15. init: (email) The email address used for initial account creation.
  16. roles: (entity_reference to user_role) The roles the user has. Allows unlimited values.
  17. default_langcode: (boolean) A flag indicating whether this is the default translation.

List of field storage configurations:

  1. user_picture: image field.

Taxonomy term entity

Module: Taxonomy (Drupal Core)
Class: Drupal\taxonomy\Entity\Term
Related article: Migrating taxonomy terms and multivalue fields into Drupal

List of base field definitions:

  1. tid: (integer) The term ID.
  2. uuid: (uuid) The term UUID.
  3. revision_id: (integer) Revision ID.
  4. langcode: (language) The term language code.
  5. vid: (entity_reference to taxonomy_vocabulary) The vocabulary to which the term is assigned.
  6. revision_created: (created) The time that the current revision was created.
  7. revision_user: (entity_reference to user) The user ID of the author of the current revision.
  8. revision_log_message: (string_long) Briefly describe the changes you have made.
  9. status: (boolean) Published.
  10. name: (string) Name.
  11. description: (text_long) Description.
  12. weight: (integer) The weight of this term in relation to other terms.
  13. parent: (entity_reference to taxonomy_term) The parents of this term. Allows unlimited values.
  14. changed: (changed) The time that the term was last edited.
  15. default_langcode: (boolean) A flag indicating whether this is the default translation.
  16. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  17. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  18. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

File entity

Module: File (Drupal Core)
Class: Drupal\file\Entity\File
Related articles: Migrating files and images into Drupal, Migrating images using the image_import plugin, and Migrating images using the image_import plugin

List of base field definitions:

  1. fid: (integer) The file ID.
  2. uuid: (uuid) The file UUID.
  3. langcode: (language) The file language code.
  4. uid: (entity_reference to user) The user ID of the file.
  5. filename: (string) Name of the file with no path components.
  6. uri: (file_uri) The URI to access the file (either local or remote).
  7. filemime: (string) The file's MIME type.
  8. filesize: (integer) The size of the file in bytes.
  9. status: (boolean) The status of the file, temporary (FALSE) and permanent (TRUE).
  10. created: (created) The timestamp that the file was created.
  11. changed: (changed) The timestamp that the file was last changed.

Module: Media (Drupal Core)
Class: Drupal\media\Entity\Media

List of base field definitions:

  1. mid: (integer) The media ID.
  2. uuid: (uuid) The media UUID.
  3. vid: (integer) Revision ID.
  4. langcode: (language) Language code (e.g. en).
  5. bundle: (entity_reference to media_type) Media type.
  6. revision_created: (created) The time that the current revision was created.
  7. revision_user: (entity_reference to user) The user ID of the author of the current revision.
  8. revision_log_message: (string_long) Briefly describe the changes you have made.
  9. status: (boolean) Published.
  10. uid: (entity_reference to user) The user ID of the author.
  11. name: (string) Name.
  12. thumbnail: (image) The thumbnail of the media item.
  13. created: (created) The time the media item was created.
  14. changed: (changed) The time the media item was last edited.
  15. default_langcode: (boolean) A flag indicating whether this is the default translation.
  16. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  17. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  18. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

List of field storage configurations:

  1. field_media_audio_file: file field.
  2. field_media_document: file field.
  3. field_media_image: image field.
  4. field_media_oembed_video: string field.
  5. field_media_video_file: file field.

Module: Comment (Drupal Core)
Class: Drupal\comment\Entity\Comment

List of base field definitions:

  1. cid: (integer) The comment ID.
  2. uuid: (uuid) The comment UUID.
  3. langcode: (language) The comment language code.
  4. comment_type: (entity_reference to comment_type) The comment type.
  5. status: (boolean) Published.
  6. uid: (entity_reference to user) The user ID of the comment author.
  7. pid: (entity_reference to comment) The parent comment ID if this is a reply to a comment.
  8. entity_id: (entity_reference to node) The ID of the entity of which this comment is a reply.
  9. subject: (string) Subject.
  10. name: (string) The comment author's name.
  11. mail: (email) The comment author's email address.
  12. homepage: (uri) The comment author's home page address.
  13. hostname: (string) The comment author's hostname.
  14. created: (created) The time that the comment was created.
  15. changed: (changed) The time that the comment was last edited.
  16. thread: (string) The alphadecimal representation of the comment's place in a thread, consisting of a base 36 string prefixed by an integer indicating its length.
  17. entity_type: (string) The entity type to which this comment is attached.
  18. field_name: (string) The field name through which this comment was added.
  19. default_langcode: (boolean) A flag indicating whether this is the default translation.

List of field storage configurations:

  1. comment_body: text_long field.

Aggregator feed entity

Module: Aggregator (Drupal Core)
Class: Drupal\aggregator\Entity\Feed

List of base field definitions:

  1. fid: (integer) The ID of the aggregator feed.
  2. uuid: (uuid) The aggregator feed UUID.
  3. langcode: (language) The feed language code.
  4. title: (string) The name of the feed (or the name of the website providing the feed).
  5. url: (uri) The fully-qualified URL of the feed.
  6. refresh: (list_integer) The length of time between feed updates. Requires a correctly configured cron maintenance task.
  7. checked: (timestamp) Last time feed was checked for new items, as Unix timestamp.
  8. queued: (timestamp) Time when this feed was queued for refresh, 0 if not queued.
  9. link: (uri) The link of the feed.
  10. description: (string_long) The parent website's description that comes from the <description> element in the feed.
  11. image: (uri) An image representing the feed.
  12. hash: (string) Calculated hash of the feed data, used for validating cache.
  13. etag: (string) Entity tag HTTP response header, used for validating cache.
  14. modified: (timestamp) When the feed was last modified, as a Unix timestamp.

Aggregator feed item entity

Module: Aggregator (Drupal Core)
Class: Drupal\aggregator\Entity\Item

List of base field definitions:

  1. iid: (integer) The ID of the feed item.
  2. langcode: (language) The feed item language code.
  3. fid: (entity_reference to aggregator_feed) The aggregator feed entity associated with this item.
  4. title: (string) The title of the feed item.
  5. link: (uri) The link of the feed item.
  6. author: (string) The author of the feed item.
  7. description: (string_long) The body of the feed item.
  8. timestamp: (created) Posted date of the feed item, as a Unix timestamp.
  9. guid: (string_long) Unique identifier for the feed item.

Custom block entity

Module: Custom Block (Drupal Core)
Class: Drupal\block_content\Entity\BlockContent

List of base field definitions:

  1. id: (integer) The custom block ID.
  2. uuid: (uuid) The custom block UUID.
  3. revision_id: (integer) The revision ID.
  4. langcode: (language) The custom block language code.
  5. type: (entity_reference to block_content_type) The block type.
  6. revision_created: (created) The time that the current revision was created.
  7. revision_user: (entity_reference to user) The user ID of the author of the current revision.
  8. revision_log: (string_long) The log entry explaining the changes in this revision.
  9. status: (boolean) Published.
  10. info: (string) A brief description of your block.
  11. changed: (changed) The time that the custom block was last edited.
  12. reusable: (boolean) A boolean indicating whether this block is reusable.
  13. default_langcode: (boolean) A flag indicating whether this is the default translation.
  14. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  15. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  16. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

List of field storage configurations:

  1. body: text_with_summary field.

Module: Contact (Drupal Core)
Class: Drupal\contact\Entity\Message

List of base field definitions:

  1. uuid: (uuid) The message UUID.
  2. langcode: (language) The message language code.
  3. contact_form: (entity_reference to contact_form) The ID of the associated form.
  4. name: (string) The name of the person that is sending the contact message.
  5. mail: (email) The email of the person that is sending the contact message.
  6. subject: (string) Subject.
  7. message: (string_long) Message.
  8. copy: (boolean) Whether to send a copy of the message to the sender.
  9. recipient: (entity_reference to user) The ID of the recipient user for personal contact messages.

Content moderation state entity

Module: Content Moderation (Drupal Core)
Class: Drupal\content_moderation\Entity\ContentModerationState

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. revision_id: (integer) Revision ID.
  4. langcode: (language) Language.
  5. uid: (entity_reference to user) The username of the entity creator.
  6. workflow: (entity_reference to workflow) The workflow the moderation state is in.
  7. moderation_state: (string) The moderation state of the referenced content.
  8. content_entity_type_id: (string) The ID of the content entity type this moderation state is for.
  9. content_entity_id: (integer) The ID of the content entity this moderation state is for.
  10. content_entity_revision_id: (integer) The revision ID of the content entity this moderation state is for.
  11. default_langcode: (boolean) A flag indicating whether this is the default translation.
  12. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  13. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.

URL alias entity

Module: Path alias (Drupal Core)
Class: Drupal\path_alias\Entity\PathAlias

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. revision_id: (integer) Revision ID.
  4. langcode: (language) Language.
  5. path: (string) The path that this alias belongs to.
  6. alias: (string) An alias used with this path.
  7. status: (boolean) Published.
  8. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  9. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

Shortcut link entity

Module: Shortcut (Drupal Core)
Class: Drupal\shortcut\Entity\Shortcut

List of base field definitions:

  1. id: (integer) The ID of the shortcut.
  2. uuid: (uuid) The UUID of the shortcut.
  3. langcode: (language) The language code of the shortcut.
  4. shortcut_set: (entity_reference to shortcut_set) The bundle of the shortcut.
  5. title: (string) The name of the shortcut.
  6. weight: (integer) Weight among shortcuts in the same shortcut set.
  7. link: (link) The location this shortcut points to.
  8. default_langcode: (boolean) A flag indicating whether this is the default translation.

Workspace entity

Module: Workspaces (Drupal Core)
Class: Drupal\workspaces\Entity\Workspace

List of base field definitions:

  1. id: (string) The workspace ID.
  2. uuid: (uuid) UUID.
  3. revision_id: (integer) Revision ID.
  4. uid: (entity_reference to user) The workspace owner.
  5. label: (string) The workspace name.
  6. parent: (entity_reference to workspace) The parent workspace.
  7. changed: (changed) The time that the workspace was last edited.
  8. created: (created) The time that the workspace was created.
  9. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.

Module: Custom Menu Links (Drupal Core)
Class: Drupal\menu_link_content\Entity\MenuLinkContent

List of base field definitions:

  1. id: (integer) The entity ID for this menu link content entity.
  2. uuid: (uuid) The content menu link UUID.
  3. revision_id: (integer) Revision ID.
  4. langcode: (language) The menu link language code.
  5. bundle: (string) The content menu link bundle.
  6. revision_created: (created) The time that the current revision was created.
  7. revision_user: (entity_reference to user) The user ID of the author of the current revision.
  8. revision_log_message: (string_long) Briefly describe the changes you have made.
  9. enabled: (boolean) A flag for whether the link should be enabled in menus or hidden.
  10. title: (string) The text to be used for this link in the menu.
  11. description: (string) Shown when hovering over the menu link.
  12. menu_name: (string) The menu name. All links with the same menu name (such as "tools") are part of the same menu.
  13. link: (link) The location this menu link points to.
  14. external: (boolean) A flag to indicate if the link points to a full URL starting with a protocol, like http:// (1 = external, 0 = internal).
  15. rediscover: (boolean) Indicates whether the menu link should be rediscovered.
  16. weight: (integer) Link weight among links in the same menu at the same depth. In the menu, the links with high weight will sink and links with a low weight will be positioned nearer the top.
  17. expanded: (boolean) If selected and this menu link has children, the menu will always appear expanded. This option may be overridden for the entire menu tree when placing a menu block.
  18. parent: (string) The ID of the parent menu link plugin, or empty string when at the top level of the hierarchy.
  19. changed: (changed) The time that the menu link was last edited.
  20. default_langcode: (boolean) A flag indicating whether this is the default translation.
  21. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  22. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  23. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

Paragraph entity

Module: Paragraphs module
Class: Drupal\paragraphs\Entity\Paragraph
Related article: Introduction to paragraphs migrations in Drupal

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. revision_id: (integer) Revision ID.
  4. langcode: (language) The paragraphs entity language code.
  5. type: (entity_reference to paragraphs_type) Paragraph type.
  6. status: (boolean) Published.
  7. created: (created) The time that the Paragraph was created.
  8. parent_id: (string) The ID of the parent entity of which this entity is referenced.
  9. parent_type: (string) The entity parent type to which this entity is referenced.
  10. parent_field_name: (string) The entity parent field name to which this entity is referenced.
  11. behavior_settings: (string_long) The behavior plugin settings
  12. default_langcode: (boolean) A flag indicating whether this is the default translation.
  13. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  14. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  15. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

List of field storage configurations:

  1. field_reusable_paragraph: entity_reference field.

Paragraphs library item entity

Module: Paragraphs Library (part of paragraphs module)
Class: Drupal\paragraphs_library\Entity\LibraryItem

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. revision_id: (integer) Revision ID.
  4. langcode: (language) Language.
  5. revision_created: (created) The time that the current revision was created.
  6. revision_uid: (entity_reference to user) The user ID of the author of the current revision.
  7. revision_log: (string_long) Briefly describe the changes you have made.
  8. status: (boolean) Published.
  9. label: (string) Label.
  10. paragraphs: (entity_reference_revisions) Paragraphs.
  11. created: (created) The time that the library item was created.
  12. changed: (changed) The time that the library item was last edited.
  13. uid: (entity_reference to user) The user ID of the library item author.
  14. default_langcode: (boolean) A flag indicating whether this is the default translation.
  15. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  16. revision_translation_affected: (boolean) Indicates if the last edit of a translation belongs to current revision.
  17. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

Profile entity

Module: Profile module
Class: Drupal\profile\Entity\Profile

List of base field definitions:

  1. profile_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. revision_id: (integer) Revision ID.
  4. type: (entity_reference to profile_type) Profile type.
  5. revision_created: (created) The time that the current revision was created.
  6. revision_user: (entity_reference to user) The user ID of the author of the current revision.
  7. revision_log_message: (string_long) Briefly describe the changes you have made.
  8. status: (boolean) Whether the profile is active.
  9. uid: (entity_reference to user) The user that owns this profile.
  10. is_default: (boolean) Whether this is the default profile.
  11. data: (map) A serialized array of additional data.
  12. created: (created) The time when the profile was created.
  13. changed: (changed) The time when the profile was last edited.
  14. revision_default: (boolean) A flag indicating whether this was a default revision when it was saved.
  15. workspace: (entity_reference to workspace) Indicates the workspace that this revision belongs to.

Available properties for other content entities

This reference includes all core content entities and some provided by contributed modules. The next article will include a reference for Drupal Commerce content entities. That being said, it would be impractical to cover all contributed modules. To get a list of yourself for other content entities, load the entity_type.manager service and call its getFieldStorageDefinitions() method passing the machine name of the entity as a parameter. Although this reference only covers content entities, the same process can be used for configuration entities.

What did you learn in today’s article? Did you know that there were so many entity properties in Drupal core? Were you aware that the list of available properties depend on factors like if the entity is fieldable, translatable, and revisionable? Did you know how to find properties for content entities from contributed modules? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Jul 02 2020
Jul 02

In today’s article we are going to provide a reference of all configuration options that can be set in migration definition files. Additional configuration options available for migrations defined as configuration will also be listed. Finally, we present the configuration options for migrations groups.

List of configuration options in YAML definition files

General configuration keys

The following keys can be set for any Drupal migration.

id key

A required string value. It serves as the identifier for the migration. The value should be a machine name. That is, lowercase letters with underscores instead of spaces. The value is for creating mapping and messages tables. For example, if the id is ud_migrations, the Migrate API will create the following migrations migrate_map_ud_migrations and migrate_message_ud_migrations.

label key

A string value. The human-readable label for the migration. The value is used in different interfaces to refer to the migration.

audit key

A boolean value. Defaults to FALSE. It indicates whether the migration is auditable. When set to TRUE, a warning is displayed if entities might be overridden when the migration is executed. For example, when doing an upgrade from a previous version of Drupal, nodes created in the new site before running the automatic upgrade process would be overridden and a warning is logged. The Migrate API checks if the highest destination ID is greater than the highest source ID.

migration_tags key

An array value. It can be set to an optional list of strings representing the tags associated with the migration. They are used by the plugin manager for filtering. For example, you can import or rollback all migrations with the Content tag using the following Drush commands provided by the Migrate Tools module:

$ drush migrate:import --tag='Content'
$ drush migrate:rollback --tag='Content'

source key

A nested array value. This represents the configuration of the source plugin. At a minimum, it contains an id key which indicates which source plugin to use for the migration. Possible values include embedded_data for hardcoded data; csv for CSV files; url for JSON feeds, XML files, and Google Sheets; spreadsheet for Microsoft Excel and LibreOffice Calc files; and many more. Each plugin is configured differently. Refer to our list of configuration options for source plugins to find out what is available for each of them. Additionally, in this section you can define source contents that can be later used in the process pipeline.

process key

A nested array value. This represents the configuration of how source data will be processed and transformed to match the expected destination structure. This section contains a list of entity properties (e.g. nid for a node) and fields (e.g. field_image in the default article content type). Refer to our list of properties for content entities including Commerce related entities to find out which properties can be set depending on your destination (e.g. nodes, users, taxonomy terms, files and images, paragraphs, etc.). For field mappings, you use the machine name of the field as configured in the entity bundle. Some fields have complex structures so you migrate data into specific subfields. Refer to our list of subfields per field type to determine which options are available. When migrating multivalue fields, you might need to set deltas as well. Additionally, you can have pseudofields to store temporary values within the process pipeline.

For each entity property, field, or pseudofield, you can use one or more process plugins to manipulate the data. Many of them are provided by Drupal core while others become available when contributed modules are installed on the site like Migrate Plus and Migrate Process Extra. Throughout the 31 days of migrations series, we provided examples of how many process plugins are used. Most of the work for migrations will be devoted to configuring the right mappings in the process section. Make sure to check our debugging tips in case some values are not migrated properly.

destination key

A nested array value. This represents the configuration of the destination plugin. At a minimum, it contains an id key which indicates which destination plugin to use for the migration. Possible values include entity:node for nodes, entity:user for users, entity:taxonomy_term for taxonomy terms, entity:file for files and images, entity_reference_revisions:paragraph for paragraphs, and many more. Each plugin is configured differently. Refer to our list of configuration options for destination plugins to find out what is available for each of them.

This is an example migration from the ud_migrations_csv_source module used in the article on CSV sources.

id: udm_csv_source_paragraph
label: 'UD dependee paragraph migration for CSV source example'
migration_tags:
  - UD CSV Source
  - UD Example
source:
  plugin: csv
  path: modules/custom/ud_migrations/ud_migrations_csv_source/sources/udm_book_paragraph.csv
  ids: [book_id]
  header_offset: null
  fields:
    - name: book_id
    - name: book_title
    - name: 'Book author'
process:
  field_ud_book_paragraph_title: book_title
  field_ud_book_paragraph_author: 'Book author'
destination:
  plugin: 'entity_reference_revisions:paragraph'
  default_bundle: ud_book_paragraph

migration_dependencies key

A nested array value. The value is used by the Migrate API to make sure the listed migrations are executed in advance of the current one. For example, a node migration might require users to be imported first so you can specify who is the author of the node. Also, it is possible to list optional migrations so that they are only executed in case they are present. The following example from the d7_node.yml migration shows how key can be configured:

migration_dependencies:
  required:
    - d7_user
    - d7_node_type
  optional:
    - d7_field_instance
    - d7_comment_field_instance

To configure the migration dependencies you specify required and optional subkeys whose values are an array of migration IDs. If no dependencies are needed, you can omit this key. Alternatively, you can set either required or optional dependencies without having to specify both keys. As of Drupal 8.8 an InvalidPluginDefinitionException will be thrown if the migration_dependencies key is incorrectly formatted.

class key

A string value. If set, it should point to the class used as the migration plugin. The MigrationPluginManager sets this key to \Drupal\migrate\Plugin\Migration by default. Whatever class specified here should implement the MigrationInterface. This configuration key rarely needs to be set as the default value can be used most of the time. In Drupal core there are few cases where a different class is used as the migration plugin:

deriver key

A string value. If set, it should point to the class used as a plugin deriver for this migration. This is an advanced topic that will be covered in a future entry. In short, it is a mechanism in which new migration plugins can be created dynamically from a base template. For example, the d7_node.yml migration uses the D7NodeDeriver to create one node migration per content type during a Drupal upgrade operation. In this case, the configuration key is set to Drupal\node\Plugin\migrate\D7NodeDeriver. There are many other derivers used by the Migrate API including D7NodeDeriver, D7TaxonomyTermDeriver, EntityReferenceTranslationDeriver, D6NodeDeriver, and D6TermNodeDeriver.

field_plugin_method key

A string value. This key must be set only in migrations that use Drupal\migrate_drupal\Plugin\migrate\FieldMigration as the plugin class. They take care of importing fields from previous versions of Drupal. The following is a list of possible values:

  • alterFieldMigration as set by d7_field.yml.
  • alterFieldFormatterMigration as set by d7_field_formatter_settings.yml.
  • alterFieldInstanceMigration as set by d7_field_instance.yml.
  • alterFieldWidgetMigration as set by d7_field_instance_widget_settings.yml

There are Drupal 6 counterparts for these migrations. Note that the field_plugin_method key is a replacement for the deprecated cck_plugin_method key.

provider key

An array value. If set, it should contain a list of module machine names that must be enabled for this migration to work. Refer to the d7_entity_reference_translation.yml and d6_entity_reference_translation.yml migrations for examples of possible values. This key rarely needs to be set. Usually the same module providing the migration definition file is the only one needed for the migration to work.

Deriver specific configuration keys

It is possible that some derivers require extra configuration keys to be set. For example, the EntityReferenceTranslationDeriver the target_types to be set. Refer to the d7_entity_reference_translation.yml and d6_entity_reference_translation.yml migrations for examples of possible values. These migrations are also interesting because the source, process, and destination keys are not configured in the YAML definition files. They are actually set dynamically by the deriver.

Migration configuration entity keys

The following keys should be used only if the migration is created as a configuration entity using the Migrate Plus module. Only the migration_group key is specific to migrations as configuration entities. All other keys apply for any configuration entity in Drupal. Refer to the ConfigEntityBase abstract class for more details on how they are used.

migration_group key

A string value. If set, it should correspond to the id key of a migration group configuration entity. This allows inheriting configuration values from the group. For example, the database connection for the source configuration. Refer to this article for more information on sharing configuration using migration groups. They can be used to import or rollback all migrations within a group using the following Drush commands provided by the Migrate Tools module:

$ drush migrate:import --group='udm_config_group_json_source'
$ drush migrate:rollback --group='udm_config_group_json_source'

uuid key

A string value. The value should be a UUID v4. If not set, the configuration management system will create a UUID on the fly and assign it to the migration entity. Refer to this article for more details on setting UUIDs for migrations defined as configuration entities.

langcode key

A string value. The language code of the entity's default language. English is assumed by default. For example: en.

status key

A boolean value. The enabled/disabled status of the configuration entity. For example: true.

dependencies key

A nested array value. Configuration entities can declare dependencies on modules, themes, content entities, and other configuration entities. These dependencies can be recalculated on save operations or enforced. Refer to the ConfigDependencyManager class’ documentation for details on how to configure this key. One practical use of this key is to automatically remove the migration (configuration entity) when the module that defined it is uninstalled. To accomplish this, you need to set an enforced module dependency on the same module that provides the migration. This is explained in the article on defining Drupal migrations as configuration entities. For reference, below is a code snippet from that article showing how to configure this key:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
  enforced:
    module:
      - ud_migrations_config_json_source

Migration group configuration entity keys

Migration groups are also configuration entities. That means that they can have uuid, langcode, status, and dependencies keys as explained before. Additionally, the following keys can be set. These other keys can be set for migration groups:

id key

A required string value. It serves as the identifier for the migration group. The value should be a machine name.

label key

A string value. The human-readable label for the migration group.

description key

A string value. More information about the group.

source_type key

A string value. Short description of the type of source. For example: "Drupal 7" or "JSON source".

module key

A string value. The machine name of a dependent module. This key rarely needs to be set. A configuration entity is always dependent on its provider, the module defining the migration group.

shared_configuration key

A nested array value. Any configuration key for a migration can be set under this key. Those values will be inherited by any migration associated with the current group. Refer to this article for more information on sharing configuration using migration groups. The following is an example from the ud_migrations_config_group_json_source module from the article on executing migrations from the Drupal interface.

uuid: 78925705-a799-4749-99c9-a1725fb54def
id: udm_config_group_json_source
label: 'UD Config Group (JSON source)'
description: 'A container for migrations about individuals and their favorite books. Learn more at https://understanddrupal.com/migrations.'
source_type: 'JSON resource'
shared_configuration:
  dependencies:
    enforced:
      module:
        - ud_migrations_config_group_json_source
  migration_tags:
    - UD Config Group (JSON Source)
    - UD Example
  source:
    plugin: url
    data_fetcher_plugin: file
    data_parser_plugin: json
    urls:
      - modules/custom/ud_migrations/ud_migrations_config_group_json_source/sources/udm_data.json

What did you learn in today’s article? Did you know there were so many configuration options for migration definition files? Were you aware that some keys apply only when migrations are defined as configuration entities? Have you used migrations groups to share configuration across migrations? Share your answers in the comments. Also, I would be grateful if you shared this blog post with friends and colleagues.

Jul 01 2020
Jul 01

On Tuesday, July 7, Agaric will host 3 free online webinars about Drupal 9. We invite the community to join us to learn more about the latest version of our favorite CMS. We will leave time at the end of each presentation for questions from the audience. All webinars will be presented online via Zoom. Fill out the form at the end of the post to reserve your seat. We look forward to seeing you.

Getting started Drupal 9

Time: 10:00 AM - 11:00 AM Eastern Time (EDT)

This webinar will cover basic site building concepts. You will learn what is a node and how they differ from content types. We are going to explain why fields are so useful for structuring your site's content and the benefits of doing this. We will cover how to use Views to create listing of content. Layout builder, blocks, taxonomies, and the user permissions system will also be explained.

Introduction to Drupal 9 migrations

Time: 11:30 AM - 12:30 AM Eastern Time (EDT)

This webinar will present an overview of the Drupal migrations system. You will learn about how the Migrate API works and what assumptions it makes. We will explain the syntax to write migrations how different source, process, and destinations plugins work. Recommended migration workflows and debugging tips will also be presented. No previous experience with the Migrate API nor PHP is required to attend.

Drupal 9 upgrades: how and when to move your Drupal 7 sites?

Time: 1:00 PM - 2:00 PM Eastern Time (EDT)

This webinar will present different tools and workflows to upgrade your Drupal 7 site to Drupal 9. We will run through what things to consider when planning an upgrade. This will include how to make site architecture changes, modules that do not have D9 counterparts, what to do when there are no automated upgrade paths.

Agaric is also offering full-day trainings for these topics later this month. Dates, prices, and registration options is available at https://agaric.coop/training

Jun 23 2020
Jun 23

Agaric is excited to announce online training on Drupal migrations and upgrades. In July 2020, we will offer three trainings: Drupal 8/9 content migrations, Upgrading to Drupal 8/9 using the Migrate API, and Getting started with Drupal 9.

We have been providing training for years at Drupal events and privately for clients. At DrupalCon Seattle 2019, our migration training was sold out with 40+ attendees and received very positive feedback. We were scheduled to present two trainings at DrupalCon Minneapolis 2020: one on Drupal migrations and the other on Drupal upgrades. When the conference pivoted to an online event, all trainings were cancelled. To fill the void, we are moving the full training experience online for individuals and organizations who want to learn how to plan and execute successful Drupal migration/upgrade projects.

Up to date with Drupal 9

Drupal is always evolving and the Migrate API is no exception. New features and improvements are added all the time. We regularly update our curriculum to cover the latest changes in the API. This time, both trainings will use Drupal 9 for all the examples! If you are still using Drupal 8, don't worry as the example code is compatible with both major versions of Drupal. We will also cover the differences between Drupal 8 and 9.

Drupal 8/9 content migrations

In this training you will learn to move content into Drupal 8 and 9 using the Migrate API. An overview of the Extract-Transform-Load (ETL) pattern that migrate implements will be presented. Source, process, and destination plugins will be explained to show how each affects the migration process. By the end of the workshop, you will have a better understanding on how the migrate ecosystem works and the thought process required to plan and perform migrations. All examples will use YAML files to configure migrations. No PHP coding required.

Date: Tuesday, July 21, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $500 USD

Click here to register

Upgrading to Drupal 8/9 using the Migrate API

In this training you will learn to use the Migrate API to upgrade your Drupal 6/7 site to Drupal 8/9. You will practice different migration strategies, accommodate changes in site architecture, get tips on troubleshooting issues, and much more. After the training, you will know how to plan and execute successful upgrade projects.

Date: Thursday, July 23, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $500 USD

Click here to register

Getting started with Drupal 9

We are also offering a training for people who want to get a solid foundation in Drupal site building. Basic concepts will be explained and put into practice through various exercises. The objective is that someone, who might not even know about Drupal, can understand the different concepts and building blocks to create a website. A simple, fully functional website will be built over the course of the day-long class.

Date: Monday, July 13, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $250 USD

Click here to register

Discounts and scholarships available

Anyone is eligible for a 15% discount on their second training. Additionally, if you are a member of an under-represented community who cannot afford the full price of the training, we have larger discounts and full scholarship available. Ask Agaric to learn more about them.

Customized training available

We also offer customized training for you or your team's specific needs. Site building, module development, theming, and data migration are some of the topics we cover. Check out our training page or ask Agaric for more details. Custom training can be delivered online or on-site in English or Spanish.

Meet your lead trainer

Mauricio Dinarte is a frequent speaker and trainer at conferences around the world. He is passionate about Drupal, teaching, and traveling. Over the last few years, he has presented 30+ sessions and full-day trainings at 20+ DrupalCamps and DrupalCons over America and Europe. In August 2019, he wrote an article every day to share his expertise on Drupal migrations.

We look forward to seeing you online in July at any or all of these trainings!

Jun 08 2020
Jun 08

While working on making a module compatible with Drupal 9 I found that the module was using an obsolete function that became a new service. It was something like this:


/**
 * My plugin.
 *
 * @SearchPlugin(
 *   id = "my_plugin",
 *   title = @Translation("My plugin")
 * )
 */
class MyPluginSearch extends SearchPluginBase implements AccessibleInterface, SearchIndexingInterface {

  /**
   * {@inheritdoc}
   */
  public static function create(
    ContainerInterface $container,
    array $configuration,
    $plugin_id,
    $plugin_definition
  ) {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition
    );
  }

  /** ... **/
  public function indexClear() {
    search_index_clear($this->getPluginId());
  }
}

The function search_index_clear is now part of the new search.index service that was added on Drupal 8.8. In order to keep this working on Drupal 8.8+ and Drupal 9 we need to inject the service in the create function. But if we do this unconditionally, we will get an error in Drupal 8.7 because that service was added on 8.8. What to do then?

Fortunately years ago I read an article that addressed a similar need. It talked about how to safely extends Drupal 8 plugin classes without fear of constructor changes. In my case I didn't want to change the constructor to keep it compatible with Drupal 8.7 and below. At the same time, I wanted to inject the new service to use it in Drupal 8.8+ and Drupal 9. I just modified a bit my code to something like this:


/**
 * My plugin.
 *
 * @SearchPlugin(
 *   id = "my_plugin",
 *   title = @Translation("My plugin")
 * )
 */
class MyPluginSearch extends SearchPluginBase implements AccessibleInterface, SearchIndexingInterface {

  /** ... */
  protected $searchIndex;

  /**
   * {@inheritdoc}
   */
  public static function create(
    ContainerInterface $container,
    array $configuration,
    $plugin_id,
    $plugin_definition
  ) {
    $instance =  new static(
      $configuration,
      $plugin_id,
      $plugin_definition
    );

    // Only inject the service in Drupal 8.8 or newer.
    if (floatval(\Drupal::VERSION) >= 8.8) {
      $instance->searchIndex = $container->get('search.index');
    }

    return $instance;
  }

  /** ... **/
  public function indexClear() {
    if (floatval(\Drupal::VERSION) >= 8.8) {
      $this->searchIndex->clear($this->getPluginId());
    }
    else {
      search_index_clear($this->getPluginId());
    }
  }
}

And that's it, Drupal 8.8 and newer will take advantage of the new service while we keep this compatible with Drupal 8.7. This will give users more time to upgrade to Drupal 8.8+ or Drupal 9.

May 05 2020
May 05

List of Drupal Commerce content entities.

For each entity we will present: the module that provides it, the class that defines it, and the available properties. For each property we will list its name, field type, a description, and a note if the field allows unlimited values (i.e. it has an unlimited cardinality). The list of properties available for a content entity depend on many factors. For example, if the entity is revisionable (e.g. revision_default), translatable (e.g. langcode), or both (e.g. revision_translation_affected). The modules that are enabled on the site can also affect the available properties. For instance, if the “Workspaces” module is installed, it will add a workspace property to many content entities. This reference assumes that Drupal was installed using the standard installation profile and only Drupal Commerce related modules that provide content entities are enabled.

It is worth noting that entity properties are divided in two categories: base field definitions and field storage configurations. Base field configurations will always be available for the entity. On the other hand, the presence of field storage configurations will depend on various factors. For one, they can only be added to fieldable entities. Attaching the fields to the entity can be done manually by the user, by a module, or by an installation profile. Again, this reference assumes that Drupal was installed using the standard installation profile with Drupal Commerce related modules enabled. By default, the commerce_product entity adds a bodyfield. For entities that can have multiple bundles, not all properties provided by the field storage configurations will be available in all bundles. For example, with the standard installation profile all content types will have a body field associated with it, but only the article content type has the field_image, and field_tags fields. If subfields are available for the field type, you can migrate into them.

If you are migrating into Drupal Commerce, make sure to check the Commerce Migrate module. It offers migrate destination field handlers for commerce fields and a plugin for commerce product types. It also provides a migration path from Commerce 1 (Drupal 7), Ubercart, and other e-commerce platforms. It is even possible to import data from other platforms like WooCommerce, Magento, and Shopify via CSV exports.

Store entity

Module: Commerce Store (part of commerce module)
Class: Drupal\commerce_store\Entity\Store

List of base field definitions:

  1. store_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. langcode: (language) Language.
  4. type: (entity_reference to commerce_store_type) Type. The store type.
  5. uid: (entity_reference to user) Owner. The store owner.
  6. name: (string) Name. The store name.
  7. mail: (email) Email. Store email notifications are sent from this address.
  8. default_currency: (entity_reference to commerce_currency) Default currency. The default currency of the store.
  9. timezone: (list_string) Timezone. Used when determining promotion and tax availability.
  10. address: (address) Address. The store address.
  11. billing_countries: (list_string) Supported billing countries. Allows unlimited values.
  12. path: (path) URL alias. The store URL alias.
  13. is_default: (boolean) Default. Whether this is the default store.
  14. default_langcode: (boolean) Default translation. A flag indicating whether this is the default translation.
  15. shipping_countries: (list_string) Supported shipping countries. Allows unlimited values.
  16. prices_include_tax: (boolean) Prices are entered with taxes included.
  17. tax_registrations: (list_string) Tax registrations. The countries where the store is additionally registered to collect taxes. Allows unlimited values.

Product entity

Module: Commerce Product (part of commerce module)
Class: Drupal\commerce_product\Entity\Product

List of base field definitions:

  1. product_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. langcode: (language) Language.
  4. type: (entity_reference to commerce_product_type) Product type.
  5. status: (boolean) Published.
  6. stores: (entity_reference to commerce_store) Stores. The product stores. Allows unlimited values.
  7. uid: (entity_reference to user) Author. The product author.
  8. title: (string) Title. The product title.
  9. variations: (entity_reference to commerce_product_variation) Variations. The product variations. Allows unlimited values.
  10. created: (created) Created. The time when the product was created.
  11. changed: (changed) Changed. The time when the product was last edited.
  12. default_langcode: (boolean) Default translation. A flag indicating whether this is the default translation.

List of field storage configurations:

  1. body: text_with_summary field.

Product variation entity

Module: Commerce Product (part of commerce module)
Class: Drupal\commerce_product\Entity\ProductVariation

List of base field definitions:

  1. variation_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. langcode: (language) Language.
  4. type: (entity_reference to commerce_product_variation_type) Product variation type.
  5. status: (boolean) Published.
  6. uid: (entity_reference to user) Author. The variation author.
  7. product_id: (entity_reference to commerce_product) Product. The parent product.
  8. sku: (string) SKU. The unique, machine-readable identifier for a variation.
  9. title: (string) Title. The variation title.
  10. list_price: (commerce_price) List price. The list price.
  11. price: (commerce_price) Price. The price
  12. created: (created) Created. The time when the variation was created.
  13. changed: (changed) Changed. The time when the variation was last edited.
  14. default_langcode: (boolean) Default translation. A flag indicating whether this is the default translation.

Product attribute value entity

Module: Commerce Product (part of commerce module)
Class: Drupal\commerce_product\Entity\ProductAttributeValue

List of base field definitions:

  1. attribute_value_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. langcode: (language) Language.
  4. attribute: (entity_reference to commerce_product_attribute) Attribute.
  5. name: (string) Name. The attribute value name.
  6. weight: (integer) Weight. The weight of this attribute value in relation to others.
  7. created: (created) Created. The time when the attribute value was created.
  8. changed: (changed) Changed. The time when the attribute value was last edited.
  9. default_langcode: (boolean) Default translation. A flag indicating whether this is the default translation.

Order entity

Module: Commerce Order (part of commerce module)
Class: Drupal\commerce_order\Entity\Order

List of base field definitions:

  1. order_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (entity_reference to commerce_order_type) Order type.
  4. order_number: (string) Order number. The order number displayed to the customer.
  5. store_id: (entity_reference to commerce_store) Store. The store to which the order belongs.
  6. uid: (entity_reference to user) Customer. The customer.
  7. mail: (email) Contact email. The email address associated with the order.
  8. ip_address: (string) IP address. The IP address of the order.
  9. billing_profile: (entity_reference_revisions) Billing information. Billing profile
  10. order_items: (entity_reference to commerce_order_item) Order items. The order items. Allows unlimited values.
  11. adjustments: (commerce_adjustment) Adjustments. Allows unlimited values.
  12. total_price: (commerce_price) Total price. The total price of the order.
  13. total_paid: (commerce_price) Total paid. The total paid price of the order.
  14. state: (state) State. The order state.
  15. data: (map) Data. A serialized array of additional data.
  16. locked: (boolean) Locked.
  17. created: (created) Created. The time when the order was created.
  18. changed: (changed) Changed. The time when the order was last edited.
  19. placed: (timestamp) Placed. The time when the order was placed.
  20. completed: (timestamp) Completed. The time when the order was completed.
  21. cart: (boolean) Cart.
  22. checkout_flow: (entity_reference to commerce_checkout_flow) Checkout flow.
  23. checkout_step: (string) Checkout step.
  24. payment_gateway: (entity_reference to commerce_payment_gateway) Payment gateway. The payment gateway.
  25. payment_method: (entity_reference to commerce_payment_method) Payment method. The payment method.
  26. coupons: (entity_reference to commerce_promotion_coupon) Coupons. Coupons that have been applied to order. Allows unlimited values.

Order item entity

Module: Commerce Order (part of commerce module)
Class: Drupal\commerce_order\Entity\OrderItem

List of base field definitions:

  1. order_item_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (entity_reference to commerce_order_item_type) Order item type.
  4. order_id: (entity_reference to commerce_order) Order. The parent order.
  5. purchased_entity: (entity_reference to node) Purchased entity. The purchased entity.
  6. title: (string) Title. The order item title.
  7. quantity: (decimal) Quantity. The number of purchased units.
  8. unit_price: (commerce_price) Unit price. The price of a single unit.
  9. overridden_unit_price: (boolean) Overridden unit price. Whether the unit price is overridden.
  10. total_price: (commerce_price) Total price. The total price of the order item.
  11. adjustments: (commerce_adjustment) Adjustments. Allows unlimited values.
  12. uses_legacy_adjustments: (boolean) Uses legacy adjustments.
  13. data: (map) Data. A serialized array of additional data.
  14. created: (created) Created. The time when the order item was created.
  15. changed: (changed) Changed. The time when the order item was last edited.

Payment method entity

Module: Commerce Payment (part of commerce module)
Class: Drupal\commerce_payment\Entity\PaymentMethod

List of base field definitions:

  1. method_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (string) Payment method type.
  4. payment_gateway: (entity_reference to commerce_payment_gateway) Payment gateway. The payment gateway.
  5. payment_gateway_mode: (string) Payment gateway mode. The payment gateway mode.
  6. uid: (entity_reference to user) Owner. The payment method owner.
  7. remote_id: (string) Remote ID. The payment method remote ID.
  8. billing_profile: (entity_reference_revisions) Billing profile. Billing profile
  9. reusable: (boolean) Reusable. Whether the payment method is reusable.
  10. is_default: (boolean) Default. Whether this is the user's default payment method.
  11. expires: (timestamp) Expires. The time when the payment method expires. 0 for never.
  12. created: (created) Created. The time when the payment method was created.
  13. changed: (changed) Changed. The time when the payment method was last edited.

Payment entity

Module: Commerce Payment (part of commerce module)
Class: Drupal\commerce_payment\Entity\Payment

List of base field definitions:

  1. payment_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (string) Payment type.
  4. payment_gateway: (entity_reference to commerce_payment_gateway) Payment gateway. The payment gateway.
  5. payment_gateway_mode: (string) Payment gateway mode. The payment gateway mode.
  6. payment_method: (entity_reference to commerce_payment_method) Payment method. The payment method.
  7. order_id: (entity_reference to commerce_order) Order. The parent order.
  8. remote_id: (string) Remote ID. The remote payment ID.
  9. remote_state: (string) Remote State. The remote payment state.
  10. amount: (commerce_price) Amount. The payment amount.
  11. refunded_amount: (commerce_price) Refunded amount. The refunded payment amount.
  12. state: (state) State. The payment state.
  13. authorized: (timestamp) Authorized. The time when the payment was authorized.
  14. expires: (timestamp) Expires. The time when the payment expires. 0 for never.
  15. completed: (timestamp) Completed. The time when the payment was completed.
  16. test: (boolean) Test. Whether this is a test payment.
  17. captured: (timestamp) Captured. The time when the payment was captured.

Module: Commerce Promotion (part of commerce module)
Class: Drupal\commerce_promotion\Entity\Promotion

List of base field definitions:

  1. promotion_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. langcode: (language) Language.
  4. name: (string) Name. The promotion name.
  5. display_name: (string) Display name. If provided, shown on the order instead of "Discount".
  6. description: (string_long) Description. Additional information about the promotion to show to the customer
  7. order_types: (entity_reference to commerce_order_type) Order types. The order types for which the promotion is valid. Allows unlimited values.
  8. stores: (entity_reference to commerce_store) Stores. The stores for which the promotion is valid. Allows unlimited values.
  9. offer: (commerce_plugin_item:commerce_promotion_offer) Offer type.
  10. conditions: (commerce_plugin_item:commerce_condition) Conditions. Allows unlimited values.
  11. condition_operator: (list_string) Condition operator. The condition operator.
  12. coupons: (entity_reference to commerce_promotion_coupon) Coupons. Coupons which allow promotion to be redeemed. Allows unlimited values.
  13. usage_limit: (integer) Usage limit. The maximum number of times the promotion can be used. 0 for unlimited.
  14. start_date: (datetime) Start date. The date the promotion becomes valid.
  15. end_date: (datetime) End date. The date after which the promotion is invalid.
  16. compatibility: (list_string) Compatibility with other promotions.
  17. status: (boolean) Status. Whether the promotion is enabled.
  18. weight: (integer) Weight. The weight of this promotion in relation to others.
  19. default_langcode: (boolean) Default translation. A flag indicating whether this is the default translation.

Module: Commerce Promotion (part of commerce module)
Class: Drupal\commerce_promotion\Entity\Coupon

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. promotion_id: (entity_reference to commerce_promotion) Promotion. The parent promotion.
  4. code: (string) Coupon code. The unique, machine-readable identifier for a coupon.
  5. usage_limit: (integer) Usage limit. The maximum number of times the coupon can be used. 0 for unlimited.
  6. status: (boolean) Status. Whether the coupon is enabled.

Log entity

Module: Commerce Log (part of commerce module)
Class: Drupal\commerce_log\Entity\Log

List of base field definitions:

  1. log_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. uid: (entity_reference to user) User. The user for the log.
  4. template_id: (string) Log template ID. The log template plugin ID
  5. category_id: (string) Log category ID. The log category plugin ID
  6. source_entity_id: (integer) Source entity ID. The source entity ID
  7. source_entity_type: (string) Source entity type. The source entity type
  8. params: (map) Params. A serialized array of parameters for the log template.
  9. created: (created) Created. The time when the log was created.

Price list entity

Module: Commerce Pricelist
Class: Drupal\commerce_pricelist\Entity\PriceList

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (string) Price list bundle.
  4. uid: (entity_reference to user) Owner. The user that owns this price list.
  5. name: (string) Name. The name of the price list.
  6. stores: (entity_reference to commerce_store) Stores. The stores for which the price list is valid. Allows unlimited values.
  7. customer: (entity_reference to user) Customer. The customer for which the price list is valid.
  8. customer_roles: (entity_reference to user_role) Customer roles. The customer roles for which the price list is valid. Allows unlimited values.
  9. start_date: (datetime) Start date. The date the price list becomes valid.
  10. end_date: (datetime) End date. The date after which the price list is invalid.
  11. weight: (integer) Weight. The weight of this price list in relation to other price lists.
  12. status: (boolean) Status. Whether the price list is enabled.
  13. changed: (changed) Changed. The time when the price list was last edited.

Price list item entity

Module: Commerce Pricelist
Class: Drupal\commerce_pricelist\Entity\PriceListItem

List of base field definitions:

  1. id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (string) Price list item bundle.
  4. uid: (entity_reference to user) Owner. The user that owns this price list item.
  5. price_list_id: (entity_reference to commerce_pricelist) Price list. The parent price list.
  6. purchasable_entity: (entity_reference to commerce_product_variation) Purchasable entity. The purchasable entity.
  7. quantity: (decimal) Quantity. The quantity tier.
  8. list_price: (commerce_price) List price. The list price.
  9. price: (commerce_price) Price. The price.
  10. status: (boolean) Status. Whether the price list item is enabled.
  11. changed: (changed) Changed. The time when the price list item was last edited.

Shipment entity

Module: Shipping
Class: Drupal\commerce_shipping\Entity\Shipment

List of base field definitions:

  1. shipment_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. type: (entity_reference to commerce_shipment_type) Shipment type.
  4. order_id: (entity_reference to commerce_order) Order. The parent order.
  5. package_type: (string) Package type. The package type.
  6. shipping_method: (entity_reference to commerce_shipping_method) Shipping method. The shipping method
  7. shipping_service: (string) Shipping service. The shipping service.
  8. shipping_profile: (entity_reference_revisions) Shipping information.
  9. title: (string) Title. The shipment title.
  10. items: (commerce_shipment_item) Items. Allows unlimited values.
  11. weight: (physical_measurement) Weight.
  12. original_amount: (commerce_price) Original amount. The original amount.
  13. amount: (commerce_price) Amount. The amount.
  14. adjustments: (commerce_adjustment) Adjustments. Allows unlimited values.
  15. tracking_code: (string) Tracking code. The shipment tracking code.
  16. state: (state) State. The shipment state.
  17. data: (map) Data. A serialized array of additional data.
  18. created: (created) Created. The time when the shipment was created.
  19. changed: (changed) Changed. The time when the shipment was last updated.
  20. shipped: (timestamp) Shipped. The time when the shipment was shipped.

Shipping method entity

Module: Shipping
Class: Drupal\commerce_shipping\Entity\ShippingMethod

List of base field definitions:

  1. shipping_method_id: (integer) ID.
  2. uuid: (uuid) UUID.
  3. langcode: (language) Language.
  4. stores: (entity_reference to commerce_store) Stores. The stores for which the shipping method is valid. Allows unlimited values.
  5. plugin: (commerce_plugin_item:commerce_shipping_method) Plugin.
  6. name: (string) Name. The shipping method name.
  7. conditions: (commerce_plugin_item:commerce_condition) Conditions. Allows unlimited values.
  8. condition_operator: (list_string) Condition operator. The condition operator.
  9. weight: (integer) Weight. The weight of this shipping method in relation to others.
  10. status: (boolean) Enabled. Whether the shipping method is enabled.
  11. default_langcode: (boolean) Default translation. A flag indicating whether this is the default translation.

Available properties for other content entities

This reference includes Drupal Commerce content entities and some provided by related contributed modules. The previous article included a reference for Drupal core content entities. That being said, it would be impractical to cover all contributed modules. To get a list of yourself for other content entities, load the entity_type.manager service and call its getFieldStorageDefinitions() method passing the machine name of the entity as a parameter. Although this reference only covers content entities, the same process can be used for configuration entities.

What did you learn in today’s article? Did you know that there were so many entity properties provided by Drupal Commerce? Were you aware that the list of available properties depend on factors like if the entity is fieldable, translatable, and revisionable? Did you know how to find properties for content entities from contributed modules? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Apr 21 2020
Apr 21

We partnered with the City of Cambridge to redesign Find It Cambridge, an online opportunity locator serving city residents, in a unique way— designing and developing out in the open, and releasing the software under an open-source license so that other cities can spin up their own Find It platforms.

Cambridge, like many cities, has a wide array of programs and events happening to serve residents. However, it can be difficult for people to find and compare the many offerings out there. The city website has a calendar of events and list of departments. These are limited to government-run programs, though. There are myriad nonprofits and community groups that run programs and events that go unlisted. Social media platforms like Facebook have filled the gap in some areas, but promotion of these opportunities relies in many ways on people's social connections— leaving those most in need of services out of the loop. Find It Cambridge solves this problem by aggregating the many different opportunities happening into one website.

The upgrade and migration from Drupal 7 to Drupal 8 (ready for Drupal 9!) afforded Cambridge and Agaric the opportunity to make the directory of organizations, events, and programs better for families and their children and all involved, grounded in research and testing.

Intuitive, Structured Authoring Experience for Service Providers

The value of a directory are the listings within it and it's truly a community effort to assemble enough accurate and up-to-date resources in a single place, for it to be useful. For Cambridge, the service providers that work at government agencies and nonprofits are the lifeblood of the directory. Without them, there would be no Find It Cambridge.

The challenge then, is building a system that is easy enough for people (many already pressed for time) to take the time to enter their information into, while structuring the data to be easily searched and filtered on.

Through user research, we mapped the information architecture to the mental models that service providers hold for their events and programs.

Input Fields Grouped by Key Questions

Most service providers thought of their events in terms of what the event is about, how to contact the organizers, who it is for, when it is happening, where it is happening, how much it costs, and if there is any sort of registration required. So we organized fields into working tabs to match: About, Contact, For whom, When, Where, Cost, and Signup.

Autosave and Soft Required Fields

Even with fields grouped by tabs, the form can take some time to complete. That's why we introduced autosave and soft save features. When working on a form, the site automatically saves the current draft every few seconds. Service providers can also save a draft to return to later. Fields that are required for publishing, are optional for a draft.

Draft States to Save Work for Later

Service providers have many responsibilities to juggle. It's important that they can start creating an event or program, save it and return to it later before publishing it.

Drupal has powerful workflow states, which we've put to use to help service providers clearly know the status of their content.

A service provider can either save their content as a draft or publish it immediately. If saved as a draft, a banner appears on the page clearly indicating that the event or program is not yet published.

Screencast of save as a draft workflow. Authors can save their work as a draft, bypassing required fields until they're ready to publish.

Authors can also create a draft alongside a published version. This allows new versions to be worked on, while maintaining the current page for site visitors.

Screencast of workflow for having a draft while a previous version of a page stays publishe Authors can have a published version of a page and also have a working draft that eventually becomes the new version.

Help text and character counts for guidance

There are particular ways to write and format content on events and programs to make the most of Find It's features. We provide help text along the way to clue providers in on the best ways to write their content. We also include a character count so providers know if they're staying within the recommended limits for certain text fields.

Bulk Select for Quick Data Entry

Certain fields have many options. In some cases the majority of them apply. For example, many educational events are for all ages up to 18. In that scenario, having a "Select All" option speeds up the data entry process. The Selectize JavaScript library adds elegant toggle and check all options to multivalue fields. We created the CheckboxesJS Drupal project so that other Drupal sites can easily incorporate these features on fields of their choosing.

Conditional Fields to Show Only What is Relevant

Some fields on Event and Programs only need to show under certain conditions. For example, if an event doesn't require registration, then there's no need to worry service providers with a registration link field. Using conditional logic keeps forms simple and streamlined.

Multiple Dates

There was a lot of discussion on whether to support repeating rules or instead allow multiple dates. We decided on multiple dates as experience has shown that even repeating events oftentimes have exceptions (and because the events we import from Cambridge Public Libraries are a list of arbitrary dates rather than a recurring rule, and somehow no one in computer science has created a library to produce a best-effort recurring rule from a list of dates).

Multiple date fields. Multiple date fields allow for flexibility on events and programs that happen more than once.

Easy but Powerful Search for Residents

Find It search is powered by Apache Solr, a popular open-source enterprise search platform. We use its numerous features to make the search results as relevant as possible for site visitors.  It's an ongoing process of tweaks; here are some of the things we've done so far.

Weighted Fields for Relevance

On content with lots of data like the events and programs of Find It, certain fields carry more importance than others. The title of an event, for example, is one of the most important. The transporation notes, on the other hand, carries less significance in search queries. When someone types the keyword "music lesson", an event with music lesson in the title or summary shows up before a program for English lessons.

Synonym Matching

When someone searches "childcare" but a program uses "child care", the search engine should know these are equivalent. The same is true for "STEM" and "science education."

Find It supports synonyms. The site manager can define synonyms so that when site visitors search for a certain term, results with matching synonyms show up as well.

Key Information in Search Results

We used the results of our user research to show the critical information people need to pin point the right opportunities: title, neighborhood, and a short summary.

Filters for Sophisticated Queries

Filters help users narrow a search query down to specific criteria. In our testing, we found that age and neighborhood were most important, especially for low-income caregivers. For those of us that rely on public transportation, events and programs need to be nearby. We placed these filters accordingly towards the top of the page.

Naming conventions in Cambridge are unique, which is true for other cities too. Residents might not know the official name of their neighborhood or live at the border between two. We've included a labeled, clickable map to help users choose the right neighborhood. We built this so that other Find It platforms can upload their own SVG map to show their neighborhood.

Informative Opportunity Pages

Find It comes out of the box with four different types of opportunities: Events, Places, Organizations and Programs.

Organization

The organization serves as the foundation for opportunities posted on a Find It page. Every event and program posted to Find It, belongs to an organization. This helps an organization's page serve as a mini-website. When an event or program is published, it automatically shows up on its organization page.

Organizations can also have "child" organizations, which is helpful for larger groups that might have distinct sub-committees or departments that have sub-departments.

Related programs field. An organization can have a parent - child relationship.

Event

An event is an opportunity with a clear start and end date. When an event is published it shows up on the Homepage, Events page, Search page and on the organization's page.

Visitors can sort opportunities by start date to find upcoming events.

Find It event page. An event's multiple dates is converted into human friendly language.

Program

A program is similar to an event. In fact, most fields are shared between the two. A program though, implies more longevity and commitment than an event. Rather than requiring a specific date or dates, a program can simply be "ongoing." There is the option to include specific dates though.

Find It program page.

Place

In the first version of Find It Cambridge, a new opportunity surfaced that didn't quite fit into the event, program, or organization categories. Parks, neighborhood pools, and other destinations were a good fit for Find It's library of opportunities. They have open hours, but many of the event fields were irrelevant. The same went for Programs. In fact, sometimes these places have events or programs happening at them.

These are community-minded destinations people can go to. In other words, places.

Find It place page.

Bring Find It to Your City!

Find It is helping Cambridge residents connect with activities and services to improve their lives. We would love to help do the same for other cities, counties, and other communities. The platform is open-source and flexible so that communities can customize it to their needs.

Whether you are city IT staff, a developer that works with cities, or are a resident that could use a Find It in your community, we'd love to talk.

Apr 10 2020
Apr 10

In the 31 days of Drupal migrations series, we explained different aspects of the syntax used by the Migrate API. In today’s article, we are going to dive deeper to understand how the API interprets our migration definition files. We will explain how to configure process plugins and set subfields and deltas for multi-value field migrations. We will also talk about process plugin chains, source constants, pseudofields, and the process pipeline. After reading this article, you will better comprehend existing migration definition files and improve your own. Let’s get started.

Understanding the syntax of Drupal migrations.

Field mappings: process plugin configuration

The Migrate API provides syntactic sugar to make migration definition files more readable. The field mappings under the process section are a good example of this. To demonstrate the syntax consider a multi-value Link field to store links to online profiles. The field machine name is field_online_profiles and it is configured to accept the URL and the link text. For brevity, only the `process` section will be shown, but it is assumed that the source includes the following columns: `source_drupal_profile`, `source_gitlab_profile`, and `source_github_profile`.


process:
  field_online_profiles: source_drupal_profile

In this case, we are directly assigning the value from source_drupal_profile in the source to the field_online_profiles in the destination entity. For now, we are ignoring the fact that the field accepts multiple values. We are setting the link text either, just the URL. Even in this example, the Migrate API is making some assumptions for us. Every field mapping requires at least one process plugin to be configured. If none is set, the get plugin is assumed. It copies a value from the source to the destination without making any changes. The previous snippet is equivalent to the next one:


process:
  field_online_profiles:
    plugin: get
    source: source_drupal_profile

The process plugin configuration options should be placed as direct children of the field that is being mapped. In the previous snippet, plugin and source are indented one level to the right under field_online_profiles. There are many process plugins provided by Drupal core and contributed modules. Their configuration can be generalized as follows:


process:
  destination_field:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
    config_3: value_3

Check out the article on using process plugins for data transformation for a working example.

Field mappings: setting sub-fields

Let's expand the example by setting the a value for the Link text in addition to the URL. To accomplish this, we will migrate data into subfields. Fields can store complex data and in many cases they have multiple components. For example, a rich text field has a subfield to store the text value and another for the text format. Address fields have 13 subfields available. Our example uses Link fields which have three subfields:

  • uri: The URI of the link.
  • title: The link text.
  • options: Serialized array of options for the link.

For now, only the uri and title subfields will be set. This also demonstrates that, depending on the field, it is not necessary to provide values for all the subfields. One more thing we will implement is to include the name of the online profile in the Link text. For example: “Drupal.org profile”.


process:
  field_online_profiles/uri: source_drupal_profile
  field_online_profiles/title:
    plugin: default_value
    default_value: 'Drupal.org profile'

If you want to set a value for a subfield, you use the field_name/subfield syntax. Then, each subfield can define its own mapping. Note that when setting the uri we are taking advantage of the get plugin considered the default to simplify the value assignment. In the case of title, the default_value process plugin is used to set a fixed value to comply with our example requirement.

When setting subfields, it is very important to understand what format is expected. You need to make sure the process plugins return data in the expected format or the migration will fail. In particular, you need to know if they return a scalar value or an array. In the case of scalar values, you need to verify if numbers or strings are expected. In the previous example, the uri subfield of the Link field expects a string containing the URL. On the other hand, File fields have a target_id subfield that expects an integer representing the File ID that is being referenced. Some process plugins might return an array or let you set subfields directly as part of the plugin configuration. For an example of the latter, have a look at the article on migrating images using the image_import plugin. image_import lets you set the alt, title, width, and height subfields for images directly in the plugin configuration. The following snippets shows a generalization for setting subfields:


process:
  destination_field/subfield_1:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/subfield_2:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2

If a field can have multiple subfields, how can I know which ones are available? For easy reference, our next blog post will include a list of subfields for different types of fields. To find out by yourself, check out this article that covers available subfields. In summary, you need to locate the class that provides the FieldType plugin and inspect its schema method. The latter defines the database columns used by the field to store its data. Because of object oriented practices, sometimes you need to look at the parent class to know all the subfields that are available. When migrating into subfields, you are actually migrating into those particular database columns. Any restriction set by the database schema needs to be respected. Link fields are provided by the LinkItem class whose schema method defines the three subfields we listed before.

If a field can have multiple subfields, how does the Migrate API know which one to set when no one is manually specified? Every Drupal field has at least one subfield. If they have more, the field type itself specifies which one is the default. For easy reference, our next blog post will indicate the default subfield for different types of fields. To find out by yourself, check out this article that covers default subfields. In summary, you need to locate the class that provides the FieldType plugin and inspect its mainPropertyName method. Its return value will be the default subfield used by the Migrate API. Because of object oriented practices, sometimes you need to look at the parent class to find the method that defines the default subfield. Link fields are provided by the LinkItem class whose mainPropertyName returns uri. That is why in the first example there was no need to specify a subfield to set the value for the link URL.

Field mappings: setting deltas for multi-value fields

Once more, let’s expand the example by setting the populating multiple values for the same field. To accomplish this, we will specify field deltas. A delta is a numeric index starting at 0 and incrementing by 1 for each subsequent element in the multi-value field. Remember that our example assumes that the source has the following columns: source_drupal_profile, source_gitlab_profile, and source_github_profile. One way to migrate all of them into the multi-value link field is:


process:
  field_online_profiles/0/uri: source_drupal_profile
  field_online_profiles/0/title:
    plugin: default_value
    default_value: 'Drupal.org profile'
  field_online_profiles/1/uri: source_gitlab_profile
  field_online_profiles/1/title:
    plugin: default_value
    default_value: 'GitLab profile'
  field_online_profiles/2/uri: source_github_profile
  field_online_profiles/2/title:
    plugin: default_value
    default_value: 'GitHub profile'

If you want to set a value for a subfield, you use the field_name/delta/subfield syntax. Then, every combination of delta and subfield can define its own mapping. Both delta and subfield are optional. If no delta is specified, 0 is assumed which corresponds to the first element of a (multi-value) field. If no subfield is specified, the default subfield is assumed as explained before. In the previous example, if there is no need to set the link text the configuration would become:


process:
  field_online_profiles/0: source_drupal_profile
  field_online_profiles/1: source_gitlab_profile
  field_online_profiles/2: source_github_profile

In this example, we wanted to highlight syntax variations that can be used with the Migrate API. Nevertheless, this way of migrating multi-value fields is not very flexible. You are required to know in advance how many deltas you want to migrate. Depending on your particular configurations, you can write complex process pipelines that take into account an unknown number of deltas. Sometimes, writing a custom migration process plugin is easier and/or the only option to accomplish a task. Even if you can write a migration with existing process plugins, that might not be the best solution. When writing migrations, strive for them to be easy to read, understand, and maintain. For reference, the generic configuration for mapping fields with deltas and subfields is:


process:
  destination_field/0/subfield_1:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/0/subfield_2:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/1/subfield_1:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/1/subfield_2:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2

Process plugin chains

So far, for every field_name/delta/subfield combination we only have used one process plugin. The Migrate API does not impose any restrictions to the number of transformations that the source data can undergo before being assigned to a destination property or field. You can have as many as needed. Chaining of process plugins works similarly to Unix pipelines in that the output of one process plugin becomes the input of the next one in the chain. When the last plugin in the chain completes its transformation, the return value is assigned. We have covered this topic in greater detail in the article on using process plugins for data transformation. For now, let’s consider an example chain of two process plugins:


process:
  title:
    - plugin: concat
      source:
        - source_first_name
        - source_last_name
      delimiter: ' '
    - plugin: callback
      callable: strtoupper

In this example, we are using the concat plugin to glue together the source_first_name and source_last_name. A space is placed in between as specified by the delimiter configuration. The result of this is later passed to the callback plugin which executes the strtoupper PHP function on the concatenated value effectively making the string uppercase. Because there are no more process plugins in the chain, the string transformed to uppercase is assigned to the title destination property. If source_first_name is ‘Mauricio’ and source_last_name is ‘Dinarte’, then title would be set to ‘MAURICIO DINARTE’. Refer to the article mentioned before for other things to consider when manipulating strings. The configuration of process plugin chains can be generalized as follows:


process:
  destination_field:
    - plugin: plugin_name
      source: source_column_name
      config_1: value_1
      config_2: value_2
    - plugin: plugin_name
      config_1: value_1
      config_2: value_2
    - plugin: plugin_name
      config_1: value_1
      config_2: value_2

It is very important to note that only the first process plugin in the chain should set a source configuration. Remember that the output of the previous process plugin is the input for the next one. Setting the source configuration in subsequent process plugins is unnecessary and can actually make the chain produce unexpected results or fail altogether.

Source constants, pseudofields, and the process pipeline

We have covered source constants, pseudo-fields, and the process pipeline in the article on using data placeholders in the migration process. This time, we are only going to give an overview to explain their syntax. Constants are arbitrary values that can be used later in the process pipeline. They are set as direct children of  the source section. Let’s consider this example:


source:
  constant:
    DRUPAL_LINK_TITLE: 'Drupal.org profile'
    GITLAB_LINK_TITLE: 'GitLab profile'
    GITHUB_LINK_TITLE: 'GitHub profile'
process:
  field_online_profiles/0/uri: source_drupal_profile
  field_online_profiles/0/title: constant/DRUPAL_LINK_TITLE
  field_online_profiles/1/uri: source_gitlab_profile
  field_online_profiles/1/title: constant/GITLAB_LINK_TITLE
  field_online_profiles/2/uri: source_github_profile
  field_online_profiles/2/title: constant/GITHUB_LINK_TITLE

To define source constants, you write a constants key and set its value to an array of name-value pairs. When you need to refer to them in the process section, you use constant/NAME and they behave like any other column present in the source. Although not required, it is customary to name constants in uppercase. This makes it easier to distinguish them from regular source columns. Notice how their use makes assigning the link titles simpler. Instead of using the default_value plugin, we read the value directly from the source constants.

Pseudofields also store arbitrary values for use later, but they are defined in the process section. Their names can be arbitrary as long as they do not conflict with a property name or field name in the destination. The value can be set to a verbatim copy from the source (a column or a constant) or they can use process plugins for data transformations. For the next example, consider that there is no need for the link text to be different among online profiles. Additionally, there is another Link field that can only store one value. This new field is used to store the URL to the primary profile. The example can be rewritten as follows:


source:
  constant:
    LINK_TITLE: 'Online profile'
process:
  pseudo_link_text:
    - plugin: get
      source: constant/LINK_TITLE
    - plugin: callback
      callable: strtoupper
  field_online_profiles/0/uri: source_drupal_profile
  field_online_profiles/0/title: '@pseudo_link_text'
  field_online_profiles/1/uri: source_gitlab_profile
  field_online_profiles/1/title: '@pseudo_link_text'
  field_online_profiles/2/uri: source_github_profile
  field_online_profiles/2/title: '@pseudo_link_text'
  field_primary_profile: '@field_online_profiles/0'

A psedofield named pseudo_link_text has been created. It has its own process pipeline to provide the link text that will be used for all online profiles. When you want to use the pseudo, you have to enclose it in quotes (') and prepend an at sign (@) to the name. The pseudo_ prefix in the name is not required. In this case it is used to make it easier to distinguish among pseudofields and regular property or field names.

The previous snippets is also a good example of how the migrate process pipeline works. When setting field_primary_profile, we are reusing a value stored in another field: the first delta of field_online_profiles. There are many things to note here:

  • The migrate process pipeline lets you reuse anything that has been defined previously in the file. It can be source constants, pseudo fields, or regular destination properties and fields. The only requirement is that whatever you want to use needs to be previously defined in the migration definition file.
  • Source columns are accessed directly by name. Source constants are accessed using the constant/NAME syntax.
  • Any element defined in the process section can be reused later in the process pipeline by enclosing its name in quotes (') and prepending an at sign (@). This applies to pseudofields and regular destination properties and fields.

When reusing an element in the process pipeline, its whole structure becomes available. In the previous example, we set field_primary_profile to '@field_online_profiles/0'. This means that all subfields in the first delta of the field_online_profiles field will be assigned to field_primary_profile. Effectively this means both the uri and title properties will be set. Be mindful that when you reuse a field, all its delta and subfields are copied along unless specifically restricted. For example, if you only want to reuse the uri of the first delta you would use '@field_online_profiles/0/uri'. In none of these scenarios, indicating that you want to reuse something guarantees that it will be stored in the new element assignment. For example, the field_primary_profile field only accepts one value. Even if we used '@field_online_profiles' to reuse all the deltas of the multi-value field, only the first one will be stored per the field's (cardinality) definition.

The Migrate API is pretty flexible and you can write very complex process pipelines. The examples we have presented today have been exaggerated to demonstrate many syntax variations. Again, when writing migrations, strive for process pipelines that are easy to read, understand, and maintain.

What did you learn in today's article? Did you know that it is possible to specify deltas and subfields in field mappings? Were you aware that process plugins can be chained for multiple data transformations? How have you used source constants and psuedofield before? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Nov 26 2019
Nov 26

Striking Chicago teachers marching in the street, carrying a banner "A Nurse in Every School, Every Day." Chicago teachers were one example of the hundreds of thousands of workers who went on strike this year winning higher wages, improved working conditions and better services for their communities. Credit: Scott Heins / Getty Images

In 2018, workers rediscovered the power of the strike with nearly 500,000 people walking off the job, compared to just 25,000 in 2017. This year has proven even more of the same, with 442,700 stopping work through September.

Portside is an independent news outlet of labor activists who volunteer their time to scour the web for thoughtful, authentic coverage and analysis of worker power. In June they won the "Labor Communicator of the Year" as a result of their efforts.

Getting meaningful information on strikes is difficult. Just six corporations own 90% of media in the US. Executives like Jeff Bezos and Rupert Murdoch have a vested interest in downplaying worker power. That coupled with sophisticated misinformation campaigns and obscure social media algorithms, it's hard to cut to what's happening on the ground.

We work with Portside to help them reach as many people as possible. In 2018 we upgraded the site from Drupal 7 to Drupal 8, optimizing the website for mobile devices and adding moderation tools to make editors' lives easier.

This year we sent out a survey to readers to learn what else would be helpful to provide the coverage so many are seeking in these politically charged moments.

Nearly 500 people responded, and the feedback was overwhelmingly positive. That said, there's always room for improvement.

In 2019, we hope to roll out the following features to help us all follow and support the labor movement and left in general.

* Improved "Related News" functionality - Right now the articles suggested are a bit hit or miss. We need time to adjust and improve our algorithm for this.
* Create "Issue" pages - the Bolivia coup, the Rojava Revolution, the Chicago Teachers' Strike... the number of progressive, transformative, and often revolutionary movements gaining traction is dizzying. Portside provides excellent coverage, but not in a long-term, coherent way. Issue pages will help us follow breaking news, with the broader context we need to be in meaningful solidarity. Editors are already hard at work cleaning up the tagging system for articles. We just need the time to design and build out these issue pages to better serve readers.
Just these two improvements will make a real difference to how well we can stay informed of what our fellow workers are up to and how we can support them.

All Improvements Are Shared Freely as Free Software

Everything we do at Agaric uses software that is free as in freedom - free to install, use and repurpose for your own needs.

Our work with Portside has already translated into improving and even creating the following Drupal modules:

* Give - on-site donation forms and reporting
* Social Post Twitter - Re-post content from your website to your Twitter account
* Social Share Facebook - Re-post contnt from your website to your Facebook page
* Minimal HTML - A text format handy for short text fields

So, when you donate to Portside's fundraising campaign, you are both supporting independent journalism and the open-source software that benefits other independent outlets and websites.

Donate to Portside

Sep 01 2019
Sep 01

Throughout the series, we explored many migration topics. We started with an overview of the ETL process and workflows for managing migrations. Then, we presented example migrations for different entities: nodes, files, images, taxonomy terms, users, and paragraphs. Next, we shifted focus to migrations from different sources: CSV, JSON, XML, Google Sheet, Microsoft Excel, and LibreOffice Calc files. Later, we explored how to manage migrations as configuration, use groups to share configuration, and execute migrations from the user interface. Finally, we gave recommendations and provided tools for debugging migrations from the command line and the user interface. Although we covered a lot of ground, we only scratched the surface. The Migrate API is so flexible that its use cases are virtually endless. To wrap up the series, we present an introduction to a very popular topic: Drupal upgrades. Let’s get started.

Note: In this article, when we talk about Drupal 7, the same applies to Drupal 6.

What is a Drupal upgrade?

The information we presented in the series is generic enough that it applies to many types of Drupal migrations. There is one particular use case that stands out from the rest: Drupal upgrades. An upgrade is the process of taking your existing Drupal site and copy its configuration and content over to a new major version of Drupal. For example, going from Drupal 6 or 7 to Drupal 8. The following is an oversimplification of the workflow to perform the upgrade process:

  • Install a fresh Drupal 8 site.
  • Add credentials so that the new site can connect to Drupal 7’s database.
  • Use the Migrate API to generate migration definition files. They will copy over Drupal 7’s configuration and content. This step is only about generating the YAML files.
  • Execute those migrations to bring the configuration and content over to Drupal 8.

Preparing your migration

Any migration project requires a good plan of action, but this is particularly important for Drupal upgrades. You need to have a general sense of how the upgrade process works, what assumptions are made by the system, and what limitations exist. Read this article for more details on how to prepare a site for upgrading it to Drupal 8. Some highlights include:

  • Both sites need to be in the latest stable version of their corresponding branch. That means the latest release of Drupal 7 and 8 at the time of performing the upgrade process. This also applies to any contributed module.
  • Do not do any configuration of the Drupal 8 site until the upgrade process is completed. Any configuration you make will be overridden, and there is no need for it anyways. Part of the process includes recreating the old site’s configuration: content types, fields, taxonomy vocabularies, etc.
  • Do not create content on the Drupal 8 site until the upgrade process is completed. The upgrade process will keep the unique identifiers from the source site: `nid`, `uid`, `tid`, `fid`, etc. If you were to create content, the references among entities could be broken when the upgrade process overrides the unique identifiers. To prevent data loss, wait until the old site's content has been migrated to start adding content to the new site.
  • For the system to detect a module’s configuration to be upgraded automatically, it has to be enabled on both sites. This applies to contributed modules in Drupal 7 (e.g., link) that were moved to core in Drupal 8. Also to Drupal 7 modules (e.g. address field) that were superseded by a different one in Drupal 8 (e.g. address). In any of those cases, as long as the modules are enabled on both ends, their configuration and content will be migrated. This assumes that the Drupal 8 counterpart offers an automatic upgrade path.
  • Some modules do not offer automatic upgrade paths. The primary example is the Views module. This means that any view created in Drupal 7 needs to be manually recreated in Drupal 8.
  • The upgrade procedure is all about moving data, not logic in custom code. If you have custom modules, the custom code needs to be ported separately. If those modules store data in Drupal’s database, you can use the Migrate API to move it over to the new site.
  • Similarly, you will have to recreate the theme from scratch. Drupal 8 introduced Twig which is significantly different to the PHPTemplate engine used by Drupal 7.

Customizing your migration

Note that the creation and execution of the migration files are separate steps. Upgrading to a major version of Drupal is often a good opportunity to introduce changes to the website. For example, you might want to change the content modeling, navigation, user permissions, etc. To accomplish that, you can modify the generated migration files to account for any scenario where the new site’s configuration diverts from the old one. And only when you are done with the customizations, you execute the migrations. Examples of things that could change include:

  • Combining or breaking apart content types.
  • Moving data about people from node entities to user entities, or vice versa.
  • Renaming content types, fields, taxonomy vocabularies and terms, etc.
  • Changing field types. For example, going from Address Field module in Drupal 7 to Address module in Drupal 8.
  • Merging multiple taxonomy vocabularies into one.
  • Changing how your content is structured. For example, going from a monolithic body field to paragraph entities.
  • Changing how your multimedia files are stored. For example, going from image fields to media entities.

Performing the upgrade

There are two options to perform the upgrade. In both cases, the process is initiated from the Drupal 8 site. One way is using the Migrate Drupal UI core module to perform the upgrade from the browser’s user interface. When the module is enabled, go to `/upgrade` and provide the database credentials of the Drupal 7 site. Based on the installed modules on both sites, the system will give you a report of what can be automatically upgraded. Consider the limitations explained above. While the upgrade process is running, you will see a stream of messages about the operation. These messages are logged to the database so you can read them after the upgrade is completed. If your dataset is big or there are many expensive operations like password encryption, the process can take too long to complete or fail altogether.

The other way to perform the upgrade procedure is from the command line using Drush. This requires the Migrate Upgrade contributed module. When enabled, it adds Drush commands to import and rollback a full upgrade operation. You can provide database connection details of the old site via command line options. One benefit of using this approach is that you can create the migration files without running them. This lets you do customizations as explained above. When you are done, you can run the migrations following the same workflow of manually created ones.

Known issues and limitations

Depending on whether you are upgrading from Drupal 6 or 7, there is a list of known issues you need to be aware of. Read this article for more information. One area that can be tricky is multilingual support. As of this writing, the upgrade path for multilingual sites is not complete. Limited support is available via the Migrate Drupal Multilingual core module. There are many things to consider when working with multilingual migrations. For example, are you using node or field translations? Do entities have revisions? Read this article for more information.

Upgrade paths for contributed modules

The automatic upgrade procedure only supports Drupal core modules. This includes modules that were added to core in Drupal 8. For any other contributed module, it is the maintainers’ decision to include an automatic upgrade path or not. For example, the Geofield module provides an upgrade path. It is also possible that a module in Drupal 8 offers an upgrade path from a different module in Drupal 7. For example, the Address module provides an upgrade path from the Address Field module. Drupal Commerce also provides some support via the Commerce Migrate module.

Not every module offers an automated upgrade path. In such cases, you can write custom plugins which ideally are contributed back to Drupal.org ;-) Or you can use the techniques learned in the series to transform your source data into the structures expected by Drupal 8. In both cases, having a broad understanding of the Migrate API will be very useful.

Upgrade strategies

There are multiple migration strategies. You might even consider manually recreating the content if there is only a handful of data to move. Or you might decide to use the Migrate API to upgrade part of the site automatically and do a manual copy of a different portion of it. You might want to execute a fully automated upgrade procedure and manually clean up edge cases afterward. Or you might want to customize the migrations to account for those edge cases already. Michael Anello created an insightful presentation on different migration strategies. Our tips for writing migrations apply as well.

Drupal upgrades tend to be fun, challenging projects. The more you know about the Migrate API the easier it will be to complete the project. We enjoyed writing this overview of the Drupal Migrate API. We would love to work on a follow up series focused on Drupal upgrades. If you or your organization could sponsor such endeavor, please reach out to us via the site’s contact form.

What about upgrading to Drupal 9?

In March 2017, project lead Dries Buytaert announced a plan to make Drupal upgrades easier forever. This was reinforced during his keynote at DrupalCon Seattle 2019. You can watch the video recording in this link. In short, Drupal 9.0 will be the latest point release of Drupal 8 minus deprecated APIs. This has very important implications:

  • When Drupal 9 is released, the Migrate API should be mostly the same as Drupal 8. Therefore, anything that you learn today will be useful for Drupal 9 as well.
  • As long as your code does not use deprecated APIs, upgrading from Drupal 8 to Drupal 9 will be as easy as updating from Drupal 8.7 to 8.8.
  • Because of this, there is no need to wait for Drupal 9 to upgrade your Drupal 6 or 7 site. You can upgrade to Drupal 8 today.

Thank you!

And that concludes the #31DaysOfMigration series. For joining us in this learning experience, thank you very much! ¡Muchas gracias! Merci beaucoup! :-D We are also very grateful to Agaric.coop, Drupalize.Me, and Centarro.io for sponsoring this series.

What did you learn in today’s blog post? Did you know the upgrade process is able to copy content and configuration? Did you know that you can execute the upgrade procedure either from the user interface or the command line? Share your answers in the comments. Also, we would be grateful if you shared this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Aug 21 2019
Aug 21

Today, we are going to talk about how to manage migrations as configuration entities. This functionality is provided by the Migrate Plus module. First, we will explain the difference between managing migrations as code or configuration. Then, we will show how to convert existing migrations. Finally, we will talk about some important options to include in migration configuration entities. Let’s get started.

Example of migration defined as configuration entity.

Drupal migrations: code or configuration?

So far, we have been managing migrations as code. This is functionality provided out of the box. You write the migration definition file in YAML format. Then, you place it in the migrations directory of your module. If you need to update the migration, you make the modifications to the files and then rebuild caches. More details on the workflow for migrations managed in code can be found in this article.

Migrate Plus offers an alternative to this approach. It allows you to manage migrations as configuration entities. You still use YAML files to write the migration definition files, but their location and workflow is different. They need to be placed in a config/install directory. If you need to update the migration,  you make the modifications to the files and then sync the configuration again. More details on this workflow can be found in this article.

There is one thing worth emphasizing. When managing migrations as code you need access to the file system to update and deploy the changes to the file. This is usually done by developers.  When managing migrations as configuration, you can make updates via the user interface as long as you have permissions to sync the site’s configuration. This is usually done by site administrators. You might still have to modify files depending on how you manage your configuration. But the point is that file system access to update migrations is optional. Although not recommended, you can write, modify, and execute the migrations entirely via the user interface.

Transitioning to configuration entities

To demonstrate how to transition from code to configuration entities, we are going to convert the JSON migration example. You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD config JSON source migration whose machine name is udm_config_json_source. It comes with four migrations: udm_config_json_source_paragraph, udm_config_json_source_image, udm_config_json_source_node_local, and udm_config_json_source_node_remote.

The transition to configuration entities is a two step process. First, move the migration definition files from the migrations folder to a config/install folder. Second, rename the files so that they follow this pattern: migrate_plus.migration.[migration_id].yml. For example: migrate_plus.migration.udm_config_json_source_node_local.yml. And that’s it! Files placed in that directory following that pattern will be synced into Drupal’s active configuration when the module is installed for the first time (only). Note that changes to the files require a new synchronization operation for changes to take effect. Changing the files and rebuilding caches does not update the configuration as it was the case with migrations managed in code.

If you have the Migrate Plus module enabled, it will detect the migrations and you will be able to execute them. You can continue using the Drush commands provided the Migrate Run module. Alternatively, you can install the Migrate Tools module which provides Drush commands for running both types of migrations: code and configuration. Migrate Tools also offers a user interface for executing migrations. This user interface is only for migrations defined as configuration though. It is available at /admin/structure/migrate. For now, you can run the migrations using the following Drush command: drush migrate:import udm_config_json_source_node_local --execute-dependencies.

Note: For executing migrations in the command line, choose between Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided by the two modules have the same name. Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands.

UUIDs for migration configuration entities

When managing migrations as configuration, you can set extra options. Some are exposed by Migrate Plus while others come from Drupal’s configuration management system. Let’s see some examples.

The most important new option is defining a UUID for the migration definition file. This is optional, but adding one will greatly simplify the workflow to update migrations. The UUID is used to keep track of every piece of configuration in the system. When you add new configuration, Drupal will read the UUID value if provided and update that particular piece of configuration. Otherwise, it will create a UUID on the fly, attach it to the configuration definition, and then import it. That is why you want to set a UUID value manually. If changes need to be made, you want to update the same configuration, not create a new one. If no UUID was originally set, you can get the automatically created value by exporting the migration definition. The workflow for this is a bit complicated and error prone so always include a UUID with your migrations. This following snippet shows an example UUID:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'

The UUID a string of 32 hexadecimal digits displayed in 5 groups. Each is separated by hyphens following this pattern: 8-4-4-4-12. In Drupal, two or more pieces of configuration cannot share the same value. Drupal will check the UUID and the type of configuration in sync operations. In this case the type is signaled by the migrate_plus.migration. prefix in the name of the migration definition file.

When using configuration entities, a single migration is identified by two different options. The uuid is used by the Drupal’s configuration system and the id is used by the Migrate API. Always make sure that this combination is kept the same when updating the files and syncing the configuration. Otherwise you might get hard to debug errors. Also, make sure you are importing the proper configuration type. The latter should not be something to worry about unless you utilize the user interface to export or import single configuration items.

If you do not have a UUID in advance for your migration, you can try one of these commands to generate it:

# Use Drupal's UUID service.
$ drush php:eval "echo \Drupal::service('uuid')->generate(). PHP_EOL;"

# Use a Drush command provided by the Devel module, if enabled.
$ drush devel:uuid

# Use a tool provided by your operating system, if available.
$ uuidgen

Alternatively, you can search online for UUID v4 generators. There are many available.

Technical note: Drupal uses UUID v4 (RFC 4122 section 4.4) values which are generated by the `uuid` service. There is a separate class for validation purposes. Drupal might override the UUID service to use the most efficient generation method available. This could be using a PECL extension or a COM implementation for Windows.

Automatically deleting migration configuration entities

By default, configuration remains in the system even if the module that added it gets uninstalled. This can cause problems if your migration depends on custom migration plugins provided by your module. It is possible to enforce that migration entities get removed when your custom module is uninstalled. To do this, you leverage the dependencies option provided by Drupal’s configuration management system. The following snippet shows how to do it:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
  enforced:
    module:
      - ud_migrations_config_json_source

You add the machine name of your module to dependencies > enforced > module array. This adds an enforced dependency on your own module. The effect is that the migration will be removed from Drupal’s active configuration when your custom module is uninstalled. Note that the top level dependencies array can have others keys in addition to enforced. For example: config and module. Learning more about them is left as an exercise for the curious reader.

It is important not to confuse the dependencies and migration_dependencies options. The former is provided by Drupal’s configuration management system and was just explained. The latter is provided by the Migrate API and is used to declare migrations that need be imported in advance. Read this article to know more about this feature. The following snippet shows an example:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
  enforced:
    module:
      - ud_migrations_config_json_source
migration_dependencies:
  required:
    - udm_config_json_source_image
    - udm_config_json_source_paragraph
  optional: []

What did you learn in today’s blog post? Did you know that you can manage migrations in two ways: code or configuration? Did you know that file name and location as well as workflows need to be adjusted depending on which approach you follow? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Workflows and benefits of managing Drupal migrations as configuration entities

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Jan 29 2019
Jan 29

Woman planting an EU flag with a security symbol in the middle into the ground.

With Europe threatening $25,000,000 fines and Facebook losing $80,000,000,000 of stock value, are you paying attention to data privacy yet? If millions and billions of dollars in news headlines never grabbed you, maybe you've noticed the dozens of e-mails from… Read more,

This goes well beyond an organization's web site, of course. Web developers may be the ones to introduce it to organizations, though, so we should be prepared. Here's the gist.

Organizations must request any personal data in clear and plain language describing the specific pieces of information and how it will be used, such that consent can be given freely and unambiguously through an affirmative action.

This means you need to be always thinking of why you are collecting information, and not collecting… Read more

Jan 29 2019
Jan 29

A woman pushing a giant rectangle with a right arrow within it.

Agaric is facilitating a full day training at DrupalCon Seattle to help you understand how to import content into your to Drupal 8 website.

This training is open for attendees with intermediate experience with Drupal- familiarity with installing a Drupal site and installing modules. We… Read more

Jan 29 2019
Jan 29

Illustration of people standing next to giant icons representing productivity.

People often ask about the free software tools Agaric uses to manage our cooperative business. In this article, we share some of the free software tools we use for office tasks and administration as well as communications. These are Agaric's chosen resources -- the tools we… Read more,

Video embed

Jan 29 2019
Jan 29

A desktop computer with social media icons above.

Being able to share an article via a social network is a common request on a project.

Fortunately for Drupal 8 there is a module for that called Social Simple. This module allows you to display a share button on a node for the most popular social… Read more

Jan 29 2019
Jan 29

The program for DrupalCon is evolving constantly. Among the changes for Nashville 2018 new tracks have been added and some have been merged. That is the case for the Symfony and PHP tracks.

Many topics clearly belong to a single track, but others could fit in more than one. When we had a dedicated Symfony track a session about Twig could be submitted to the Symfony or front end tracks. A session about Drupal Console could be… Read more

Jan 29 2019
Jan 29

Over 8 years have passed since there was a DrupalCamp in tropical Nicaragua. With the help of a diverse group of volunteers, sponsors, and university faculty staff, we held our second one. DrupalCamp Lagos y Volcanes ("Lakes & Volcanoes") was a great success with over 100 people attending in 2 days. It was a big undertaking so we followed giants' footsteps to prepare for our event. Lots of the ideas were taken from some of the organizers' experience while attending Drupal events.… Read more

Jan 29 2019
Jan 29

Once a text field has data stored, it is not very easy or obvious how to change its maximum length. In the UI there is a message warning you that the field cannot be changed, because there is existing data. Sometimes it is necessary to change these values. It seems that there are a few ways and some resources to do this in Drupal 7, but I could not find a way to do this in Drupal 8. I decided to create a small function to do it:

Caution: Any change in the database needs to be done carefully… Read more

Jan 29 2019
Jan 29

When you think of training, perhaps you remember an event that you were sent to where you had to learn something boring for your job. The word training does not usually make people smile and jump for joy, that is unless you are talking about Drupal training. These gatherings spread the Drupal knowledge and increase diversity in the community of Drupal developers.

Join us for the next Drupal Global Training Day with our online full day session on getting started with Drupal… Read more

Jan 29 2019
Jan 29

Drupal 8 has a great AJAX Form API which includes some tools to create modal dialogs using the jQuery modal library. The Examples module even demonstrates how to create a custom form and display it in a modal window. But what if what you want to do is display an already created form in a modal? How do we do that? Let's see how to do it with an example. Let's display the node add form in a modal window.

The first thing… Read more

Jan 29 2019
Jan 29

TL;DR: For PHP Hexadecimals, Decimals and Octals are all Integers, so they must be declared as @param integer

While I was working on a patch I had to write the docblock of a function which received a hexadecimal number and I wasn't sure what I was supposed to put in the @type param.

I went to Drupal's API documentation and comments standards… Read more

Jan 29 2019
Jan 29

While creating content, there are pieces of information that are only relevant when other fields have a certain value. For example, if we want to allow the user to upload either an image or a video, but not both, you can have another field for the user to select which type of media they want to upload. In these scenarios, the Javascript States API for Drupal 8 can be used to conditionally hide and show the input elements for image and video conditionally.

Note: Do not confuse the… Read more

Jan 29 2019
Jan 29

CKEditor is well-known software with a big community behind it and it already has a ton of useful plugins ready to be used. It is the WYSIWYG text editor which ships with Drupal 8 core.

Unfortunately, the many plugins provided by the CKEditor community can't be used directly in the CKEditor that comes with Drupal 8. It is necessary to let Drupal know that we are going to add a new button to the CKEditor.

Why Drupal needs to know about our plugins

Drupal allows us to… Read more

Aug 14 2018
Aug 14

Agaric is facilitating a full day training at Drupal Europe in Darmstadt, Germany to help you understand how to import content into your to Drupal 8 website.

This training is open for attendees with intermediate experience with Drupal- familiarity with installing a Drupal site and installing modules. We will use the Migrate API and related modules, which allows users to migrate content without writing any code.

With two instructors and a small group size we will ensure no one gets left behind. Instead everyone will get the attention they need.

Attendees will learn to:

  • Import data from CSV and JSON files.
  • Transform the data to populate taxonomy, date, image, file, and address fields.
  • Get content into Paragraphs.
  • Debug migration errors.

For a sneak peak of what you will learn, attend our Drupal Migration Q&A on Friday August 17th at 14:00 UTC (10 AM ET) on Zoom at https://zoom.us/j/3307127772

Regular training price is €476, however we are offering an early bird registration price of €276 from now through Sunday, August 19th. Space is limited so we do encourage registering early to take advantage of the price and ensure your spot.

As a web development cooperative that champions free software, we're passionate about migrations. It is a way to better understand Drupal's codebase, tap into the power of new features and build community. We have successfully migrated multiple sites to Drupal 8, including large projects with custom modules and hope to share that experience with you.

Register for Migration Training

Not able to make it to the training? No worries, we offer trainings on a regular basis. Sign up for our low-traffic, announcement only list to stay informed of future trainings. These will not be sent more than a few times a year.

Daily Business Operations Using Free Software

Aug 12 2018
Aug 12
Aug 11 2018
Aug 11

Update: Unfortunately we had to cancel this event due to low number of registrants. We do hope to offer this training at BADCamp and DrupalCon Seattle. We can also offer our training directly to organizations.

Agaric is facilitating a full day training at Drupal Europe in Darmstadt, Germany to help you understand how to import content into your to Drupal 8 website.

This training is open for attendees with intermediate experience with Drupal- familiarity with installing a Drupal site and installing modules. We will use the Migrate API and related modules, which allows users to migrate content without writing any code.

With two instructors and a small group size we will ensure no one gets left behind. Instead everyone will get the attention they need.

Attendees will learn to:

  • Import data from CSV and JSON files.
  • Transform the data to populate taxonomy, date, image, file, and address fields.
  • Get content into Paragraphs.
  • Debug migration errors.

For a sneak peak of what you will learn, attend our Drupal Migration Q&A on Friday August 17th at 14:00 UTC (10 AM ET) on Zoom at https://zoom.us/j/3307127772

Regular training price is €476, however we are offering an early bird registration price of €276 from now through Sunday, August 19th. Space is limited so we do encourage registering early to take advantage of the price and ensure your spot.

As a web development cooperative that champions free software, we're passionate about migrations. It is a way to better understand Drupal's codebase, tap into the power of new features and build community. We have successfully migrated multiple sites to Drupal 8, including large projects with custom modules and hope to share that experience with you.

Embracing Data Privacy

May 25 2018
May 25

Creating a New Social Simple Button

May 04 2018
May 04
Dec 23 2017
Dec 23

The program for DrupalCon is evolving constantly. Among the changes for Nashville 2018 new tracks have been added and some have been merged. That is the case for the Symfony and PHP tracks.

Many topics clearly belong to a single track, but others could fit in more than one. When we had a dedicated Symfony track a session about Twig could be submitted to the Symfony or front end tracks. A session about Drupal Console could be submitted to the Symfony, the PHP, or back end tracks. In an effort to reduce confusion in the call for proposal process, the track team has agreed on the following:

  • The back end development track is for sessions focused on Drupal coding and development practices.
  • The PHP track is for sessions that cover the broader PHP ecosystem. These sessions can be related to Drupal, but focused on an external project/library like composer, or PHPUnit, Symfony components.
  • The Symfony track merged with the PHP track.

Undoubtedly Symfony plays a key role in Drupal. Symfony 4 has just been released and it would be great to learn about what the future looks like. We want to learn about what is new in the latest version and what benefits adopting it would bring. We are also interested in sessions that would allow developers to learn from each other. What does a Symfony developer need to know to write Drupal code? What does a Drupal developer needs to know about Symfony to become a better developer? In other words - how to make proper use of Symfony components and related best practices.

Other session ideas include best practices on using Composer, PHPUnit, and third party libraries; new features in PHP 7; functional, asynchronous, and reactive programming; machine learning; micro services; etc.

If you want to attend DrupalCon Nashville, but the cost of attending is too high there are some ways to reduce the expenses:

  • Getting a session selected gives you a DrupalCon ticket.
  • You can get a $350 stipend to help cover expenses if your session is selected and you identify yourself within at least one of the "Big Eight" Social Identifiers. This is part of an effort to increase diversity at DrupalCon.
  • If you volunteer as a mentor, you can get a free ticket. No need to be a speaker for this one.
  • There are grants and scholarships that can provide a ticket and/or money to cover expenses. No need to be a speaker for this one.

The track team for DrupalCon Nashville 2018 is here to help you during the session submission process. We can help review proposals, suggest topics, and clear any doubt you might have. For the PHP track, Chad Hester, Tim Millwood, and myself are ready to help.

For people who have not presented before and for those of underrepresented groups, the Drupal Diversity and Inclusion Initiative has created a channel in Slack to help them with the proposal project. Mentoring is available at drupal.org/slack? in the #ddi-session-help channel.

The call for proposals closes in less than a month on January 17. Do no leave things until the last minute. We look forward to your session submissions!

Dec 08 2017
Dec 08

Over 8 years have passed since there was a DrupalCamp in tropical Nicaragua. With the help of a diverse group of volunteers, sponsors, and university faculty staff, we held our second one. DrupalCamp Lagos y Volcanes ("Lakes & Volcanoes") was a great success with over 100 people attending in 2 days. It was a big undertaking so we followed giants' footsteps to prepare for our event. Lots of the ideas were taken from some of the organizers' experience while attending Drupal events. Others came from local free software communities who have organized events before us. Let me share what we did, how we did it, and what the results were.

Saturday group photo

Diversity

In line with DrupalCon, we used the "Big Eight" social identifiers to define diversity and encourage everyone to have a chance to present. Among other statistics, we are pleased that 15% of the sessions and 33% of the trainings were presented by women. We would have liked higher percentages, but it was a good first step. Another related fact is that no speaker presented more than one session. We had the opportunity to learn from people with different backgrounds and expertise.

https://twitter.com/drupalni/status/931566029771329536
https://twitter.com/drupalni/status/931550669953294339

Ticket cost

BADCamp, Drupal's largest event outside of DrupalCons, is truly an inspiration when it comes to making affordable events. They are free! We got close. For $1 attendees had access to all the sessions and trainings, lunch both days, a t-shirt, and unlimited swag while supplies lasted. Of course, they also had the networking opportunities that are always present at Drupal events. Even though the camp was almost free, we wanted to give all interested people a chance to come and learn so we provided scholarships to many attendees.

https://drive.google.com/open?id=1iYKwlLZIlieqVHrwUzOBuEU11MQIaMI3

Scholarships

The camp offered four types of scholarships:

  • Ticket cost: we would waive the $1 entry fee.
  • Transportation: we would cover any expense for someone to come from any part of the country.
  • Lodging: we would provide a room for people to stay overnight if they would come from afar.
  • Food: we would pay for meals during the two days of the camp.

About 40% of the people who attended did not pay the entry fee. We also had people traveling from differents parts of the country. Some stayed over. Others travelled back and forth each day. Everyone who requested a scholarship received it. It felt good to provide this type of opportunities and recipients were grateful for it.

https://twitter.com/drupalni/status/931940088522698752
https://drive.google.com/open?id=1uILaPOJOs7oIE1kjkm0aVhlSQMyqBkfo

Sponsors

As you can imagine, events like these need funding and we are extremely grateful to our sponsors:

These are people who attended from afar. Some were scholarship recipients. Others got educational memberships.

https://drive.google.com/open?id=16hVraHW2uVq_IBR_z78sdHp9gAayT1zp

Session recordings

Although we worked hard to make it possible for interested people to attend, we knew that some would not be able to make it. In fact, having sessions recorded would make it possible for anyone who understands Spanish to benefit from what was presented at the camp.

We used Kevin Thull’s recommended kit to record sessions. My colleague Micky Metts donated the equipment and I did the recording. I had the opportunity to be at some camps that Kevin recorded this year and he was very kind in teaching me how to use the equipment. Unfortunately, the audio is not clear in some sessions and I completely lost one. I have learned from the mistakes and next time it should be better. Check out the camp playlist in Drupal Nicaragua’s YouTube channel for the recordings.

Thank you Kevin. It was through session recordings that I improved my skills when I could not afford to travel to events. I’m sure I am not the only one. Your contributions to the Drupal community are invaluable!

https://drive.google.com/open?id=1cepjh_WJbTTnwSrlYbp2vXjG9Xdqc8jw
https://twitter.com/drupalni/status/932000997387526144

Sprints and live commit!

Lucas Hedding lead a sprint on Saturday morning. Most sprinters were people who had never worked with Drupal before the camp. They learned how to contribute to Drupal and worked on a few patches. One pleasant surprise was when Lucas went on stage with one of the sprinters and proceeded with the live commit ceremony. I was overjoyed that even with a short sprint an attendee’s contribution was committed. Congrats to Jorge Morales for getting a patch committed on his first sprint! And thanks to Holger Lopez, Edys Meza, and Lucas Hedding for mentoring and working on the patch.

https://twitter.com/drupalni/status/931968505406218240

Swag

Northern Lights DrupalCamp decided to change the (physical) swag for experiences. What we lived was epic! For our camp, we went for a low cost swag. The only thing we had to pay for was t-shirts. Other local communities recommended us to have them and so we did. The rest was a buffet of the things I have collected since my first DrupalCon, Austin 2014: stickers, pins, temporary tattoos. It was funny trying to explain where I had collected each item. I could not remember them all, but it was nice to bring back those memories. We also had hand sanitizer and notebooks provided by local communities. Can you spot your organization/camp/module/theme logo on our swag table?

https://twitter.com/drupalni/status/931555663972634624
https://twitter.com/drupalni/status/931893854311239680
https://drive.google.com/open?id=1hx8bFDO79gN8Y2IKxdyrUpKXx-H0TsDN

Free software communities

We were very lucky to have the support of different local communities. We learned a lot from their experiences organizing events. They also sent an army of volunteers and took the microphone to present on different subjects. A special thank you to the WordPress Nicaragua community who helped us immensely before, during, and after the event. It showed that when communities work together, we make a bigger impact.

https://drive.google.com/open?id=1C3jLFblkPwcQ8ZQwgU-edS7Q2DArLw51
https://twitter.com/drupalni/status/932025833207861248

Keeping momentum

Two weeks after the camp, we held two Global Training Days workshops. More than 20 people attended. I felt honored when some attendees shared that they had travelled from distant places to participate. One person travelled almost 8 hours. But more than distance, it was their enthusiasm and engagement during the workshops that inspired us. The last month has been very exhausting, but the local community is thrilled with the result.

https://twitter.com/lucashedding/status/937693447804383232
https://twitter.com/drupalni/status/937034655718666241

A blooming community

The community has come a long way since I got involved in 2011. We have had highs and lows. Since Lucas and myself kickstarted the Global Training Days workshops in 2014 we have seen more interest in Drupal. By the way, this edition marked our third anniversary facilitating the workshop! But despite all efforts, people would not stay engaged for long after initially interacting with the community. Things have changed.

In the last year interest in Drupal has increased. We have organized more events and more people have attended. Universities and other organizations are approaching us requesting trainings. And what makes me smile most… the number of volunteers is at its all-time peak. In the last month alone, the number of volunteers have almost doubled. The DrupalCamp and the Global Training Days workshops contributed a lot to this.

We recognize that the job is far from complete and we already have plans for 2018. One of the things that we need to do is find job opportunities. Even if people enjoy working with Drupal they need to make a living. If you are an organization looking for talent consider Nicaragua. We have very great developers. Feel free to get in touch to put you in contact with them.

https://drive.google.com/open?id=1yukr-xzALMowbeJlFOdwPGiO5MC5YWMm
https://drive.google.com/open?id=1kSPBMDNSIrGRb5BeLXKVE2xxrfOUJphA
https://twitter.com/drupalni/status/931958170259468289

A personal thank you

I would like to take this opportunity to say thanks to Felix Delattre. He started the Drupal community in Nicaragua almost a decade ago. He was my mentor. He gave me my first Drupal gig. At a time when there was virtually no demand for Drupal talent in my country, that project helped me realize that I could make a living working with Drupal. But most importantly, Felix taught me the value of participating in the community. I remember creating my drupal.org account after he suggested it in a local meetup.

His efforts had a profound effect on the lives of many, even beyond the borders of my country or those of a single project. Felix was instrumental in the development of local communities across Central and South America. He also started the OpenStreetMap (OSM) community in Nicaragua. I still find it impressive how OSM Nicaragua have mapped so many places and routes. In some cities, their maps are more accurate and complete than those of large Internet corporations. Thank you Felix for all you did for us!

Sunday group photo

We hope to have you in 2018!

The land of lakes and volcanoes awaits you next year. Nicaragua has a lot to offer and a DrupalCamp can be the perfect excuse to visit. ;-) Active volcanoes, beaches to surf, forests rich in flora and fauna are some of the charms of this tropical paradise.

Let’s focus on volcanoes for a moment. Check out this website for a sneak peek into one of our active volcanoes. That is Masaya, where you can walk to the border of the crater and see the flow of lava. Active volcanoes, dormant volcanoes, volcanoes around a lake, volcanoes in the middle of a lake, lagoons on top of volcanoes, volcanoes where you can “surf” down the slope... you name it, we have it.

We would love to have you in 2018!

https://twitter.com/drupalni/status/932024521674063872

In this album there will be more photos of the event.

Dec 04 2017
Dec 04

Once a text field has data stored, it is not very easy or obvious how to change its maximum length. In the UI there is a message warning you that the field cannot be changed, because there is existing data. Sometimes it is necessary to change these values. It seems that there are a few ways and some resources to do this in Drupal 7, but I could not find a way to do this in Drupal 8. I decided to create a small function to do it:

Caution: Any change in the database needs to be done carefully. Before you continue please create a backup of your database.

/**
 * Update the length of a text field which already contains data.
 *
 * @param string $entity_type_id
 * @param string $field_name
 * @param integer $new_length
 */
function _module_change_text_field_max_length ($entity_type_id, $field_name, $new_length) {
  $name = 'field.storage.' . $entity_type_id . "." . $field_name;

  // Get the current settings
  $result = \Drupal::database()->query(
    'SELECT data FROM {config} WHERE name = :name',
    [':name' => $name]
  )->fetchField();
  $data = unserialize($result);
  $data['settings']['max_length'] = $new_length;

  // Write settings back to the database.
  \Drupal::database()->update('config')
    ->fields(['data' => serialize($data)])
    ->condition('name', $name)
    ->execute();

  // Update the value column in both the _data and _revision tables for the field
  $table = $entity_type_id . "__" . $field_name;
  $table_revision = $entity_type_id . "_revision__" . $field_name;
  $new_field = ['type' => 'varchar', 'length' => $new_length];
  $col_name = $field_name . '_value';
  \Drupal::database()->schema()->changeField($table, $col_name, $col_name, $new_field);
  \Drupal::database()->schema()->changeField($table_revision, $col_name, $col_name, $new_field);

  // Flush the caches.
  drupal_flush_all_caches();
}

This method needs the name of the entity, the name of the field, and the name and the new length.

And we can use it like this:

   _module_change_text_field_max_length('node', 'field_text', 280);
  

Usually, this code should be placed in (or called from) a hook_update so it will be executed automatically in the update.

And if the new length is too long to be placed in a regular input area, you can use the Textarea widget for text fields which will allow you to use the larger text area form element for text fields.

Nov 23 2017
Nov 23

When you think of training, perhaps you remember an event that you were sent to where you had to learn something boring for your job. The word training does not usually make people smile and jump for joy, that is unless you are talking about Drupal training. These gatherings spread the Drupal knowledge and increase diversity in the community of Drupal developers.

Join us for the next Drupal Global Training Day with our online full day session on getting started with Drupal on November 29th. It will be held online from 9 AM to 4 PM EST.

Sign up now.

A link to the live workshop on Zoom will be provided when you sign up!

The Drupal Association coordinates four dates each year as Global Training Days, designed to offer free and low-cost training events to new-to-Drupal developers and to create more Drupal talent around the world. The community is growing exponentially as more people learn how fun and easy it is to get involved and be productive. Volunteer trainers host these global events in person and online. In 2016, a Global Training Days Working Group was established to run this program. There is a Global Training Days group on Drupal.org that lists trainings around the world.

Mauricio Dinarte will be leading the training online on November 29th. As an introduction to Drupal a person needs to learn certain things that are specific to Drupal and some are not that intuitive. It is important to cover the very basics in terminology and process. An introductory class can include many things, but this list is what Mauricio covers during the day long event:

  • Drupal installation requirements and process
  • Nodes
  • Content types
  • Fields
  • Blocks
  • Theme regions
  • Views
  • User and permissions
  • Menus
  • Taxonomy

The outcome of the day of training is that everyone walks away understanding the main moving parts of Drupal and a bit about what they do. Of course you will not become a developer overnight, but you will have enough information to build a simple site and then explore more of Drupal on your own.

You can follow up with many online tutorials and by joining the Drupal group in your area and attending the meetings. At meetings you will connect with other people at different levels of skill and you will be helped and helpful at the same time! If there is no Drupal group in your area, I suggest you start one. It can start as easily as posting online that you will be at a specific location doing Drupal at a certain time of day - you will be surprised at who may show up. If no one shows up the first time, try again or try a different location. One of the best things about Drupal is the community and how large and connected we are. If you start a group, people will usually help it grow.

Bringing new people to Drupal is not only good for increasing the size of the member base, it also brings diversity and reaches people that may never have had an opportunity or access to a free training. Drupal trainings are usually held at a university in or near a city which attracts people from different backgrounds and cultures. We can also reach people that are not in a city or near a school by sharing online.

Have you ever thought about volunteering at a Global Training Days event? We have a blog about organizing your own Global Training Days workshop that can get you started. This is a great way to get to know the people in the community better, up your skills and perhaps share something you have learned. I learned much about programming by assisting developers at sprints and trainings. This is where the real fun begins. Learning does not have to be stressful, and in the Drupal community people are friendly and welcoming. No question is stupid and even those with no experience have valuable skills. Developers love people without prior experience because they make the perfect testing candidates for UI and UX. The down side is that Drupal is so captivating that you will probably not remain a newbie for very long, so enjoy it while it lasts.

One of the true highlights of Global Training Days is seeing all the people around the world gain valuable skills and share knowledge. We hope you can join us.

Nov 22 2017
Nov 22

Drupal 8 has a great AJAX form API which includes some tools to create modal dialogs using the jQuery modal library. The Examples module even demonstrates how to create a custom form and display it in a modal window. But what if what you want to do is display an already created form in a modal? How do we do that? Let's see how to do it with an example. Let's display the node add form in a modal window.

The first thing that we need to do is create a link which will trigger the modal when the user clicks it. The only special things that this link needs to have are a few attributes that will let Drupal know to display the contents of the link in a dialog:

<a href="http://agaric.com/node/add/article"
  class="use-ajax"
  data-dialog-type="modal"
  data-dialog-options="{'width':800,'height':500}">
    Create Node
</a>

Drupal also needs to include the JavaScript libraries which will read these attributes and make them work, so let's add the following libraries to your module's dependencies (in your equivalent to this example's modal_form_example.libraries.yml file).

  dependencies:
     'core/drupal.dialog.ajax',
     'core/jquery.form',

If you are unsure about how to add libraries on Drupal 8 you can consult the documentation to either add it in a theme or add it in a custom module. At the end of the post I will provide a repository with the code where I added the libraries in a block.

And that's it! If you click the link, the form will be displayed in a modal dialog! Drupal will automatically detect that you are sending an AJAX request and will display just the form so you won't need to worry about removing the rest of the blocks or hiding undesired markup.

The last thing missing, is what will happen if the user creates a node? By default, the node will redirect the user to another page but if we want to just close the modal dialog and leave the user on the same page we need to tell the form to do that. For this we are going to alter the form and add an AJAX command letting Drupal know that we want to close the dialog as soon as the node is created. In the .module file of a custom module we will add this code:

use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Ajax\AjaxResponse;
use Drupal\Core\Ajax\CloseModalDialogCommand;
use Drupal\Core\Ajax\RedirectCommand;


/**
 * Implements hook_form_alter().
 */
function modal_form_example_form_node_article_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  $form['actions']['submit']['#submit'][] = '_modal_form_example_ajax_submit';
  $form['actions']['submit']['#attributes']['class'][] = 'use-ajax-submit';
}

/**
 * Close the Modal and redirect the user to the homepage.
 *
 * @param array $form
 *   The form that will be altered.
 * @param \Drupal\Core\Form\FormStateInterface $form_state
 *   FormState Object.
 */
function _modal_form_example_ajax_submit(array $form, FormStateInterface &$form_state) {
  $response = new AjaxResponse();
  $response->addCommand(new CloseModalDialogCommand());
  $form_state->setResponse($response);
}

The first function adds an extra submit function (which will be executed after Drupal finishes processing the node) and the second function adds the command to close the Dialog when the node has been created.

We can do this with practically any form in Drupal and you can add extra commands to do more complex things. Here are two resources:

Nov 14 2017
Nov 14

TL;DR: For PHP Hexadecimals, Decimals and Octals are all Integers, so they must be declared as @param integer

While I was working on a patch I had to write the docblock of a function which received a hexadecimal number and I wasn't sure what I was supposed to put in the @type param.

I went to Drupal's API documentation and comments standards page to see which is the best type for this param and I found the following:

Data types can be primitive types (int, string, etc.), complex PHP built-in types (array, object, resource), or PHP classes.

Alright, a hexadecimal number is not a complex PHP built-in type nor a PHP Class so it must be a primitive type, so I went to the PHP documentation page to see which primitives PHP has and I found the following:

  • boolean
  • integer
  • float (floating-point number, aka double)
  • String

So there wasn't a specific reference for a Hexadecimal number...

The solution:

In the end Pieter Frenssen helped me (Thanks!) with this, and he showed me that in PHP, it doesn't matter what the base number is and it can be an octal, hexadecimal or a decimal, for PHP they all are integers (which makes sense but I wanted to be sure) and he shared this small snippet where we can see that PHP sees the numbers as integers and the base doesn't matter:

$ php -a
Interactive shell

php > var_dump(gettype(0x0f));
string(7) "integer"

php > var_dump(0x08 === 8);
bool(true)

So if you are writing the documentation of a function in which one of its params is a hexadecimal number you must declare it as Integer.

Nov 07 2017
Nov 07

While creating content, there are pieces of information that are only relevant when other fields have a certain value. For example, if we want to allow the user to upload either an image or a video, but not both, you can have another field for the user to select which type of media they want to upload. In these scenarios, the Javascript States API for Drupal 8 can be used to conditionally hide and show the input elements for image and video conditionally.

Note: Do not confuse the Javascript States API with the storage State API.

The basics: conditional fields in node forms

Let’s see how to accomplish the conditional fields behavior in a node form before explaining the implementations for paragraphs. For this example, let’s assume a content type has a machine name of article with three fields: field_image, field_video, and field_media_type. The field_image_or_video field is of type List (text) with the following values: Image and Video.

/**
 * Implements hook_form_alter().
 */
function nicaragua_form_alter(&$form, \Drupal\Core\Form\FormStateInterface $form_state, $form_id) {
  if ($form_id == 'node_article_form' || $form_id == 'node_article_edit_form') {
    $form['field_ image']['#states'] = [
      'visible' => [
        ':input[name="field_image_or_video"]' => ['value' => 'Image'],
      ],
    ];

    $form['field_ video']['#states'] = [
      'visible' => [
        ':input[name="field_image_or_video"]' => ['value' => 'Video'],
      ],
    ]; 
  }
}

Note that in Drupal 8, the node add and edit form have different form ids. Hence, we check for either one before applying the field states. After checking for the right forms to alter, we implement the fields’ states logic as such:

$form[DEPENDEE_FIELD_NAME]['#states'] = [
  DEPENDEE_FIELD_STATE => [
    DEPENDENT_FIELD_SELECTOR => ['value' => DEPENDENT_FIELD_VALUE],
  ],
];

DEPENDENT_FIELD_SELECTOR is a CSS selector to the HTML form element rendered in the browser. Not to be confused with a nested Drupal form structure.

Conditional fields in Drupal 8 paragraphs

Although hook_form_alter could be used in paragraphs as well, their deep nesting nature makes it super complicated. Instead, we can use hook_field_widget_form_alter to alter the paragraph widget before it is added to the form. In fact, we are going to use the widget specific hook_field_widget_WIDGET_TYPE_form_alter to affect paragraphs only.

For this example, let’s assume a content type has a machine name of campaign with an entity reference field whose machine name is field_sections. The paragraph where we want to apply the conditional logic has a machine name of embedded_image_or_video with the following fields: field_image, field_video, and field_image_or_video. The field_image_or_video field is of type List (text) with the following values: Image and Video.

/**
 * Implements hook_field_widget_WIDGET_TYPE_form_alter().
 */
function nicaragua_field_widget_paragraphs_form_alter(&$element, \Drupal\Core\Form\FormStateInterface $form_state, $context) {
  /** @var \Drupal\field\Entity\FieldConfig $field_definition */
  $field_definition = $context['items']->getFieldDefinition();
  $paragraph_entity_reference_field_name = $field_definition->getName();

  if ($paragraph_entity_reference_field_name == 'field_sections') {
    /** @see \Drupal\paragraphs\Plugin\Field\FieldWidget\ParagraphsWidget::formElement() */
    $widget_state = \Drupal\Core\Field\WidgetBase::getWidgetState($element['#field_parents'], $paragraph_entity_reference_field_name, $form_state);

    /** @var \Drupal\paragraphs\Entity\Paragraph $paragraph */
    $paragraph_instance = $widget_state['paragraphs'][$element['#delta']]['entity'];
    $paragraph_type = $paragraph_instance->bundle();

    // Determine which paragraph type is being embedded.
    if ($paragraph_type == 'embedded_image_or_video') {
      $dependee_field_name = 'field_image_or_video';
      $selector = sprintf('select[name="%s[%d][subform][%s]"]', $paragraph_entity_reference_field_name, $element['#delta'], $dependee_field_name);

      // Dependent fields.
      $element['subform']['field_image']['#states'] = [
        'visible' => [
          $selector => ['value' => 'Image'],
       ],
      ];

      $element['subform']['field_video']['#states'] = [
        'visible' => [
          $selector => ['value' => 'Video'],
        ],
      ];
    }
  }
}

Paragraphs can be referenced from multiple fields. If you want to limit the conditional behavior you can check the name of the field embedding the paragraph using:

$field_definition = $context['items']->getFieldDefinition();
$paragraph_entity_reference_field_name = $field_definition->getName();

If you need more information on the field or entity where the paragraph is being embedded, the field definition (instance of FieldConfig) provides some useful methods:

$field_definition->getName(); // Returns the field_name property. Example: 'field_sections'.
$field_definition->getType(); // Returns the field_type property. Example: 'entity_reference_revisions'.
$field_definition->getTargetEntityTypeId(); // Returns the entity_type property. Example: 'node'.
$field_definition->getTargetBundle(); // Returns the bundle property. Example: 'campaign'.

In Drupal 8 it is a common practice to use the paragraph module to replace the body field. When doing so, a single field allows many different paragraph types. In that scenario, it is possible that different paragraph types have fields with the same name. You can add a check to apply the conditional logic only when one specific paragraph type is being embedded.

$widget_state = \Drupal\Core\Field\WidgetBase::getWidgetState($element['#field_parents'], $paragraph_entity_reference_field_name, $form_state);
$paragraph_instance = $widget_state['paragraphs'][$element['#delta']]['entity'];
$paragraph_type = $paragraph_instance->bundle();

The last step is to add the Javascript states API logic. There are two important things to consider:

  • The paragraph widgets are added under a subform key.
  • Because multiple paragraphs can be referenced from the same field, we need to consider the order (i.e. the paragraph delta). This is reflected in the DEPENDENT_FIELD_SELECTOR.
$element['subform'][DEPENDEE_FIELD_NAME]['#states'] = [
  DEPENDEE_FIELD_STATE => [
    DEPENDENT_FIELD_SELECTOR => ['value' => DEPENDENT_FIELD_VALUE],
  ],
];

When adding the widget, the form API will generate markup similar to this:

<select data-drupal-selector="edit-field-sections-0-subform-field-image-or-video"
  id="edit-field-sections-0-subform-field-image-or-video--vtQ4eJfmH7k"
  name="field_sections[0][subform][field_image_or_video]"
  class="form-select required"
  required="required"
  aria-required="true">
    <option value="Image" selected="selected">Image</option>
    <option value="Video">Video>
</select>

So we need a selector like select[name="field_sections[0][subform][field_image_or_video]"] which can be generated using:

$selector = sprintf('select[name="%s[%d][subform][%s]"]', $paragraph_field_name, $element['#delta'], $dependee_field_name);

By using $element['#delta'] we ensure to apply the conditional field logic to the proper instance of the paragraph. This works when a field allows multiple paragraphs, including multiple instances of the same paragraph type.

You can get the example code here.

Warning: Javascript behavior does not affect user input

It is very important to note that the form elements are hidden and shown via javascript. This does not affect user input. If, for example, a user selects image and uploads one then changes the selection to video and sets one then both the image and video will be stored. Switching the selection from image to video and vice versa does not remove what the user had previous uploaded or set. Once the node is saved, if there are values for the image and the video both will be saved. One way to work around this when rendering the node is to toggle field visibility in the node Twig template. In my session "Twig Recipes: Making Drupal 8 Render the Markup You Want" there is an example on how to do this. Check out the slide deck and the video recording for reference.

What do you think of this approach to add conditional field logic to paragraphs? Let me know in the comments.

Nov 03 2017
Nov 03

CKEditor is well-known software with a big community behind it and it already has a ton of useful plugins ready to be used. It is the WYSIWYG text editor which ships with Drupal 8 core.

Unfortunately, the many plugins provided by the CKEditor community can't be used directly in the CKEditor that comes with Drupal 8. It is necessary to let Drupal know that we are going to add a new button to the CKEditor.

Why Drupal needs to know about our plugins

Drupal allows us to create different text formats, where depending on the role of the user (and so what text formats they have available) they can use different HTML tags in the content. Also, we can decide if the text format will use the CKEditor at all and, if it does, which buttons will be available for that text format.

That is why Drupal needs to know about any new button, so it can build the correct configuration per text format.

Adding a new button to CKEditor

We are going to add the Media Embed plugin, which adds a button to our editor that opens a dialog where you can paste an embed code from YouTube, Vimeo, and other providers of online video hosting.

First of all, let's create a new module which will contain the code of this new button, so inside the /modules/contrib/ folder let's create a folder called wysiwyg_mediaembed. (If you're not intending to share your module, you should put it in /modules/custom/— but please share your modules, especially ones making CKEditor plugins available to Drupal!)

cd modules/contrib/
mkdir wysiwyg_mediaembed

And inside let's create the info file: wysiwyg_mediaembed.info.yml

name: CKEditor Media Embed Button (wysiwyg_mediaembed)
type: module
description: "Adds the Media Embed Button plugin to CKEditor."
package: CKEditor
core: '8.x'
dependencies:
  - ckeditor

Adding this file will Drupal allows us to install the module, if you want to read more about how to create a custom module, you can read about it here.

Once we have our info file we just need to create a Drupal plugin which will give info to the CKEditor about this new plugin, we do that creating the following class:

touch src/Plugin/CkEditorPlugin/MediaEmbedButton.php

With this content:

namespace Drupal\wysiwyg_mediaembed\Plugin\CKEditorPlugin;

use Drupal\ckeditor\CKEditorPluginBase;
use Drupal\editor\Entity\Editor;

/**
 * Defines the "wysiwyg_mediaembed" plugin.
 *
 * @CKEditorPlugin(
 *   id = "mediaembed",
 *   label = @Translation("CKEditor Media Embed Button")
 * )
 */
class MediaEmbedButton extends CKEditorPluginBase {

  /**
   * Get path to library folder.
   * The path where the library is, usually all the libraries are
   * inside the '/libraries/' folder in the Drupal root.
   */
  public function getLibraryPath() {
    $path = '/libraries/mediaembed';
    return $path;
  }

  /**
   * {@inheritdoc}
   * Which other plugins require our plugin, in our case none.
   */
  public function getDependencies(Editor $editor) {
    return [];
  }

  /**
   * {@inheritdoc}
   * The path where CKEditor will look for our plugin.
   */
  public function getFile() {
    return $this->getLibraryPath() . '/plugin.js';
  }

  /**
   * {@inheritdoc}
   *  
   *  We can provide extra configuration if our plugin requires
   *  it, in our case we no need it.
   */
  public function getConfig(Editor $editor) {
    return [];
  }

  /**
   * {@inheritdoc}
   * Where Drupal will look for the image of the button. 
   */
  public function getButtons() {
    $path = $this->getLibraryPath();
    return [
      'MediaEmbed' => [
        'label' => $this->t('Media Embed'),
        'image' => $path . '/icons/mediaembed.png',
      ],
    ];
  }
}

The class's code is pretty straightforward: it is just a matter of letting Drupal know where the library is and where the button image is and that's it.

The rest is just download the library and put it in the correct place and activate the module. If all went ok we will see our new button in the Drupal Text Format Page (usually at: /admin/config/content/formats).

This module was ported because we needed it in a project, so if you want to know how this code looks all together, you can download the module from here.

Now that you know how to port a CKEditor plugin to Drupal 8 the next time you can save time using Drupal Console with the following command:

drupal generate:plugin:ckeditorbutton

What CKEditor plugin are you going to port?

Sep 01 2017
Sep 01

In the next few weeks, Mauricio Dinarte (dinarcon on drupal.org) will be traveling to deliver his expertise to multiple Drupal events in Europe and America. He is on a mission to continue sharing the knowledge gained from many years as an active member of the Drupal community. Over the last few years he has presented numerous sessions and full day trainings at more than 18 Drupal camp— so you may have been at one of his presentations about Drupal basic concepts, twig recipes, or D8 migrations!

In addition to touring, Mauricio is very active in both local and global communities. He serves as a lead organizer of the Nicaraguan Drupal community where he has trained dozens of people, some of whom have made Drupal their career. He is also a member of the Drupal Global Training - Community Working Group and was recently added to Drupal's MAINTENANERS.txt as part of the mentoring team.

Here is where Mauricio will be presenting next:

Driven by passion to teach others what he has learned, Mauricio's skills go way beyond coding and he has been on a mission to take part in as many International Drupal Cons and Camps as humanly possible. If you happen to be in any of the cities on Mauricio's itinerary, please say hello to him and shake his hand for me. He has touched the lives of many developers and would-be developers, and started some on a path to follow in his footsteps.

Jul 27 2017
Jul 27

Former school teacher turned technologist, Clayton now applies his background in linguistics, community organizing and web development as a user experience architect for Agaric and co-founder of Drutopia.

Clayton has worked with universities, open source companies, tech diversity advocates, prison abolitionists, and others to translate their organizational goals into impactful digital tools with meaningful content.

Aside from content strategy and information architecture, Clayton also enjoys being a goofy dad and always appreciates a good paraprosdokian.

[email protected] Clayton Dewey's Public GPG Key

Social Media Mastodon

Jun 30 2017
Jun 30

Last week we were asked to disable the "threads" in the comments and display them as a plain list in one of our projects, so checking in the Comment field settings we found an option which says:

"Threading: Show comment replies in a threaded list."

Which basically display the comments as a threaded instead of a list of comments. This option is marked on by default, so it seemed that it was just a matter of unchecking this option, but there was a problem with this, that setting did display the comments as a plain list, but in the code they were still being saved as a thread.

The main problem with this is if a comment is deleted and it has replies (even if the option of the thread is unchecked) all the replies are going to be deleted as well.

This is not a new problem there is an 11 year old issue and it seems that this has been happening since Drupal 4. The way to fix this so far was by using a contrib module called: Flat Comments The problem is that this module hasn't been ported to D8 yet.

We decided to fix that and port the module to help others with this problem. You can check the code here: https://github.com/agaric/flat_comments and we are in contact with the module's maintainer to create the D8 branch in drupal.org soon.

Expecting that this can help someone else.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web