Basic Migration of File Fields to Media Entities Fields in Drupal 8

Parent Feed: 

The Migrate File to Media module provides an easy way to migrate old file fields like images, files, videos, etc, into the media entities with a few drush commands.

So you can have an understanding of how the migration process works, In this tutorial, we will run through a few quick step-by-step instructions on how to migrate specific image field types to image entity reference fields.

Media Module

The core media module is what creates the “media” entity types. If you haven’t set this up yet, or if you are not too familiar with media entity setup, I highly recommend following this tutorial “Drupal 8 Basic Media And Media Browser Setup For Beginners”, before you continue with this tutorial. 

If you still want to continue with a basic setup of media entities, simply enable/install the core media module that already comes with Drupal 8 core (NOT the media_entity module). Upon installation, it will automatically create a few media entity bundles for you, including the image media bundle we will be using for this tutorial.

Migrate File to Media module

Now for the migration, install the “Migrate File to Media module”

composer require drupal/migrate_file_to_media

Enable the “migrate_file_to_media” module:

drush en migrate_file_to_media

It will warn you that it will enable the following (“migrate”,”migrate_drupal”, “migate_plus” and “migrate_tools”) if they’re not enabled yet.

Once we have all of this in place, we can start using some simple commands to do the work for us. However, first, we need to create some mapping so that the migration module knows from what source fields to what destination fields.

Destination

We can’t migrate a source without a road map and destination. In the example of this tutorial, we will be performing a migration of the field “field_image” (image field type), in the content bundle “article” (node entity type) into a new media entity reference field that we will be creating using drush commands.

When migrating a site with a vast amount of content, I recommend to become familiar with the site’s content structure, the document where all the images are located, and in what content type bundles.

Generate the destination media field in the article node types using drush:

drush migrate:file-media-fields node article image image

Command bisection:

drush migrate:file-media-fields [entity_type] [bundle] [source_field_type] [destination_media_bundle]

  • [entity_type]: (The parent entity type) In the case of our example, it is a node content type. But other bundle types, like paragraphs or taxonomies for example, can be used.
  • [bundle]: (The parent bundle type) In this example, the “article”
  • [source_field_type]: (The source field type) This will be the source field type. Not to be confused with the name of the field, but the type of field. When running the command, this will check for all the fields of type “image” as in this tutorial.
  • [destination_media_bundle]: (The destination field media bundle) This will be the destination field bundle type. This will create a media entity reference field of type “images” for every image field found on the parent node bundle type. It will also give it the same name with the suffix of “_media” as in this tutorial.

Roadmap

Ok, so we created the destination. Now let’s create the roadmap so that the migration knows where and how to migrate data from the old to new by generating some YML files.

Before generating the YML files, we need to generate/create a custom module where the new files will reside: (Just generate the module, don’t enable it yet.)

Basic module generate command, run on web-root:

drupal generate:module
# Note: follow the instructions on the screen and give it the machine name “basic_migration_example”

- Or: Module generate command with all options for this tutorial, run on your web-root:

drupal generate:module --module="Basic Migration Example 101" --machine-name="basic_migration_example" --module-path="modules/custom" --description="Basic Migration Tutorial 101" --core="8.x" --package="Custom" --features-bundle=”no” --test=”yes” --dependencies=”no” --twigtemplate=”no” --composer --module-file

Once the module is generated, clear the cache:

drush cr

Now we are ready to start generating the YML files, run:

drush generate yml-migrate_file_to_media_migration_media

You will get something like this. For this tutorial please follow the instructions as follows:

Welcome to yml-migrate_file_to_media_migration_media generator!

–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

 Module machine name:
 ➤ basic_migration_example

 Module name:
 ➤ Basic Migration Example 101

 Plugin label [Example]:
 ➤ article_media_basic_migration

 Plugin ID [example]:
 ➤ article_media_basic_migration

 Migration Group [media]:
 ➤ ( Press Enter Button to use suggested "media" )

 Entity Type [node]:
 ➤ ( Press Enter Button to use suggested "node")

 Source Bundle:
 ➤ article

 Source Field Names (comma separated) [field_image]:
 ➤ ( Press Enter Button to use suggested "field_image" )

 Target Media Type [image]:
 ➤ ( Press Enter Button to use suggested "image" )

 Target Field [field_media_image]:
 ➤ ( Press Enter Button to use suggested "field_media_image" )

 Language Code [en]:
 ➤ ( Press Enter Button to use suggested "en" )

 Translation languages (comma separated) [none]:
 ➤ ( Press Enter Button "none" )

The new files will be created in your new module folder /config/install.

There will be two YML files generated per migration ID, these file names will be suffixed with _step1 and _step2. You can open these files and adjust them as needed, but for this tutorial, the configuration we gave it on the YML generation process is just what we need.

If you want to look at YML examples, you will find some under the migrate_file_to_media module folder: modules/contrib/migrate_file_to_media/migrate_file_to_media_example/

Once you install/enable the new module we just created for this migration, all the YML configurations will be loaded to your Drupal 8 database setup.

Editing Migration Configurations after installation:

Once loaded into the system, if you need to make modifications, you need to get familiarized with the configuration management interface and how it all works to make changes. Here is a good read on that: https://www.ostraining.com/blog/drupal/config/

Install a New Module and YML files:

Ok, so now that the YML files are created, let’s enable/install the new module, and the new YML configuration files will be added to the system as well.

Enable new custom module:

drush en basic_migration_example

And now, check the migration status by running: (This will also give you the generated migration ids)

drush migrate:status

You will get a result of something like this:

Group Migration ID Status Total Imported Unprs
media article-media-basic-migration_step1 Idle 3 0 3
media article-media-basic-migration_step2 Idle 3 0 3
           
Default  ***Ignore the “Default (default)” group*** 0 0 0

Duplicate File Detection (!)

– This is an important step, DO NOT SKIP – In order to run migrations, first you need to run a file duplication check for all migrations.

drush migrate:duplicate-file-detection [migration-id]

For our tutorial, run file duplication checks run “_step1” of the migration. You will have to do this for every first instance of your migrations one by one as I have not been able to figure out another way to make this command just run through all in one command.

For this tutorial, run:

drush migrate:duplicate-file-detection article_media_basic_migration_step1

Migration time:

Now, if all is set up correctly, we can run a migration:

Migrate all migration of media group by group id:

drush migrate:import --group=media 

- Or: Run single migration by ids:

drush migrate:import article_media_basic_migration_step1 
drush migrate:import article_media_basic_migration_step2 

If it all goes well, you should see that the imported will match the total and unprocessed will be 0. Run:

drush migrate:status

Results should look like this:

Group Migration ID Status Total Imported Unprs
media article-media-basic-migration_step1 Idle 3 3 0
media article-media-basic-migration_step2 Idle 3 3 0
           
Default  ***Ignore the “Default (default)” group*** 0 0 0

Take a look at the content of your article. You will now see that the new fields have been populated with the right image media entity reference. Just adjust your displays to show the new media field and hide or remove them altogether with the old image field when all your migration is complete.

A few more drush migration commands that may come useful:

  • drush migrate –help
  • drush migrate:rollback
  • drush migrate:stop
  • drush migrate:reset-status
  • drush migrate:import  [_OPTIONS_] :
    • –feedback - Frequency of progress messages, in seconds or items processed ( great for debugging)
    • –update - In addition to processing unimported items from the source, update previously-imported items with new data
    • –group - Name of the migration group to run
    • –idlist - A comma-delimited list of ids to import or rollback. If unspecified, migrate imports all pending items or rolls back all items for the content set.
    • –limit - Limit on the length of each migration process, expressed in seconds or number of items
    • –rollback - Rollback specified migration(s) if applicable
    • –stop - Stop specified migration(s) if applicable
    • Less commonly used options:
      • –file_function - Override file function to use when migrating images
      • –force - Force an operation to run, even if all dependencies are not satisfied
      • –needs-update - Reimport up to 10K records where needs_update=1. This option is only needed when your Drupal DB is on a different DB server from your source data. Otherwise, these records get migrated with just migrate-import.
      • –instrument - Capture performance information (timer, memory, or all)

Further information on Drush Migrate Tools commands visit: https://www.drupal.org/node/1561820

Happy coding!

Original Post: 

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web