Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough

Migrate

Migrate

Posted on: Thursday, August 16th 2012 by Brandon Tate

Time and time again, I’ve had to import content into a Drupal site. In the past I would rely on Feeds to do the job for me. However, over time, I found Feeds to come up short when it came to complex imports. So, I went looking elsewhere for a better solution and came across Migrate and haven’t looked back since. I’ve developed four import solutions for different sites and have only come across one problem with Migrate which I’ll get to later. First, let's get into the great features of Migrate.

I’ll start by creating a Film migration example. This will be extending an abstract class provided by Migrate named Migration - in most cases, you'll do nearly all your work in extending this class. The Migration class handles the details of performing the migration for you - iterating over source, creating destination objects, and keeping track of the relationships between them.

Sources

First, lets start with defining the source data and where it's coming from. In all cases, I’ve had to import CSV or Tab delimited files, so my import source has been MigrateSourceCSV. This tells migrate that your import source is coming from a CSV or Tab file. When importing tab files though, you need to specify in the options array the delimeter to be “\t”. As well, all files I’ve imported include a header, within the options array you can specify this option to ensure that the header row is not imported. Next, we will want to specify an array describing the file's columns where keys are integers (or may be omitted), values are an array of field name then description. However, if your file contains a header column this may be left empty (I've provided this array below). Migrate can also import from SQL, Oracle, MSSQL, JSON and XML. More Info on sources.

class FilmExampleMigration extends Migration {
  public function __construct() {
    parent::__construct();
    $options = array();
    $options['header_rows'] = 1;
    $options['delimiter'] = "\t";
    $columns[0] = array('Film_ID', 'Film ID');
    $columns[1] = array('Film_Body', 'Film Body');
    $columns[2] = array('Origin', 'Film Origin');
    $columns[3] = array('Producer_ID', 'Film Producer');
    $columns[4] = array('Actor_ID', 'Film Actors');
    $columns[5] = array('Image', 'Film Image');
    $this->source = new MigrateSourceCSV(variable_get('file_private_path', conf_path() . '/files-private') . '/import_files/example.tab', $columns, $options);

Destinations

Next, we will need to define the destination. Migrate can import into Nodes, Taxonomy Terms, Users, Files, Roles, Comments and Tables. As well, there is another module called Migrate Extras which extends the stock ability of Migrate. I personally have implemented Nodes, Terms and Tables so I will discuss those here. Migrate to a node is done by including MigrateDestinationNode with the node type as the parameter. Terms are defined by including MigrateDestinationTerm with the vocabulary name as the parameter. Lastly, Tables is defined by MigrateDestinationTable where the table name is the parameter. Migrating into a table also requires that the table is already created within your database. More Info on destinations.

//term
$this->destination = new MigrateDestinationTerm('category');

//node
$this->destination = new MigrateDestinationNode('film');

//table
$this->destination = new MigrateDestinationTable('film_category_reference_table');

Migrate Map

With Migrate, the MigrateMap class tracks the relationship of source records to their destination records as well as messages generated during the migration. Your source can have a single map key or multi key. Below is an example of both.

//single key
$this->map = new MigrateSQLMap(
     $this->machineName,
       array(
          'Film_ID' => array(
            'type' => 'varchar',
            'length' => 255,
            'not null' => TRUE,
          ),
    MigrateDestinationNode::getKeySchema()
);

//multi key - importing to a table 
$this->map = new MigrateSQLMap(
    $this->machineName,
      array(
        'Film_ID' => array(
           'type' => 'varchar',
           'length' => 255,
           'not null' => TRUE,
         ),
        'Origin_ID' => array(
          'type' => 'varchar',
          'length' => 255,
          'not null' => TRUE,
        )            
    ),
    MigrateDestinationTable::getKeySchema('viff_guest_program_xref')
);

Dependencies

Some imports depend on other imports so Migrate handles this by including the ability to define hard or soft dependencies. Hard dependencies give you the ability to prevent a child import to run before the parent import has run successfully. That means, no errors or anything can occur during that parent import before the child import can run. A soft dependency ensures that the parent import has run before the child but does allow errors to occur within the parent.

//hard dependencies
$this->dependencies = array('Producers', 'Actors');
//soft dependencies
$this->softDependencies = array('Producers', 'Actors');

Field Mappings

Next up, we need to associate source fields to destination fields using what Migrate calls field mappings. You can read up on Migrate’s mappings here. In a simple case, usually text or an integer is provided and it is easily mapped into Drupal using the following line:

$this->addFieldMapping('field_body', 'Film_Body');
$this->addFieldMapping('field_id', 'Film_ID');

Field mappings also allows you to provide default values, such as:

//default image
$this->addFieldMapping('field_image', 'Film_Image')->defaultValue(‘public://default_images/default_film.png’);
//default body text to full html
$this->addFieldMapping('field_body', 'Film_Body')->defaultValue(‘full_html’);

In some cases you need to reference one node to another. This can easily be done using the sourceMigration function. In this example, the ID provided in Program_ID is associated to the Drupal entity that will be created, using the entity ID that was created in the MigrateProgram migration.

$this->addFieldMapping('field_producer', 'Producer_ID')->sourceMigration('MigrateProducer');

Another useful ability is to explode multiple values into a single field. Imagine a Film has multiple actors and the data is defined as “value1::value2::value3”. We would handle this use case using the following:

$this->addFieldMapping('field_actors', Actors')->separator('::')->sourceMigration('MigrateActor');

Taxonomy values are commonly used in migrates but in most cases the client’s CSV file does not contain the term ID needed, instead the term name is provided. In this case, we need to tell the field mapping that the name is being provided instead of the ID:

$this->addFieldMapping('field_genre', 'Genre')->arguments(array('source_type' => 'term'));

Migrate Hooks

In most cases Migrate’s field mappings can handle all situations of getting the data in the system. However, sometimes you need access to the row being imported or the entity type being created. Migrate gives you three functions for this, prepareRow($row), prepare(&$node, $row) and complete(&$node, $row). PrepareRow() allows you to manipulate the row before the rows starts to be imported into the system. You can modify row attributes or even append more row columns as needed. Prepare() allows you to modify the node before it gets saved. Complete() is essentially the same as prepare() but is fired at the end of the row import process.

//prepare example
function prepare(&$node, $row) {
  // concatinate date + time
  $start_time_str  = $row->Event_Date .' '. $row->Event_Time;
  $start_timestamp = strtotime($start_time_str);
  $start_time      = date('Y-m-d\TG:i:s', $start_timestamp);

  $end_time_str  = $row->Event_Date_End .' '. $row->Event_End_Time;
  $end_timestamp = strtotime($end_time_str);
  $end_time      = date('Y-m-d\TG:i:s', $end_timestamp);

  $node->field_event_starttime[LANGUAGE_NONE][0]['value'] = $start_time;
  $node->field_event_endtime[LANGUAGE_NONE][0]['value']   = $end_time;
}

//prepareRow example
public function prepareRow($row){
  //prepend image location onto filenames
  $prefix = 'public://import/images/';
  $row['Image'] = $prefix . $row['Image'];
}

Migrate Drush Commands

Last but not least, I wanted to touch on Drush. Luckily for us command line lovers Migrate has implemented a bunch of useful drush commands that allow us to import, rollback and do just about anything the Migrate UI can do. Here is the list below:

migrate-audit (ma)	View information on problems in a migration.         	
migrate-deregister      Remove all tracking of a migration                   	
migrate-fields-desti     List the fields available for mapping in a destination. 
migrate-fields-sourc   List the fields available for mapping from a source. 	
migrate-import (mi)    Perform one or more migration processes              	
migrate-mappings      View information on all field mappings in a migration.  
migrate-reset-status   Reset a active migration's status to idle            	
migrate-rollback (mr) 	Roll back the destination objects from a given migration
migrate-status (ms)    List all migrations with current status.             	
migrate-stop (mst)	Stop an active migration operation                   	
migrate-wipe (mw) 	Delete all nodes from specified content types.  

I mentioned earlier that there is only one problem I’ve seen with Migrate. This issue is related to CSV and Tab files. Migrate is expecting the files to be continuous. This allows migrate to rollback and re-import at a whim. However, if the import file contains only the new set of data you wish to import, and not the old data that is already imported, you lose the roll back ability because the mappings are lost. As well, the Migrate UI becomes pretty confusing as none of the total rows, imported and un-imported columns make sense since the ID’s don’t relate to data stored within migrates mapping tables. This is the only issue I’ve come across Migrate but still prefer it over other options.

Overall, I’m very impressed with Migrate and its ability to offer such a verbose option of sources, destinations, field mappings and hooks that allow you to ensure the data will make it into your Drupal site no matter what the situation. Feel free to comment and offer suggestions on your experiences with Migrate. For more information on Migrate, check the documentation. Also, I've attached the Film example migration file here.

Author: 
Original Post: 

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web