Jun 18 2018
Jun 18
June 18th, 2018

Last month, Ithaca College introduced the first version of what will represent the biggest change to the college’s website technology, design, content, and structure in more than a decade—a redesigned and rebuilt site that’s more visually appealing and easier to use.

Over the past year, the college and its partners Four Kitchens and design firm Beyond have been hard at work within a Team Augmentation capacity to support the front-to-back overhaul of Ithaca.edu to better serve the educational and community goals of Ithaca’s students, faculty, and staff. The results of the team’s efforts can be viewed at https://www.ithaca.edu.  

Founded in 1892, Ithaca College is a residential college dedicated to building knowledge and confidence through a continuous cycle of theory, practice, and performance. Home to some 6,500 students, the college offers more than 100 degree programs in its schools of Business, Communications, Humanities and Sciences, Health Sciences and Human Performance, and Music.

Students, faculty, and staff at Ithaca College create an active, inclusive community anchored in a keen desire to make a difference in the local community and the broader world. The college is consistently ranked as one of the nation’s top producers of Fulbright scholars, one of the most LGBTQ+ friendly schools in the country, and one of the top 10 colleges in the Northeast.

We emphasized applying automation and continuous integration to focus the team on the efficient development of creative and easy to test solutions.

On the backend, the team—including members of Ithaca’s dev org working alongside Four Kitchens—built a Drupal 8 site. The transition to Drupal 8 keeps the focus on moving the college to current technology for sustainable success. Four Kitchens emphasized applying automation and continuous integration to focus the team on the efficient development of creative and easy to test solutions. To achieve that, the team set up automation in Circle CI 2.0 as middleware between the GitHub repository and hosting in PantheonGitHub was used throughout the project to implement, automate, and optimize visual regression, advanced communication between systems and a solid workflow using throughout the project to ensure fast and effective release cycles. Learn from the experiences obtained from implementing the automation pipeline in the following posts:

The frontend focused heavily on the Atomic Design approach. The frontend team utilized Emulsify and Pattern Lab to facilitate pattern component-based design and architecture. This again fostered long-term ease of use and success for Ithaca College.

The team worked magic with content migration. Using the brainchild of Web Chef, David Diers, the team devised a plan to migrate of portions of the site one by one. Subsites corresponding to schools or departments were moved from the legacy CMS to special Pantheon multidevs that were built off the live environment. Content managers then performed a moderated adaptation and curation process to ensure legacy content adhered to the new content model. A separate migration process then imported the content from the holding environment into the live site. This process allowed Ithaca College’s many content managers to thoroughly vet the content that would live on the new site and gave them a clear path to completion. Learn more about migrating using Paragraphs here: Migrating Paragraphs in Drupal 8

Steady scrum rhythm, staying agile, and consistently improving along the way.

In addition to the stellar dev work, a large contributor to the project’s success was establishing a steady scrum rhythm, staying agile, and consistently improving along the way. Each individual and unit solidified into a team through daily 15-minute standups, weekly backlog grooming meetings, weekly ‘Developer Showcase Friday’ meetings, regular sprint planning meetings, and biweekly retrospective meetings. This has been such a shining success the internal Ithaca team plans to carry forward this rhythm even after the Web Chefs’ engagement is complete.     

Engineering and Development Specifics

  • Drupal 8 site hosted on Pantheon Elite, with the canonical source of code being GitHub and CircleCI 2.0 as Continuous Integration and Delivery platform
  • Hierarchical and decoupled architecture based mainly on the use of group entities (Group module) and entity references that allowed the creation of subsite-like internal spaces.
  • Selective use of configuration files through the utilization of custom and contrib solutions like Config Split and Config Ignore modules, to create different database projections of a shared codebase.
  • Migration process based on 2 migration groups with an intermediate holding environment for content moderation.
  • Additional migration groups support the indexing of not-yet-migrated, raw legacy content for Solr search, and the events thread, brought through our Localist integration.
  • Living style guide for site editors by integrating twig components with Drupal templates
  • Automated Visual Regression
Aerial view of the Ithaca College campus from the Ithaca College homepage. From the Ithaca College Homepage.

A well-deserved round of kudos goes to the team. As a Team Augmentation project, the success of this project was made possible by the dedicated work and commitment to excellence from the Ithaca College project team. The leadership provided by Dave Cameron as Ithaca Product Manager, Eric Woods as Ithaca Technical Lead and Architect, and John White as Ithaca Dev for all things legacy system related was crucial in the project’s success. Ithaca College’s Katherine Malcuria, Senior Digital User Interface Designer,  led the creation of design elements for the website. 

Katherine Malcuria, Senior Digital User Interface Designer, works on design elements of the Ithaca.edu website

Ithaca Dev Michael Sprague, Web Chef David Diers, Architect,  as well as former Web Chef Chris Ruppel, Frontend Engineer, also stepped in for various periods of time on the project.  At the tail end of the project Web Chef, Brian Lewis, introduced a new baby Web Chef to the world, therefore the amazing Randy Oest, Senior Designer and Frontend Engineer, stepped in to assist in pushing this to the finish line from a front-end dev perspective. James Todd, Engineer, pitched in as ‘jack of all trades’ connoisseur helping out where needed.

The Four Kitchens Team Augmentation team for the Ithaca College project was led by Brandy Jackson, Technical Project Manager, playing the roles of project manager, scrum master, and product owner interchangeably as needed. Joel Travieso, Senior Drupal Engineer, was the technical lead, backend developer, and technical architect. Brian Lewis, Frontend Engineer, meticulously worked magic in implementing intricate design elements that were provided by the Ithaca College design team, as well a 3rd party design firm, Beyond, at different parts of the project.

A final round of kudos goes out to the larger Ithaca project team, from content, to devOps, to quality assurance, there are too many to name. A successful project would not have been possible without their collective efforts as well.

The success of the Ithaca College Website is a great example of excellent team unity and collaboration across multiple avenues. These coordinated efforts are a true example of the phrase “teamwork makes the dream work.” Congratulations to all for a job well done!

Special thanks to Brandy Jackson for her contribution to this launch announcement. 

Four Kitchens

The place to read all about Four Kitchens news, announcements, sports, and weather.

May 23 2018
May 23

On 19th May 2018, the day of the Royal Wedding and the FA-Cup final, there was a lot going on in the UK. As a Chelsea fan, I wonder if there could be a better feeling than watching the FA Cup final at Wembley against Manchester United?! But, instead, a small group of Drupal users gathered together in Edinburgh for DrupalCamp Scotland. I was invited to attend and chosen to speak about Drupal migration, rather than going to Wembley... I can confirm that it was an excellent decision and I had a great time there!

DrupalCamp ScotlandMost of the attendees at DrupalCamp Scotland

DrupalCamp Scotland kick-started with a welcome speech from Jennifer Milne, Deputy CIO of the University of Edinburgh. She spoke a bit about the history of the university, and I was amazed to learn that it is the sixth oldest university in the UK! It was very nice to see the Deputy CIO and Director of the University of Edinburgh so excited and happy to support the Open source Technology Community, including Drupal.

After that, Billy Wardrop did a quick presentation about Drupal Europe. He explained the event details and how you can get involved. More importantly, he noted how vital the event is for Drupal users in Europe. Why? Because, it is more than a technical conference; Europe's largest Drupal event in 2018 will be a festival that promotes real feeling between people who have the opportunity to meet each other and to have a great time together. Drupal Europe will introduce additional dimensions including summits for industry verticals, workshops, and a broad set of session tracks, including Back-end development, Front-end, Design/UX, Site Building, Project Management.

The first talk at DrupalCamp Scotland was given by Jeffrey A. "JAM" McGuire, from Open Strategy Partners. He was speaking about how we can do ‘Authentic Communication’ in business and how this maps to how we should can be more purposeful and generate greater impact in our Drupal contributions.

Authentic CommunicationThe essential components of authentic communication

He was also telling us how “Empathy” and “Context” have to work together in business communications:

Empathy and ContextThe importance of Empathy and Context

Finally “JAM” talked about how to plan your own Contribution narrative. He explained that your “Contribution Narrative” must encapsulate the following:

  • Goals: what do you want or need from contribution
    • more features, quality, increase adoption
    • Help others, educate people
  • Who: could help you get there
  • Vibrancy signals: how a free and open source projects appears worthy of engagement for someone: regular releases, state of issue queue, quality of docs and release notes, responsiveness in channels.
  • Why: the logical/emotional reasons someone should contribute
  • How to contribute: another vibrancy signal: you care enough to be explicit about how people should engage. Telling them your expectations is also setting their expectations.
  • Where to connect: how you prefer to communicate, where your people hang out.

Contribution NarrativePlanning your Contribution Narrative

Once you know and have mapped out all of this, you can weave it into all your comms and hopefully attract new contributors.” Jeffrey A. ‘Jam’ McGuire

The second session was from Julia Pradel and Christian Fritsch from Thunder CMS Team. They talked about the feature provided by Thunder CMS as a Drupal distribution, explaining who is using this feature globally and how useful it is for media and publishing products. Then they talked about the SharpEye theme testing tool, which they have also introduced to Drupal. SharpEye a visual regression tool that takes screenshots and compares them in automated tests. We open sourced the tool and you can download it here.

Thunder CMSJulia and Christian's talk: Photos from Thunder CMS

The third talk was given by me! I was talking about migration API in Drupal 8, specifically: How to migrate data from a csv file and what is gained by migrating everything into Drupal. I have begun writing a blog series on the same theme, if you would like to find out more.

After my speech, we had a great lunch and enjoyed the surprising beauty of the Scottish summer.

Lunchtime Sunshine at DrupalCamp ScotlandNetworking over lunch in the Scottish Sunshine

After Lunch, the fourth session of the day was taken by Audra Martin MerrickAs the former VP Strategy & Operations for The Economist, Audra was talking about Content and Community, or Content and its Audience. She explained how to use social media efficiently, to connect the right content with the right audience.


How to convene an audienceHow do we convene an audience?

The fifth talk of the session came from Alice G Herbison. She talked about challenges identified from her own experience, as well as the edge case scenarios to be mindful of when conducting User Research. Her advice: she explained how to use tree testing for user experience and why it is important.

Defining Research GoalsThe importance of User Research

We had quick coffee break and got back to the sixth session of the day, which was taken by Martin Fraser. He was talking all about css: examples of people writing bad css, advice on how they can write good css, and how css can promote user interactivity, instead of using Javascript.

Poor old CSSPoor Old CSS

The seventh and final session of the day came from Paul Linney. He was talking about Voyage to Drupal 8. As he is currently working on some sizeable government projects, he discussed the problems he faced with Drupal 7 features and custom module, and how they improved the performance when they changed custom module with object-oriented programming. To support his important projects, he is looking into Drupal 8 instead.

Looking to Drupal 8Looking to Drupal 8

The closing note was delivered by Duncan Davidson. He was one of the main organisers who made DrupalCamp Scotland possible.

I really enjoyed my time at DrupalCamp Scotland and would like to give special thanks to Billy, Duncan and all of the other organisers. Huge thanks to University of Edinburgh and Makkaru for sponsoring the event and great thanks to the attendees also. On a personal note, I thank Paul Johnson and CTI for supporting me to go there and speak.

If you would like to learn more about my insights into Drupal 8 migrations, follow my blog series.

Apr 26 2018
Apr 26

In our previous blog post, we gave a brief intro to some terms that we believe are necessary to understand the basics of Drupal.   Here we have what we believe to be the next round of terms that we consider necessary to understanding those basics. Recently, we had the opportunity to assist Matrix AMC in migrating from Drupal 6 to Drupal 8.  They were unable to use their website because of the version of Drupal that their website was hosted on was out of date and no longer supported by the Drupal community. While these specific terms are consistent across Drupal versions, they are crucial to understanding the importance of being up to date in with your version of Drupal.

Key Terms:     

  1. Block – the boxes visible in the regions of a Drupal website.
    1. Most blocks (e.g. recent forum topics) are generated on-the-fly by various Drupal modules, but they can be created in the administer blocks area of a Drupal site.
  2. Region – defined areas of a page where content can be placed. Different themes can define different regions so the options are often different per-site. Basic regions include:
    1. Header
    2. Footer
    3. Content
    4. Left sidebar
    5. Right Sidebar
  3. Roles – a name for a group of users, to whom you can collectively assign permissions. There are two predefined, locked roles for every new Drupal installation:
    1. Authenticated User- anyone with an account on the site.
    2. Anonymous User- those who haven’t yet created accounts or are not logged in.
  4. WYSIWYG – What You See Is What You Get; An acronym used in computing to describe a method in which content is edited and formatted by interacting with an interface that closely resembles the final product.
  5. Book – a set of pages tied together in a hierarchical sequence, perhaps with chapters, sections, or subsections.  Books can be used for manuals, site resource guides, Frequently Asked Questions (FAQs), etc.
  6. Breadcrumbs – the set of links, usually near the top of the page, that shows the path you followed to locate the current page.
    1. The term is borrowed from Hansel and Gretel, who left crumbs of bread along their path so they could find their way back out of the forest.
  7. Form mode – this is a way to customize the layout of an entity’s edit form.
  8. Multisite – a feature of Drupal that allows one to run multiple websites from the same Drupal codebase.
  9. Patch – a small piece of software designed to update or fix problems with a computer program or its supporting data.
    1. This includes fixing bugs, replacing graphics and improving the usability or performance.
  10. User – the user interacting with Drupal. This user is either anonymous or logged into Drupal through its account.

Refer to Drupal.org for any other questions!

Mar 28 2018
Mar 28

When it comes to considering what is the best CMS for a website, most don’t know up from down or Drupal from WordPress.  At Mobomo, we consider ourselves Drupal experts and have guided many of our clients through a Drupal migration. Drupal is a content management system that is at the core of many websites.  Drupal defines itself as “an open source platform for building amazing digital experiences.” These simple Drupal terms, or taxonomies, make it sound easy, but it can, in fact, be very confusing. Listed below are some popular terms defined to help make the start of the migration process what it should be, simple and easy:

Key Terms:

  1. Taxonomy – this is the classification system used by Drupal. This classification system is very similar to the Categories system you’ll find in WordPress.
  2. Vocabularies – a category, or a collection of terms.
  3. Terms – items that go into a vocabulary.
  4. Tags – this is a generic way to classify your content and this is also the default setting when you first begin.
  5. Menus – these refer both to the clickable navigation elements on a page, as well as to Drupal’s internal system for handling requests. When a request is sent to Drupal, the menu system uses the provided URL to determine what functions to call.  
    • There are 4 types:
      • Main
      • Management
      • Navigation
      • User
  6. Theme – this refers to the look and feel of a site and it is determined by a combined collection of template files, in addition to configuration and asset files.  Drupal modules define themeable functions which can be overridden by the theme file.  The header, icons, and block layout are all contained within a theme
  7. Content-Type – Every node, see below for definition, belongs to a content type.  This defines many different default settings for nodes of that type.  Content Types may have different fields, as well as modules may define their own content types.
  8. Fields – These are elements that can be attached to a node or other Drupal entities. Fields typically have text, image, or terms.
  9. Node – A piece of content in Drupal that has a title, an optional body, and perhaps other fields. Every node belongs to a particular content type (see above), and can be classified using the taxonomy system. Examples of nodes are polls and images.
  10. Views – This refers to a module that allows you to click and configure the interface for running database queries. It can give the results in many formats.
  11. Views Display – A views display is created inside of a view to show the objects fetched by the view in a variety of ways.
  12. Module – A code that extends Drupal features and functionality. Drupal core comes with required modules, some of which are optional. A large number of “contrib,” or non-core, modules are listed in the project directory.
    • Core- has features that are available within Drupal by default
    • Custom- a module that is custom developed for a purpose that may not be available within the core system.  
    • Contributed- A module that is made available to others within the Drupal community after it was created as a custom module. There are more than 40,000 modules available today.

Any other questions? Check out Drupal.org!

Mar 28 2018
Mar 28

In our Drupal 7 site we have an enhanced textfield that autocompletes already stored values. When migrating to Drupal 8 we could normalize this by using a vocabulary and terms instead.

So in our current Drupal 7 setup we have something like:

Content type: Employee

nid name field_role 1 John Web developer 2 Emil Web developer 3 Henrik Web designer 4 Karl Web designer 5 David Sales 6 John Sales

And in Drupal 8 we want this instead:

Vocabulary: Role

tid title 1 Web developer 2 Web designer 3 Sales

Content type: Employee

nid name field_role 1 John 1 2 Emil 1 3 Henrik 2 4 Karl 2 5 David 3 6 John 3

Using entity_generate process plugin

The first approach we can take is to use the entity_generate plugin provided by Migrate Plus module.

In our migration file:

process:
  field_role:

    # Plugin to use
    plugin: entity_generate

    # Field from source configuration
    source: field_role

    # Value to compare in the bundle
    value_key: name

    # Bundle key value
    # If you get errors consider using only bundle
    bundle_key: vid

    # Bundle machine name
    bundle: role

    # Type of entity
    entity_type: taxonomy_term

    # Set to true to ignore case on lookup
    ignore_case: true

Running this migration will take the value of the textfield and try to lookup the term by name on the destination and if it does not exist the term will be created.

In the second part of this article (coming soon) we will take a look at how we can deal with translations.

Feb 01 2018
Feb 01
February 1st, 2018

Paragraphs is a powerful Drupal module that makes gives editors more flexibility in how they design and layout the content of their pages. However, they are special in that they make no sense without a host entity. If we talk about Paragraphs, it goes without saying that they are to be attached to other entities.
In Drupal 8, individual migrations are built around an entity type. That means we implement a single migration for each entity type. Sometimes we draw relationships between the element being imported and an already imported one of a different type, but we never handle the migration of both simultaneously.
Migrating Paragraphs needs to be done in at least two steps: 1) migrating entities of type Paragraph, and 2) migrating entities referencing imported Paragraph entities.

Migration of Paragraph entities

You can migrate Paragraph entities in a way very similar to the way of migrating every other entity type into Drupal 8. However, a very important caveat is making sure to use the right destination plugin, provided by the Entity Reference Revisions module:

destination: plugin: ‘entity_reference_revisions:paragraph’ default_bundle: paragraph_type destination:plugin:entity_reference_revisions:paragraphdefault_bundle:paragraph_type

This is critical because you can be tempted to use something more common like entity:paragraph which would make sense given that Paragraphs are entities. However, you didn’t configure your Paragraph reference field as a conventional Entity Reference one, but as an Entity reference revisions field, so you need to use an appropriate plugin.

An example of the core of a migration of Paragraph entities:

source: plugin: url data_fetcher_plugin: http data_parser_plugin: json urls: 'feed.url/endpoint' ids: id: type: integer item_selector: '/elements' fields: - name: id label: Id selector: /element_id - name: content label: Content selector: /element_content process: field_paragraph_type_content/value: content destination: plugin: 'entity_reference_revisions:paragraph' default_bundle: paragraph_type migration_dependencies: { } plugin:urldata_fetcher_plugin:httpdata_parser_plugin:jsonurls:'feed.url/endpoint'    type:integeritem_selector:'/elements'    name:id    label:Id    selector:/element_id    name:content    label:Content    selector:/element_contentfield_paragraph_type_content/value:contentdestination:plugin:'entity_reference_revisions:paragraph'default_bundle:paragraph_typemigration_dependencies:{  }

To give some context, this assumes the feed being consumed has a root level with an elements array filled with content arrays with properties like element_id and element_content, and we want to convert those content arrays into Paragraphs of type paragraph_type in Drupal, with the field_paragraph_type_content field storing the text that came from the element_content property.

Migration of the host entity type

Having imported the Paragraph entities already, we then need to import the host entities, attaching the appropriate Paragraphs to each one’s field_paragraph_type_content field. Typically this is accomplished by using the migration_lookup process plugin (formerly migration).

Every time an entity is imported, a row is created in the mapping table for that migration, with both the ID the entity has in the external source and the internal one it got after being imported. This way the migration keeps a correlation between both states of the data, for updating and other purposes.

The migration_lookup plugin takes an ID from an external source and tries to find an internal entity whose ID is linked to the external one in the mapping table, returning its ID in that case. After that, the entity reference field will be populated with that ID, effectively establishing a link between the entities in the Drupal side.

In the example below, the migration_lookup returns entity IDs and creates references to other Drupal entities through the field_event_schools field:

field_event_schools: plugin: iterator source: event_school process: target_id: plugin: migration_lookup migration: schools source: school_id field_event_schools:  plugin:iterator  source:event_school  process:    target_id:      plugin:migration_lookup      migration:schools      source:school_id

However, while references to nodes or terms basically consist of the ID of the referenced entity, when using the entity_reference_revisions destination plugin (as we did to import the Paragraph entities), two IDs are stored per entity. One is the entity ID and the other is the entity revision ID. That means the return of the migration_lookup processor is not an integer, but an array of them.

process: field_paragraph_type_content: plugin: iterator source: elements process: temporary_ids: plugin: migration_lookup migration: paragraphs_migration source: element_id target_id: plugin: extract source: '@temporary_ids' index: - 0 target_revision_id: plugin: extract source: '@temporary_ids' index: - 1 field_paragraph_type_content:  plugin:iterator  source:elements  process:    temporary_ids:      plugin:migration_lookup      migration:paragraphs_migration      source:element_id    target_id:      plugin:extract      source:'@temporary_ids'      index:        -0    target_revision_id:      plugin:extract      source:'@temporary_ids'      index:        -1

What we do then is, instead of just returning an array (it wouldn’t work obviously), use the extract process plugin with it to get the integer IDs needed to create an effective reference.

Summary

In summary, it’s important to remember that migrating Paragraphs is a two-step process at minimum. First, you must migrate entities of type Paragraph. Then you must migrate entities referencing those imported Paragraph entities.

More on Drupal 8

Top 5 Reasons to Migrate Your Site to Drupal 8

Creating your Emulsify 2.0 Starter Kit with Drush

Web Chef Joel Travieso
Joel Travieso

Joel focuses on the backend and architecture of web projects seeking to constantly improve by considering the latest developments of the art.

Web Chef Dev Experts
Development

Blog posts about backend engineering, frontend code work, programming tricks and tips, systems architecture, apps, APIs, microservices, and the technical side of Four Kitchens.

Read more Development
Sep 28 2017
Sep 28
September 28th, 2017

If your site was built with Drupal within the last few years, you may be wondering what all the D8 fuss is about. How is Drupal 8 better than Drupal 6 or 7? Is it worth the investment to migrate? What do you need to know to make a decision? In this post we’ll share the top five reasons our customers—people like you—are taking the plunge. If you know you’re ready, tell us.

  1. Drupal 8 has a built-in services-based, API architecture. That means you can build new apps to deliver experiences across lots of devices quickly and your content only needs to live in one place. D8’s architecture means you don’t have to structure your data differently for each solution—we’ve helped clients build apps for mobile, Roku, and Amazon Alexa using this approach (read how we helped NBC). If you’re on Drupal 6 now, a migration to Drupal 8 will allow you to do unleash the power of your content with API integration.
  2. You can skip Drupal 7 and migrate straight to D8. If you’re on Drupal 6, migrating directly to Drupal 8 is not just doable—it’s advisable. It will ensure every core and contributed module, security patch, and improvement is supported and compatible for your site for longer.
  3. The Drupal 8 ecosystem is ready. One of the reasons people love Drupal is for the amazing variety of modules available. Drupal 8 is mature enough now that most of the major Drupal modules you have already work for D8 sites.
  4. Drupal 8 is efficient. Custom development on Drupal 8 is more efficient than previous versions—we’ve already seen this with our D8 clients and others in the Drupal community are saying the same thing. When you add that to the fact that Drupal 8 is the final version to require migration—all future versions will be minor upgrades—you’ve got a solid business reason to move to Drupal 8 now.
  5. It’s a smart business decision. Drupal 6 is no longer supported—and eventually Drupal 7 will reach “end of life”—which means any improvements or bug fixes you’re making to your existing site will need to be re-done when you do make the move. Migrating to Drupal 8 now will ensure that any investments you make to improving or extending your digital presence are investments that last.

If you’re still not sure what you need, or if you would like to discuss a custom review and recommendation, get in touch. At Four Kitchens, we provide a range of services, including user experience research and design, full-stack development, and support services, each with a strategy tailored to your success.

LET’S TALK!

Web Chef Todd Ross Nienkerk
Todd Ross Nienkerk

Todd Ross Nienkerk is the CEO and co-founder of Four Kitchens. He was born in a subterranean cave in the future.

Jul 13 2017
Jul 13
July 13th, 2017

When creating the Global Academy for continuing Medical Education (GAME) site for Frontline, we had to tackle several complex problems in regards to content migrations. The previous site had a lot of legacy content we had to bring over into the new system. By tackling each unique problem, we were able to migrate most of the content into the new Drupal 7 site.

Setting Up the New Site

The system Frontline used before the redesign was called Typo3, along with a suite of individual, internally-hosted ASP sites for conferences. Frontline had several kinds of content that displayed differently throughout the site. The complexity with handling the migration was that a lot of the content was in WYSIWYG fields that contained large amounts of custom HTML.

We decided to go with Drupal 7 for this project so we could more easily use code that was created from the MDEdge.com site.

“How are we going to extract the specific pieces of data and get them inserted into the correct fields in Drupal?”

The GAME website redesign greatly improved the flow of the content and how it was displayed on the frontend, and part of that improvement was displaying specific pieces of content in different sections of the page. The burning question that plagued us when tackling this problem was “How are we going to extract the specific pieces of data and get them inserted into the correct fields in Drupal?”

Before we could get deep into the code, we had to do some planning and setup to make sure we were clear in how to best handle the different types of content. This also included hammering out the content model. Once we got to a spot where we could start migrating content, we decided to use the Migrate module. We grabbed the current site files, images and database and put them into a central location outside of the current site that we could easily access. This would allow us to re-run these migrations even after the site launched (if we needed to)!

Migrating Articles

This content on the new site is connected to MDEdge.com via a Rest API. One complication is that the content on GAME was added manually to Typo3, and wasn’t tagged for use with specific fields. The content type on the new Drupal site had a few fields for the data we were displaying, and a field that stores the article ID from MDedge.com. To get that ID for this migration, we mapped the title for news articles in Typo3 to the tile of the article on MDEdge.com. It wasn’t a perfect solution, but it allowed us to do an initial migration of the data.

Conferences Migration

For GAME’s conferences, since there were not too many on the site, we decided to import the main conference data via a Google spreadsheet. The Google doc was a fairly simple spreadsheet that contained a column we used to identify each row in the migration, plus a column for each field that is in that conference’s content type. This worked out well because most of the content in the redesign was new for this content type. This approach allowed the client to start adding content before the content types or migrations were fully built.

Our spreadsheet handled the top level conference data, but it did not handle the pages attached to each conference. Page content was either stored in the Typo3 data or we needed to extract the HTML from the ASP sites.

Typo3 Categories to Drupal Taxonomies

To make sure we mapped the content in the migrations properly, we created another Google doc mapping file that connected the Typo3 categories to Drupal taxonomies. We set it up to support multiple taxonomy terms that could be mapped to one Typo3 category.
[NB: Here is some code that we used to help with the conversion: https://pastebin.com/aeUV81UX.]

Our mapping system worked out fantastically well. The only problem we encountered was that since we were allowing three taxonomy terms to be mapped to one Typo3 category, the client noticed some use cases where too many taxonomies were assigned to content that had more than one Typo3 category in certain use cases. But this was a content-related issue and required them to re-look at this document and tweak it as necessary.

Slaying the Beast:
Extracting, Importing, and Redirecting

One of the larger problems we tackled was how to get the HTML from the Typo3 system and the ASP conference sites into the new Drupal 7 setup.

The ASP conference sites were handled by grabbing the HTML for each of those pages and extracting the page title, body, and photos. The migration of the conference sites was challenging because we were dealing with different HTML for different sites and trying to get get all those differences matched up in Drupal.

Grabbing the data from the Typo3 sites presented another challenge because we had to figure out where the different data was stored in the database. This was a uniquely interesting process because we had to determine which tables were connected to which other tables in order to figure out the content relationships in the database.

The migration of the conference sites was challenging because we were dealing with different HTML for different sites and trying to get get all those differences matched up in Drupal.

A few things we learned in this process:

  • We found all of the content on the current site was in these tables (which are connected to each other): pages, tt_content, tt_news, tt_news_cat_mm and link_cache.
  • After talking with the client, we were able to grab content based on certain Typo3 categories or the pages hierarchy relationship. This helped fill in some of the gaps where a direct relationship could not be made by looking at the database.
  • It was clear that getting 100% of the legacy content wasn’t going to be realistic, mainly because of the loose content relationships in Typo3. After talking to the client we agreed to not migrate content older than a certain date.
  • It was also clear that—given how much HTML was in the content—some manual cleanup was going to be required.

Once we were able to get to the main HTML for the content, we had to figure out how to extract the specific pieces we needed from that HTML.

Once we had access to the data we needed, it was a matter of getting it into Drupal. The migrate module made a lot of this fairly easy with how much functionality it provided out of the box. We ended up using the prepareRow() method a lot to grab specific pieces of content and assigning them to Drupal fields.

Handling Redirects

We wanted to handle as many of the redirects as we could automatically, so the client wouldn’t have to add thousands of redirects and to ensure existing links would continue to work after the new site launched. To do this we mapped the unique row in the Typo3 database to the unique ID we were storing in the custom migration.

As long as you are handling the unique IDs properly in your use of the Migration API, this is a great way to handle mapping what was migrated to the data in Drupal. You use the unique identifier stored for each migration row and grab the corresponding node ID to get the correct URL that should be loaded. Below are some sample queries we used to get access to the migrated nodes in the system. We used UNION queries because the content that was imported from the legacy system could be in any of these tables.

SELECT destid1 FROM migrate_map_cmeactivitynode WHERE sourceid1 IN(:sourceid) UNION SELECT destid1 FROM migrate_map_cmeactivitycontentnode WHERE sourceid1 IN(:sourceid) UNION SELECT destid1 FROM migrate_map_conferencepagetypo3node WHERE sourceid1 IN(:sourceid) … SELECTdestid1FROMmigrate_map_cmeactivitynodeWHEREsourceid1IN(:sourceid)UNIONSELECTdestid1FROMmigrate_map_cmeactivitycontentnodeWHEREsourceid1IN(:sourceid)UNIONSELECTdestid1FROMmigrate_map_conferencepagetypo3nodeWHEREsourceid1IN(:sourceid)

Wrap Up

Migrating complex websites is rarely simple. One thing we learned on this project is that it is best to jump deep into migrations early in the project lifecycle, so the big roadblocks can be identified as early as possible. It also is best to give the client as much time as possible to work through any content cleanup issues that may be required.

We used a lot of Google spreadsheets to get needed information from the client. This made things much simpler on all fronts and allowed the client to start gathering needed content much sooner in the development process.

In a perfect world, all content would be easily migrated over without any problems, but this usually doesn’t happen. It can be difficult to know when you have taken a migration “far enough” and you are better off jumping onto other things. This is where communication with the full team early is vital to not having migration issues take over a project.

Web Chef Chris Roane
Chris Roane

When not breaking down and solving complex problems as quickly as possible, Chris volunteers for a local theater called Arthouse Cinema & Pub.

Jul 05 2017
Jul 05
July 5th, 2017

We’re happy to announce the new Global Academy for continuing Medical Education (GAME) site! GAME, by Frontline, provides doctors and medical professionals with the latest news and activities to sharpen their skills and keep abreast on the latest medical technologies and techniques.

As a followup to our launch of Frontline Medical communication’s MDEdge portal last October, the new GAME site takes all of the strengths of MDEdge—strong continuing education materials, interactive video reviews, content focused on keeping medical professionals well-trained—and wraps that in a fresh new package. The new GAME site is optimized for performance so that visitors can learn from their phones on-the-go, in the field on their tablets, or at their desktops in the office between meetings. Behind the scenes, site administrators have an interface that streamlines their workflow and allows them to focus on creating content.
[NB: Read our MDEdge launch announcement, here.]

The Project

Four Kitchens worked with the Frontline and GAME teams to…

  • migrate a bevy of static and dynamic content from their existing Typo3 CMS site and ten external ASP-based conference sites.
  • create a method to streamline canonical content sharing between the GAME site and the MDEdge portal through web standard APIs, and a mirror API for automated content creation from the portal to the GAME site.
  • create a single domain home for conferences originally resting on multiple source domains, redirecting as needed while keeping the source domains public for advertising use without requiring extra domain hosting.
  • provide functional test coverage across the platform for high-value functionality using Behat and CircleCI.
  • revise the design and UX of the site to help engage users directly with the content they were seeking.

Engineering and Development Specifics

Check out the new Global Academy for continuing Medical Education (GAME) site today!

  • built on Drupal 7
  • hosted on Pantheon Elite
  • code standards enforced with ESLint and PHP_CodeSniffer
  • site migration via custom migration module plugins and Google Docs mapping
  • custom MDEdge and other 3rd party integrations
  • style guide produced and reviewed using Emulsify

The Team

The Four Kitchens team of Web Chefs included James Todd as technical lead, Chris Roane as lead engineer, Randy Oest as the designer and frontend engineer, and Scott Riley as the project manager. Additional engineering work was completed by Diego Tejera, Justin Riddiough, and Web Chef Patrick Coffey.

Web Chef James Todd
James Todd

James tinkers with hardware, software, and everything in between.

Mar 09 2016
Mar 09

Migrate is one of the most established modules in the Drupal ecosystem. So much so that with Drupal 8, a decision has been made to get some of its functionality ported and added to Drupal core. An important reason was that the traditional upgrade between major releases was replaced with a migration of Drupal 6 or 7 content and configuration to Drupal 8.

Drupal 8 logo

Not all the functionality of the Migrate module has been moved into core though. Rather, we have the basic framework within the core migrate module and the functionality serving the upgrade path from Drupal 6 and 7 within the core migrate_drupal module. What’s left can be found in a host of contrib modules. The most important is Migrate Tools which contains, among other things, drush commands and the UI for running migrations. Additionally, the Migrate Source CSV, Migrate Source XML and Migrate Source JSON modules provide plugins for the most used types of migration sources.

In this article we are going to look at how migration works in Drupal 8 by migrating some content into node entities. For simplicity, the data we play with resides in tables in the same database as our Drupal installation. If you want to see the final code, you can check it out in this repository.

Drupal 8 Migration Theory

The structure of a migration in Drupal 8 is made up of three main parts: the source, the process and the destination, all three being built using the new Drupal 8 plugin system. These three parts form a pipeline. The source plugin represents the raw data we are migrating and is responsible for delivering individual rows of it. This data is passed to one or more process plugins that perform any needed manipulation on each row field. Finally, once the process plugins finish preparing the data, the destination plugins save it into Drupal entities (either content or configuration).

Creating a migration involves using such plugins (by either extending or directly using core classes). All of these are then brought together into a special migration configuration entity. This is shipped as module config that gets imported when the module is first enabled or can be constructed using a template (a case we won’t be exploring today). The migration configuration entity provides references to the main plugins used + additional metadata about the migration.

Movie Migration

Let us now get down to it and write a couple of migrations. Our data model is the following: we have two tables in our database: movies and movies_genres. The former has an ID, name and description while the latter has an ID, name and movie ID that maps to a movie from the first table. For the sake of simplicity, this mapping is one on one to avoid the complication of a third table. Here is the MySQL script that you can use to create these tables and populate with a few test records (if you want to follow along):

CREATE TABLE `movies` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `description` text,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

CREATE TABLE `movies_genres` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `movie_id` int(11) DEFAULT NULL,
  `name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8;

INSERT INTO `movies` (`id`, `name`, `description`)
VALUES
        (1, 'Big Lebowsky', 'My favorite movie, hands down.'),
        (2, 'Pulp fiction', 'Or this is my favorite movie?');

INSERT INTO `movies_genres` (`id`, `movie_id`, `name`)
VALUES
        (1, 1, 'Comedy'),
        (2, 1, 'Noir'),
        (3, 2, 'Crime');
        

What we want to achieve is migrate the movies into basic Drupal Article nodes and the genres into taxonomy terms of the Tags vocabulary (which the Article nodes reference via a field). Naturally, we also want the migration to mirror the association between the movies and the genres.

Genres

Let us first take care of the genre migration because in Drupal, the movies will depend on them (they will reference them).

Inside the config/install folder of our module, we need the following configuration entity file called migrate.migration.genres.yml:

id: genres
label: Genres
migration_group: demo
source:
  plugin: genres
  key: default
destination:
  plugin: entity:taxonomy_term
process:
  vid:
    plugin: default_value
    default_value: tags
  name: name

The first three elements of this configuration are the migration ID, label and group it belongs to. The following three keys are responsible for the three main parts of the migration pipeline mentioned earlier. Let’s talk about the latter three separately.

Source

Under the source key we specify which plugin the migration should use to represent and deliver the source data (in our case the plugin with the ID of genres). The key key is used to specify which database should our source query use (that is where our data is).

So in the Plugin/migrate/source folder of our module, let’s create our SQL based source plugin for our genres:

Genres.php

namespace Drupal\demo\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SqlBase;


class Genres extends SqlBase {

  
  public function query() {
    $query = $this->select('movies_genres', 'g')
      ->fields('g', ['id', 'movie_id', 'name']);
    return $query;
  }

  
  public function fields() {
    $fields = [
      'id' => $this->t('Genre ID'),
      'movie_id' => $this->t('Movie ID'),
      'name' => $this->t('Genre name'),
    ];

    return $fields;
  }

  
  public function getIds() {
    return [
      'id' => [
        'type' => 'integer',
        'alias' => 'g',
      ],
    ];
  }
}

Since we are using a SQL source, we need to have our own source plugin class to provide some information as to how the data needs to be read. It’s not enough to use an existing one from core. The query() method creates the query for the genre data, the fields() method defines each individual row field and the getIds() method specifies the source row field that acts as the unique ID. Nothing complicated happening here.

We are extending from SqlBase which, among other things, looks for the plugin configuration element named key to learn which database it should run the query on. In our case, the default one, as we detailed in the migration configuration entity.

And this is all we need for our simple source plugin.

Destination

For the migration destination, we use the default core taxonomy term destination plugin which handles everything for us. No need to specify anything more.

Process

Under the process key of the migration, we list each destination field we want populated and one or more process plugins that transform the source row fields into data for the destination fields. Since we want the genres to be all terms of the Tags vocabulary, for the vid field we use the default_value plugin which accepts a default_value key that indicates the value each record will have. Since all will be in the same vocabulary, this works well for us.

Lastly, for the term name field we can simply specify the source row field name without an explicit plugin name. This will, under the hood, use the get plugin that simply takes the data from the source and copies it over unaltered to the destination.

For more information on how you can chain multiple process plugins in the pipeline or what other such plugins you have available from core, I recommend you check out the documentation.

Movies

Now that our genres are importable, let’s take a look at the movies migration configuration that resides in the same folder as the previous (config/install):

migrate.migration.movies.yml

id: movies
label: Movies
migration_group: demo
source:
  plugin: movies
  key: default
destination:
  plugin: entity:node
process:
  type:
    plugin: default_value
    default_value: article
  title: name
  body: description
  field_tags:
    plugin: migration
    migration: genres
    source: genres
migration_dependencies:
  required:
    - genres

We notice the same metadata as before, the three main parts of the migration (source, process and destination) but also the explicit dependency which needs to be met before this migration can be successfully run.

Source

Like before, let’s take a look at the movies source plugin, located in the same place as the genres source plugin (Plugin/migrate/source ):

Movies.php:

namespace Drupal\demo\Plugin\migrate\source;

use Drupal\migrate\Plugin\migrate\source\SqlBase;
use Drupal\migrate\Row;


class Movies extends SqlBase {

  
  public function query() {
    $query = $this->select('movies', 'd')
      ->fields('d', ['id', 'name', 'description']);
    return $query;
  }

  
  public function fields() {
    $fields = [
      'id' => $this->t('Movie ID'),
      'name' => $this->t('Movie Name'),
      'description' => $this->t('Movie Description'),
      'genres' => $this->t('Movie Genres'),
    ];

    return $fields;
  }

  
  public function getIds() {
    return [
      'id' => [
        'type' => 'integer',
        'alias' => 'd',
      ],
    ];
  }

  
  public function prepareRow(Row $row) {
    $genres = $this->select('movies_genres', 'g')
      ->fields('g', ['id'])
      ->condition('movie_id', $row->getSourceProperty('id'))
      ->execute()
      ->fetchCol();
    $row->setSourceProperty('genres', $genres);
    return parent::prepareRow($row);
  }
}

We have the same three required methods as before, that do the same thing: query for and define the data. However, here we also use the prepareRow() method in order to alter the row data and available fields. The purpose is to select the ID of the movie genre that matches the current row (movie). That value is populated into a new source field called genres, which we will see in a minute how it’s used to save the Tags taxonomy term reference.

Destination

In this case, we use the node entity destination plugin and we need nothing more.

Process

There are four fields on the Article node we want populated with movie data. First, for the node type we use the same technique as before for the taxonomy vocabulary and set article to be the default value. Second and third, for the title and body fields we map the movie name and description source fields unaltered.

Lastly, for the tags field we use the migration process plugin that allows us to translate the ID of the genre (that we added earlier to the genres source row field) into the ID of its corresponding taxonomy term. This plugin does this for us by checking the migration mapping of the genres and reading these IDs. And this is why the genres migration is also marked as a dependency for the movies import.

Activating and Running the Migration

Now that we have our two migration configuration entities and all the relevant plugins, it’s time to enable our module for the first time and have the configuration imported by Drupal. If your module was already enabled, uninstall it and then enable it again. This will make the config import happen.

Additionally, in order to run the migrations via Drush (which is the recommended way of doing it), install the Migrate Tools module. Then all that’s left to do is to use the commands to migrate or rollback the movies and genres.

To see the available migrations and their status:

drush migrate-status

To import all migrations:

drush migrate-import --all

To roll all migrations back:

drush migrate-rollback --all

Conclusion

And there we have it – a simple migration to illustrate how we can now import, track and roll back migrations in Drupal 8. We’ve seen how the plugin system is used to represent all these different components of functionality, and how the migration definition has been turned into configuration that brings these elements together.

There is much more to learn, though. You can use different source plugins, such as for data in CSV or JSON, complex process plugins (or write your own), or even custom destination plugins for whatever data structure you may have. Good luck!

Daniel Sipos

Meet the author

Daniel Sipos is a Drupal developer who lives in Brussels, Belgium. He works professionally with Drupal but likes to use other PHP frameworks and technologies as well. He runs webomelette.com, a Drupal blog where he writes articles and tutorials about Drupal development, theming and site building.
Oct 09 2015
Oct 09

When migrating a site from Drupal 6 to Drupal 8, we had to write some very basic Plugins. Since plugins and some of their related pieces are new to Drupal 8, here is a walk-through of how we put it together:

Use case

In Drupal 6, the contrib Date module provided a date field that had both a start and end date. So, the beginning of Crazy Dan’s Hot Air Balloon Weekend Extravaganza might be July 12, with an end date of July 14. However, the datetime module in Drupal 8 core does not allow for end dates. So, we had to use two distinct date fields on the new site: one for the start date, and one for the end date.

Fields in D6:
1.  field_date: has start and end date

Fields in D8:
1.  field_date_start: holds the start date
2.  field_date_end: holds the end date

Migration overview

A little background information before we move along: migrations use a series of sequential plugins to move your data: builder, source, process, and finally, destination.

Since we are moving data from one field into two, we had to write a custom migration process plugin. Process plugins are where you can manipulate the data that is being migrated.

Writing the process plugin (general structure)

The file system in Drupal 8 is organized very differently than in Drupal 7. Within your custom module, plugins will always go in [yourmoduledir]/src/Plugin. In this case, our migrate process plugin goes in [yourmodulename]/src/Plugin/migrate/process/CustomDate.php.

Here is the entire file, which we’ll break down below.

<?php
/**
 * @file
 * Contains \Drupal\custom_migrate\Plugin\migrate\process\CustomDate.
 */

Standard code comments.

namespace Drupal\custom_migrate\Plugin\migrate\process;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\Row;

Instead of using functions like include(); or include_once(); to add various PHP files, now we “include” them by referencing their namespaces. Or rather Drupal knows which files to autoload based on the namespace.  This way, if a class is ever moved to a new directory, we won’t have to change code elsewhere, as long as the namespace stays the same. We will allow our code to be used the same way, by defining its namespace.

/**
* This plugin converts Drupal 6 Date fields to Drupal 8.
*
* @MigrateProcessPlugin(
*   id = "custom_date"
* )
*/

This class comment includes an annotation. When the Migrate module is looking for all available migration plugins, it scans the file system, looking for annotations like this. By including it, you let the migration module discover your migrate process plugin with the unique id ‘custom_date’.

Our new class will inherit from the ProcessPluginBase class, which is provided by the Migrate module in core. Let’s step back and look at that class. This is it’s definition:

abstract class ProcessPluginBase extends PluginBase implements MigrateProcessInterface { ... }

Since this is an abstract class, it can never be instantiated by itself. So, you never call new ProcessPluginBase(). Instead we create our own class that inherits it, by using the keyword extends:

class CustomDate extends ProcessPluginBase { ... }

The ProcessPluginBase class has two public methods, which will be available in child classes, unless the child class overrides the methods. In our case, we will override transform(), but leave multiple() alone, inheriting the default implementation of that one.

(A note about abstract classes: If there were any methods defined as abstract, our child class would be required to implement them. But we don’t have to worry about that in this case!) To override transform() and create our own logic, we just copy the method signature from the parent class:

public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property)

Writing the process plugin (our specific data manipulation)

In order to write custom code, let’s review our use case of dates again. Since this is our mapping:

OLD D6 field_date (from component) -> NEW D8 field_date_from
OLD D6 field_date (to component) -> NEW D8 field_date_to

We will migrate field_date twice per node. The first time, we will pull the from date. The second time, we will pull the to date. Since our process plugin needs to be aware of which piece we’re looking for in that particular run, we will allow the process plugin to have additional configuration. In our case, we will call this configuration date_part, which can be either from or to, and defaults to from:

$date_part = isset($this->configuration['date_part']) ? $this->configuration['date_part'] : 'from';

Depending on which date part we’re looking for, we’ll grab the appropriate date from the D6 source field, which is stored in the array $value.

$value = ($date_part == 'from') ? $value['value'] : $value['value2'];

And we’ll return the string, which will populate the new field:

return $value;

That’s it for writing our process plugin! Now we just have to use it.

Using the process plugin

In our migration definition, we need to call this plugin and feed it the correct information. So back in [yourmodulename]/config/install/migrate.migration.d6_node.yml, we map the new and old fields:

field_date_start:
  plugin: custom_date
  source: field_date
  date_part: from
field_date_end:
  plugin: custom_date
  source: field_date
  date_part: to

Which reads like this: For field_date_start on the new site, pass the old field_date to the custom_date process plugin, with the configuration date_part = ‘from’. Do this all again for field_date_end, but with the configuration date_part = ‘to’. Both of our D8 fields get filled out, each getting its data from a single field on the old D6 site.

migration

Time to fly! (Image courtesy of Wikimedia)


Feedback

Hopefully this helps. If you have any corrections, improvements, questions, or links to how you use plugins, leave them in the comments!

Aug 26 2015
Aug 26

We’re working on our first Drupal 8 project here at Advomatic, and Jim and I have been tasked with implementing a content migration from the client’s existing Drupal 6 site.

My first assignment was to write a plugin which rewrites image assist tags in node body fields as regular HTML image tags. Fortunately, lots of smart people had already solved this problem for Drupal 6 to Drupal 7 migrations (I adapted my plugin from Olle Jonsson’s script on Github), so the biggest hurdle was learning how to implement this thing in Drupal 8.

This is the true story of how we made it work.

Note: You’ll need to use Drush 8 for working with Drupal 8. I’d recommend following Karen Stevenson’s great tutorial from the Lullabot blog to help set up multiple versions of Drush on your system.

Initial setup

As of this writing, you’ll need some very specific Git checkouts of Drupal core and the migration helper modules, or you’re going to immediately encounter a pile of fatal errors. These are working for us:

Enable those modules and their dependencies, then set up your own custom module for your plugin code. The very cool Drupal Console module is a quick way to generate the boilerplate files you’ll need.

Write some code

Migration template

Your migration is likely going to need to provide migration templates for various node types, as well as one that handles all nodes. This plugin for handling image assist tags needs to run on all imported nodes, so we start by copying /core/modules/node/migration_templates/d6_node.yml over to our module and adjusting it a little to instruct it to run the ImgAssist plugin (see line 36 here).

Migrate plugin

There are example process plugins in “process” folders around the installation, and looking at those was a great way to figure out how to write ours. Jim made note of these commands to use for finding example code:

find ./ -type d -name 'migration_templates'
find ./ -type d -name 'process'

Our ImgAssist migrate process plugin starts with the Drupal 6 node body and teaser values, and then it runs through a few steps to create their Drupal 8 counterparts:

Running a node migration, step-by-step

  • 1. Get the D6 site running locally.
  • 2. Install Drupal 8 dev at the commit noted above.
  • 3. Install migrate_plus and migrate_upgrade at the commits noted above, and enable your custom module.
  • 4. Add your D6 database connection information to settings.php (you can follow the Drupal 7 directions here).
  • 5. Run these Drush commands:
    • drush8 migrate-upgrade --legacy-db-url=mysql://dbusername:[email protected]/D6databasename --legacy-root=http://d6site.local --configure-only
    • drush8 migrate-status (just to make sure your custom migration template is registering)
    • drush8 migrate-import yourmodule_d6_node

You’ll probably get an error the first time running migrate-import since the node migration depends on a few others to run first, such as d6_user. Run the dependency migrations as needed, then try the custom node import again.

If you have a lot of nodes, the import process will take a few minutes. I actually wrote this entire blog post while waiting for imports to run. Go do something fun for a minute, you’ve earned it.

Eventually, migrate-import will finish running, and you’ll be all set! You can compare the node on your D8 site against the node on the D6 site and see that the tag has been replaced. Hooray!

If it didn’t work: read on. It’s totally fine, you’ve got this.

So what if you have to roll it back?

drush migrate-rollback hasn’t been implemented in D8 just yet (but it is getting close). A workaround is to use drush scr to run a script which deletes your newly-imported nodes. We’ve been using this: https://gist.github.com/sarahg/993b97d6733003814fda

Then, you’ll need to uninstall your custom module, remove all of its config entities from the database, and drop its database tables. You can do that with queries like these:

DELETE from config where name=“migrate.migration.yourmodule_d6_node”;
DROP table migrate_map_yourmodule_d6_node;
DROP table migrate_message_yourmodule_d6_node;

To make this a little easier, you could add these queries to a hook_uninstall function in your module. I’m not one for making things easy (I’m working on this migration before there’s even a Drupal 8 release candidate, after all), so I’ve just been using drush sql-cli.

Now you can adjust your code as needed, re-enable your module and give it another shot (you can just skip ahead to the “drush8 migrate-import yourmodule_d6_node” step at this point).

Further reading

It took a lot of research to figure out how to get migrate working in Drupal 8 this early in the game. These articles were immensely helpful (thanks bloggers and documenters!).

Aug 26 2015
Aug 26

We’re wrapping up our first Drupal 8 project here at Advomatic, and Jim and I have been tasked with implementing a content migration from the client’s existing Drupal 6 site.

My first migration job was to write a plugin which rewrites image assist tags in node body fields as regular HTML image tags. Fortunately, lots of smart people had already solved this problem for Drupal 6 to Drupal 7 migrations (I adapted my plugin from Olle Jonsson’s script on Github), so the biggest hurdle was learning how to implement this thing in Drupal 8.

This is the true story of how we made it work.

Note: You’ll need to use Drush 8 for working with Drupal 8. I’d recommend following Karen Stevenson’s great tutorial from the Lullabot blog to help set up multiple versions of Drush on your system.

Initial setup

When we first ran our migrations, a few months ago, we needed specific checkouts from dev branches of core and migrate modules. However, as of our final migration run yesterday (10/6/15), we were able to use:

Enable those modules and their dependencies, then set up your own custom module for your plugin code. The very cool Drupal Console module is a quick way to generate the boilerplate files you’ll need.

Write some code

Migration template

Your migration is likely going to need to provide migration templates for various node types, as well as one that handles all nodes. This plugin for handling image assist tags needs to run on all imported nodes, so we start by copying /core/modules/node/migration_templates/d6_node.yml over to our module and adjusting it a little to instruct it to run the ImgAssist plugin (see line 36 here).

Migrate plugin

There are example process plugins in “process” folders around the installation, and looking at those was a great way to figure out how to write ours. Jim made note of these commands to use for finding example code:

find ./ -type d -name 'migration_templates'
find ./ -type d -name 'process'

Our ImgAssist migrate process plugin starts with the Drupal 6 node body and teaser values, and then it runs through a few steps to create their Drupal 8 counterparts:

  • 1. Read through the body value and pick out [img_assist] tags.
  • 2. Split those tags into usable pieces.
  • 3. Build the HTML image tag.
  • 4. Replace the original content containing img_assist tags with the rewritten version, using the built-in transform function.

Running a node migration, step-by-step

  • 1. Get the D6 site and your D8 site running locally.
  • 3. Enable migrate_plus, migrate_upgrade and your custom module.
  • 4. Add your D6 database connection information to settings.php (you can follow the Drupal 7 directions here).
  • 5. Run these Drush commands:
    • drush8 migrate-upgrade --legacy-db-url=mysql://dbusername:[email protected]/D6databasename --legacy-root=http://d6site.local --configure-only
    • drush8 migrate-status (just to make sure your custom migration template is registering)
    • drush8 migrate-import yourmodule_d6_node

You’ll get a notice the first time running migrate-import since the node migration depends on a few others to run first, such as d6_user. Run the dependency migrations as needed, then try the custom node import again.

If you have a lot of nodes, the import process will take a few minutes. I actually wrote this entire blog post while waiting for imports to run. Go do something fun for a minute, you’ve earned it.

Eventually, migrate-import will finish running, and you’ll be all set! You can compare the node on your D8 site against the node on the D6 site and see that the tag has been replaced. Hooray!

If it didn’t work: read on. It’s totally fine, you’ve got this.

So what if you have to roll it back?

drush migrate-rollback hasn’t been implemented in D8 just yet (but it is getting close). A workaround is to use drush scr to run a script which deletes your newly-imported nodes. We’ve been using this: https://gist.github.com/sarahg/993b97d6733003814fda

Then, you’ll need to uninstall your custom module, remove all of its config entities from the database, and drop its database tables. You can do that with queries like these:

DELETE from config where name=“migrate.migration.yourmodule_d6_node”;
DROP table migrate_map_yourmodule_d6_node;
DROP table migrate_message_yourmodule_d6_node;

To make this a little easier, you could add these queries to a hook_uninstall function in your module. I’m not one for making things easy (I’m working on this migration before there’s even a Drupal 8 release candidate, after all), so I’ve just been using drush sql-cli.

Now you can adjust your code as needed, re-enable your module and give it another shot (you can just skip ahead to the “drush8 migrate-import yourmodule_d6_node” step at this point).

Further reading

It took a lot of research to figure out how to get migrate working in Drupal 8 this early in the game. These articles were immensely helpful (thanks bloggers and documenters!).

May 18 2013
May 18

Average: 5 (2 votes)

DrupalCamp AustinWe're super-excited to announce that we've been invited to present a half-day workshop during DrupalCamp Austin. The Camp takes place the weekend of June 21-23, 2013 and we'll be presenting "Getting Stuff into Drupal - Basics of Content Migration" from 1:30pm until 5:30pm on Saturday the 22nd. The workshop will cost $75 and we'll be covering the basics of three of the most common ways of importing content into Drupal: the Feeds, Migrate, and the Drupal-to-Drupal data migration (based on Migrate) modules. Interested? Check out all the details and then register today.

Over the past few years, we've performed various types of migrations into Drupal from all sorts of sources: static web sites, spreadsheets, other content management systems, and older versions of Drupal sites. Using this experience, we've developed an example-based workshop that demonstrates some of our go-to tools for bringing content into Drupal.

The workshop will be short on lecturing, and long on real-world examples. We'll import spreadsheet data using Feeds, a Drupal 6 site into Drupal 7 using Drupal-to-Drupal migration, and a custom migration using the Migrate module.

We're always looking for new and exciting workshops to offer - please take a few minutes and take this short survey to help us determine potential topics for future workshops.

Trackback URL for this post:

http://drupaleasy.com/trackback/591

Sep 10 2012
Sep 10

As we are all aware, Drupal is an excellent content management system which can be used to build complex and scalable web sites. A few years ago when I was only managing my own personal Drupal sites, I was enchanted by the variety of modules available to add functionality to my site. When I had a new idea, there was usually a module which could handle the functionality. Since I wasn't picky, I was also happy to just use a community contributed theme. Then, I could just focus on creating content and move on with my life. Installation was easy - all I had to do was click a few buttons and my host would automatically install Drupal on my shared hosting account.

Fast forward a few years, and my desires have become increasingly complex. I'm not satisfied with sites that are just "OK." No, I want my sites to look great and also handle more and more functionality. At the same time, I'm managing more projects - both for clients here at OpenSourcery, and for my own personal use. With only so many hours in the day, I'm always looking for ways to optimize my workflow. Thankfully, I am confident that Drupal 8 is going to deliver many improvements which - at the end of the day - mean better tools for developers and more value for customers. Here is why.

1. Drupal 8 Developers are Smart People.

While working at the Drupal Association, I had the opportunity to meet several people who are responsible for making Drupal 8 a reality. Dries Buytaert is the founder of Drupal and is responsible for managing contributions to Drupal 8 core. He, Nat Catchpole, and Angie Byron work very closely to oversee its development. Having met both Dries and Angie (although not Nat), I'm confident with their ability to lead Drupal 8 into success. Let's not forget that these developers have a long standing track record of excellence in the Drupal community.

2. The Industry is Ready.

While it's easy to see the performance of the core contributors, it's even easier to see the success record of the community as a whole. Millions of websites across the web are dependent on Drupal - and so are the companies who build, maintain, and own these sites. This fact alone really secures the future of Drupal: With so much momentum, many, many people have interest in keeping the ball rolling. We've proven that the open source model works, and everybody is more excited than ever to contribute.

3. Technical Improvements. Drupal 8 is going to make development faster than ever. Although I don't think that Drupal will be "easier" to use and understand, I do think that it will be "better" in the sense that professional developers will be more capable of doing their jobs effectively. Here are some technical changes which should help us out.

A) Improved Deployment Tools One of the most challenging aspects of working as a professional Drupal developer is handling deployments. When we build a web site - or make changes to an existing site - we don't do it live on the web. Sites have to be duplicated, worked on, and tested BEFORE any changes are made to a live site. Then, we have to figure out the best way to move those changes online. With inconsistent hosting environments and inherently unpredictable behavior of modules such as Features, this process becomes tricky. Changes that may appear to be simple can be complicated by the problem of deployment. Drupal 8 will offer a more consistent way to transfer configuration settings - built right into core.

B) Improvements for Front End Developers Making a website look good is not easy. However, it's a very important aspect of web projects today. Sites can't just work, they need to work with style. In fact, many of the changes clients request are changes to the look and feel of their sites, even after it's functioning 100%. But, because each website has its own theme, colors, and layouts, it can be difficult for developers to quickly make changes, because they have to spend time learning how each particular site works. Drupal 8 attempts to not only improve the way Drupal renders pages, but to standardize how layouts are dictated. The goal is to allow developers to make style changes more quickly and with less hassle.

Aug 16 2012
Aug 16

Migrate

Posted on: Thursday, August 16th 2012 by Brandon Tate

Time and time again, I’ve had to import content into a Drupal site. In the past I would rely on Feeds to do the job for me. However, over time, I found Feeds to come up short when it came to complex imports. So, I went looking elsewhere for a better solution and came across Migrate and haven’t looked back since. I’ve developed four import solutions for different sites and have only come across one problem with Migrate which I’ll get to later. First, let's get into the great features of Migrate.

I’ll start by creating a Film migration example. This will be extending an abstract class provided by Migrate named Migration - in most cases, you'll do nearly all your work in extending this class. The Migration class handles the details of performing the migration for you - iterating over source, creating destination objects, and keeping track of the relationships between them.

Sources

First, lets start with defining the source data and where it's coming from. In all cases, I’ve had to import CSV or Tab delimited files, so my import source has been MigrateSourceCSV. This tells migrate that your import source is coming from a CSV or Tab file. When importing tab files though, you need to specify in the options array the delimeter to be “\t”. As well, all files I’ve imported include a header, within the options array you can specify this option to ensure that the header row is not imported. Next, we will want to specify an array describing the file's columns where keys are integers (or may be omitted), values are an array of field name then description. However, if your file contains a header column this may be left empty (I've provided this array below). Migrate can also import from SQL, Oracle, MSSQL, JSON and XML. More Info on sources.

class FilmExampleMigration extends Migration {
  public function __construct() {
    parent::__construct();
    $options = array();
    $options['header_rows'] = 1;
    $options['delimiter'] = "\t";
    $columns[0] = array('Film_ID', 'Film ID');
    $columns[1] = array('Film_Body', 'Film Body');
    $columns[2] = array('Origin', 'Film Origin');
    $columns[3] = array('Producer_ID', 'Film Producer');
    $columns[4] = array('Actor_ID', 'Film Actors');
    $columns[5] = array('Image', 'Film Image');
    $this->source = new MigrateSourceCSV(variable_get('file_private_path', conf_path() . '/files-private') . '/import_files/example.tab', $columns, $options);

Destinations

Next, we will need to define the destination. Migrate can import into Nodes, Taxonomy Terms, Users, Files, Roles, Comments and Tables. As well, there is another module called Migrate Extras which extends the stock ability of Migrate. I personally have implemented Nodes, Terms and Tables so I will discuss those here. Migrate to a node is done by including MigrateDestinationNode with the node type as the parameter. Terms are defined by including MigrateDestinationTerm with the vocabulary name as the parameter. Lastly, Tables is defined by MigrateDestinationTable where the table name is the parameter. Migrating into a table also requires that the table is already created within your database. More Info on destinations.

//term
$this->destination = new MigrateDestinationTerm('category');

//node
$this->destination = new MigrateDestinationNode('film');

//table
$this->destination = new MigrateDestinationTable('film_category_reference_table');

Migrate Map

With Migrate, the MigrateMap class tracks the relationship of source records to their destination records as well as messages generated during the migration. Your source can have a single map key or multi key. Below is an example of both.

//single key
$this->map = new MigrateSQLMap(
     $this->machineName,
       array(
          'Film_ID' => array(
            'type' => 'varchar',
            'length' => 255,
            'not null' => TRUE,
          ),
    MigrateDestinationNode::getKeySchema()
);

//multi key - importing to a table 
$this->map = new MigrateSQLMap(
    $this->machineName,
      array(
        'Film_ID' => array(
           'type' => 'varchar',
           'length' => 255,
           'not null' => TRUE,
         ),
        'Origin_ID' => array(
          'type' => 'varchar',
          'length' => 255,
          'not null' => TRUE,
        )            
    ),
    MigrateDestinationTable::getKeySchema('viff_guest_program_xref')
);

Dependencies

Some imports depend on other imports so Migrate handles this by including the ability to define hard or soft dependencies. Hard dependencies give you the ability to prevent a child import to run before the parent import has run successfully. That means, no errors or anything can occur during that parent import before the child import can run. A soft dependency ensures that the parent import has run before the child but does allow errors to occur within the parent.

//hard dependencies
$this->dependencies = array('Producers', 'Actors');
//soft dependencies
$this->softDependencies = array('Producers', 'Actors');

Field Mappings

Next up, we need to associate source fields to destination fields using what Migrate calls field mappings. You can read up on Migrate’s mappings here. In a simple case, usually text or an integer is provided and it is easily mapped into Drupal using the following line:

$this->addFieldMapping('field_body', 'Film_Body');
$this->addFieldMapping('field_id', 'Film_ID');

Field mappings also allows you to provide default values, such as:

//default image
$this->addFieldMapping('field_image', 'Film_Image')->defaultValue(‘public://default_images/default_film.png’);
//default body text to full html
$this->addFieldMapping('field_body', 'Film_Body')->defaultValue(‘full_html’);

In some cases you need to reference one node to another. This can easily be done using the sourceMigration function. In this example, the ID provided in Program_ID is associated to the Drupal entity that will be created, using the entity ID that was created in the MigrateProgram migration.

$this->addFieldMapping('field_producer', 'Producer_ID')->sourceMigration('MigrateProducer');

Another useful ability is to explode multiple values into a single field. Imagine a Film has multiple actors and the data is defined as “value1::value2::value3”. We would handle this use case using the following:

$this->addFieldMapping('field_actors', Actors')->separator('::')->sourceMigration('MigrateActor');

Taxonomy values are commonly used in migrates but in most cases the client’s CSV file does not contain the term ID needed, instead the term name is provided. In this case, we need to tell the field mapping that the name is being provided instead of the ID:

$this->addFieldMapping('field_genre', 'Genre')->arguments(array('source_type' => 'term'));

Migrate Hooks

In most cases Migrate’s field mappings can handle all situations of getting the data in the system. However, sometimes you need access to the row being imported or the entity type being created. Migrate gives you three functions for this, prepareRow($row), prepare(&$node, $row) and complete(&$node, $row). PrepareRow() allows you to manipulate the row before the rows starts to be imported into the system. You can modify row attributes or even append more row columns as needed. Prepare() allows you to modify the node before it gets saved. Complete() is essentially the same as prepare() but is fired at the end of the row import process.

//prepare example
function prepare(&$node, $row) {
  // concatinate date + time
  $start_time_str  = $row->Event_Date .' '. $row->Event_Time;
  $start_timestamp = strtotime($start_time_str);
  $start_time      = date('Y-m-d\TG:i:s', $start_timestamp);

  $end_time_str  = $row->Event_Date_End .' '. $row->Event_End_Time;
  $end_timestamp = strtotime($end_time_str);
  $end_time      = date('Y-m-d\TG:i:s', $end_timestamp);

  $node->field_event_starttime[LANGUAGE_NONE][0]['value'] = $start_time;
  $node->field_event_endtime[LANGUAGE_NONE][0]['value']   = $end_time;
}

//prepareRow example
public function prepareRow($row){
  //prepend image location onto filenames
  $prefix = 'public://import/images/';
  $row['Image'] = $prefix . $row['Image'];
}

Migrate Drush Commands

Last but not least, I wanted to touch on Drush. Luckily for us command line lovers Migrate has implemented a bunch of useful drush commands that allow us to import, rollback and do just about anything the Migrate UI can do. Here is the list below:

migrate-audit (ma)	View information on problems in a migration.         	
migrate-deregister      Remove all tracking of a migration                   	
migrate-fields-desti     List the fields available for mapping in a destination. 
migrate-fields-sourc   List the fields available for mapping from a source. 	
migrate-import (mi)    Perform one or more migration processes              	
migrate-mappings      View information on all field mappings in a migration.  
migrate-reset-status   Reset a active migration's status to idle            	
migrate-rollback (mr) 	Roll back the destination objects from a given migration
migrate-status (ms)    List all migrations with current status.             	
migrate-stop (mst)	Stop an active migration operation                   	
migrate-wipe (mw) 	Delete all nodes from specified content types.  

I mentioned earlier that there is only one problem I’ve seen with Migrate. This issue is related to CSV and Tab files. Migrate is expecting the files to be continuous. This allows migrate to rollback and re-import at a whim. However, if the import file contains only the new set of data you wish to import, and not the old data that is already imported, you lose the roll back ability because the mappings are lost. As well, the Migrate UI becomes pretty confusing as none of the total rows, imported and un-imported columns make sense since the ID’s don’t relate to data stored within migrates mapping tables. This is the only issue I’ve come across Migrate but still prefer it over other options.

Overall, I’m very impressed with Migrate and its ability to offer such a verbose option of sources, destinations, field mappings and hooks that allow you to ensure the data will make it into your Drupal site no matter what the situation. Feel free to comment and offer suggestions on your experiences with Migrate. For more information on Migrate, check the documentation. Also, I've attached the Film example migration file here.

Apr 03 2012
Apr 03

Posted Apr 3, 2012 // 0 comments

Here's a series of cartoon sketches I did to illustrate some aspects of a successful Drupal migration. These were used at a DrupalCon presentation given recently, "Managing Highly Successful Drupal Migrations" by Frank Febbraro and Mike Morris.

Migration Means a LOT of Moving Parts

Don't let the strays get away from you!

Project Owners and Their High Expectations

Be clear on scope, budget, and schedule.

Project Owners

Sometimes People Will Make Comparisons to their Existing Systems

Plan to do lots of educating about technology.

Comparisons

We Don't use a Fancy Estimating Machine to Determine a Project's Cost

...but we have some formulas to make it easier.

Estimating projects

Does Your Site Seem Outdated?

Now might be the perfect time for a redesign.

Plan for Many Lines of Communication

Don’t underestimate the amount you can communicate in a 15 minute phone call (but don’t forget to take notes and right down decisions).

Lines of Communication

Content Shifts During Migration - Cleanup Needed!

Stuff happens.

Content Cleanup

Don't Skimp on Training Editors

Teach the basics...and more.

Training

Picking a Launch Date

Prepare for delays and what else it might effect (other projects, vacations).

Pick a Launch Date

Prepare for Traffic to Your Site

Load testing advised.

Prepare for Traffic

And It's Finally Launch Day!

Celebrate - but make sure you plan ahead like being ready for search engine indexing bots.

Launch Day

When User Interface Designer Laura Schoppa sits down to talk creative strategy with our clients, she brings to the table more than 12 years of professional experiences designing and developing interfaces for websites and web-based ...

Dec 19 2010
Dec 19
Printer-friendly versionPDF version

If you spend enough time working with Drupal you'll eventually end up needing to migrate data or posts from other platforms into a Drupal site. I have had three separate situations where I needed to accomplish this.

  1. Importing raw data into a site from comma separated value (CSV) files in order to display the data in tables that could be viewed on the site.
  2. Importing a WordPress blog.
  3. Importing several blogs from Squarespace.

I have previously covered the first situation in a post titled Moving Beyond Nodes. Please see that post if you're interesting in finding out more about how I accomplished those data imports. The second situation I addressed by utilizing the WP2Drupal module. The third situation I handled by utilizing the Import Typepad module.

WP2Drupal

As fate would have it the WP2Drupal module is currently listed as abandoned. That doesn't mean it can't/won't work for you if you are still on Drupal 6 and WordPress 2.3. For the record the maintainer of WP2Drupal recommends the WordPress Import module which will have a Drupal 7 release. WP2Drupal works by connecting to the database of the WordPress site so you will need to have access to the database settings (host, username and password) of the site you are migrating from. The module will import blog posts, categories, pages, comments and trackbacks from your WordPress site. 

Some things to consider prior to importing.

  • Mapping of WP users to Drupal users
  • Taxonomy vocabulary that will hold the WP categories and tags
  • Node types for blog entries and pages
  • Input format for posts and comments
  • Whether or not the old URLs must be redirected to new ones

Think about all these things before you start the migration. Users, content types and taxonomies are especially important considerations. If things aren't setup correctly and you go ahead with the import many headaches can result. For that reason it's a best practice to make a database backup prior to actually migrating. You can always restore the original database and redo the import if things go wrong.

Import Typepad

Although this module is meant for Typepad/Movable Type blogs I used it to import "journals" (which are basically individual blogs) from Squarespace. To get started you need an export file for each journal that is generated on your Squarespace account. To generate the export file you need to put the journal into structure editing mode, then select Configure This Page, and scroll down to the Data Export section. There is a button there that says "Export Blog Data". Pushing that button generates the file that you will download and then upload using the Import Typepad module. The data that will be imported will be the posts (title and description), and the categories. The date and time of the posting will be preserved as well.

With your various export files in hand you then need to navigate to /admin/content/import_typepad on your Drupal site. There are three things that you need to have set up prior to importing.

  1. Content types
  2. Taxonomy vocabulary
  3. Users

Unlike the WP module there is no automatic user mapping. You will simply select an existing user to assign import posts to. The actual import itself is a 2 step process. In step 1 you select the export file to upload, select the content type to import to, and also select the taxonomy to import categories to. In the next step you see a preview of what the imported content will look like on your site, along with the categories that will be imported. You also map the content to a Drupal user at that time. When you're ready you click an Import button and then wait for a message indicating that the import is complete.

Post Import Tasks

In both cases you'll want to spend time reviewing the content that you imported to make sure things look like you want them to. You may want to adjust input formats or issues related to how the content is displayed on your site. You will also want to use the Views module to create blocks and pages to display groups of posts that you have imported. I have found Views Bulk Operations to be a very helpful tool to correct and update large groups of nodes.

If you have migration tips, and war stories or direct experience with the modules mentioned here, and you would like to share your experiences feel free to do so in the comment section below.

Video Links

YouTube Version

Flash Version

Quicktime Version

Jul 27 2010
Jul 27
stepping through the node import wizard

I recently undertook a migration from oscommerce to ubercart. The scope of this migration was limited to the transfer of products (and categories) - I didn't try and migrate customers and previous orders.

Here's an overview of the procedure I followed:

  • generate a CSV file of categories in oscommerce
  • import into drupal / ubercart using taxonomy_csv module
  • generate a CSV file of product data from oscommerce
  • import into drupal / ubercart using (patched) node_import module

My life was made relatively easy by the fact that although the categories in oscommerce had a hierarchical structure, it was very simple. There were only a handful of top-level categories, and the tree was only one deep. Here's what the schema for categories looks like in oscommerce:

mysql> SHOW TABLES LIKE 'categor%';
+-----------------------------------+
| Tables_in_mysite_osc1 (categor%) |
+-----------------------------------+
| categories                        |
| categories_description            |
+-----------------------------------+
mysql> DESCRIBE categories;
+------------------+-------------+------+-----+---------+----------------+
| FIELD            | TYPE        | NULL | KEY | DEFAULT | Extra          |
+------------------+-------------+------+-----+---------+----------------+
| categories_id    | INT(11)     | NO   | PRI | NULL    | AUTO_INCREMENT |
| categories_image | VARCHAR(64) | YES  |     | NULL    |                |
| parent_id        | INT(11)     | NO   | MUL | 0       |                |
| sort_order       | INT(3)      | YES  |     | NULL    |                |
| date_added       | datetime    | YES  |     | NULL    |                |
| last_modified    | datetime    | YES  |     | NULL    |                |
+------------------+-------------+------+-----+---------+----------------+
mysql> DESCRIBE categories_description;
+-----------------+-------------+------+-----+---------+-------+
| FIELD           | TYPE        | NULL | KEY | DEFAULT | Extra |
+-----------------+-------------+------+-----+---------+-------+
| categories_id   | INT(11)     | NO   | PRI | 0       |       |
| language_id     | INT(11)     | NO   | PRI | 1       |       |
| categories_name | VARCHAR(32) | NO   | MUL |         |       |
+-----------------+-------------+------+-----+---------+-------+

I had 115 categories, 5 of which were top-level, and the remaining 110 were all children of one of those 5 parents. This meant it was simple to generate the CSV for the import of the category hierarchy into drupal manually; I simply went through my top-level categories and got a list of all their children. All I wanted for the import was the category names. (n.b. I could also ignore language as this site's monolingual.) I did some simple queries to get the names of categories where the parent was one of my top-level categories (which all had a parent_id of 0), e.g.

SELECT categories_name FROM categories c 
                    INNER JOIN categories_description cd ON c.categories_id = cd.categories_id 
                    WHERE c.parent_id = 31;

...and prepared a CSV file, a snippet of which is below (where Empire and Europe and Colonies are top-level) I then imported my categories using taxonomy_csv, set to mode Hierarchical tree structure or one term by line structure. node_import can also import taxonomy terms, but unless I'm mistaken it doesn't support hierarchical taxonomies.

"Empire",
,"Aden"
,"Antigua"
,"Ascension"
,"Australia"
,"B.O.I.C."
,"Bahamas"
...snip...
"Europe and Colonies",
,"Austria"
,"Baltic"
,"Benelux"
,"Eastern Europe"
,"France"
...etc...

Next was my products. I also wanted to use a CSV file to import these into ubercart, so I had to generate a CSV file from the oscommerce database. I wrote a quick php cli script which queries the database, (optionally) grabs product images using CURL from the webserver oscommerce is running on, and outputs a nice CSV file and a folder full of product images (which need to be put in the right place on the drupal/ubercart server). Here's the script:

#!/usr/bin/php
<?php
 
define('CSV_OUTPUT',             '/tmp/osc_products.csv'              );
define('DB_HOST',                'localhost'                          );
define('DB_NAME',                'mysite_osc1'                        );
define('DB_USER',                'root'                               );
define('DB_PASS',                'top-secret'                         );
define('LIMIT',                  1000                                 );
define('GRAB_IMAGES',            false                                );
define('IMAGE_REMOTE_PATH',      'http://shop.example.com/images/'    );
define('IMAGE_LOCAL_DIR',        '/tmp/osc_images/'                   );
 
$query = <<<EOQ
SELECT * FROM 
  products p 
    INNER JOIN products_description pd ON p.products_id = pd.products_id 
    INNER JOIN products_to_categories ptc ON p.products_id = ptc.products_id 
    INNER JOIN categories_description cd ON ptc.categories_id = cd.categories_id 
  WHERE p.products_status = 1
EOQ;
$query .= ' LIMIT ' . LIMIT;
 
$db = db_connect(DB_HOST, DB_NAME, DB_USER, DB_PASS);
$products = db_query($query, $db);
$counter = 0;
 
/* example of results:
(
    [[products_id]] => 5656
    [[products_quantity]] => 1
    [[products_model]] => 
    [[products_image]] => AdenStH.jpg
    [[products_price]] => 10.0000
    [[products_date_added]] => 2009-03-16 12:34:00
    [[products_last_modified]] => 
    [[products_date_available]] => 
    [[products_weight]] => 0.00
    [[products_status]] => 1
    [[products_tax_class_id]] => 0
    [[manufacturers_id]] => 0
    [[products_ordered]] => 0
    [[language_id]] => 1
    [[products_name]] => N05656 - Ascension : Multifranked to St. Helena 1939
    [[products_description]] => St. Helena receiver dated 1941! Curiosity!!
    [[products_url]] => 
    [[products_viewed]] => 159
    [[categories_id]] => 409
    [[categories_name]] => Ascension
)
*/
 
// prepare files
$handle = fopen(CSV_OUTPUT, 'w');
 
$columns = array('sku', 'name', 'date', 'description', 'image', 'price', 'category');
fputcsv($handle, $columns);
 
 
if (GRAB_IMAGES) {
  if (!is_dir(IMAGE_LOCAL_DIR)) { 
    mkdir(IMAGE_LOCAL_DIR);
  }
}
 
while (($product = db_object($products)) && ($counter < LIMIT)) {
  //print_r($product);
  $counter ++;
 
  $sku = substr($product->products_name, 0, 6);
  $product_name = substr($product->products_name, 9);
  $image_name = clean_filename($product->products_image);
 
  $data_to_write = array(
                          $sku,
                          $product_name,
                          $product->products_date_added,
                          $product->products_description,
                          $image_name,
                          $product->products_price,
                          $product->categories_name
                        );
  $data_to_write = array_map('trim', $data_to_write);
 
  if (GRAB_IMAGES) {
    if (grab_image($product->products_image, $image_name)) {
      echo 'grabbed ' . $product->products_image . " ($image_name)\n";
    }
    else {
      echo 'failed to grab ' . $product->products_image . "\n";
    }
  }
 
  fputcsv($handle, $data_to_write);
}
 
fclose($handle);
echo "# iterated over $counter products\n### END\n";
exit;
 
/** helper functions **/
 
function db_error($message) {
  echo "db_error: $message\n" . mysql_error() . "\n";
}
 
function db_connect($db_host, $db_name, $db_user, $db_pass) {
  $db = mysql_connect($db_host, $db_user, $db_pass) or db_error('Unable to connect to database');
  mysql_select_db($db_name, $db);
  return $db;
}
 
function db_query($query, $db) {
  $result = mysql_query($query, $db) or db_error($sql);
  return $result;
}
 
function db_object($result) {
  return mysql_fetch_object($result);
}
 
function grab_image($image_name, $new_name) {
  $url = IMAGE_REMOTE_PATH . $image_name;
   // use curl to grab the image from the server
  $ch = curl_init($url);
 
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
  curl_setopt($ch, CURLOPT_HEADER, FALSE);
 
  $data = curl_exec($ch);
  curl_close($ch);
  if (strlen($data) > 0) {
    $retval = file_put_contents(IMAGE_LOCAL_DIR . $new_name, $data);
  }
  else {
    $retval = false;
  }
  return $retval;
}
 
function clean_filename($old_name) {
  $bad_stuff =  array('.JPG', ' ', '&');
  $good_stuff = array('.jpg', '',  '-');
  $new_name = str_replace($bad_stuff, $good_stuff, $old_name);
  return $new_name;
}

You'll see my products had an SKU we'd put into the first part of the title in oscommerce - this should be easy to remove if it's not applicable. The image grabbing requires php-curl - you could just grab a whole image directory off the server running oscommerce, but I wanted to be careful to only migrate images actually being used. There are obviously many other ways of doing this.

I had to apply a couple of patches to node_import (rc4) to get it working for me:

...I also set escape char to \ (when using fputcsv in my script). After all that, I was able to follow the node_import wizard, mapping fields in the CSV to fields in my ubercart product content type, after which the imported successfully digested my CSV, and all my products appeared with their images in my new ubercart site (I obviously had to put the images my script had grabbed into the right place on the ubercart webserver as well).

Jul 17 2009
Jul 17

At The Economist, we've been struggling some to keep our database changes 100% in code. We're required to automate everything using hook_update and hook_install. We don't want to be pushing database imports around, and we want our build scripts to create a current database for testing purposes. Also, we want to see how the database is changing over time. How we do it? The answer might not surprise you--the Rake of Drupal is Drush.

The relative difficultly of this highlights an aspect of Drupal that's lacking in Rails: anyone can build a site without knowing PHP, and many devs (me included, on smaller projects) are perfectly happy clicking around the admin area, building views, creating blocks, and learning modules rather than writing code. Because there's no true site management area in Rails, if you want to make changes like these, you can either manipulate the database in SQL (and lose versioning) or write migrate scripts in Ruby. As a result, they have a much more robust and proven toolkit for these tasks--the drawback is that non-developers are shut out of application development.

Bit of background--Rake is a build tool for Ruby, which is used by Rails to execute schema changes by way of the command rake migrate. For each change, you write an "up" and "down" method. The up method progresses the system one version forward, the down one version back, so you have to write code that installs a feature and can also uninstall it. Our approach is a bit simpler due to API limitations for things like this; we just write upgrade functions.

The use of the install and update functions for contrib modules and your own functions will almost certainly differ. Contrib modules need an install function that will configure the module entirely on a clean system plus update functions that will move things around when you install a new version. Frankly, I'd like it if Drupal executed every update function on module installation and skipped hook_install altogether; the premise being that your lowest numbered update function (6001 for Drupal 6 modules) should install the lowest version of your schema. The only situation in which that would be inopportune is a module with hundreds of complex update functions that constantly back out of prior changes, and it'd only be really poor if it took so long that an unwitting site admin would cancel the module's installation using the browser's stop button because it took so long. Probably unlikely for most modules. We use our update functions like this; they're run progressively on installation. It saves us considerable development time versus building a purely clean install function.

The one potential pitfall to running update functions in this way is if the environment changes, a function might go missing or accept different parameters, which would cause our installation to break--this would be especially obnoxious on our Hudson Continuous Integration environment. So in this way we might find ourselves spending time refactoring old update functions, but this hasn't happened yet. This fact makes our approach much less tenable for contrib modules, which might depend throughout its update hooks on a certain environment configuration (e.g. one version of a function should exist in hook_update_6001 and another in hook_update_6010, but obviously only the most recent version actually exists). And contrib modules generally can't refactor their previous update functions either.

These problems aren't so severe in Rails because the migration system is generally simpler and on many applications would just involve creating database tables, for which there is seriously beautiful syntax--typical of Ruby. Managing your site's changes over time is critical to any framework--these tools aren't shipped with Drupal, like rake migrate is with Rails, but with a little bit of work, you can get very close!

Key Concepts in Automating Upgrades in Drupal

  • A current and widely discussed limitation of Drupal is its lack of API functions. There's sometimes no way around directly manipulating the database, but the Install Profile API is a great start.
  • Build your views and CCK types in the admin UI, then export them to code and install the exports using hook_update and the Install Profile API.
  • Download, install, and get to know Drush really well. Knowing and using Drush will make you efficient anyways.
  • Write build scripts in bash, make, or ant to install a ready-to-go version of your site with no database import required. This will make automating your unit tests much easier.
  • One bootstrap module with update functions that install other modules is effective. We use a Google spreadsheet to claim update numbers for our bootstrap module and have placed the URL to this document at the bottom of install file. You can and should install contrib modules this way too, but drupal_install_modules() will not fail if one or more dependencies listed in the install file are unmet, so be careful!
  • Get in the habit of writing your updates in code first, then running drush updatedb from the command line rather than making the changes in the admin area and then putting them in code to run later or on a CI server. The exceptions I make are for Views and CCK, which need to be exported to array structures.

Code snippets

Install function which iterates over hook_update functions

function ec_channel_install() {
  // Execute our defined updates.
  $version = 6001;
  while (function_exists('ec_channel_update_' . $version)) {
    call_user_func('ec_channel_update_' . $version++);
  }
  // hook_install has no return value
}

Create a content type

function ec_channel_update_6001() {
 
  install_include(array('node'));
 
  // Create the Channel Page content type.
  $props = array(
    'description' => t('Channel page, contains news packages, stories and blocks.'),
    'has_title' => TRUE,
    'title_label' => t('Channel Name'),
    'has_body' => FALSE,
  );
  install_create_content_type('channel', 'Channel Page', $props);
  return array();
}

Install Some Modules

function ec_channel_update_6003() {
  drupal_install_modules(array('content_multigroup', 'fieldgroup', 'link'));
  return array();
}

Import CCK fields

function ec_channel_update_6002() {
  $ret = array();
 
  // Economist Story node reference field.
  install_include(array('content', 'content_copy'));
  install_content_copy_import_from_file(drupal_get_path('module', 'ec_channel') .'/'.'freeform_story_link_econ_story.type', 'freeform_story_link');
  return $ret;
}

Related

Jul 28 2008
Jul 28

I'm just finishing something of a large migration from an old Xoops site to Drupal. I've not had to get too down and dirty with the Xoops database, for which I'm quite happy as it wasn't so coherent after lots of upgrades and customisations. I've made a small migration handbook for myself which could be handy for folks.

Migrating the data in on it's own would have been pretty simple from just following the basic node import note. What was done here was much more, and interesting because of it.

The sites data was spread all over the database tables, and often not recorded in it's own field but just within the content itself. So the migration was built up with lots of scripts to pull out and enter into CCK fields all the links, youtube videos, pictures and files.

All of a sudden a wealth of resources that people have added to the site and lost in a mountain of data suddenly become easily to search, filter and display in different forms. Well worth the extra time it took to run all the content through.

Jan 19 2007
Jan 19

As I grow more and more liking on Drupal, I decided to truly integrate my WordPress-powered blog with my main site. This is quite challenging, since database schemas differ between different content management systems.

With the wp2drupal module, this can be easily achieved, and the only thing that matters would be how much would be migrated. My case was a pretty exceptional one, since I maintain the WordPress blog on a different subdomain with customized permalinks.

I chose to migrate the blog posts into story nodes to further integrate and blur the line between a real site and a blog. In the end, I ended up manually migrating those permalinks (which was obnoxious), retagging every post, and getting rid of useless posts.

The earlier you take the jump to switch to Drupal, the easier it'll be, and in the end I learned this the hard way by migrating from WordPress 2.0 to Drupal 5.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web