Feb 28 2014
Feb 28

In part 1 of this series, I introduced the workflow we use for managing i18n-friendly configuration using Features. One of the cornerstones of the solution is the Features Translations module (hereafter called FT for brevity), which stores string translations in the feature.

One former limitation of FT is that it exported all translated strings in a given language/text group set, even if those strings had never been changed manually after importing them from the Drupal Translations site. For the Default text group, the resulting feature file was huge. With a deployment workflow that involves drush fr-all -y each time, the deployment time became unbearably slow.

We fortunately found some time to fix this problem, by optimizing the translation sets to only changed strings. The general idea of the solution is extremely simple: we just want to export those string translations that are different from their original translation, or that don't exist in core or contrib. Our reference is the .po files served by Drupal Translations.

You can try it right now:

  • Update FT to 7.x-1.0-beta3 or later.
  • Enable the Localization update module
  • Set the local po directory path at admin/config/regional/language/update
  • Update your contrib module string translation using drush l10n-update and ensure the .po files are correctly saved where you pointed
  • Update your feature that contains translations using drush fu my-translation-feature -y

That's it! You should notice a significant decrease in the exported string translations. Many thanks to fellow Meedani Mohammed El-Sawy for the neat implementation.

Jun 02 2013
Jun 02

Last week, I described a technique to query and display nodes in all available translations. This worked well enough, but a performance-minded reader pointed out that the query generated by Views (that includes N self-joins for N enabled languages) would not scale to a large number of nodes.

My usual approach when implementing new ideas is to ensure the logic works first, and only handle optimization when needed. It's a strategy that has worked well for me in the past. So I set out to test this hypothesis, and to optimize the query if it was needed. Here's what happened:

The first obstacle was to generate a large set of nodes and their translations. Devel Generate, the Devel sub-module that generates Drupal objects for development purposes, does not support content translation at the time of writing. I submitted a D7 patch to the 2 years old feature request to achieve this. I tested it with 10K nodes, and it seems to work well. Your review is appreciated!

Having generated 10K nodes and their translations to Arabic and French (30K nodes in total), I cloned the Proverbs view from last time to query and display this content. The result was quite explicit: the view page never finished loading! Clearly, the Views-generated query was not scaling. And for good reason: 3 SQL JOINS of 30,000 records each is a performance black hole. Optimization was needed.

My goal for optimizing the query was to retain all the advantages that Views offers in terms of theming query results, integrating into Drupal pages, etc. - these are indispensable features when creating real-world applications. In short, I wanted to transparently override the Views-generated query. To do so, I needed to:

  • Remove the peformance-killing JOINs from the query
  • Perform an optimized query to find node translations
  • Re-insert the results from the optimized query into the Views results, to allow it to proceed with the display

The code I used follows. I will explain the important parts below.

Remove the peformance-killing JOINs from the query

The function demo_i18n_views_query_alter() removes from the Views query object all references to the SQL JOINs, which are called "relationships" in Views parlance. Views core invokes this hook just before converting the query object into an SQL statement. The resulting query that Views will execute looks like this:

SELECT node.nid AS nid, node.created AS node_created
FROM {node} node
WHERE (( (node.status = '1') AND (node.type IN  ('multilingual_node')) AND (node.tnid = node.nid OR node.tnid = 0) ))

Perform an optimized query to find node translations

The query as modified above will only return nodes that are translation sources. It's now up to me to query the node translations, by waiting for Views to execute the modified query, and then gathering the nids to find their translations (as stored in {node}.tnid). This is a simple query using the SQL IN operator. I call this hand-made query in the demo_i18n_views_post_execute() function, which is invoked by Views after it executes its own query.

Re-insert the results from the optimized query into the Views results

The challenge with the new query is that it returns one node translation per row, as opposed to the original query which returned all translations on the same row. In addition, the results need to be copied into the view::result object, with the right key names that Views expects. In order to find the right key names, I first displayed the results from the unmodified Views query and noted the result keys. With this information, I then proceeded to loop over the optimized query results, and find the corresponding entry in the Views result array that would receive them. This loop is also implemented in the demo_i18n_views_post_execute() function.

The results were impressive! The view page loaded in very acceptable time (ApacheBench reports a mean time of ~1350ms, against ~650ms in the case of a view with just 4 nodes), and Views happily themed the translated nodes as if it had queried them itself. You can see this code in action on my i18n demo site.

The approach of hand-crafting Views queries has been on my mind for a long time, and I'm glad I took the first step. So far, I am not sure that a generic module can be created out of this, mainly due to the necessity to transform the result set after the optimized query is run. In any case, I'll be applying this technique in my projects!

May 25 2013
May 25

Here's a little puzzle: display a table of nodes, each row containing the same content in all available translations.

Then, a couple of days ago, someone asked me if I had solved it. I hadn't thought of that puzzle since then, but I would have felt bad answering no. So, with 3 years of i18n work under my belt, I decided to give it another go. I did find a solution this time, but it's not optimal, and it required coding. You can find a demo of the solution online. Demo of solution

The basic idea is to select the nodes in their source language, then relate each node to all its translations. To do this, the view is built by filtering on Content translation: Source translation, then adding one Content translation: Translations relationship per language. Nodes and translations

Now this view works pretty well, except for nodes that are not translated: although they are picked up by the SQL statement, the related nodes in each language are empty, since the tnid is not set for untranslated nodes. That's where I had to write a new join handler that not only joins the source language node to its translation, but also joins it to itself in case there are no translations. The following code silently replaces the standard join handler for Content translation: Translations with this new one:

The resulting query will look like the following - note the JOIN clauses:

SELECT node_node_1.title AS node_node_1_title, 
            node_node_1.nid AS node_node_1_nid, 
            node_node_1.language AS node_node_1_language, 
            node_node_2.title AS node_node_2_title, 
            node_node_2.nid AS node_node_2_nid, 
            node_node_2.language AS node_node_2_language, 
            node_node.title AS node_node_title, 
            node_node.nid AS node_node_nid, 
            node_node.language AS node_node_language, 
            node.created AS node_created
FROM {node} node
LEFT JOIN {node} node_node 
           ON (node.nid = node_node.tnid OR (node_node.tnid = 0 AND node.nid = node_node.nid)) AND node_node.language = 'ar'
LEFT JOIN {node} node_node_1 
           ON (node.nid = node_node_1.tnid OR (node_node_1.tnid = 0 AND node.nid = node_node_1.nid)) AND node_node_1.language = 'en'
LEFT JOIN {node} node_node_2 
           ON (node.nid = node_node_2.tnid OR (node_node_2.tnid = 0 AND node.nid = node_node_2.nid)) AND node_node_2.language = 'fr'
WHERE (( (node.status = '1') AND (node.type IN  ('proverb')) AND (node.tnid = node.nid OR node.tnid = 0) ))

The careful reader will have noticed that there's one extra database JOIN in my solution: the one that joins the source language node to itself. If you have a suggestion to remove it, please let me know!

AttachmentSize 104.25 KB 22.44 KB
Mar 19 2013
Mar 19

This week, I'll describe a particularly challenging component I had to deal with: inoffensive-sounding menu items. Should be easy, right? Well, it wasn't.

I won't get into the basics of creating multilingual menu items here. There is good documentation on how to get that set up using the i18n submodule i18n_menu. To provide a context for this post, I'll just mention that we are manually creating menu items, through the admin UI. We have two types of menu items:

  • Ones that should appear in both languages, localized, and pointing to the same place. Those are created with the language set to "Language neutral".
  • Ones that should only appear in a specific language. We set that language explicitly on the menu item's form.

Internally, the i18n_menu modifies the core menu_links and menu_custom tables by adding, among others, a language attribute to save the above info. In addition, the module creates a new text group called menu to save menu translations. In this group, each translation is identified by a menu name and mlid to refer back to the original menu item.

The first problem occurred even before we introduced Features into the mix. The decision to use the mlid as part of the translation identifier means that, across site stages and instances, menu items should have the same database primary key in order to be correctly translated. This design decision introduced a lot of instability in our configurations, for example preventing us from creating new menu items on the fly, for testing or customization purposes. In essence, we would have needed to stick to an install profile approach to manage the site configuration - which is a desirable goal in itself, but one we didn't pursue. In the mean time, our menu item string translations were only reliably showing on the machine where translation occurred, but not elsewhere. This clearly had to be fixed.

We found the module Entity menu links to solve one half of this problem. This module creates a uuid for each menu item, ensuring that uuid remains unchanged throughout the lifetime of the menu item. This module also modifies the core menu_links table by adding to it a uuid attribute.

Still, the string translations were being saved with the mlid. How to convince i18n_menu to use the uuid instead? That's where we had to write some code. The i18n architecture uses the concept of i18n objects that correspond to Drupal site components. We used the following code to override the i18n object info for menu items to use the uuid as a key:

/**
 * Implements hook_i18n_object_info_alter().
 */
function checkdesk_core_i18n_object_info_alter(&$info) {
  // Use UUID field to identify menu_link.
  $info['menu_link']['key'] = 'uuid';
  $info['menu_link']['load callback'] = 'checkdesk_core_i18n_menu_link_uuid_load';
}

/**
 * Callback to load menu_link by UUID.
 */
function checkdesk_core_i18n_menu_link_uuid_load($uuid) {
 if (!empty($uuid)) {
    $query = db_select('menu_links', 'ml');
    $query->leftJoin('menu_router', 'm', 'm.path = ml.router_path');
    $query->fields('ml');
    // Weight should be taken from {menu_links}, not {menu_router}.
    $query->addField('ml', 'weight', 'link_weight');
    $query->fields('m');
    $query->condition('ml.uuid', $uuid);
    if ($item = $query->execute()->fetchAssoc()) {
      $item['weight'] = $item['link_weight'];
      _menu_link_translate($item);
      return $item;
    }
  }
  return FALSE;
}

Unfortunately, the strings were still showing up with the mlid on the translation interface. After many a WTF incantation, we found that i18n_menu was not honouring the i18n object key attribute while creating the string identifiers. We submitted a patch to fix this.

At this point, we had successfully modified the indexing mechanism of i18n_menu to use uuids, as shown below. Our testing with manual .po files revealed that regardless of mlid differences, menu item translations were being successfully moved from one instance to another.

i18n_menu with UUID identifiers

Now menu item translations were made reliable across instances, but they still weren't being saved in a feature. The core Features module does support menu items, but it does not export the additional attributes we introduced above, namely language and uuid. We also need to export customized because i18n will not translate menu items that are not marked as customized.

We submitted a simple patch to Features that allows a hook_query_TAG_alter to extend the relevant query and return extra fields to be exported. Our implementation of this hook looks like this:

/**
 * Implements hook_query_TAG_alter() for `features_menu_link`.
 */
function checkdesk_core_query_features_menu_link_alter($query) {
  // Add missing attributes for translation.
  $query->fields('menu_links', array('uuid', 'language', 'customized'));
}

After this change, the exported menu links look like this (note the last 3 attributes on each entry):

/**
 * @file
 * checkdesk_core_feature.features.menu_links.inc
 */

/**
 * Implements hook_menu_default_menu_links().
 */
function checkdesk_core_feature_menu_default_menu_links() {
  $menu_links = array();

  // Exported menu link: main-menu:node/add/discussion
  $menu_links['main-menu:node/add/discussion'] = array(
    'menu_name' => 'main-menu',
    'link_path' => 'node/add/discussion',
    'router_path' => 'node/add/discussion',
    'link_title' => 'Create story',
    'options' => array(
      'attributes' => array(
        'title' => '',
      ),
      'alter' => TRUE,
    ),
    'module' => 'menu',
    'hidden' => '0',
    'external' => '0',
    'has_children' => '0',
    'expanded' => '0',
    'weight' => '-47',
    'uuid' => 'edc54df9-4aa8-bf84-dd89-ca0a351af23b',
    'language' => 'und',
    'customized' => '1',
  );
  // Exported menu link: main-menu:node/add/media
  $menu_links['main-menu:node/add/media'] = array(
    'menu_name' => 'main-menu',
    'link_path' => 'node/add/media',
    'router_path' => 'node/add/media',
    'link_title' => 'Submit report',
    'options' => array(
      'attributes' => array(
        'title' => '',
      ),
      'alter' => TRUE,
    ),
    'module' => 'menu',
    'hidden' => '0',
    'external' => '0',
    'has_children' => '0',
    'expanded' => '0',
    'weight' => '-49',
    'uuid' => '0bc3af5d-28a8-c864-bd93-f17d8bea2366',
    'language' => 'und',
    'customized' => '1',
  );
  ...
}

With these patches, we were able to reliably persist multilingual menu links using Features. The menu item translations are saved in the translations component of the feature, as described in part 1 of this series. They look like this:

/**
 * @file
 * checkdesk_core_feature.features.translations.inc
 */

/**
 * Implements hook_translations_defaults().
 */
function checkdesk_core_feature_translations_defaults() {
  $translations = array();
  $translations['ar:menu']['a6b48d33d248c146aa8193cb6f618651'] = array(
    'source' => 'Create story',
    'context' => 'item:edc54df9-4aa8-bf84-dd89-ca0a351af23b:title',
    'location' => 'menu:item:edc54df9-4aa8-bf84-dd89-ca0a351af23b:title',
    'translation' => 'أنشئ خبر',
    'plid' => '0',
    'plural' => '0',
  );
  $translations['ar:menu']['8578b45ff528c4333ef4034b3ca1fe07'] = array(
    'source' => 'Submit report',
    'context' => 'item:0bc3af5d-28a8-c864-bd93-f17d8bea2366:title',
    'location' => 'menu:item:0bc3af5d-28a8-c864-bd93-f17d8bea2366:title',
    'translation' => 'أضف تقرير',
    'plid' => '0',
    'plural' => '0',
  );
  ...
}

Now I need your help to review and support (and possibly enhance) the patches submitted to i18n and Features. Please visit them here:

As you know, the more people show interest in a patch, the more likely it will go in quickly. Your help is appreciated!

Next time, I'll describe other components of the multilingual puzzle: taxonomy terms, static pages, etc.

AttachmentSize 81.31 KB
Mar 05 2013
Mar 05

In my role as development team leader, I am responsible for the application architecture that allows other team members to focus on building functionality with minimum friction and rework. As such, one of my biggest tasks is to ensure that new features and configurations can be reliably deployed to the various stages: development, testing and production.

My current project is an Arabic/English application built on Drupal 7, that is deployed in multisite fashion to several partners. I use Features as a base configuration management system, and a number of extension modules to help me manage specific site components. The need to manage the configuration of multilingual components makes the task more complex, and in this series of posts I hope to describe a full recipe that's allowing our distributed team to commit code without overwriting existing settings.

Here are some of the main architectural components on the site:

  • Menus and menu links
  • Taxonomies
  • Static pages
  • Content types
  • Views
  • Rules
  • Heartbeat messages

All these components need to be shown in multiple languages, with the help of the Internationalization module and friends.

Although our application can be delivered in multiple languages, we do all our development on the English UI. When we switched the default language to Arabic, we found that all our menus, taxonomies, and field translations were no longer showing. And for good reason: Drupal does not store the source language of strings in its database, so i18n has to guess the source language - and by default, it considers the site's default language to be the source. Fortunately, you can explicitly specify the source language via the variable i18n_string_source_language. We decided to hard-code the value of this variable in settings.php like so:

// settings.php

// Hardcode i18n_string_source_language to prevent nasty surprises.
$conf['i18n_string_source_language'] = 'en';

// Load per-site configurations.
require 'settings.local.php';

The last line is just a directive that allows us to version-control settings.php for global configurations and loads settings.local.php for instance-specific settings such as database connection.

Because we're deploying across 3 stages, with several instances at each stage, we cannot afford to manually import .po files each time we create or modify a translation. In order to keep the UI translation workflow sane, we designated a specific stage as the recipient of all translation work, allowing the configuration manager (yours truly) to solve the problem of automating the deployment of these translations to other stages and instances. To this end, I wrote a Features plugin called Features Translations that persists the selected translations within a feature, as shown in the screenshot below:

Translations in the Features UI

A file called feature_name.features.translations.inc gets exported to the feature, looking like this:

/**
 * @file
 * checkdesk_core_feature.features.translations.inc
 */

/**
 * Implements hook_translations_defaults().
 */
function checkdesk_core_feature_translations_defaults() {
  $translations = array();
  $translations['ar:default'][] = array(
    'source' => 'The optional description of the taxonomy vocabulary.',
    'context' => '',
    'location' => '',
    'translation' => 'الوصف الاختياري لمعجم الوسوم.',
    'plid' => '0',
    'plural' => '0',
  );
  $translations['ar:default'][] = array(
    'source' => 'You are not authorized to access this page.',
    'context' => '',
    'location' => '',
    'translation' => 'غير مسموح لك بالوصول إلى هذه الصفحة.',
    'plid' => '0',
    'plural' => '0',
  );
  $translations['ar:default'][] = array(
    'source' => 'Function',
    'context' => '',
    'location' => '',
    'translation' => 'الوظيفة',
    'plid' => '0',
    'plural' => '0',
  );
  [...]
  return $translations;
}

In the next part, I'll discuss our handling of multilingual menu items, which took a considerable amount of effort and patches to Features and i18n!

AttachmentSize 69.53 KB

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web