Mar 26 2018
Mar 26
Michael Meyers

I’m excited to announce that Michael Meyers has joined the Tag1 team as Managing Director. Michael was one of our very first clients 10 years ago, we’ve worked together on many projects over the years, and we look forward to working even more closely with him now that he’s a part of the Tag1 team. Michael has extensive experience building market leading high-growth technology companies and is particularly well known in the Drupal Community for his role in driving innovation of the Drupal platform. Michael brings over 20 years of experience managing global technology teams building high traffic, high performance mobile and web applications. Tag1 recently celebrated our 10th anniversary, in that time we’ve established ourselves as the leading provider of highly available, scalable, secure, high performance systems and as the organization other agencies and the users of Drupal turn to for help with their most challenging problems. We will be working with Michael to expand on our success to date and to help lead Tag1 into the future.

Roots in Success

Michael joins Tag1 from Acquia, where he spent the last 5 years on the leadership team as VP of Developer Relations, Developer Marketing, and helped launch the Developer Products group. He also ran Acquia’s Large Scale Drupal (LSD) program, a strategic alliance that engaged Acquia’s largest clients and partners in open source collaborations to enhance Drupal. Tag1 participated in many aspects of Large Scale Drupal including speaking at events, and mentoring teams at LSD hackathons.

Prior to joining Acquia, Michael co-founded & was CTO of NowPublic.com, the first venture-backed Drupal based startup. With the help of Tag1, he grew NowPublic from an idea into a top 500 website. The open source performance optimized Pressflow distribution of Drupal that Tag1 co-maintains was based in large part on the work we did together scaling NowPublic. Later, as CTO of The Clarity Digital Group which acquired NowPublic, Michael and the Tag1 team worked closely together to rebuild Examiner.com — creating the first Drupal-based top 100 Internet site. The Examiner team (including a number of Tag1 employees) was the leading contributor to Drupal 7, and helped meaningfully shape the technology that today powers millions of websites.

Continuing to Build Upward

Michael now has a key role on our management team, helping us usher in the next chapter of growth for our company. I will continue in my role as CEO, ensuring Tag1 stays true to its roots and the strategy that has driven our success to date, with Michael leading our new business development & marketing efforts, focusing on growing our team and service offerings, and building on the success of our first SaaS product, Tag1 Quo.

The depth and breadth of Tag1’s Drupal expertise and contributions to Drupal is largely unparalleled, certainly among the best in the world. In the coming year, we plan to double down on our commitment to Drupal and the Drupal Community. Everyone at Tag1 is really excited about Drupal 8, and decoupled Drupal 8 in particular, which has been a huge success for our clients like Symantec where Drupal 8 has been leveraged to manage content between many disparate systems. For those of you coming to DrupalCon Nashville, we hope you will join Michael, BKJ Digital, and Symantec at our talk “Symantec - From Early Drupal Adoption to the Latest Drupal Innovations”. Michael will highlight how we enable a large Fortune 500 enterprise to drive success with agile practices at scale, foster collaboration across internal groups and external partners with true CI/CD, simplify infrastructure management by treating configuration as code, and enforce security and architecture through gatekeeping and code reviews.

Evolution of Tag1 Products and Services

Over the years Tag1 has greatly expanded its capabilities and offerings — in the face of the most difficult challenges, we’ve never let a client down, and as a result we are often asked by the organizations we work with to take on more and more full service development (from requirements gathering and UX/Design, to development & theming, QA, through to ongoing support and maintenance). Leveraging our broad and deep Drupal expertise, and development process best practices, we provide mentorship and gatekeeping for client teams, and even other services companies. Last year we introduced a SaaS subscription based product, Tag1 Quo, that enables organizations to monitor their Drupal sites (across all versions of the platform) for security issues and updates ensuring they patch systems as quickly as possible and avoid problems before they happen.

We’ve expanded beyond Drupal as well, and are increasingly working with Node.js, Django, Angular, and WordPress for projects where they are the best fit. We have also increased our involvement in stack independent projects like automating development, performance testing, and QA processes, managing infrastructure as code through advanced devops systems, and taking on a broader array of projects including big data, business intelligence & analytics systems.

The Next Episode

Together with Michael, we will lead Tag1 into its second decade, by increasing our commitment to Drupal while also continuing to expand beyond Drupal, working on a greater variety of projects, growing our team, and bringing our reputation of driving success to more and more clients. We’d love to work with you in 2018. Whether you need help with your existing Drupal systems, are exploring upgrade paths, or require help with projects outside of Drupal, please reach out to Michael to discuss how we can help.

On behalf of Tag1, please join me in welcoming Michael to the team!

Nov 29 2017
Nov 29

TL;DR

Introducing a new, easy to use, Drupal 8 module for background images: https://www.drupal.org/project/background_image

If your site is designed around utilizing background images, then this module is for you! Whether you need a surgical implementation that only administrators/developers can implement or provide the ability to allow users to attach their own background images to entities, this module has you covered.

If you're just looking on how to setup this module, you can skip ahead to the Installation section.

The Problem

Over the past month, I have attempted to solve a lot of problems in regards to displaying a user provided image (via Drupal's UI) on a rendered page in the form of background-image: url(...);) in Drupal 8.

In Drupal 7, there were several modules that attempted to do just that. However, much of time, it was simply easier to implement an entirely custom solution. While this did take some effort to do it just right (using custom image fields and then preprocessing via a sub-theme), it was relatively easy enough to accomplish if you knew what needed to be done.

Now, you may be thinking "Great another module that does the same as another module, but only slightly different." You would be kind of right, but it's actually quite a bit different than any previous implementation.

In Drupal 7, many of the "solutions" were primarily focused on using custom field types or field formatters. This required the same type of field or formatter settings to be applied to every single entity/bundle you needed this functionality on. It was still time consuming to implement in a consistent fashion across all the moving parts (which is why a custom implementation was often easier).

A solution in Drupal 8 can now be implemented in an entirely different way. One that eases the setup process and also brings an easier to use interface for your users.

Tag1's backend and frontend Engineers are among the top contributors worldwide to Drupal 8’s code-base.

Responsibility

I think one of the biggest challenges with this entire topic has always been "Whose responsibility is it to handle the UI and then output of said background image?" In the past, there has never really been a clear delineation of whether this responsibility specifically fell to a module or to a theme. Sure, in Drupal 7, some of the modules used fields, but often times you still had to customize the output (usually in a theme) given your specific needs.

In Drupal 8, themes now have very limited "power" in what hooks and plugins they are actually allowed to participate in. This inherently and dramatically forces the responsibility toward needing a dedicated module to handle all of the intricacies required for properly displaying a background image provided by a user.

Entities

Drupal 8 brought with it many great and wonderful things, the least of which is being able to create new entities (out-of-the-box) in a very easy to implement way. No need to depend on a contrib module or figure out complex database schemas. Core just handles the bulk of what you need if you just configure it right.

This module implements a truly dedicated entity. No more blanket custom "Background Image" node bundle entities. No new fields. No field formatters. No more wasting precious time.

This major difference is the primary reason behind creating an entirely new module and not attempting to "port" an existing one. Never mind the added inherit benefits of being able to do things like entity specific permissions.

Cacheability

As mentioned above, often times it was simply easier to implement custom solutions in Drupal 7. This usually meant going in the theme and preprocessing what you needed. However, gone are the days that simple "theme hacks" in the preprocess layer can accomplish what they used to.

Many of those "easier solutions" in Drupal 7 are simply and naturally hindered by the much needed and massive improvement to the cacheability layer in Drupal 8. Thus, themes can no longer inject a background image (on a page level) by simply retrieving the "image field" value for the current page's entity (node) object. Render cache will just "stick" to whatever image value was requested first.

This can be a big "gotcha" for Drupal 8 beginners. They aren't aware of the powerful caching layer that's used by default in Drupal. The exact reasons behind why this happens is extremely complex, but suffice it to say: there is no context for the specific entity's image at a page rendering level.

Sure, one could simply add the specific entity's cache context to the page, but then you would be rendering/caching every single entity (variation). That is not a good solution and actually defeats the entire purpose of (render) caching. This is where implementing a dedicated entity in Drupal 8 is extremely powerful: entities are also a separate cacheable object.

Optimization

Another beautiful feature of Drupal 8 are the two newly added modules: Breakpoint and Responsive Image. I wanted to take advantage of both of these (not required, but highly recommended) to optimize a background image even further; especially when dealing with 2x multipliers for retina displays.

However, doing so in regards to background-image: url(...); would mean that there would be a need to generate custom CSS for each and every image to account for the responsive media queries. CSS generation in Drupal has typically been string concatenations, but that's a lot of strings to ensure everything is "just right".

Cut to... Twig! Drupal 8 also already comes with Twig and it felt like a natural approach when needing to generate the multitude of rules without also adding a preprocessor requirement (which has always been a highly debated topic).

Note: if you enable the Advanced CSS/JS Aggregation module, this generated CSS will also be minified when the site's CSS is being preprocessed.

By using a .css.twig template, generating each media query rule was actually quite a delight to implement:

{# Media Queries #}
{% for media in media_queries %}
@media {{ media.query }} {
  .{{ background_image_class }}::after {
    background-image: url("{{ media.url }}");
  }
}
{% endfor %}

Obviously, generating this CSS each time would be a huge resource drain, especially when an image or its settings change. First, I attempted to use core's \Drupal\Core\Asset\AssetDumper. Storing it in this fashion seemed to be OK, but I soon realized that to get the actual CSS filename meant I had to generate the file right away. This caused issues with having to wait for the initial page response (think BigPipe).

Instead, I took a page from core's own image styles playbook and decided to generate this CSS only upon an actual CSS resource request from the browser. If it didn't actually exist at the specified location it would then, and only then, generate it on the fly.

Currently, this generated CSS can be regenerated upon a sitewide cache rebuild. There are plans to optimize this even further by providing a way in the administrative UI to clear only these generated files.

Preload Image/Color

While media queries certainly help reduce the unnecessary loading of a massive image on a small device, it doesn't help with network latency and what seems to be FOUC. This problem can be even more drastic when the page is using a dark image/style and you have a white background, waiting for the image to load.

Sites like Medium use various techniques to optimize the display of images. One of their solutions is easily doable in Drupal considering it comes standard with image styles: generate an extremely tiny version of the image, preload it and then use it as the initial background image. Since image styles' foundation utilizes \Drupal\Core\ImageToolkit\ImageToolkitInterface, we can also extract the average "color" of the image by resizing it all the way down to a 1x1 image.

This means that what an end user visually sees, depending on network latency, is done so in a very progressive manner:

  • Average background color that resembles the image
  • Preload image (100 x 67) low quality/blurred version of the image
  • Fallback (mobile) image, normal quality
  • Hi-resolution image

Inheritability

Depending on a site's design, there is often a need to "inherit" an image or its settings based on various conditions. For some sites (like this one), a "global" background image is needed. For others, individual entities need there own images/settings (see the various blog posts here). There are several "types" of background images supported by this module, with even more planned:

  • Global
  • Entity Bundle
  • Entity
  • View
  • Routes or Paths (or both, planned)

The background image types above are listed in order of how they are inherited. Each background image provides the ability to inherit or override an image or individual settings entirely. This allows for true customization without the "all or nothing" approach in previous Drupal 7 implementations.

Extensibility

Sometimes your specific "needs" aren't met by what you get "out-of-the-box" by a module. In the case of a background image, you may need to programmatically add additional information to the overlay text.

An example of this can be found at the top of this page where the "Author's information" is added to the background image overlay text (so it also displays when the image is using the full viewport setting). This can easily be accomplished using one of the many alter hooks this module provides:

<?php
/**
 * Implements hook_background_image_text_build_alter().
 *
 * {@inheritdoc}
 */
function MODULE_background_image_text_build_alter(array &$element, array &$context) {
  /** @var \Drupal\node\NodeInterface $entity */
  $entity = $context['entity'];

  /** @var \Drupal\Core\Datetime\DateFormatterInterface $date_formatter */
  $date_formatter = \Drupal::service('date.formatter');

  /** @var \Drupal\Core\Render\RendererInterface $renderer */
  $renderer = \Drupal::service('renderer');
  
  // Immediately return if this is not a blog post.
  $entity_type = $entity ? $entity->getEntityTypeId() : FALSE;
  $bundle = $entity ? $entity->bundle() : FALSE;
  if (!$entity || $entity_type !== 'node' || $bundle !== 'blog') {
    return;
  }

  // Retrieve the blog author and the time the blog post was created.
  $user = $entity->getOwner();
  $created = $entity->getCreatedTime();

  // Retrieve the user profile picture by using the "compact" view mode.
  $picture = \Drupal::entityTypeManager()->getViewBuilder('user')->view($user, 'compact');

  // Bubble the picture's cache metadata to the element.
  $meta = BubbleableMetadata::createFromRenderArray($element)->merge(BubbleableMetadata::createFromRenderArray($picture));
  $meta->applyTo($element);

  // Append the custom user "profile".
  // Note: it isn't recommended to use custom markup like this, however this is being processed
  // via a text formatter and not as a MarkupInterface object, so raw HTML is required.
  $element['#text'] .= '<div class="user-profile">';
  $element['#text'] .= $renderer->render($picture);
  $element['#text'] .= '<div>';
  $element['#text'] .= '<h3 class="name">' . $user->get('field_full_name')->value . '</h3>';
  $element['#text'] .= '<h4 class="title">' . $user->get('field_job_title')->value . '</h4>';
  $element['#text'] .= '<time class="date" datetime="' . $date_formatter->format($created, 'custom', 'c') . '">' . $date_formatter->format($created, 'custom', 'F d, Y') . '</time>';
  $element['#text'] .= '</div></div>';
}

Installation

You can download and install this module from https://www.drupal.org/project/background_image. Once installed, there are a few steps you will have to do before it will actually work.

Recommended Modules:

The following are a list of additional modules that you may find useful when working with this module:

  • Advanced CSS/JS Aggregation - Useful for minimizing the generated CSS for each background image.
  • ImageMagick - Useful in cases where PHP memory limits bug out due to large image processing.
  • Image Style Quality - Useful for further optimizing the preload background image style quality (recommend configuring somewhere around 30%)
  • Image Optimize - Useful for further optimizing all background images (progressive loading)
  • Inline Entity Form - Currently a soft dependency if you wish to embed the background image form on other entities.
  • Token - Provides a way to view which tokens are available to use when adding overlay text for a background image.

Note: the Background Image module merely provides the "muscle" behind handling background images for a site. While this module does generate CSS, it is merely for the display/positioning logic of the background images themselves. It does not actually stylistically theme the background image or other elements around it or within (like the overlay text).

Note: all subsequent administrative screenshots will be using the Seven admin theme (included with core) and all front-end facing screenshots will be a custom sub-theme of the Drupal Bootstrap base theme.

Step 1: Initial Configuration

  1. Navigate to /admin/config/media/background_image/settings on your site and choose which entity bundles you wish to enable support for background images on:
    Background Image: Administrative Settings
    • Enable - Indicates whether this specific entity bundle supports background images at all.
    • Embed - Embeds the background image add/edit form inside the specific entity bundle's add/edit form (currently requires the Inline Entity Form module, with plans to remove this soft dependency in the future). The dropdown next to it indicates which "Group" the form is added to. The "Default" group will place it at the bottom of the entire form. The "Advanced" group will place it in the "advanced" group that is present on some entities and some themes may move this to a sidebar (like in Seven).
    • Require - Explicitly requires an image of some kind.
  2. Click Save configuration.

Step 2: New Region

The Background Image module automatically and dynamically creates a dedicated background_image region in all active themes. This means you don't have to add it to your theme's .info.yml file.

Where you actually render this region is entirely up to you, however it should be noted that this region should be a top level region, positioned outside of all other regions or containers. It should also be positioned somewhere between your header (navigation) and your main content. This is so that when the "full viewport" setting is used, it can push the content (but not the navigation) down out of view.

Once you determine where you want to place this region, alter your theme's page.html.twig file to include the following code so it can actually render this region:

{# Background Image #}
{% block background_image %}
  {% if page.background_image %}
    {{ page.background_image }}
  {% endif %}
{% endblock %}

The following code is from a custom Drupal Bootstrap sub-theme and may give insight into where this region should be placed:

{#
/**
 * @file
 * page.html.twig
 * ...
 */
#}
{% set container = theme.settings.fluid_container ? 'container-fluid' : 'container' %}
{# Navbar #}
{% if page.navigation or page.navigation_collapsible %}
  {% block navbar %}
    {%
      set navbar_classes = [
        'navbar',
        theme.settings.navbar_inverse ? 'navbar-inverse' : 'navbar-default',
        theme.settings.navbar_position ? 'navbar-' ~ theme.settings.navbar_position|clean_class : container,
      ]
    %}
    <header{{ navbar_attributes.addClass(navbar_classes) }} id="navbar" role="banner">
      {% if not navbar_attributes.hasClass(container) %}
        <div class="{{ container }}">
      {% endif %}
      <div class="navbar-header">
        {{ page.navigation }}
        {# .btn-navbar is used as the toggle for collapsed navbar content #}
        {% if page.navigation_collapsible %}
          <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-collapse">
            <span class="sr-only">{{ 'Toggle navigation'|t }}</span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
          </button>
        {% endif %}
      </div>

      {# Navigation (collapsible) #}
      {% if page.navigation_collapsible %}
        <div id="navbar-collapse" class="navbar-collapse collapse">
          {{ page.navigation_collapsible }}
        </div>
      {% endif %}
      {% if not navbar_attributes.hasClass(container) %}
        </div>
      {% endif %}
    </header>
  {% endblock %}
{% endif %}

{# Background Image #}
{% block background_image -%}
  {%- if page.background_image -%}
    {{- page.background_image -}}
  {%- endif -%}
{%- endblock %}

{# Main #}
{% block main %}
  <div role="main" class="main-container {{ container }} js-quickedit-main-content">
    <div class="row">

      {# Header #}
      {% if page.header %}
        {% block header %}
          <div class="col-sm-12" role="heading">
            {{ page.header }}
          </div>
        {% endblock %}
      {% endif %}

      {# Sidebar First #}
      {% if page.sidebar_first %}
        {% block sidebar_first %}
          <aside class="col-sm-3" role="complementary">
            {{ page.sidebar_first }}
          </aside>
        {% endblock %}
      {% endif %}

      {# Content #}
      {%
        set content_classes = [
          page.sidebar_first and page.sidebar_second ? 'col-sm-6',
          page.sidebar_first and page.sidebar_second is empty ? 'col-sm-9',
          page.sidebar_second and page.sidebar_first is empty ? 'col-sm-9',
          page.sidebar_first is empty and page.sidebar_second is empty ? 'col-sm-12'
        ]
      %}
      <section{{ content_attributes.addClass(content_classes) }}>

        {# Highlighted #}
        {% if page.highlighted %}
          {% block highlighted %}
            <div class="highlighted">{{ page.highlighted }}</div>
          {% endblock %}
        {% endif %}

        {# Help #}
        {% if page.help %}
          {% block help %}
            {{ page.help }}
          {% endblock %}
        {% endif %}

        {# Content #}
        {% block content %}
          <a id="main-content"></a>
          {{ page.content }}
        {% endblock %}
      </section>

      {# Sidebar Second #}
      {% if page.sidebar_second %}
        {% block sidebar_second %}
          <aside class="col-sm-3" role="complementary">
            {{ page.sidebar_second }}
          </aside>
        {% endblock %}
      {% endif %}
    </div>
  </div>
{% endblock %}

{% if page.footer %}
  {% block footer %}
    <footer class="footer {{ container }}" role="contentinfo">
      {{ page.footer }}
    </footer>
  {% endblock %}
{% endif %}

For customizability, the module does not assume that a background image will always be rendered in the provided background_image region or in which order these elements are rendered in.

Instead, it provides multiple background image based blocks that allow you to control what is displayed and where. The purpose of doing this is two-fold:

  1. It allows each element of the background image to be cached independently using the various cache contexts that this module provides. This is particularly useful if you "inherit", say, the background image, but only the overlay text has changed. It doesn't invalidate the entire background image and force a new CSS file with this image to be generated.
  2. Future proofing. Over time, new features will be added, like providing credit/attribution for an image (which is planned), and providing individual blocks for each of these rendering elements will allow features like this to be added in a backwards compatible way without "damaging" existing sites.
Once you have added the region to your page.html.twig template, you will need to clear the cache. You can use drush cr or navigate to /admin/config/development/performance and click Clear all caches:
Background Image: Clear cache

Step 3: Permissions

Navigate to /admin/people/permissions and configure the following permissions:
Background Image: Permissions

Step 4: Add Blocks

  1. Navigate to /admin/structure/block on your site and click the Place Block for the Background Image region
    Background Image: Place Block
  2. Add both the Background Image and Background Image - Text blocks:
    Background Image: Blocks

    Warning: do not set any of the "Visibility" settings unless you know what you are doing. The Background Image module automatically determines a block's #access based on whether there is a background image for the current request or not.


  3. Verify that the blocks were added to the background_image region (should look similar to the following):
    Background Image: Blocks Added

    Note: due to how core's config installation of blocks work, this module cannot provide any "default block placement"; you must place these blocks yourself for the module to work properly.

Step 5: Add Global Background Image

  1. Navigate to /admin/config/media/background_image on your site and click Add background image:
    Background Image: Add
  2. Enter a label (e.g. "Site wide") for the "Global" background image type and then click Continue:
    Background Image: Add Global
  3. Select an a background image (I'm going to use https://unsplash.com/photos/JgOeRuGD_Y4 for this example):
    Background Image: Add Image
  4. By default, the Background Image module determines if the image is "Dark" based on the preload background color which is calculated from the average color of a 1x1 resize of the image. You can choose to override these default settings here:
    Background Image: Dark & Preload Background Color settings
  5. Full Viewport setting:
    Background Image: Full Viewport setting
  6. Blur settings:
    Background Image: Blur settings

    The radius and speed blur settings are pretty self explanatory, but a couple of the blur "type" options need a little extra to help differentiate between the two:

    • Scrolling - The current scrolltop determines the radius of the blur (without going over the maximum blur radius). This means that when the scrolltop is 0, the blur is 0.
    • Scrolling (Full Viewport) - If the "Full Viewport" setting is enabled, then this will behave like the above. If it's not, then it will behave as if the "Persistent" option was selected.
  7. Overlay text:
    Background Image: Overlay text setting

    This is formatted text and allows various tokens to be used. You can configure this however you wish, but the following screenshot is how this site's global background image is configured to give an example of just how easy and power this is to use with tokens:

    <h2>[node:field_lead]</h2>
    <h1>[node:title][user:field_full_name][view:title]</h1>
    

    The first line shows a custom "lead" field that is on each node bundle on our site. This simply allows us to show larger verbiage that doesn't affect the node title. The second line is actually the page title for various entity types (so we can display outside the main content container). By default, if a token doesn't get replaced with a real value, it's removed.

    This allows us to use multiple tokens here depending on which entity type is being viewed. Thus, subsequent "child" background images do not actually need to override this value since it automatically inherits this value from the global background image and displays the proper title.

    Note: to allow users (or yourself) to view which tokens are available (link that displays a dialog like in the screenshot), you will need to install the Token contrib module.

  8. Once you've configured the global background image, go ahead and click Save.
  9. You will be redirected back to the background image listing page (/admin/config/media/background_image) and see the "Global: Site wide" background image you just created:
    Background Image: Administrative Listing

Step 6: Test

Navigate to your home page, you should see something like the following:

Without the "Full Viewport" setting enabled (blurred):

Background Image: Front without the Full Viewport setting enabled (blurred)

With the "Full Viewport" setting enabled:

Background Image: Front with the Full Viewport setting enabled

Reminder: this module does not generate stylistic CSS. Your sub-theme is responsible for styling different colors depending on whether an image is "dark" or not. The following screenshots are to prove that this is the "intended behavior" (without sub-theme styling). Please do not open an issue saying that it is "broken"; it's not.

Mark Carver, Drupal Bootstrap maintainer and Senior Frontend Performance Engineer, builds maintainable and responsive Drupal front-ends for Tag1's clients.

Future Roadmap

I really hope that if you've made it this far, you're as excited about this module as I am! I know it doesn't cover every use case and there are likely to be a few things that still need some fine tuning once it has survived its real world debut. Just know that this is just the beginning.

I've already mentioned a few planned features that will make their way into the module, but here are even more ideas that have been floating around in my head while I've been working on it:

  • Decouple the soft dependency on the Inline Entity Form module for embedding the Background Image form on other entities.
  • More UI based configuration - There's currently a lot of hardcoded config values, but it's all developed in a way that will allow for each config value to be easily configurable from the UI (or manually in a config file). Settings like determining the fallback and preload image styles, which responsive image configuration to use, retina media queries rules, the base CSS class to use, etc.
  • Routes and/or Path based background image types
  • The ability to credit/attribute the owner of the background image.
  • New image effects/filters (like the scrolling blur effect you see here).
  • Media API integration (still waiting for the initiative to land in core, in a stable manner).
  • And more... submit a feature request!
Apr 13 2017
Apr 13

Apache JMeter and I have a long and complicated relationship. It is definitely a trusted and valuable tool, but I am also quite confident that certain parts of it will make an appearance in my particular circle of hell. Due to this somewhat uncomfortable partnership, I am always interested in new tools for applying load to an infrastructure and monitoring the results. Locust.io is not exactly a new tool, but I have only recently begun to use it for testing.

What Is Locust?

Locust is a load-testing framework which allows you to write your load plan in regular Python. This is a welcome experience if you have ever had to manually edit a JMeter JMX file. Not only is it a more pleasant experience, but writing executors in Python makes it easy to create a very flexible load plan.

     Idea For A Circle Of Hell: Given a slightly corrupted JMX file that must be loadable and cannot easily be replaced, attempt to look through it to find the error preventing loading. Every time you save the file, some other tag corrupts slightly. Who needs eternal damnation, give me a large JMX file and some failing drives…

The other advantage of Locust is that it has a quite nice flask-based UI (that you can extend fairly easily) and it is quite easy to distribute load generation among multiple locust instances or servers.

Simple Load Plan

In the grand tradition of blog entries like this, let's build a completely impractical, simplistic example.

from locust import HttpLocust, TaskSet, task

class test_user(TaskSet):
    @task
    def front_page(self):
        self.client.get("/")
        
    @task
    def about_page(self):
        self.client.get("/about/")

class TestUserRunner(HttpLocust):
    task_set = test_user

The above code imports the required pieces to build a Locust test plan, these being the TaskSet, HttpLocust instance, and the task decorator. The class you create by inheriting the TaskSet class represents a type of user for the most part (Anonymous, Authenticated, Editorial, etc). In reality it is just a set of individual tasks, supporting elements and supporting methods. However, that matches the reality of a user rather well, so in general I define separate task sets for different user types.

The majority of this code is fairly self-explanatory. You can make requests via the client.get call and individual tasks are marked with the ‘@task’ decorator. These tasks will be what the main testing routine executes and you can weight each task differently, if you choose to. For example, in the above code we might want to weight the front_page higher than the about_page, since the front page will likely see more traffic. You do this by passing the task decorator a weight (where a higher weight equals increased likelihood of running), like so:

@task(10)
 
def front_page(self):

    self.client.get("/")

Running Our Simple Load Plan

Executing our load plan is not difficult. We save the code to plan.py (or any other name that is NOT locust.py) and run:

locust -f plan.py --host=<our test target>

We then open a browser and go to localhost:8089. You will be prompted for the number of users to spawn and how many users to spawn per second. Once you fill this out you can start the test. You will see something like the following:

Locust dashboard

This screen will allow you to monitor your load test, download csv files containing results, stop the load test, and see failures. A quick note on ramp-up: You may notice that you get results up until the point where all of your requested users are launched, then the results are cleared. This is so your final results only include numbers from when all users were launched, but it can take you by surprise if you are not expecting it.

Fetching Static Assets

While the above example can test Drupal's ability to build and deliver a page, it doesn't do much to actually test the site or infrastructure. This is partly because we aren't fetching any static assets. This is where things get a bit interesting. In JMeter, you would check a box at this point. Locust on the other hand trusts you to handle this yourself. It has a very high opinion of you.

Fortunately, it isn’t that hard. There are a few different tools you can use to parse a returned page and pull out static resources. I am going to use BeautifulSoup because I find the name amusing.

     NOTE: It is decisions like this that make me think I need to start planning for which circle of hell I end up in.

For my load tests I wrote a helper function called “fetch_static_assets”. The function is below:

def fetch_static_assets(session, response):
    resource_urls = set()
    soup = BeautifulSoup(response.text, "html.parser")
 
    for res in soup.find_all(src=True):
        url = res['src']
        if is_static_file(url):
            resource_urls.add(url)
        else:
            print "Skipping: " + url

The function is_static_file is quite important. The BeautifulSoup is going to return all URLs to you. Some of these may be broken, some may be off-site, etc. I recommend defining the is_static_file function and have it return false. Then look at what URLs are being skipped and slowly add patterns that match your static files and/or the URLs you want to fetch as sub-requests. In particular for a staging site, you don’t necessarily want to apply load to everything linked from your page. Here is an example of a very simplistic is_static_file function:

def is_static_file(f):
    if "/files" in f:
        return True
    else:
        return False

The rest of the fetch_static_assets function is below:

    for url in set(resource_urls):
        if "amazonaws.com" in url:
            session.client.get(url, name="S3 Static File")
            print "S3: " + url
        else:
            session.client.get(url, name="Static File")
            print "Regular: " + url

You can pass these static files into client.get for monitoring, but I would recommend setting the name to something consistent or else it is going to make your results quite messy. As you can see, I am tagging S3 URLs separately from regular static files in this example. Since you're defining all of this yourself, you have the flexibility to do basically whatever you want when you are parsing the page response and requesting sub-resources.

Below is an example of using this static asset function:

@task(10)
def frontpage(self):
    response = self.client.get("/")
    fetch_static_assets(self, response)

Logging Into Drupal

So, our load test can now fetch static assets. It can even fetch static assets of our choice and tag them as we would like. However, we are basically just testing the Drupal page cache at this point or perhaps Varnish or NGINX or even a CDN. Could be useful...probably isn’t though. To really be useful, we are going to have to login to the site. Fortunately this isn’t that difficult with Locust and we can use BeautifulSoup again. We are going to use the on_start method now. This is a special method of a Locust TaskSet that gets called on the start of the task set. It is not creatively named. Our example on_start is below:

def on_start(l):
    response = l.client.get("/user/login", name="Login")
    soup = BeautifulSoup(response.text, "html.parser")
    drupal_form_id = soup.select('input[name="form_build_id"]')[0]["value"]
    r = l.client.post("/user/login", {"name":"nnewton", "pass":"hunter2", "form_id":"user_login_form", "op":"Log+in", "form_build_id":drupal_form_id})

And there it is. Once this TaskSet logs in, Locust will keep the session cookie for the duration of that run. All requests from this TaskSet will be considered a logged in user. It is not uncommon for a test plan to have two TaskSets at the outside, one to cover the anonymous use-case and one for logged in users.

Locust is definitely a bit more difficult to approach than the JMeter GUI, but I have found it to be much easier to deal with when you are attempting to represent a somewhat complicated user pattern. In our next blog on this topic, we will be discussing how to use SaltStack and EC2 to automate Locust testing from multiple endpoints, i.e. spinning up a group of test VMs to run the Locust test plan and report back to a reporting node.

Download the demo load test described in this post.

Take the headache out of Drupal security with Tag1 Quo - Drupal Monitoring Made Easy.

Mar 15 2017
Mar 15

Though it came and went largely unnoticed, February 24th, 2017 marked an important anniversary to tens of thousands of Drupal website owners. February 24th 2017 was the 1-year anniversary of the End-of-Life (EOL) announcement for Drupal 6 as no longer supported by the Drupal community.

It is widely known that major Drupal version upgrades require non-trivial resources. Not only do they require significant planning, technical expertise, and budget, but the path is often determined by funding and availability of maintainers of popular contributed functionality (modules). Add the complexity of upgrading custom development, and the conditions create significant challenges for small to medium websites without large operating budgets. As evidence of this, our research indicates there are at least 150,000 publicly accessible sites still running Drupal 6.

Tag1 Quo is the only Drupal monitoring solution that supports Drupal 6 LTS, Drupal 7, and Drupal 8 under one dashboard.

For most D6 site managers, the most critical (and stressful) impact of EOL is the discontinuation of Drupal 6 security patches by the Drupal security team. When a major version reaches EOL, the Drupal security team ceases to release patches, or serve public Security Advisories for that version. Unless those sites are maintained by a skilled developer with the time to monitor upstream projects, review patches, and backport them by hand, Drupal 6 site managers find themselves in a vulnerable spot: ongoing, publicly announced vulnerabilities may be directly exposed on their site.

To its credit, the Drupal security team developed a plan so as not to abandon D6 site owners to the wilderness. Under the Long Term Support (LTS) initiative, they selected Tag1 and other qualified vendors to provide Drupal 6 patches as a paid service to site owners, under the condition that those patches also be made available to the public.

Tag1 Quo: A Year of LTS

With the EOL deadline rapidly approaching, Tag1—like many Drupal consulting firms—had clients still on Drupal 6. We were happy to sign on as an LTS provider to support our clients formally under the LTS initiative. It didn’t take us long to decide on automating patch delivery and empowering customers with some useful management tools. A few months into EOL, Tag1 Quo was launched with automated detection and notification of outstanding patches, and a unified view of security updates across all of their Drupal websites.

The vision was simple:Tag1 Quo Security Patch Notification

  • Provide D6 sites with a dashboard to quickly assess the status of their modules and themes, providing automated patches and pre-packaged releases delivered to their inbox, tested by our team of Drupal experts.
  • Make it platform and hosting-agnostic to provide maximum flexibility to the varied workflows and infrastructure of our customers.
  • Make it simple to setup and run from any site, in any environment, returning clear status reports and ongoing updates with the install of one module and a few clicks.
  • Price it aggressively: for less than the cost of 1 hour of senior developer time per month, a D6 customer could lease Quo as their security concierge, monitoring for patches around the clock.

Because of customers of Tag1 Quo and the LTS program, we’ve delivered on that vision. Paying customers of LTS have financed 25 Drupal 6 patches, written and contributed back to the community. While Drupal 8 continues to mature and add contributed functionality, the D6 LTS initiative is still going strong, giving site managers breathing room to fundraise, budget, and plan for their next big upgrade.

Enterprise Security with Tag1 Quo

Like many power users of Drupal, at Tag1 we maintain internal Drupal 6, 7, and 8 sites, as well as client sites on all of those versions. As we began designing and building Tag1 Quo, we quickly realized that the tools we wanted and needed for managing updates across sites were tools that would come in handy for other enterprise users:

  • agencies
  • universities
  • large corporations, and
  • infrastructure providers

In January 2017, we launched support for Drupal versions 7 and 8 on Tag1 Quo. With discounted rates for multiple sites, Tag1 Quo customers can now manage multiple sites via a centralized security dashboard with email notifications, across Drupal versions 6 through 8.

Tag1 Quo dashboard

Quo also provides powerful filtering tools across all sites. Filter by site, project, module version to see all instance of a particular module, across all sites. At-a-glance status color-coding tells you if your module has available updates, security-related or otherwise.

Filtering on modules in Tag1 Quo

Click in on a module to access a direct link to the latest release and access project metadata such as package, info, schema, and dependencies.Tag1 Quo module details

In managing our own sites, we’ve found that combining these tools in one central system has rapidly increased our turnaround on identifying and patching vulnerabilities, while lowering our management overhead. Eating our own dogfood has been satisfying: Tag1 Quo has freed up valuable developer time and budget to focus on feature development. If you are an agency maintaining client sites, or an IT department managing multiple corporate properties, you must have a security updates monitoring strategy and we’re confident that enterprise Tag1 Quo provides a solution.

Making Drupal maintenance easy forever

For years, the community has wrestled with the problem of expensive upgrades referenced in the beginning of this blog. How can Drupal continue to be a leader in innovation without becoming cost prohibitive to non-enterprise users? Last week, Dries published an important blog Making Drupal upgrades easy forever that announces an exciting, new approach for Drupal upgrades, based on a policy change initiated by Tag1’s Nat Catchpole.

Writes Dries:

we will continue to introduce new features and backwards-compatible changes in Drupal 8 releases. In the process, we sometimes have to deprecate old systems. Instead of removing old systems, we will keep them in place and encourage module maintainers to update to the new systems. This means that modules and custom code will continue to work. The more we innovate, the more deprecated code there will be in Drupal 8...Eventually, we will reach a point where we simply have too much deprecated code in Drupal 8. At that point, we will choose to remove the deprecated systems and release that as Drupal 9. This means that Drupal 9.0 should be almost identical to the last Drupal 8 release, minus the deprecated code.

For site owners and decision makers, this change is potentially earth-shattering. It replaces the monumental major version upgrade with incremental minor-version updates. Drupal 6 sites planning a Drupal 7 upgrade might want to revisit that plan. Drupal 7 sites waiting to upgrade directly to Drupal 9 may also want to reconsider. Site managers will need to invest more time on planning around minor releases: contributed code they rely on will be ported more frequently (though less dramatically). These changes are good for the Drupal ecosystem but issues of backward compatibility, legacy APIs, and deprecated code will likely require additional diligence.

We’ve built Tag1 Quo with an eye to this new future, with current and upcoming features to help site owners manage this complexity. If you are still on Drupal 6, Tag1 Quo has your back. If you are still on Drupal 7 when it goes EOL, Tag1 Quo will be there. And if you are somewhere in-between D8.7 and D8.11, Tag1 Quo will also be there for you, too.

In March 2017, get $50 credit towards your subscription with

Oct 25 2016
sam
Oct 25

When we left off last time, we’d assembled a definition of what versions are. Now, we’re going to dive into how we use them in Tag1 Quo: comparing them to one another!

The general goal is straightforward enough: we want to know if, say, 6.x-1.0 is less than 6.x-1.1. (Yup!) Or if 6.x-1.0-alpha1 is less than 6.x-1.0. (Also yup!) Let’s rewrite these two examples as tuple comparisons:

{6,1,0,4,0,0} < {6,1,1,4,0,0} = TRUE
{6,1,0,0,0,0} < {6,1,1,0,0,0} = TRUE

To determine if one tuple is less than the other, we proceed pairwise through the tuple’s values, comparing the integers at the same position from each, until we find different values. Whichever tuple’s value at that position is less is considered to be the lesser version. (Uniformity in this comparison operation is why the mapping for prerelease types assigns unstable to 0, rather than 4.)

However, this simple comparison operation doesn’t actually meet Quo’s requirements. Remember, Quo’s crucial question is not whether there are any newer versions, but whether there are newer security releases that are likely to apply to the version we’re investigating.

So, say we’re looking at 6.x-1.1 for a given extension, and there exists a 7.x-2.2 that’s a security release. While the latter is obviously less than the former:

{6,1,1,4,0,0} < {7,2,2,4,0,0} = TRUE

We don’t care, because these releases are on totally different lines of development.

...right? I mean, it’s probably true that whatever security hole existed in 7.x-2.1 doesn’t exist in 6.x-1.1. Maybe? Sort of. Certainly, you can't upgrade to 7.x-2.1 directly from 6.x-1.1, as that's changing major versions. But Quo came to be as part of the D6LTS promise - that IF there are security holes in later versions, we'll backport them to 6.x - so it's certainly possible that the problem might still exist. It all depends on what you take these version numbers to mean.

Yeah, we need to take a detour.

Versions are meaningless

As you become accustomed to a version numbering scheme - Drupal, semver, rpm, whatever - the meanings of the version components gradually work their way to the back of your mind. You don’t really “read” versions, so much as “scan and decode” them, according to these osmosed semantics. This peculiar infection of our subconscious makes it far too easy to forget a simple fact:

Version numbers have absolutely no intrinsic meaning. They have no necessary relationship to the code they describe.

Maybe this is obvious. Maybe it isn’t. If not, consider: what would prevent you from writing a module for Drupal 7 APIs, but then tagging and releasing it as 8.x-1.0? Or, for that matter, writing a module with no functions, but prints “spork” on inclusion of its .module file? (Answer: nothing.) Also, Donald Knuth uses a π-based numbering system for TeX’s versions, adding one more digit with each successive release. The version looks like a number, but the only property that matters is its length. Versions are weird.

This nebulous relationship is both the blessing and curse of versions. The curse is obvious: we can’t actually know anything with certainty about code just by looking at, or comparing, version numbers. But the blessing is more subtle: a well-designed version numbering system provides a framework for consistently encoding all of our intended semantics, together. Both of those words have specific meaning here:

  • “Together,” as in, it combines all the different aspects of changes to code that are important for Quo’s purposes: independent lines of development, Drupal core version compatibility, D6LTS’ own patch addenda, etc.

  • “Consistent,” as in, a numerical coördinate system - rather than an ad-hoc collection of flags, strings, and numbers - is a formal mathematical system without weird, nasty combinations of states.

The blessing outweighs the curse because, even if versions may lie to us about what the code actually is, they provide a formal structure in which it’s easy to understand what it should be. And, in the wild west of organic open source software growth, knowing with certainty about what things should be is a pretty good goal. It makes tasks concrete enough that you can actually build a business and product - like Tag1 Quo! Which takes us back to the main road after our detour - what’s the answer to this question?

{6,1,1,4,0,0} < {7,1,2,4,0,0}

The strictly mathematical answer is “yes.” But, for the question we’re actually interested in. we generally assume that security releases are only necessary when they’re on both the same core version, in the same line of development (major version). So, we say “no” here. And we’d also say “no” if the core version were the same:

{6,1,1,4,0,0} < {6,1,2,4,0,0}

This one is a little iffier, though. While porting from one Drupal core version to the next almost always involves a significant rewrite, that’s not necessarily the case for major versions. The security release may actually apply. It’s the kind of thing we need to investigate when deciding whether or not to release our D6LTS patch versions.

Today, Quo assumes security releases on different lines of development aren’t applicable to one another, but what’s important is that we know that’s an assumption. By representing the versions in a fully abstracted coördinate system as we have, rather than (for example) formally encoding assumptions about e.g. “lines of development” into the data itself, we allow ourselves the flexibility of changing those assumptions if they turn out to be wrong. Being that Quo’s entire business proposition turns on answering questions about versions correctly, it pays to build on a clear, flexible foundation.

This post rounded out the broader theory and big-picture considerations for versions in Quo. In the next post, I’ll get more into the nitty gritty - how Quo implements these ideas in a way that is practical, fast, and scalable.

 

Oct 25 2016
sam
Oct 25

When we left off last time, we’d assembled a definition of what versions are. Now, we’re going to dive into how we use them in Tag1 Quo: comparing them to one another!

The general goal is straightforward enough: we want to know if, say, 6.x-1.0 is less than 6.x-1.1. (Yup!) Or if 6.x-1.0-alpha1 is less than 6.x-1.0. (Also yup!) Let’s rewrite these two examples as tuple comparisons:

{6,1,0,4,0,0} < {6,1,1,4,0,0} = TRUE
{6,1,0,0,0,0} < {6,1,1,0,0,0} = TRUE

To determine if one tuple is less than the other, we proceed pairwise through the tuple’s values, comparing the integers at the same position from each, until we find different values. Whichever tuple’s value at that position is less is considered to be the lesser version. (Uniformity in this comparison operation is why the mapping for prerelease types assigns unstable to 0, rather than 4.)

However, this simple comparison operation doesn’t actually meet Quo’s requirements. Remember, Quo’s crucial question is not whether there are any newer versions, but whether there are newer security releases that are likely to apply to the version we’re investigating.

So, say we’re looking at 6.x-1.1 for a given extension, and there exists a 7.x-2.2 that’s a security release. While the latter is obviously less than the former:

{6,1,1,4,0,0} < {7,2,2,4,0,0} = TRUE

We don’t care, because these releases are on totally different lines of development.

...right? I mean, it’s probably true that whatever security hole existed in 7.x-2.1 doesn’t exist in 6.x-1.1. Maybe? Sort of. Certainly, you can't upgrade to 7.x-2.1 directly from 6.x-1.1, as that's changing major versions. But Quo came to be as part of the D6LTS promise - that IF there are security holes in later versions, we'll backport them to 6.x - so it's certainly possible that the problem might still exist. It all depends on what you take these version numbers to mean.

Yeah, we need to take a detour.

Versions are meaningless

As you become accustomed to a version numbering scheme - Drupal, semver, rpm, whatever - the meanings of the version components gradually work their way to the back of your mind. You don’t really “read” versions, so much as “scan and decode” them, according to these osmosed semantics. This peculiar infection of our subconscious makes it far too easy to forget a simple fact:

Version numbers have absolutely no intrinsic meaning. They have no necessary relationship to the code they describe.

Maybe this is obvious. Maybe it isn’t. If not, consider: what would prevent you from writing a module for Drupal 7 APIs, but then tagging and releasing it as 8.x-1.0? Or, for that matter, writing a module with no functions, but prints “spork” on inclusion of its .module file? (Answer: nothing.) Also, Donald Knuth uses a π-based numbering system for TeX’s versions, adding one more digit with each successive release. The version looks like a number, but the only property that matters is its length. Versions are weird.

This nebulous relationship is both the blessing and curse of versions. The curse is obvious: we can’t actually know anything with certainty about code just by looking at, or comparing, version numbers. But the blessing is more subtle: a well-designed version numbering system provides a framework for consistently encoding all of our intended semantics, together. Both of those words have specific meaning here:

  • “Together,” as in, it combines all the different aspects of changes to code that are important for Quo’s purposes: independent lines of development, Drupal core version compatibility, D6LTS’ own patch addenda, etc.

  • “Consistent,” as in, a numerical coördinate system - rather than an ad-hoc collection of flags, strings, and numbers - is a formal mathematical system without weird, nasty combinations of states.

The blessing outweighs the curse because, even if versions may lie to us about what the code actually is, they provide a formal structure in which it’s easy to understand what it should be. And, in the wild west of organic open source software growth, knowing with certainty about what things should be is a pretty good goal. It makes tasks concrete enough that you can actually build a business and product - like Tag1 Quo! Which takes us back to the main road after our detour - what’s the answer to this question?

{6,1,1,4,0,0} < {7,1,2,4,0,0}

The strictly mathematical answer is “yes.” But, for the question we’re actually interested in. we generally assume that security releases are only necessary when they’re on both the same core version, in the same line of development (major version). So, we say “no” here. And we’d also say “no” if the core version were the same:

{6,1,1,4,0,0} < {6,1,2,4,0,0}

This one is a little iffier, though. While porting from one Drupal core version to the next almost always involves a significant rewrite, that’s not necessarily the case for major versions. The security release may actually apply. It’s the kind of thing we need to investigate when deciding whether or not to release our D6LTS patch versions.

Today, Quo assumes security releases on different lines of development aren’t applicable to one another, but what’s important is that we know that’s an assumption. By representing the versions in a fully abstracted coördinate system as we have, rather than (for example) formally encoding assumptions about e.g. “lines of development” into the data itself, we allow ourselves the flexibility of changing those assumptions if they turn out to be wrong. Being that Quo’s entire business proposition turns on answering questions about versions correctly, it pays to build on a clear, flexible foundation.

This post rounded out the broader theory and big-picture considerations for versions in Quo. In the next post, I’ll get more into the nitty gritty - how Quo implements these ideas in a way that is practical, fast, and scalable.

 

Drupal 6 reached end of life, but Quo has security patches your site needs to remain in the game.

Oct 20 2016
sam
Oct 20

When Tag1 decided to build Tag1 Quo, we knew there was one question we’d have to answer over, and over, and over again: is there a security update available for this extension? Answering that question - at scale, for many websites, across many extensions, through all the possible versions they might have - is the heart of what Quo does.

The problem seems simple enough, but doing it at such scale, for “all” versions, and getting it right, has some deceptive difficulties. Given a site with an extension at a particular version, we need to know where it sits on the continuum of all versions that exist for that extension (we often refer to that as the “version universe,”), and whether any of the newer versions contain security fixes.

There are a few different approaches we could’ve taken to this problem. The one we ultimately settled on was a bit more abstracted than what might initially seem necessary. It was also not a “typical” Drupal solution. In this blog series, I’ll cover both the theoretical foundation of the problem, and the approach we took to implementation.

What’s a version?

Let's start at the beginning.

Quo works by having existing Drupal 6 (for now!) sites install an agent module, which periodically sends a JSON message back to the Quo servers indicating what extensions are present on that site, and at what versions. Those “versions” are derived from .info files, using functions fashioned after the ones used in Drupal core. Once that JSON arrives at Quo’s servers, we have to decide how to interpret the version information for each extension.

All versions arrive as strings, and the first decision Quo has to make is whether that string is “valid” or not. Validation, for the most part, means applying the same rules as applied by the Drupal.org release system. Let’s demonstrate that through some examples:

6.x-1.0

Drupal extension versions are a bit different than most software versions in the wild, because the first component, 6.x, explicitly carries information about compatibility, not with itself, but with its ecosystem: it tells us the version of Drupal core that the extension is supposedly compatible with.

The next part, 1.0, holds two bits of information: the major (1) and minor (0) versions. It’s wholly up to the author to decide what numbers to use there. The only additional meaning layered on is that releases with a 0 major version, like 6.x-0.1, are considered to be prerelease, and thus don’t receive coverage from the Drupal.org security team. Of course, Quo’s raison d’etre is that Drupal 6 is no longer supported by the security team, so that distinction doesn’t really matter anymore.

Now, 6.x-1.0 is an easy example, but it doesn’t cover the range of what’s possible. For example:

6.x-1.0-alpha1

This adds the prerelease field - alpha1. This field is optional, but if it does appear, it indicates the version to be some form of prerelease - and thus, as above, not covered by the security team. Drupal.org’s release system also places strong restrictions on the words that can appear there: alpha, beta, rc, and unstable. Additionally, the release system requires that the word must be accompanied by a number - 1, in this case.

There are a couple of notably different forms that valid Drupal versions can come in. There can be dev releases:

6.x-1.x-dev

Dev releases can only have the compatibility version and a major version, and cannot have a prerelease field. They’re supposed to represent a “line” of development, where that line is then dotted by any individual releases with the same major version.

And of course, Drupal core itself has versions:

6.43

Core versions have the same basic structure as the major version/minor version structure in an extension. Here, the 43 is a minor version - a dot along the development line of the 6 core version.

These examples illustrate what’s allowed by the Drupal.org release system, and thus, the shape that all individual extensions’ version universe will have to take. All together, we can say there are five discrete components of a version:

  • Core version
  • Major version
  • Minor version
  • Prerelease type
  • Prerelease number

Viewed in this way, it’s a small step to abstracting the notion of version away from a big stringy blob, and towards those discrete components. Specifically, we want to translate these versions into a 5-dimensional coördinate system, or 5-tuple: {core, major, minor, prerelease_type, prerelease_num}, where each of these dimensions has an integer value. Four of the components are already numbers, so that’s easy, but prerelease type is a string. However, because there’s a finite set of values that can appear for prerelease type, it’s easy to map those strings to integers:

  • Unstable = 0
  • Alpha = 1
  • Beta = 2
  • Rc = 3
  • (N/A - not a prerelease) = 4

With this mapping for prerelease types, we can now represent 6.x-1.0-alpha1 as {6,1,0,1,1}, or 6.x-2.3 as {6,2,3,4,0}.

However, that’s not quite the end of the story. The Quo service is all about delivering on the Drupal 6 Long Term Support (D6LTS) promise: providing and backporting security fixes for Drupal 6 extensions, now that they’re no longer supported by the security team.

Because such fixes are no longer official, we can’t necessarily expect there to be proper releases for them. At the same time, it’s still possible for maintainers to release new versions of 6.x modules, so we can’t just reuse the existing numbering scheme - the maintainer might later release a conflicting version.

For example, if D6LTS providers need to patch version 6.x-1.2 of some module, then we can’t release the patched version as 6.x-1.3, because we don’t have the authority to [get maintainers to] roll official releases (we’re not the security team, even though several members work for Tag1), and the maintainer could release 6.x-1.3 at any time, with or without our fix. Instead, we have to come up with some new notation that works alongside the existing version notation, without interfering with it.

Converting to the coördinate system gives us a nice tip in the right direction, though - we need a sixth dimension to represent the LTS patch version. And “dimension” isn’t metaphorical: for any given version, say {6, 1, 0, 1, 1} (that is, 6.x-1.0-alpha1), we may need to create an LTS patch to it, making it {6, 1, 0, 1, 1, 1}. And then later, maybe we have to create yet another: {6, 1, 0, 1, 1, 2}.

Now, we also have to extend the string syntax to support this sixth dimension - remember, strings are how the agent reports a site’s extension versions! It’s easy enough to say “let’s just add a bit to the end,” like we did with the coördinates, but we’re trying to design a reliable system here - we have to understand the implications of such changes.

Fortunately, this turns out to be quite easy: {6, 1, 0, 1, 1, 1} becomes 6.x-1.0-alpha1-p1; {6, 2, 3, 4, 0, 1} becomes 6.x-2.3-p1. This works well specifically because the strings in the prerelease type field are constrained to unstable, alpha, beta, and rc - unlike in semver, for example:

     A pre-release version MAY be denoted by appending a hyphen and a series
     of dot separated identifiers immediately following the patch version.
     Identifiers MUST comprise only ASCII alphanumerics and hyphen
     [0-9A-Za-z-].

     ...identifiers consisting of only digits are compared numerically and
     identifiers with letters or hyphens are compared lexically in ASCII
     sort order.

In semver, prerelease information can be any alphanumeric string, can be repeated, and are compared lexicographically (that is, alpha < beta < rc < unstable). If Drupal versions were unbounded in this way, then a -p1 suffix would be indistinguishable from prerelease information, creating ambiguity and making conflicts possible. But they’re not! So, this suffix works just fine.

Now, a coordinate system is fine and dandy, but at the end of the day, it’s just an abstracted system for representing the information in an individual version. That’s important, but the next step is figuring out where a particular version sits in the universe of versions for a given extension. Specifically, the question Quo needs to ask is if there’s a “newer” version of the component available (and if so, whether that version includes security fixes). Basically, we need to know if one version is “less” than another.

And that’s where we’ll pick up in the next post!

Oct 20 2016
sam
Oct 20

When Tag1 decided to build Tag1 Quo, we knew there was one question we’d have to answer over, and over, and over again: is there a security update available for this extension? Answering that question - at scale, for many websites, across many extensions, through all the possible versions they might have - is the heart of what Quo does.

The problem seems simple enough, but doing it at such scale, for “all” versions, and getting it right, has some deceptive difficulties. Given a site with an extension at a particular version, we need to know where it sits on the continuum of all versions that exist for that extension (we often refer to that as the “version universe,”), and whether any of the newer versions contain security fixes.

There are a few different approaches we could’ve taken to this problem. The one we ultimately settled on was a bit more abstracted than what might initially seem necessary. It was also not a “typical” Drupal solution. In this blog series, I’ll cover both the theoretical foundation of the problem, and the approach we took to implementation.

What’s a version?

Let's start at the beginning.

Quo works by having existing Drupal 6 (for now!) sites install an agent module, which periodically sends a JSON message back to the Quo servers indicating what extensions are present on that site, and at what versions. Those “versions” are derived from .info files, using functions fashioned after the ones used in Drupal core. Once that JSON arrives at Quo’s servers, we have to decide how to interpret the version information for each extension.

All versions arrive as strings, and the first decision Quo has to make is whether that string is “valid” or not. Validation, for the most part, means applying the same rules as applied by the Drupal.org release system. Let’s demonstrate that through some examples:

6.x-1.0

Drupal extension versions are a bit different than most software versions in the wild, because the first component, 6.x, explicitly carries information about compatibility, not with itself, but with its ecosystem: it tells us the version of Drupal core that the extension is supposedly compatible with.

The next part, 1.0, holds two bits of information: the major (1) and minor (0) versions. It’s wholly up to the author to decide what numbers to use there. The only additional meaning layered on is that releases with a 0 major version, like 6.x-0.1, are considered to be prerelease, and thus don’t receive coverage from the Drupal.org security team. Of course, Quo’s raison d’etre is that Drupal 6 is no longer supported by the security team, so that distinction doesn’t really matter anymore.

Now, 6.x-1.0 is an easy example, but it doesn’t cover the range of what’s possible. For example:

6.x-1.0-alpha1

This adds the prerelease field - alpha1. This field is optional, but if it does appear, it indicates the version to be some form of prerelease - and thus, as above, not covered by the security team. Drupal.org’s release system also places strong restrictions on the words that can appear there: alpha, beta, rc, and unstable. Additionally, the release system requires that the word must be accompanied by a number - 1, in this case.

There are a couple of notably different forms that valid Drupal versions can come in. There can be dev releases:

6.x-1.x-dev

Dev releases can only have the compatibility version and a major version, and cannot have a prerelease field. They’re supposed to represent a “line” of development, where that line is then dotted by any individual releases with the same major version.

And of course, Drupal core itself has versions:

6.43

Core versions have the same basic structure as the major version/minor version structure in an extension. Here, the 43 is a minor version - a dot along the development line of the 6 core version.

These examples illustrate what’s allowed by the Drupal.org release system, and thus, the shape that all individual extensions’ version universe will have to take. All together, we can say there are five discrete components of a version:

  • Core version

  • Major version

  • Minor version

  • Prerelease type

  • Prerelease number

Viewed in this way, it’s a small step to abstracting the notion of version away from a big stringy blob, and towards those discrete components. Specifically, we want to translate these versions into a 5-dimensional coördinate system, or 5-tuple: {core, major, minor, prerelease_type, prerelease_num}, where each of these dimensions has an integer value. Four of the components are already numbers, so that’s easy, but prerelease type is a string. However, because there’s a finite set of values that can appear for prerelease type, it’s easy to map those strings to integers:

With this mapping for prerelease types, we can now represent 6.x-1.0-alpha1 as {6,1,0,1,1}, or 6.x-2.3 as {6,2,3,4,0}.

However, that’s not quite the end of the story. The Quo service is all about delivering on the Drupal 6 Long Term Support (D6LTS) promise: providing and backporting security fixes for Drupal 6 extensions, now that they’re no longer supported by the security team.

Because such fixes are no longer official, we can’t necessarily expect there to be proper releases for them. At the same time, it’s still possible for maintainers to release new versions of 6.x modules, so we can’t just reuse the existing numbering scheme - the maintainer might later release a conflicting version.

For example, if D6LTS providers need to patch version 6.x-1.2 of some module, then we can’t release the patched version as 6.x-1.3, because we don’t have the authority to [get maintainers to] roll official releases (we’re not the security team, even though several members work for Tag1), and the maintainer could release 6.x-1.3 at any time, with or without our fix. Instead, we have to come up with some new notation that works alongside the existing version notation, without interfering with it.

Converting to the coördinate system gives us a nice tip in the right direction, though - we need a sixth dimension to represent the LTS patch version. And “dimension” isn’t metaphorical: for any given version, say {6, 1, 0, 1, 1} (that is, 6.x-1.0-alpha1), we may need to create an LTS patch to it, making it {6, 1, 0, 1, 1, 1}. And then later, maybe we have to create yet another: {6, 1, 0, 1, 1, 2}.

Now, we also have to extend the string syntax to support this sixth dimension - remember, strings are how the agent reports a site’s extension versions! It’s easy enough to say “let’s just add a bit to the end,” like we did with the coördinates, but we’re trying to design a reliable system here - we have to understand the implications of such changes.

Fortunately, this turns out to be quite easy: {6, 1, 0, 1, 1, 1} becomes 6.x-1.0-alpha1-p1; {6, 2, 3, 4, 0, 1} becomes 6.x-2.3-p1. This works well specifically because the strings in the prerelease type field are constrained to unstable, alpha, beta, and rc - unlike in semver, for example:

     A pre-release version MAY be denoted by appending a hyphen and a series
     of dot separated identifiers immediately following the patch version.
     Identifiers MUST comprise only ASCII alphanumerics and hyphen
     [0-9A-Za-z-].

     ...identifiers consisting of only digits are compared numerically and
     identifiers with letters or hyphens are compared lexically in ASCII
     sort order.

In semver, prerelease information can be any alphanumeric string, can be repeated, and are compared lexicographically (that is, alpha < beta < rc < unstable). If Drupal versions were unbounded in this way, then a -p1 suffix would be indistinguishable from prerelease information, creating ambiguity and making conflicts possible. But they’re not! So, this suffix works just fine.

Now, a coordinate system is fine and dandy, but at the end of the day, it’s just an abstracted system for representing the information in an individual version. That’s important, but the next step is figuring out where a particular version sits in the universe of versions for a given extension. Specifically, the question Quo needs to ask is if there’s a “newer” version of the component available (and if so, whether that version includes security fixes). Basically, we need to know if one version is “less” than another.

And that’s where we’ll pick up in the next post!

 

Tag1 Quo is the only Drupal monitoring solution that supports Drupal 6 LTS, Drupal 7, and Drupal 8 under one dashboard.

Aug 30 2016
Aug 30

Long Term Support for Drupal 6 might be my favorite new feature included in Drupal 8. (I know, that might be stretching things for the fundamentally awesome step forward that Drupal 8 is, but bear with me.)

Long Term Support for Drupal 6 Long Term Support for Drupal 6

If you're like me, you have loved the power of building websites for people that expose their ideas or services to the world. If you're like me, you've ended up "owning" a number of these websites that you somehow ended up supporting along the way too. And if you're like me, you've ended up with lots of Drupal 6 websites to support, even though D6 hit End-of-Life on February 24th, 2016.

There are a lot of D6 sites out there with no money for an upgrade, but which still have a niche to fill or useful information for the world. Those can be an albatross around our necks and a time sink. We don't have the resources to update (and their owners don't either) but we can't set the site owners adrift.

When previous versions of Drupal hit end-of-life, it was always a catastrophe for those of us with sites out there. Upgrade or else. Very costly in time and effort or money. But this time, with the release of Drupal 8, the Drupal Security Team and a number of vendors teamed up to actually offer real commercial support for Drupal 6. Yay! Yay!

One of those vendors is Tag1 Consulting with our slick new Tag1 Quo service, which monitors both Drupal.org security releases and your D6 site to make sure you know what the security issues are. We even work with the community and the Security Team to backport discovered D7 security fixes. (Full disclosure: I got to work on development of the very nice D8 Quo site.)

The Tag1 Quo Dashboard The Tag1 Quo Dashboard

I got to beta test Tag1 Quo with several of my old sites this year, and it was surprisingly easy. I just installed the D6 Tag1 Quo module on each D6 site, and immediately got a summary (on the Tag1 Quo site) of the security status of each site. Then, when Drupal 7 Security Advisories have gone out in recent months, I get an email with a patch or even a tarball for the affected module.

Drupal 6 Long Term Support is a huge step forward for our community, and great kudos are in order for the Security Team for arranging and supporting it and the vendors for providing it.

Tag1 has you covered with our Drupal security monitoring solution, Tag1 Quo.

Aug 22 2016
Aug 22

Tag1 Quo: Drupal 6 Long Term SupportIt’s been an exciting summer, building our first product with Drupal 8. When we originally made the decision to offer Long Term Support for Drupal 6, we were thinking about a few of our clients that were a little behind on their upgrade plans, and had envisioned a mostly manual process. However, once we took the plunge and signed up new clients, we had more modules and themes to track than could easily be done manually, and it remained critically important we never miss an upstream release.

Tag1 Quo Dashboard, multi-site up-to-date status The Tag1 Quo Dashboard, managing multiple websites.

This ultimately led to building Tag1 Quo, a product built on top of Drupal 8 to automatically track upstream releases and security advisories, comparing them against subscriber code bases to determine which need to be backported. This automation was combined with an administrative dashboard and email notification system resulting in a fancy system that quickly delivers all applicable patches to new and existing customers, ensuring everyone stays up to date while also making it easy for us to track ongoing security issues.

Why Drupal 8

Drupal 8 LogoThe first step was architecting the central service where all this information was going to be collected, stored, parsed and shared. After a little debate, we ultimately decided on Drupal 8 for a variety of reasons: there are huge improvements in hosting web services, we have an unparalleled team of Drupal 8 experts, and we generally wanted more real world Drupal 8 experience.

There have been days (and occasionally weeks) I’ve regretted the decision. For developers intimately familiar with earlier versions of Drupal, taking the plunge into 8 can feel intimidating and overwhelming. Fortunately it quickly becomes familiar and you realize it’s still very much Drupal, and you’re just working with a more powerful set of the same essential building blocks. Quite frequently the same problems are solved differently which doesn’t always equate to better, but nor does it always mean it’s worse; I found I had to remind myself occasionally to maintain a good attitude, and ultimately learned a lot in the progress and more often than not found myself preferring the Drupal 8 way.

Ultimately, it’s been a fantastic experience. I mean, I wouldn’t have wanted to do it with any other team. There were weeks we were tracking down and fixing core bugs, many that were both non-trivial and yet basic/common functionality. We carefully maintain an ever-growing directory of core patches waiting to get committed upstream. We also found that a number of key contrib modules weren’t quite stable, leading us to help fix bugs and add the features we need, always sharing them upstream. Through our development cycle, many of our patches have already been merged benefiting us and anyone else using Drupal 8.

Once I got used to the Drupal 8 file structure, and began to wrap my head around the object-oriented paradigm, there’s a number of wonderful improvements that make developing with Drupal 8 a joy. I’ve personally loved managing the site with the new configuration management system -- all configuration changes are made locally in a private development environment, reviewed, merged, and flow upstream in a controlled and auditable fashion to the shared development server, staging server, and finally deployed to the production server.

There is a certain irony in building this Drupal 8 website, which is parsing data from the Drupal 7 powered Drupal.org, and ingesting JSON data sent from and ultimately for supporting Drupal 6 websites. Nat Catchpole, one of the Drupal 8 branch maintainers involved in building Tag1 Quo, stated this eloquently in a tweet:

First proper 8.x project involves parsing HTML from 7.x d.o for 6.x LTS support. I might be stuck in a loop.

— catch (@catch56) May 20, 2016

Drupal 6 Long Term Support

Getting Started With Tag1 Quo Getting started with Tag1 Quo

At this point, it’s time to explain just exactly what Drupal 6 Long Term Support is. The idea is simple: Drupal is an open source project and as the project moves forward community volunteers simply can’t support all old versions. Shortly after the release of Drupal 8, Drupal 6 was “end of lifed”, which means Drupal’s volunteer security team is no longer actively reviewing, patching, or maintaining it in any way. That’s where we come in at Tag1: we monitor Drupal 7 and Drupal 8 security releases for core and contrib modules, and backport them to Drupal 6 if they affect any of our clients.

Simple, right? Except not really as it can quickly become complicated figuring out who’s installed what version of each module; is the 6.x-1.x branch affected or the 6.x-3.x branch? was the module renamed when it was removed from core in Drupal 7? or when it was merged into core in Drupal 8? and so on.

So, we automated it. We wrote and maintain a simple Drupal tag1quo module with a trivially simple configuration of a single token which then securely pushes information about your website to our central server. At the same time, we track upstream security releases and security advisories from Drupal.org, both through parsing RSS feeds and scraping web pages.

And at the heart of all of this we created a special field type that strictly parses version strings in a way that allows reliable and quick comparisons. This both allows us to flag upstream releases needing review, as well as when we need to notify users of patches that affect them.

Example Tag1 Quo security advisory Example Tag1 Quo security advisory, received as email.

As upstream security releases are made, we review them to determine if they also apply to the Drupal 6 version of the code. When they do and we have subscribers using the module, we carefully backport patches, test them, share them with other D6 LTS providers for additional testing, and ultimately manage a coordinated release.

For the end user, all of this effort is hidden. Notification emails show up in your inbox with simple and clear instructions on how to apply an update.

Tag1 Quo: D6 LTS And Then Some

While we automated our D6 LTS support offering, we realized we had something that was useful for far more than just Drupal 6 Long Term Support. While the core functionality remains about keeping your website secure, we also highlight modules needing non-security updates, or those missing version information (such as those installed directly from source). We take the guesswork out of which updates affect you and simplify it with a pretty, graphical dashboard allowing you to quickly monitor all your website from on a single overview page. More complex searching and filtering across sites and projects is also provided.

At this time, we offer three levels of service:

Pro is our recommended option, as it includes our pro-active Drupal 6 Long Term Support. We monitor all your modules and themes for upstream Drupal 7 and Drupal 8 security releases as described earlier. Our expert engineers carefully review each upstream security release to determine whether or not your Drupal 6 code is also vulnerable. If it is, we backport, test, and deliver patches to fix all identified security issues. You’re covered.

Tag1 Quo viewing single site details. Viewing one website's details with Tag1 Quo.

Users desiring more direct support from Tag1 and an adaptable pricing structure for larger numbers of websites will be interested in our Enterprise level offering.

Finally, we also developed an option for those on a tighter budget that can’t afford to subscribe to Drupal 6 Long Term Support but still want to keep as up-to-date and secure as possible. Our Basic offering delivers all patches affecting your website that were paid for by our Pro and Enterprise subscribers. We don’t monitor all of your modules and themes, but your more popular modules can still be kept up to date and secure.

What’s next?

We have big plans for Tag1 Quo. We’ll continue to revisit our roadmap in future blogs, but briefly now this includes adding support for monitoring Drupal 7 and Drupal 8 websites. And when Drupal 9 is released and Drupal 7 is end of lifed, we’ll be there to support it. But before that, Tag1 Quo is still a hugely useful tool for proactively keeping your site up to date. Coming soon is a feature to help with planning upgrades, tracking your modules against what’s been ported to the version of Drupal you’re looking to migrate. We’re also working to support other open source CMS’s, starting with WordPress 4.5 and 4.6.

Tag1 Consulting

Tag1 Consulting LogoWhile we’re very proud of Tag1 Quo, we remain a consulting company at heart, and we’d love to hear about how we can help you with your project. Whether you’re building your own Drupal 8 website (or product), upgrading your Drupal 6 website, or making improvements to your existing website, we’d love to be involved! Our specialities include Security, Performance and Scalability, and Architecture. We’ve always preferred to do it right the first time, but we can also help get you out of a jam.

It’s exciting to have put so much effort into a product over the summer, and to finally have something we’re proud of that we can share. If you have a Drupal 6 website you should sign up today, affordably keeping your website up to date and secure!

May 21 2015
ChX
May 21

Drupal 7

In Drupal 7, a hook_node_access implementation could return NODE_ACCESS_IGNORE, NODE_ACCESS_ALLOW and NODE_ACCESS_DENY. If any of them returned NODE_ACCESS_DENY then access was denied. If neither did but one returned NODE_ACCESS_ALLOW then access was allowed. If neither of these values were returned by any implementation then the decision was made based on other rules but at the end of the day some code needed to grant access explicitly or access was denied. Other entities didn’t have access control.

Also, blocks had some sort of access control in a very strange way: hook_block_list_alter is used -- even by core -- to remove the non-visible blocks.

Drupal 8

Drupal 8 brings a unified entity API -- even blocks become entities. It also uses many of the same objects and concepts for routing access. Instead of constants, we now use objects implementing the AccessResultInterface. You can get an instance by calling the rather self descriptive AccessResult::allowed(), AccessResult::forbidden(), AccessResult::neutral() methods. If you are handed an AccessResultInterface object you can figure out which one it is by calling the isAllowed, isForbidden, isNeutral methods on it. Only one of them can return TRUE. Access results can be cached and so have relevant cache contexts and tags -- this is why we bother with Neutral. This caching metadata is properly merged when doing various operations on access results.

These objects are returned by hook_entity_access implementations for entity access check and by the check method of services tagged with access_check used for routing access

Let’s first consider a node access case. The entity access checker fires hook_entity_access and hook_ENTITY_TYPE_access (in this case hook_node_access) and tries to put together the results in a sane way. If we do not want to change the behavior from D7, then any D7-deny (now called Forbidden) should result in a Forbidden end result. If there are no Forbiddens then any Allowed should result in a final Allowed return value. Finally, if there were neither Forbiddens nor Allows then only Neutral were present (if anything at all) so return a Neutral. This is called the orIf operation, if you have two result objects then run $result1->orIf($result2) to get the end result. The other doesn’t matter, $result2->orIf($result1) is the same.

Let’s take a look at the AccessResult::allowedIfHasPermission method. What results do we want? Obviously, if the permission is present we want an Allowed to be returned. But if the permission is not there? It can not result in Forbidden because then any hook_node_access simply returning the result of this method would immediately deny access to the node. It really can only return Neutral: the lack of a permission means we this method can’t form an opinion about access. (Using this method is strongly preferred to if ($user->hasPermission()) { return AccessResult::allowed();} because it adds the right caching metadata).

Let’s say we want to determine whether a field is editable! This requires the entity to be editable and the field to be editable. For the sake of simplicity, let’s presume both are controlled by a permission. So let’s say the user has “edit entity” permission but doesn’t have the “edit field” permission. So according to the previous paragraph AccessResult::allowedIfHasPermission($account, ‘edit entity’) is Allowed while AccessResult::allowedIfHasPermission($account, ‘edit field) is Neutral. We can not return Allowed in this case! So we can’t use orIf -- we need another operation: this is called andIf. Much like orIf we want any Forbidden input to result in a Forbidden output and again the same as orIf we want two Allowed to result in Allowed. The only difference is in the case detailed above: when one is Allowed, the other Neutral, here orIf results in Allowed but andIf results in Neutral.

If you want to use your knowledge and familiarity with AND/OR then consider first the iron rule of “any Forbidden input results in Forbidden” and only if that rule didn’t apply then consider Allowed as TRUE and Neutral as FALSE and apply the normal AND/OR to these values. This logic is called Kleene's weak three-valued logic where the “third value” is Forbidden. Most misunderstanding and confusion results from trying to treat Forbidden as FALSE instead of being the contagious “third value” it is. The name Neutral might make you think “oh, three values are not a problem, I will just erase any N I see and the resulting two values can be evaluated like normal AND/OR” this is absolutely incorrect! In fact, if you have two variables and both are Allowed / Forbidden then the results for $x->orIf($y) will be the exact same as $x->andIf($y)! The outcome will only differ if either $x or $y is Neutral (and the other is Allowed).

Routing vs Entity Access

We have clarified the necessity for two operators and we have two subsystems, each using one of them: routing uses andIf, entity uses orIf. The difference is subtle -- as we have seen the only difference is the end of result of two access checks where one is Neutral, the other is Allowed. This becomes a Neutral result in routing and an Allowed result in entity access.

Making a Decision

All this three value logic is nice but at the end of the day, this is all internal. The user wants to see a page, can we show it to them or do we show a 403 page? Can we show this entity? The answer cannot be “I do not know”, it must be yes or no. So the system looks at the result and says yes if it is explicitly allowed, otherwise says no. For routing (remember, andIf) this is very simple: if every access checker answered Allowed then and only then we can answer yes. For entity access (remember, orIf) there should be no Forbidden answers and at least one Allowed to say yes.

Apr 22 2015
Apr 22

Nedjo Rogers is a Senior Performance Engineer with Tag1 based out of Victoria, Canada. He’s been an active Drupal contributor since 2003, has served as an advisory board member of the Drupal Association, and has led Drupal development projects for clients including Sony Music, the Smithsonian Institute, the Linux Foundation, and a number of nonprofit organizations. He’s also the co-founder of Chocolate Lily, where he builds web tools for nonprofits, including the Drupal distribution Open Outreach.

Nedjo and I chatted shortly after he returned from a bike trip, touching on a range of topics including his entry into Drupal, working at Tag1, his nonprofit distribution, Drupal 8, corporate influence, CMI, Backdrop, and mining activism.

Dylan: How did you get started in web development and Drupal?

Nedjo: In 1996 I was working at an environmental group focused on the impacts of mining. My first work was mapping--providing communities with information about proposed mining projects in their area and the impacts of existing mines. We also had an international project, working in solidarity with organizations and communities impacted by the activities of Canadian mining companies in Peru and other Latin American countries.

In 1997 I started building an organizational website using Microsoft Front Page. Then, working with an international network of mining activist organizations, I produced a complex relational database application to track the worldwide activities of mining companies. Our ambitious aim was to enable communities to pool their information, supporting each other. I built the database using the software I had installed on my work computer: Microsoft Access.

In short: I was trying to build liberatory, activist internet resources, but using proprietary tools. I didn't know better. I'd never heard of free software.

By 2003, when I was working at a small technical support NGO, I'd had my introduction to the free software community, thanks initially to Paul Ramsay of PostGIS renown. At the time, Drupal was a small and relatively little known project. The available contributed modules numbered in the low dozens. Compared to other options I evaluated, the core software itself did very little. But what it did do, it did well and cleanly. I set about writing and contributing modules and dabbling in Drupal core.

Dylan: Your Drupal bio says you have a masters degree in geography. When you started working in technology was it an explicit part of your job responsibilities or was it more of something you fell into because someone needed to do it?

Nedjo: Decidedly the latter. In my MA, I took exactly one course in anything related to technology: geographic information systems (GIS). I was teased initially when I started hacking code. "Nerdjo" was my nickname of that time.

I've always had a love-hate relationship with the idea of working in IT. In many ways, I love the work. At its best, it's a constant source of intellectual challenge. But technology per se I don't care about at all. Outside of the Drupal community, few of my friends know in any detailed sense what I do.

Dylan: Yet based on what I’ve read about you, I'd venture a guess that your education, your intellectual interests, your views have had a big influence on the work you’ve done in the Drupal community. Is that true?

Nedjo: Very much so. Software, code, documentation--my sense is that all of it begins with the basic questions we ask ourselves.

Like many others, I came to Drupal with a purpose: to build tools that empower local, change-making organizations and movements. To put emerging communication technologies into the hands of activists and community builders. To do so, we need to work with those communities to shape technology to the needs and abilities of those who will use it.

I believe of Drupal that it retains, at least latently, a lot of the original principles of its genesis as a student project: a core focus on communities and collaboration. So, for me, working to bring a grounded and critical perspective to topics like structure and architecture has been a core part of my service to the project.

Working at Tag1

Dylan: Can you tell me about how you started working with Tag1 and what your experience working here has been like?

Tag1 partner Jeremy Andrews and I were longtime colleagues at CivicSpace, and I worked closely with Tag1 manager Peta Hoyes when we were both at CivicActions, so when Peta reached out to me with the opportunity to work with Tag1 I already knew it was going to be a great place to work.

My first assignment was on archetypes.com, a complex custom build that included some intriguing challenges. The project came to Tag1 through a route that’s now become familiar. We were brought on to work on performance issues on the site, which was built originally by a different team. We recommended architectural improvements that would address performance bottlenecks and then were engaged to guide that process, working with the in house Archetypes dev team.

It was my first chance to work with Tag1 engineer Fabian Franz, and I was immediately struck by his creative genius. I also got to collaborate again with Károly Négyesi (ChX), who years earlier was an important mentor for me, encouraging me to participate in the planning sprint for fields in Drupal core. Károly was also on hand to witness my near arrest by a half dozen of Chicago’s finest on Larry Garfield’s mother’s back porch while attending a Star Trek meetup, but that’s a longer story ;)

Recent highlights at Tag1 include collaborating with staff at the Linux Foundation to upgrade and relaunch their identity management system and working with Mark Carver and others to complete a redesign for Tag1’s own Drupal Watchdog site.

Tag1 is a good fit with my interests and commitments and I love the chance to work as part of a creative team where I’m always learning.

Nonprofit Distribution

Dylan: You’ve developed a Drupal distribution for nonprofits, Open Outreach and a set of features called Debut. Tell me about those projects.

Nedjo: In 2010 or so, while working at CivicActions, I completed six months of intensive work as tech lead of a project for the Smithsonian, a site on human origins for the US National Museum of Natural History.

We'd repeatedly worked 12 to 16 hours days to complete a grueling series of sprints. The site was a resounding success, but for me it sparked a crisis of faith. If we were here for free software, what were we doing pouring our efforts into what was essentially closed, proprietary development?

More troubling for me personally was the realization that I'd steadily lost touch with the reasons that brought me to Drupal in the first place: the small, activist groups I care about and am part of. If I wasn't going to focus solidly on building technology specifically for them, exactly who was?

I left CivicActions and soon afterwards, my partner, Rosemary Mann, also left her job of ten years as the executive director of a small group working with young, often marginalized parents. With our two kids, we pulled up roots and spent five months in Nicaragua. While there, I volunteered with an activist popular health organization.

When we returned, both at a crossroads, we decided to start up a Drupal distribution for the activist and community groups that we cared about. Groups that, we felt, were increasingly being left behind as Drupal shops pursued high-paying client contracts.

When we started into the work, via a small partnership, Chocolate Lily, several people said they didn’t understand our business model. By focusing on small, low resourced organizations, we'd never generate the lucrative contracts that Drupal shops depend on.

And we said, yeah, that's kind of the point.

Our first Open Outreach site was for the organization that Rosemary worked for for ten years, the Young Parents Support Network.

When building Open Outreach, we put a lot of energy into interoperability. At that time, the Kit specification, aimed at enabling the features of different distributions to work together, was a recent initiative. We very intentionally divided our work into two parts. Everything that was general beyond our specific use case (small NGOs and activist groups) we put into a generic set of features, Debut. The idea was that these features could be used independently of our distribution.

We also put energy into various other efforts that seemed to be oriented towards interoperability. This included the Apps module, which, at least in theory, included interoperability in its mandate. Why put energy in this direction?

Features itself - focused on capturing configuration in a form that can be reused on multiple sites - can be seen as extending the "free software" ethos from generic platform tools (modules) to applied use cases. But so long as features were limited to a particular distribution, they were in essence crippled. They were proprietary, in the sense that they were tied to a given solution set. Only by explicitly tackling the problem of interoperability could we truly bring the powerful tools being developed in Drupal to organizations other than those that could pay tens or hundreds of thousands of dollars on custom development.

The original authors of Features at Development Seed recognized this problem and took steps to address it. But when they largely left Drupal development, their insights and efforts languished.

I won’t make any great claims for what we achieved with Debut. To say that it was not broadly adopted is definitely an understatement ;) But in my more optimistic moments I like to think of it as a step in a broader process that will take time to bear fruit.

As Victor Kane recently put it, "We truly need open source propagation of reusable sets of Drupal configuration, and to reclaim the power of generically abstracting solutions from single site projects with the aim of re-use, with the creation of install profiles and distributions."

Drupal 8

Dylan: You are getting at some issues that I want to circle back to shortly. I’d like to shift to Drupal 8. What do you find most exciting or interesting about Drupal 8?

Nedjo: I've written or drafted several contributed modules now with Drupal 8. Coming from previous Drupal versions, I found it challenging at first to wrap my head around the new approaches. But as I got comfortable with them, I gained a lot of appreciation for the basic architecture and the flexibility and power that it enables.

Only a couple of months in, I already feel like I'm writing my cleanest Drupal code ever, thanks to the solid framework. While the previous procedural code left a lot up to the individual developer in terms of how best to structure a project, Drupal 8's well developed design patterns provide a lot of guidance.

One key feature of Drupal 8's architecture is "dependency injection", which roughly means that the elements (classes) that a particular solution needs are passed in dynamically as customizable “services” rather than being hard-coded. This design choice means a considerable leap in abstraction. At the same time, it opens the way for very specific and efficient customizations and overrides.

A concrete example, returning to the question of interoperability. A key barrier to features being interoperable is conflicting dependencies. This issue arises from the fact that some types of components need to be supplied once and used in many different features. Examples are user roles and "field bases" in Drupal 7. (In Drupal 8, what was a field base is now a "field storage".) In practice, most distributions put these shared components into a common feature that becomes a dependency for most or all other features in the distribution. For example, two different distributions might both include a core feature that provides a “body” field and an “administrator” role and that is a dependency of other features in the respective distributions.

This means you can’t easily mix features from one distribution with those of another, because their dependencies conflict.

In Drupal 7, I wrote the Apps Compatible module to try to address this problem. Apps Compatible makes it possible for multiple features and distros to share a certain limited set of shared configuration without conflicting dependencies. However, due to architectural shortcomings in Drupal 7, it does so via a bunch of awkward workarounds, and only for a very limited set of types of components (entities).

In Drupal 8, I've drafted an equivalent, Configuration Share. Using the dependency injection system, I was able very lightly to override a single method of the 'config.installer' service. In doing so, I achieved the same aim I’d attempted in Apps Compatible--except that this time it was done cleanly, in a few short lines of code, and for all types of configuration types at once.

Wow.

I also wrote the initial architecture of the Drupal 8 Features module, which - thanks to some fresh approaches building on Drupal 8 architectural improvements - promises to be a vast improvement over previous versions. Mike Potter at Phase2 is taking my start and adding on a lot of great ideas and functionality, while respecting and extending the basic architecture.

So, yeah, so far I’ve loved working in Drupal 8. I honour the creative energy that's gone into it. At the same time, I have a critical perspective on some directions taken in the version and the project as a whole. To me, those perspectives are not in any way contradictory. In fact, they stem from the same set of values and analysis.

Corporate Influence

Dylan: That’s a good segue back to an earlier point you made that is worth revisiting. A not-insignificant portion of early adoption and funding of Drupal work was driven by small, scrappy nonprofits, working with dev shops with social change in their missions. Many of those early adopting orgs were attracted to Drupal not only for the economic accessibility of an open source CMS but for the philosophical values of community and collaboration and mutual empowerment through shared interest. You’ve been vocal about your concerns regarding enterprise influence. Assuming that Drupal does not want to lose these users, what can or should be done?

Nedjo: You're correct to highlight the important and ongoing contributions of contributors and groups motivated by the ideals and principles of free software and more broadly of progressive social activism. We can think here of CivicSpace, a social change project that was a key incubator for a lot of what shaped Drupal core.

Back in 2009, working at CivicActions, I had the rare opportunity of doing six months of solid work on Drupal core and contrib for a corporate sponsor, Sony Music.

As technical lead of a fantastic team including Nathaniel Catchpole and Stella Powers, and in close collaboration with great folks at Sony Music and Lullabot, I got to dig into practically every area of Drupal code related to language support and internationalization. At the end of the project, on my own time, I wrote a detailed case study crediting Sony Music with their amazing contribution.

So, when it comes to corporate sponsorship, you could say I'm an early adopter and promoter.

Just not an uncritical one.

In a thoughtful 2008 post, Acquia co-founder Jay Batson wondered whether the balance between volunteer and commercially sponsored development “will change for Drupal as Acquia appears.” His starting point was a blog post on “The effects of commercialization on open source communities”.

Jay bet that sponsored development would not become “the most”, while, commenting on Jay’s post, Drupal founder and Acquia co-founder Dries Buytaert was “100% confident that the Drupal community will continue to do the bulk of the work.”

Dries noted though that “What is important to company A, is not necessarily important to company B. … At Acquia, we have our own itch to scratch, and we'll contribute a lot of code to scratch it properly ;-)”

I recently read a relevant article from 2011, “Control in Open Source Software Development”, in an industry journal. The authors outlined a series of strategies that companies could follow to ensure they achieved significant control over open source projects and thus "help companies achieve their for-profit objectives using open source software."

Looking at the strategies examined, we can see many of them at work in the Drupal project, including sponsoring the work of lead developers.

The question is not whether the “itch scratching” Dries described affects the functionality and focus of the project. Of course it does. That’s the whole point.

It’s how.

Besides sponsorship, there are also indirect but potentially equally powerful influences. Through our work, we're exposed to certain types of problems. If, as our skill level and reputation rise, we end up working on larger budget projects for large corporations, governments, and enterprise-level NGOs, we develop expertise and solutions that address those clients' priorities. This focus can carry over in both explicit and more subtle ways to our work in other areas, even if we're doing that work as volunteers.

I see this influence on a daily basis in my own work. I couldn’t count the number of customizations I’ve done for paid client projects and then carried forward as a volunteer, putting in the unpaid time to turn them into a contributed patch or module.

So, seven years on, it’s worth returning to the questions Jay raised. Have we as contributors in effect been relegated to a role of outsourced and unpaid bugfixers, or do we retain a central role in shaping our collective project? What is our project? What do we want to achieve with it--ultimately, who it is for?

CMI

Dylan: You’ve raised some concerns about the direction that the Configuration Management Initiative (CMI) in D8 is going...

Nedjo: Yes, I think we can see a lot of these questions - of influence, target users, and enterprise interests - at play in Drupal 8 configuration management approaches.

When he introduced Drupal 8 initiatives in his 2011 keynote address at DrupalCon Chicago, Dries explained how he’d consulted twenty of the largest companies and organizations using Drupal about their pain points, and configuration management was one of two things that stood out. So from the start, the focus of the CMI was, as Dries put it, to fix “the pains of large organizations adopting Drupal”.

The CMI began with a particular limited use case: the staging or deployment problem. Initiative lead Greg Dunlop was explicit about this focus. Greg hoped that further use cases - like those of distributions - could and would be added as we moved forward.

Recently, I began tackling in a serious way the possibilities and limitations of Drupal 8 configuration management when it comes to distributions. My starting place is a very practical one: I co-maintain a distribution, Open Outreach, that's aimed at small, low-resourced organizations. I need to know: will Drupal 8 be a sound basis for a new version of our distribution?

As I dug into the code, two things happened. First, as I mentioned before, I was impressed with the architecture of Drupal 8. Second, I was concerned about implications for small sites generally and for distributions in particular.

Briefly (and please bear with me, this gets a bit technical): In Drupal 7, the system of record for exportable configuration like views or image styles provided by a module is, by default, the module itself. If a site builder has explicitly overridden an item, the item’s system of record becomes the site (in practice, the site database).

An item that has not been overridden is on an update path and automatically receives improvements as they are added to the original module. An overridden item is in effect forked--it no longer receives updates and is maintained on a custom basis. This model prioritizes the free software sharing of updatable configuration items among multiple sites, while enabling customization via explicit overriding (forking).

In Drupal 8 as currently built, the system of record for all configuration is the site. There is no support for receiving configuration updates from modules. In effect, the free software use case of sharing configuration among many sites has, for the moment, lost out to the use case that's built into Drupal core: staging configuration between different versions of the same site.

While staging can, in theory, benefit anyone, in practice, there are pretty steep technical hurdles. To fully use the staging system as currently built into Drupal 8 core, you need to have access to a server that supports multiple databases and multiple install directories; know how to set up and manage multiple versions of a single site; move configuration and modules and files between different versions of the site; and make sense of a UI that presents machine names of configuration items and, even more dauntingly, line-by-line diffs of raw data representations of arbitrary types of configuration.

In other words, the configuration management system - which includes a lot of great ideas and improvements - serves a staging use case that has a lot of built-in assumptions about the technical capacity of the target users, and so far lacks support for the more broadly accessible use case of receiving updates to configuration supplied by extensions (modules and themes).

It looks to me like a case where focusing too closely on a particular use case (staging) has raised some unanticipated issues for other use cases, including distributions.

None of this is in any way an irreversible problem. The use case problem is one that Greg Dunlap pretty much foresaw when he commented on his initial limited focus. He noted it was only a starting place, with lots of followup to do. The contributed Configuration Update Manager module provides a good start. I've sketched in potential next steps in Configuration Synchronizer.

But my view is all of this belongs in core. Only there can we ensure that the free software use case of sharing configuration among multiple sites is on at least an equal footing with that of staging config on a single site.

Backdrop

Dylan: Is Backdrop part of the answer?

Nedjo: It's a welcome addition and a healthy sign. For Open Outreach, we're evaluating Backdrop alongside Drupal. We're even considering building versions for both.

Backdrop inherits some of the same issues as Drupal 8 when it comes to configuration management. But there are encouraging signs that the lead Backdrop developers are both aware of these issues and open to fixing them in core.

Mining Activism

Dylan: You mentioned that mining activism was your entry point into web technologies. Are you still involved in activist work?

Nedjo: I was out the loop for several years in terms of mining activism, but recently I've gotten involved again. Three years ago I co-founded a small organization in my hometown of Victoria, the Mining Justice Action Committee (MJAC). Last month I coordinated a visit by a Guatemalan indigenous leader, Angelica Choc, whose husband was killed in connection with his resistance to a Canadian mining company's operations. Angelica is part of a groundbreaking legal case to hold the company HudBay Minerals accountable for impacts related to its presence in Guatemala. She was a focus of the film Defensora.

At the event, I provided interpretation from Spanish to English as Angelica shared her story with the more than 100 people who filled a church hall. As I spoke her words, I was in tears. From the pain that she and family and community have suffered, but just as much from her courage, vision, and resilience.

My partner, Rosemary, is also an active member of MJAC, and last year she built its organizational website using our distribution, Open Outreach. In small ways, circles are completed.

Dylan: Last question. How was your bike ride?

Nedjo: My younger child is eighteen now and plans to leave home this summer, so these last months with them at home have a special intensity. Ardeo and I biked along a former rail line dotted with trestles that carried us high above the Cowichan River, swollen with spring runoff. In several places, winter storms had brought down trees that blocked the trail and we had to unhitch our panniers and work together to hoist bikes over the sprawling trunks. Dusk was falling as we reached our campground beside the river.

These times of connection - personal and ecological - are so essential. They remind us that technology, or our work in the world, is at most a detail, a small means to help us realize what really matters. What truly has value.

You seem to have caught me in a philosophical mood. We'd better stop here before I sound more sappy than I already do!

Dylan: Thanks for your time, Nedjo!

Apr 14 2015
Apr 14

I want to share two stories with you.

I started with Drupal in 2005. I started my first Drupal job in 2006 at $40/hr which was a pay cut. I quickly got a raise to $50/hr. I released Coder module in late 2006 and talked at OSCMS (the predecessor to DrupalCon) in 2007 and I began to be known in the Drupal community. Sometime in 2008 I started working on search. And because of my contributed work and reputation I was offered a 2 week job. I was asked how much I wanted. I remember I was making $50/hr. I said I'd like $100/hr. The client was the New York Observer and the person hiring me was Jeff Robbins of Lullabot. He knew my ask was low. He said, how about $110? Wow!

The second story is from DrupalCon San Francisco. In Dries' keynote he said, "if Drupal has changed your life, please stand up." I stood up and so did hundreds of others. I felt chills. Drupal changed my life because of this great career I now have. I work from home. I work with teams of smart, kind, diverse people. I work for interesting and great clients. And I make a nice paycheck.

So here I am to say something about Drupal 8 and make a challenge to you. Through the Drupal Association the community is trying to raise $250,000 to use as grants and bounties to help finish Drupal 8. I've worked many years full time with Drupal 7 and made a nice paycheck during that time. I also contributed heavily to Drupal 6 and Drupal 7.

But here we are in 2015 and I have contributed very little to Drupal 8. I want D8 released for selfish reasons. I want to start using it in my day job and secure the next several years of work.

So I contributed $1000 and have challenged others that are like me, that work in Drupal full time, who make a good paycheck from Drupal, and who have contributed less to Drupal core than I have to contribute one day's pay to #d8accelerate. Will you stand up with me and take my challenge?

Donate here.

Feb 26 2015
Feb 26

This is the third in a series of blog posts about the relationship between Drupal and Backdrop CMS, a recently-released fork of Drupal. The goal of the series is to explain how a module (or theme) developer can take a Drupal project they currently maintain and support it for Backdrop as well, while keeping duplicate work to a minimum.

  • In part 1, I introduced the series and showed how for some modules, the exact same code can be used with both Drupal and Backdrop.
  • In part 2, I showed what to do when you want to port a Drupal module to a separate Backdrop version and get it up and running on GitHub.
  • In part 3 (this post), I'll wrap up the series by explaining how to link the Backdrop module to the Drupal.org version and maintain them simultaneously.

Linking the Backdrop Module to the Drupal.org Version and Maintaining Them Simultaneously

In part 2 I took a small Drupal module that I maintain (User Cancel Password Confirm) and ported it to Backdrop. In the end, I wound up with two codebases for the same module, one on Drupal.org for Drupal 7, and one on GitHub for Backdrop.

However, the two codebases are extremely similar. When I fix a bug or add a feature to the Drupal 7 version, it's very likely that I'll want to make the exact same change (or at least an extremely similar one) to the Backdrop version. Wouldn't it be nice if there were a way to pull in changes automatically without having to do everything twice manually?

If you're a fairly experienced Git user, you might already know that the answer is "yes". But if you're not, the process isn't necessarily straightforward, so I'm going to document it step by step here.

Overall, what we're doing is simply taking advantage of the fact that when we imported the Drupal.org repository into GitHub in part 2, we pulled in the entire history of the repository, including all of the Drupal commits. Because our Backdrop repository knows about these existing commits, it can also figure out what's different and pull in the new ones when we ask it to.

In what follows, I'm assuming a workflow where changes are made to the Drupal.org version of the module and pulled into Backdrop later. However, it should be relatively straightforward to reverse these instructions to do it the other way around (or even possible, but perhaps less straightforward, to have a setup where you can do it in either direction).

  1. To start off, we need to make our local clone of the Backdrop repository know about the Drupal.org repository. (A local clone is obtained simply by getting the "clone URL" from the GitHub project page and copying it locally, for example with the command shown below.)
    git clone [email protected]:backdrop-contrib/user_cancel_password_confirm.git
    

    First let's check what remote repositories it knows about already:

    $ git remote -v
    origin    [email protected]:backdrop-contrib/user_cancel_password_confirm.git (fetch)
    origin    [email protected]:backdrop-contrib/user_cancel_password_confirm.git (push)
    

    No surprise there; it knows about the GitHub version of the repository (the "origin" repository that it was cloned from).

    Let's add the Drupal.org repository to this list and check again:

    $ git remote add drupal http://git.drupal.org/project/user_cancel_password_confirm.git
    $ git remote -v
    drupal    http://git.drupal.org/project/user_cancel_password_confirm.git (fetch)
    drupal    http://git.drupal.org/project/user_cancel_password_confirm.git (push)
    origin    [email protected]:backdrop-contrib/user_cancel_password_confirm.git (fetch)
    origin    [email protected]:backdrop-contrib/user_cancel_password_confirm.git (push)
    

    The URL I used here is the same one I used in part 2 to import the repository to GitHub (that is, it's the public-facing Git URL of my project on Drupal.org, available from the "Version control" tab of the drupal.org project page, after unchecking the "Maintainer" checkbox - if it’s present - so that the public URL is displayed). I've also chosen to give this repository the name "drupal". (Usually the convention is to use "upstream" for something like this, but in GitHub-land "upstream" is often used in a slightly different context involving development forks of one GitHub repository to another. So for clarity, I'm using "drupal" here. You can use anything you want to.)

  2. Next let's pull in everything from the remote Drupal repository to our local machine:
    $ git fetch drupal
    remote: Counting objects: 4, done.
    remote: Compressing objects: 100% (2/2), done.
    remote: Total 3 (delta 0), reused 0 (delta 0)
    Unpacking objects: 100% (3/3), done.
    From http://git.drupal.org/project/user_cancel_password_confirm
    * [new branch]          7.x-1.x -> drupal/7.x-1.x
    * [new branch]          master  -> drupal/master
    * [new tag]             7.x-1.0-rc1 -> 7.x-1.0-rc1
    

    You can see it has all the branches and tags that were discussed in part 2 of this series. However, although I pulled the changes in, they are completely separate from my Backdrop code (the Backdrop code lives in "origin" and the Drupal code lives in "drupal").

    If you want to see a record of all changes that were made to port the module to Backdrop at this point, you could run git diff drupal/7.x-1.x..origin/1.x-1.x to examine them.

  3. Now let's fix a bug on the Drupal.org version of the module. I decided to do a simple documentation fix: Fix documentation of form API functions to match coding standards

    I made the code changes on my local checkout of the Drupal version of the module (which I keep in a separate location on my local machine, specifically inside the sites/all/modules directory of a copy of Drupal so I can test any changes there), then committed and pushed them to Drupal.org as normal.

  4. Back in my Backdrop environment, I can pull those changes in to the "drupal" remote and examine them using git log:
    $ git fetch drupal
    remote: Counting objects: 5, done.
    remote: Compressing objects: 100% (3/3), done.
    remote: Total 3 (delta 2), reused 0 (delta 0)
    Unpacking objects: 100% (3/3), done.
    From http://git.drupal.org/project/user_cancel_password_confirm
      7a70138..997d82d  7.x-1.x     -> drupal/7.x-1.x
    
    $ git log origin/1.x-1.x..drupal/7.x-1.x
    commit 997d82dce1a4269a9cee32d3f6b2ec2b90a80b33
    Author: David Rothstein 
    Date:   Tue Jan 27 13:30:00 2015 -0500
    
            Issue #2415223: Fix documentation of form API functions to match coding standards.
    

    Sure enough, this is telling me that there is one commit on the Drupal 7.x-1.x version of the module that is not yet on the Backdrop 1.x-1.x version.

  5. Now it's time to merge those changes to Backdrop. We could just merge the changes directly and push them to GitHub and be completely done, but I'll follow best practice here and do it on a dedicated branch with a pull request. (In reality, I might be doing this for a more complicated change than a simple documentation fix, or perhaps with a series of Drupal changes all at once rather than a single one. So I might want to formally review the Drupal changes before accepting them into Backdrop.)

    By convention I'm going to use a branch name ("drupal-2415223") based on the Drupal.org issue number:

    $ git checkout 1.x-1.x
    Switched to branch '1.x-1.x'
    
    $ git checkout -b drupal-2415223
    Switched to a new branch 'drupal-2415223'
    
    $ git push -u origin drupal-2415223
    Total 0 (delta 0), reused 0 (delta 0)
    To [email protected]:backdrop-contrib/user_cancel_password_confirm.git
    * [new branch]          drupal-2415223 -> drupal-2415223
    Branch drupal-2415223 set up to track remote branch drupal-2415223 from origin.
    
    $ git merge drupal/7.x-1.x
    Auto-merging user_cancel_password_confirm.module
    Merge made by the 'recursive' strategy.
    user_cancel_password_confirm.module |   10 ++++++++--
    1 file changed, 8 insertions(+), 2 deletions(-)
    

    In this case, the merge was simple and worked cleanly. Of course, there might be merge conflicts here or other changes that need to be made. You can do those at this time, and then git push to push the changes up to GitHub.

  6. Once the changes are pushed, I went ahead and created a pull request via the GitHub user interface, with a link to the Drupal.org issue for future reference (I could have created a corresponding issue in the project's GitHub issue tracker also, but didn't bother):
    • Fix documentation of form API functions to match coding standards (pull request) (diff)

    Merging this pull request via the GitHub user interface gets it onto the official 1.x-1.x Backdrop branch, and into the Backdrop version of the module.

    Here's the commit for Drupal, and the same one for Backdrop:

    http://cgit.drupalcode.org/user_cancel_password_confirm/commit/?id=997d8...
    https://github.com/backdrop-contrib/user_cancel_password_confirm/commit/...

Using the above technique, it's possible to have one main issue (in this case on Drupal.org) for any change you want to make to the module, do essentially all the work there, and then easily and quickly merge that change into the Backdrop version without the hassle of repeating lots of manual, error-prone steps.

Hopefully this technique will be useful to developers who want to contribute their work to Backdrop while also continuing their contributions to Drupal, and will help the two communities continue to work together. Thanks for reading!

Further Backdrop Resources

Do you have any thoughts or questions, or experiences of your own trying to port a module to Backdrop? Leave them in the comments.

Feb 17 2015
Feb 17

This is the second in a series of blog posts about the relationship between Drupal and Backdrop CMS, a recently-released fork of Drupal. The goal of the series is to explain how a module (or theme) developer can take a Drupal project they currently maintain and support it for Backdrop as well, while keeping duplicate work to a minimum.

  • In part 1, I introduced the series and showed how for some modules, the exact same code can be used with both Drupal and Backdrop.
  • In part 2 (this post), I'll explain what to do when you want to port a Drupal module to a separate Backdrop version and get it up and running on GitHub.
  • In part 3, I'll explain how to link the Backdrop module to the Drupal.org version and maintain them simultaneously.

Porting a Drupal Module to Backdrop and Getting it Up and Running on GitHub

For this post I’ll be looking at User Cancel Password Confirm, a very small Drupal 7 module I wrote for a client a couple years back to allow users who are canceling their accounts to confirm the cancellation by typing in their password rather than having to go to their email and click on a confirmation link there.

We learned in part 1 that adding a backdrop = 1.x line to a module’s .info file is the first (and sometimes only) step required to get it working with Backdrop. In this case, however, adding this line to the .info file was not enough. When I tried to use the module with Backdrop I got a fatal error about a failure to open the required includes/password.inc file. What's happening here is simply that Backdrop (borrowing a change that's also in Drupal 8) reorganized the core directory structure compared to Drupal 7 to put most core files in a directory called "core". When my module tries to load the includes/password.inc file, it needs to load it from core/includes/password.inc in Backdrop instead.

This is a simple enough change that I could just put a conditional statement into the Drupal code so that it loads the correct file in either case. However, over the long run this would get unwieldy. Furthermore, if I had chosen a more complicated module to port, one which used Drupal 7's variable or block systems (superseded by the configuration management and layout systems in Backdrop) it is likely I'd have more significant changes to make.

So, this seemed like a good opportunity to go through the official process for porting my module to Backdrop.

Backdrop contrib modules, like Backdrop core, are currently hosted on GitHub. Regardless of whether you're already familiar with GitHub from other projects, there are some steps you should follow that might not be familiar, to make sure your Backdrop module's repository is set up properly and ultimately to get it included on the official list of Backdrop contributed projects.

Importing to GitHub

The best way to get a Drupal module into GitHub is to import it; this preserves the pre-Backdrop commit history which becomes important later on.

Before you do this step, if you're planning to port a Drupal module that you don't maintain, it's considered best practice to notify the current maintainer and see if they'd like to participate or lead the Backdrop development themselves (see the "Communicating" section of the Drupal 7 to Backdrop conversion documentation for more information). In my case I'm already the module maintainer, so I went ahead and started the import:

  1. Go to the GitHub import page and provide the public URL of the Drupal project's Git repository (which I got from going to the project page on Drupal.org, clicking the "Version control" tab, and then - assuming you are importing a module that you maintain - making sure to uncheck the "Maintainer" checkbox so that the public URL is displayed). Drupal.org gives me this example code:


    git clone --branch 7.x-1.x http://git.drupal.org/project/user_cancel_password_confirm.git

    So I just grabbed the URL from that.

  2. Where GitHub asks for the project name, use the same short name (in this case "user_cancel_password_confirm") that the Drupal project uses.
  3. Import the project into your own GitHub account for starters (unless you're already a member of the Backdrop Contrib team - more on that later).

Here's what it looks like:
GitHub import

Submitting this form resulted in a new GitHub repository for my project at https://github.com/DavidRothstein/user_cancel_password_confirm.

As a final step, I edited the description of the GitHub project to match the description from the module's .info file ("Allows users to cancel their accounts with password confirmation rather than e-mail confirmation").

Cleaning Up Branches and Tags

Next up is some housekeeping. First, I cloned a copy of the new repository to my local machine and then used git branch -r to take a look around:


$ git clone [email protected]:DavidRothstein/user_cancel_password_confirm.git
$ git branch -r
origin/7.x-1.x
origin/HEAD -> origin/master
origin/master

Like many Drupal 7 contrib projects, this has a 7.x-1.x branch where all the work is done and a master branch that isn't used. When I imported the repository to GitHub it inherited those branches. However, for Backdrop I want to do all work on a 1.x-1.x branch (where the first "1.x" refers to compatibility with Backdrop core 1.x).

  1. So let's rename the 7.x-1.x branch:


    $ git checkout 7.x-1.x
    Branch 7.x-1.x set up to track remote branch 7.x-1.x from origin.
    Switched to a new branch '7.x-1.x'
    $ git branch -m 7.x-1.x 1.x-1.x
    $ git push --set-upstream origin 1.x-1.x
    Total 0 (delta 0), reused 0 (delta 0)
    To [email protected]:DavidRothstein/user_cancel_password_confirm.git
    * [new branch] 1.x-1.x -> 1.x-1.x
    Branch 1.x-1.x set up to track remote branch 1.x-1.x from origin.

  2. And delete the old one from GitHub:


    $ git push origin :7.x-1.x
    To [email protected]:DavidRothstein/user_cancel_password_confirm.git
    - [deleted] 7.x-1.x

  3. We want to delete the master branch also, but can't do it right away since GitHub treats that as the default and doesn't let you delete the default branch.

    So I went to the module's GitHub project page, where (as the repository owner) I have a "Settings" link in the right column; via that link it's possible to change the default branch to 1.x-1.x through the user interface.

    Now back on my own computer I can delete the master branch:


    $ git push origin :master
    To [email protected]:DavidRothstein/user_cancel_password_confirm.git
    - [deleted] master

  4. On Drupal.org, this module has a 7.x-1.0-rc1 release, which was automatically imported to GitHub. This won't be useful to Backdrop users, so I followed the GitHub instructions for deleting it.
  5. Finally, let's get our local working copy somewhat in sync with the changes on the server. The cleanest way to do this is probably just to re-clone the repository, but you could also run git remote set-head origin 1.x-1.x to make sure your local copy is working off the same default branch.

The end result is:


$ git branch -r
origin/1.x-1.x
origin/HEAD -> origin/1.x-1.x

Just what we wanted, a single 1.x-1.x branch which is the default (and which was copied from the 7.x-1.x branch on Drupal.org and therefore contains all its history).

Updating the Code for Backdrop

Now that the code is on GitHub, it's time to make it Backdrop-compatible.

To do this quickly, you can just make commits to your local 1.x-1.x branch and push them straight up to the server. In what follows, though, I'll follow best practices and create a dedicated branch for each change (so I can create a corresponding issue and pull request on GitHub). For example:


$ git checkout -b backdrop-compatibility
$ git push -u origin backdrop-compatibility

Then make commits to that branch, push them to GitHub, and create a pull request to merge it into 1.x-1.x.

  1. To get the module basically working, I'll make the simple changes discussed earlier:
    • Add basic Backdrop compatibility (issue) (diff)

    If you look at the diff, you can see that instead of simply adding the backdrop = 1.x line to the .info file, I replaced the core = 7.x line with it (since the latter is Drupal-specific and does not need to be in the Backdrop version).

    With that change, the module works! Here it is in action on my Backdrop site:

    Cancel account using password

    (Also visible in this screenshot is a nice effect of Backdrop's layout system: Editing pages like this one, even though they are using the default front-end Bartik theme, have a more streamlined, focused layout than normal front-end pages of the site, without the masthead and other standard page elements.)

  2. Other code changes for this small module weren't strictly necessary, but I made them anyway to have a fully-compatible Backdrop codebase:
    • Replace usage of "drupal" with "backdrop" in the code (issue) (diff)
    • Use method on the user account object to determine its ID (issue) (diff)
  3. Next up, I want to get my module listed on the official list of Backdrop contributed projects (currently this list is on GitHub, although it may eventually move to the main Backdrop CMS website).

    I read through the instructions for applying to the Backdrop contributed project group. They're relatively simple, and I've already done almost everything I need above. The one thing I'm missing is that Backdrop requires a README.md file in the project root with some standard information in it (I like that they're enforcing this; it should help developers browsing the module list a lot), and it also requires a LICENSE.txt file. These were both easy to create following the provided templates and copying some information from the module's Drupal.org project page:

    Once that's done, and after reading through the rest of the instructions and making sure I agreed with them, I proceeded to create an issue:

    Application to join contrib team

    In my case it was reviewed and approved within a few hours (perhaps helped by the fact that I was porting a small module), and I was given access to the Backdrop contributed project team on GitHub.

  4. To get the module transferred from my personal GitHub account to the official Backdrop contrib list, I followed GitHub's instructions for transferring a repository.

    They are mostly straightforward. Just make sure to use "backdrop-contrib" as the name of the new owner (who you are transferring the repository to):

    Transfer repository to backdrop-contrib

    And make sure to check the box that gives push access to your repository to the "Authors" team within the Backdrop Contrib group (if you leave it as "Owners", you yourself wouldn't be able to push to it anymore):

    Grant access to the Authors team

    That's all it took, and my module now appears on the official list.

    You'll notice after you do this that all the URLs of your project have changed, although the old ones redirect to the new ones. That's why if you follow many of the links in this post, which point to URLs like https://github.com/DavidRothstein/user_cancel_password_confirm, you'll see that they actually redirect you to https://github.com/backdrop-contrib/user_cancel_password_confirm.

    For the same reason, you can keep your local checkout of the repository pointed to the old URL and it will still work just fine, although to avoid any confusion you might want to either do a fresh clone at this point, or run a command like the following to update the URL:

    git remote set-url origin [email protected]:backdrop-contrib/user_cancel_password_confirm.git
    

With the above steps, we’re all set; the module is on GitHub and can be developed further for Backdrop there.

But what happens later on when I make a change to the Drupal version of the module and want to make the same change to the Backdrop version (certainly a common occurrence)? Do I have to repeat the same changes manually in both places? Luckily the answer is no. In part 3 of this series, I’ll explain how to link the Backdrop module to the Drupal.org version and maintain them simultaneously. Stay tuned!

Further Backdrop Resources

Do you have any thoughts or questions, or experiences of your own trying to port a module to Backdrop? Leave them in the comments.

Feb 09 2015
ChX
Feb 09

While coding the MongoDB integration for Drupal 8 I hit a wall first with the InstallerKernel which was easy to remedy with a simple core patch but then a similar problem occurred with the TestRunnerKernel and that one is not so simple to fix: these things were not made with extensibility in mind. You might hit some other walls -- the code below is not MongoDB specific. But note how unusual this is: you won’t hit similar problems often. Drupal 8 very extensible but it has its limits. But when you need to bypass those limits, PHP has a solution for you to extend classes (to some extent) that were not meant to be extended. Yes, it’s a hack. But when all else fails...

Be very careful because the next version of Drupal core or other software you are commando-extending might not work the same internally and so your hack will break.

With that caveat, let me introduce you to reflection. While some might think it was only meant to investigate objects, already in PHP 5 there was a ReflectionProperty::setValue method. In PHP 5.3 ReflectionProperty::setAccessible was added.

$r = new \ReflectionObject($kernel);
$services = $r->getProperty('serviceYamls');
$services->setAccessible(TRUE);
$value = $services->getValue($kernel);
$value['app'][] = 'modules/mongodb/mongodb.services.yml';
$services->setValue($kernel, $value);

Let’s investigate carefully what happens here because it is not evident and the documentation is lacking. $services is a ReflectionProperty object and while it was created by a getProperty call on a ReflectionObject encapsulating the $kernel object, the $services object is completely ignorant that it belongs to the $kernel object. We could’ve created it from the InstallerKernel class for the same result, it doesn’t matter.

Once this is understood, there are few surprises in this block of code: the property is protected so we need to make it accessible. The setAccessible() call will not make $kernel->serviceYamls public because, again, the $services ReflectionProperty object is ignorant of the $kernel object it was created from. It will, however, enable us to call getValue and setValue on it. But again, we need to pass $kernel to both the getValue and the setValue call.

So this way and only this way we can change a protected object property.

Even better, this works for private properties as well. So if you need to extend a class which has private properties (many Symfony classes do this) then instead of extending, wrap the object (this is called the delegate pattern) and use this method to access its private properties. As an example, let’s extend Symfony\Component\DependencyInjection\Scope (this is the shortest class I could find with a private property and an interface, the example is totally nonsensical and has nothing to do with the Scope class, it’s just an example):

class MyScope implements ScopeInterface {
  public function __construct($name, $parentName = ContainerInterface::SCOPE_CONTAINER) {
    $this->scope = new Scope($name, $parentName);
  }
  public function getName() {
    // Awful hack breaking encapsulation.
    $reflection_scope = new \ReflectionObject($this->scope);
    $name = $reflection_scope->getProperty('name');
    $name->setAccessible(TRUE);
    $value = ‘my’ . $name->getValue($this->scope);
    $name->setValue($this->scope, $value);
    return $value;
  }
  public function getParentName() {
    // This is proper: delegate the call.
    return $this->scope->getParentName();
  }
}
Feb 03 2015
Feb 03

Part 1 - Reuse the Same Code

In mid-January, the first version of Backdrop CMS was released. Backdrop is a fork of Drupal that adds some highly-anticipated features and API improvements to the core Drupal platform while focusing on performance, usability, and developer experience.

When an open-source fork makes the news, it's often because it was born from a fierce, acrimonious battle (example: Joomla forking from Mambo); the resulting projects compete with each other on the exact same turf and developers are forced to choose sides. Backdrop's goal, however, is not to destroy or replace the original Drupal project. Rather, it aims to be a "friendly fork" that focuses on Drupal's traditional audience of site builders and developers, an audience which the Backdrop founders believe are being slowly left behind by the Drupal project itself.

Because of this, I expect that many existing Drupal developers will not want to choose between the platforms, but instead will continue working with Drupal while also beginning to use Backdrop. In this series of blog posts, I will explain how a module (or theme) developer can take a Drupal project they currently maintain and support it for Backdrop as well, while keeping duplicate work to a minimum.

  • In part 1 (this post), I'll show how for some modules, the exact same code can be used with both Drupal and Backdrop.
  • In part 2, I'll explain what to do when you want to port a Drupal module to a separate Backdrop version and get it up and running on GitHub.
  • In part 3, I'll explain how to link the Backdrop module to the Drupal.org version and maintain them simultaneously.

Sometimes the Same Exact Code can be Used With Both Drupal and Backdrop

To start things off let's look at Field Reference Delete, a Drupal 7 module I maintain which does some behind-the-scenes cleanup in your database when entities such as nodes or taxonomy terms are deleted on the site. It's a moderately-complex module which makes heavy use of Drupal's field and entity systems.

To make this or any Drupal module work with Backdrop, there is one step that is always required: Adding a backdrop = 1.x line to the .info file to inform Backdrop core that the code is Backdrop-compatible.

Easy enough. The big question is what changes are required beyond that?

Checking for Changes

Although Backdrop is not 100% compatible with Drupal 7 (due to the features and API improvements it adds) it aims to be backwards-compatible as much as possible, for example by including a compatibility layer to allow Drupal functionality that is deprecated in Backdrop to still work correctly.

The Backdrop change records and module conversion guide provide technical advice for developers on how and when to upgrade their code. The biggest changes are probably the configuration management system (Backdrop’s replacement for Drupal 7’s variable API and other configuration that was previously stored in the database) and layout system (which removes much of the functionality of the Drupal 7 Block module in favor of a newer, more powerful Layout module).

If your Drupal 7 module makes heavy use of these systems, it’s likely you’ll want to make some changes in order to work with Backdrop. However, the compatibility layer means that you might not actually need to. For example, Backdrop retains Drupal 7’s variable API (although it is marked as deprecated and is not as powerful as the configuration system which replaces it). So your code might still work even if it uses this system. It really depends on the details of how your module works, so the best advice is to test it out and see what (if anything) is broken.

It’s also worth noting that because Backdrop was forked from an early version of Drupal 8 (not from Drupal 7) it inherited a smattering of changes that were added to Drupal 8 early in the release cycle. Not all of these have made it into the list of Backdrop change records yet, although work is ongoing and people are adding them as they are noticed.

In the case of Field Reference Delete, I tested the module on Backdrop and it worked fine. I also skimmed the change records and module conversion guide mentioned above and didn't see anything that obviously needed to change. Even though entities in Backdrop have been converted to classed objects and the field API has been converted to use the configuration management system, Field Reference Delete’s heavy use of the entity and field systems still didn’t require that any changes be made. All I had to do to get the module working was add that one backdrop = 1.x line to the .info file.

Adding the One Line of Code on Drupal.org

Interestingly enough, since Drupal will happily ignore a backdrop = 1.x line in a module's .info file, it's possible to simply add that code to the Drupal.org version of the module and use the same version of the module for either Drupal or Backdrop. I did that in this issue; the resulting diff is simply this:


diff --git a/field_reference_delete.info b/field_reference_delete.info
...
name = Field reference delete
description = Immediately removes references to a deleted entity from fields stored in an SQL database.
core = 7.x
+backdrop = 1.x

Drupal uses the core = 7.x line to determine Drupal compatibility, and Backdrop uses the backdrop = 1.x line to determine Backdrop compatibility. The 7.x-1.0-beta1 release of Field Reference Delete contains the above change and can be used equally well on a Drupal or Backdrop site. Simple!

There are some downsides to doing this, however:

  • Although no changes to the module code may be strictly required, there are usually optional (and non-backwards-compatible) changes that can be made to better integrate with new Backdrop features.
  • It is hard for Backdrop users and developers to find the module and know that it's compatible with Backdrop. I tried to improve its discoverability by adding a "Backdrop compatibility" tag to the above-mentioned issue, and I also put a note on the project page explaining that Backdrop is supported. These aren't ideal, but should help; perhaps a standard will eventually catch on but there isn't a formal one yet.

Despite these disadvantages, for the time being I'd like to just have one copy of the code for this particular module (hosted in one place), and it's nice to know that's possible.

In part 2 of this series, I’ll take a look at a different module I maintain where it isn’t possible to use the same exact code for Drupal and Backdrop, and I’ll show how I went through the official process for porting my module to Backdrop and getting it hosted on GitHub. Stay tuned!

Further Backdrop Resources

Do you have any thoughts or questions, or experiences of your own trying to port a module to Backdrop? Leave them in the comments.

Dec 15 2014
Dec 15

Earlier this year we undertook a project to upgrade a client's infrastructure to all new servers including a migration from old Puppet scripts which were starting to show their age after many years of server and service changes. During this process, we created a new set of Puppet scripts using Hiera to separate configuration data from modules. The servers in question were all deployed with CentOS, and it soon became obvious that we needed a modular way in Puppet to install and configure yum package repositories from within our various Puppet modules.

Searching through the Puppet Forge uncovered a handful of modules created to deal with yum repos, however most were designed to implement a single repository, or were not easily configurable via Hiera, which was one of our main goals for the new Puppet scripts. So, we decided to create our own module, and the yumrepos Puppet module was born.

The idea behind the module is to provide a clean and easy way to pull in common CentOS/RHEL yum repos from within Puppet. By wrapping each repo with its own class (e.g. yumrepos::epel and yumrepos::ius), we gain the ability to override default class parameters with Hiera configuration, making the module easy to use in most any environment. For example, if you have your own local mirror of a repo, you can override the default URL parameter either in Hiera, or from the class declaration without having to directly edit any files within the yumrepos module. This is as easy as:

  1. In your calling class, declare the yumrepo class you need. In this example, we'll use EPEL: class { 'yumrepos::epel': }
  2. In your Hiera configuration, you can configure the repo URL with: yumrepos::epel::epel_url: http://your.local.mirror/path/to/epel/

Currently the yumrepos module provides classes for the following yum repos:

  • Drupal Drush 5
  • Drupal Drush 6
  • EPEL
  • IUS Community (Optionally: IUS Community Archive)
  • Jenkins
  • Jpackage
  • Percona
  • PuppetLabs
  • RepoForge
  • Varnish 3
  • Zabbix 2.4

Each repo contains the GPG key for the repo (where available) and defaults to enabling GPG checks. Have another repo you'd like to see enabled? Feel free to file an issue or pull request at https://github.com/tag1consulting/puppet-yumrepos

Additionally, yumrepos classes accept parameters for package includes or excludes so that you can limit packages on a per-repo basis. These translate to the includepkgs and exclude options within a yum repo configuration. Similar to overriding the default repo URL, these options can be overridden by passing parameters within the class declaration or by setting the appropriate variables within Hiera.

Once we had the yumrepos module in place, we had an easy way to configure yum repos from within our other modules. Stay tuned to the blog; we'll have more information about the overall Puppet scripts coming soon.

Dec 02 2014
Dec 02

I was drawn to Behavior Driven Development the moment I was pointed toward Behat not just for the automation but because it systematized and gave me a vocabulary for some things I already did pretty well. It let me teach some of those skills instead of just using them. At DrupalCon Amsterdam, Behat and Mink architect Konstantin Kudryashov gave a whole new dimension to that. His presentation, Doing Behaviour-Driven Development with Behat, was a straightforward, well-sequenced, and insightful explanation about why automation is so not the point, even though he began by writing automation tools.

Many Drupal shops have found themselves battered in the bizarre project management triangle, where clients push for more and more features within fixed budget often within wildly unrealistic timelines. Based on research conducted by Inviqa, where he works as the BDD Practice Manager, as well as a now-classic study by the Standish group showing that for all that pressure for more and more features, nearly 50% are never used, he makes the following claim:

Your clients don’t care about your scope. Your clients don’t care about their budget. They care about their business. And the reason they try to get so many features on a finite budget is that they don’t believe that you care about their business.

They don’t know how what you do will help them, yet they take a chance. But because they don’t know and they don’t believe, when they come to you they’re really saying, “I don’t know if you can help me or not, but I’m willing to spend this amount of money to see if you can. And I don’t know which of the things you do will really make a difference, so I want everything I can get in hopes that something will help."

The talk focuses on quality vs. quantity in software development and how BDD methodology and tools are intended to drive a quality-first development process.

[embedded content]

The research was interesting, for sure, but for me, not entirely necessary. I already had a big case of confirmation bias.

I entered web development in the mid-90s, the flat-file days before Content Management Systems were popular. I had pretty okay Unix system administration experience and was developing standards-oriented HTML 2.0 skills, but at heart I was we now call a “content strategist.” My formal, pre-technical background is in literature and education, so it makes a certain amount of sense that I entered the tech world content-first.

From the very beginning of the Web, it seemed clear to me that the point of sites was to communicate. There are significant technical considerations about how to support the communication, but ultimately the goals and missions should guide what is built, when it’s built and how it’s constructed.

I emphasized content above all back then, but after my first update of a large site from Drupal 4.7 to 5, I started to explain to new clients that Drupal is free as in puppies, and you’d better plan a maintenance budget. They would need to either trust or understand the kind of ongoing consequences of technical choices. You don’t just take home a new web site, you have to care for it, feed it, make arrangements for its care when you plan to go out of town. Despite good intentions, I had implemented my share of those unused features for a variety of reasons (all of which made sense at the time).

Still, it wasn’t until the late 00s, when I started to interact with Drupal agencies, that I realized achieving business goals and improving organizational communication weren’t even goals for some projects. As Drupal rescue projects began to come in, I realized that if Drupal was free as in puppies, then clients needed to steer clear of the puppy mills. I was feeling that pressure, especially when bidding on a project, to promise more and more features for a site without regard to the consequences - for delivery now and for maintenance later.

At one point, I toyed with addressing that pressure by joining a larger team. I remember interviewing for a job with a Drupal agency that involved immersive on-site consulting with clients. I asked during that interview how much they interfaced with their clients’ business process. My interviewer responded, “We don’t have the luxury of understanding our clients’ business.” I appreciated the honesty and withdrew my application.

I knew the value I offered my client was less about technical prowess and more about listening well to their needs and desires, then translating those into deliverables. That’s what the Behavior Driven Development methodology helps me with - ways to keep the whole team focused so they work on what really matters.

This image is from a great blog write-up by agile coach Dave Moran about where unused features come from, features which lead to considerable baggage as people implement, maintain, and sometimes even port features which are never used.

Sep 24 2012
Sep 24

Drupal’s highly dynamic and modular nature means that many of the central core and contrib subsystems and modules need to maintain a large amount of meta-data.

Rebuilding the data every request would be very expensive, and usually when one part of the data is needed during part of the request, another part will be needed later during the same request. Since just about every request needs to check variable_get(), load entities with fields attached etc., the meta-data needs to be loaded too.

The pattern followed by most subsystems is to put the data into a single large cache item and statically cache it. The more modules you have, the larger these cache items become — since more modules mean more variables, hook_schema() and hook_theme() implementations, etc. And the same happens via configuration with field instances, content types and default views.

This affects many of the central core subsystems — without which it’s impossible to run a Drupal site — as well as some of the most popular contrib modules. The theme, schema, path alias, variables, field API and modules system all have similar issues in core. Views and CCK have similar issues in contrib.

With just a stock Drupal core install, none of this is too noticeable, but once you hit 100 or 200 installed modules, suddenly every request needs to fetch and unserialize() potentially dozens of megabytes of data. Some of the largest cache items like the theme registry can grow too large for MAX_ALLOWED_PACKET or the memcache default slab size. Since the items are statically cached, these caches can easily add 30MB or 40MB to PHP memory usage combined.

The full extent of this problem became apparent when I profiled WebWise Symantec Connect site (Drupal.org case study). Symantec Connect currently runs on Drupal 6, and as a complex site with a lot of social functionality has a reasonably large number of installed modules.

To find the size of these cache items, the simplest method is to profile the request with XHProf. Here’s a detail of the _theme_load_registry() function from Symantec Connect before this issue was dealt with. As you can see from this screenshot, the theme registry is taking up over 3mb of memory usage, and it’s taking over 6ms to pull it from cache.

XHProf screenshot

When reviewing Drupal 7 code it was found that in all cases the problems with oversized cache items were still there.

To tackle this and similar large cache items, I started working on a solution with the following requirements:

Keep the single round trip to the cache backend.
Massively reduce the amount of data loaded on each request.
Avoid excessive cache writes or segmentation.
Fix the performance issue without breaking APIs.

Webwise agreed to fund core patches for the most serious issues; between that funding, my own time, and plenty of Drupal community review we eventually ended up with CacheArray as a new core API - merged into Drupal 8 and also backported to Drupal 7.

CacheArray allows for the following:
Implements ArrayAccess to allow (mostly) backwards compatibility with the vast Arrays of Doom™.
Begins empty and progressively fills in array keys/values are they’re requested.
Caches any newly found array keys at the end of the request with one write to the cache system, so they’re available for the next one.
Items that would have been loaded into the ‘real’ array every request, but are never actually used, skip being loaded, resulting in a much smaller cache item.

For many systems such as the theme registry and the schema cache, much less than half of the data is actually used during most requests to the site. Theme functions may be defined for an admin page which is never visited. Often the schema definition for a table is only required for a handful of tables on a site, for example if drupal_write_record() is used to write to them. By excluding these unused items from the runtime caching, we’re able to significantly reduce the amount of data that needs to be lugged around every request.

Revisiting the XHProf screenshots, let’s look at Connect with the Theme Registry replaced by a CacheArray, we can see that the time spent to fetch the cache item was reduced to 1.7ms and the memory usage to 360kb (although actual site traffic determines how large the cache item will eventually grow).

XHProf screenshot

So far, the following core issues have been fixed in Drupal 8, most of these are already backported to Drupal 7 or are in the process of being so:

Introduce DrupalCacheArray and use it for drupal_get_schema()

Theme registry can grow too large for MySQL max_allowed_packet and memcache default slab size

Be more sparing with locale cache clears

system_list() memory usage

Still in progress (reviews/re-rolls welcome!):

variable_set() should rebuild the variable cache, not just clear it

Avoid slow query for path alias whitelists

And related issues — not using CacheArray but resulting from the same research:

_content_type_info() memory usage improvements (CCK - fixed)
_field_info_collate_fields() memory usage (core, in progress).
views_get_default_view() - race conditions and memory usage.
views_fetch_data() cache item can reach over 10mb in size

Since CacheArray implements ArrayAccess which is only available in PHP 5.2+, it’s unfortunately not possible to backport it to Drupal 6 core. However Tag1 Consulting maintains a fork of Pressflow which contains many of these improvements. Since Pressflow core requires PHP 5.2 these can also be submitted as pull requests for the main distribution.

Apr 26 2012
Apr 26

At Tag1 Consulting we do a lot of work on increasing web site performance, especially around Drupal sites. One of the common tools we use is memcached combined with the Drupal Memcache module. In Drupal, there are a number of different caches which are stored in the (typically MySQL) database by default. This is good for performance as it cuts down on potentially large/slow SQL queries and PHP execution needed to display content on a site. The Drupal Memcache module allows you to configure some or all of those caches to be stored in memcached instead of MySQL, typically these cache gets/puts in memcache are much faster than they would be in MySQL, and at the same time it decreases work load on the database server. This is all great for performance, but it involves setting up an additional service (memcached) as well as adding a PHP extension in order to communicate with memcached. I've seen a number of guides on how to install these things on Fedora or CentOS, but so many of them are out-dated or give instructions which I wouldn't suggest such as building things from source, installing with the 'pecl' command (not great on a package based system), or using various external yum repositories (some of which don't mix well with the standard repos). What follows is my suggested method for installing these needed dependencies in order to use memcached with Drupal, though the same process should be valid for any other PHP script using memcache.

PECL Packages

For the Drupal Memcache module, either the PECL memcache or PECL memcached (note the 'd'!) extensions can be used. While PECL memcached is newer and has some additional features, PECL memcache (no 'd'!) tends to be better tested and supported, at least for the Drupal Memcache module. Yes, the PECL extension names are HORRIBLE and very confusing to newcomers! I almost always use the PECL memcache extension because I've had some strange behavior in the past using the memcached extension; likely those problems are fixed now, but it's become a habit and personal preference to use the memcache extension.

Installing and Configuring memcached

The first step is to get memcached installed and configured. CentOS 5 and 6 both include memcached in the base package repo, as do all recent Fedora releases. To install memcached is simply a matter of:
# yum install memcached

Generally, unless you really know what you're doing, the only configuration option you'll need to change is the amount of memory to allocate to memcached. The default is 64MB. That may be enough for small sites, but for larger sites you will likely be using multiple gigabytes. It's hard to recommend a standard size to use as it will vary by a large amount based on the site. If you have a "big" site, I'd say start at 512MB or 1GB; if you have a smaller site you might leave the default, or just bump it to 512MB anyway if you have plenty of RAM on the server. Once it's running, you can watch the memory usage and look for evictions (removal of a cache item once the cache is full) to see if you might want to increase the memory allocation.

On all Fedora / CentOS memcached packages, the configuration file is stored in /etc/sysconfig/memcached. By default, it looks like this:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""

To increase the memory allocation, adjust the CACHESIZE setting to the number of MB you want memcached to use.

If you are running memcached locally on your web server (and only have one web server), then I strongly recommend you also add an option for memcached to listen only on your loopback interface (localhost). Whether or not you make that change, please consider locking down the memcached port(s) with a firewall. In order to listen only on the 127.0.0.1 interface, you can change the OPTIONS line to the following:

OPTIONS="-l 127.0.0.1"

See the memcached man page for more info on that or any other settings.

Once you have installed memcached and updated the configuration, you can start it up and configure it to start on boot:

# service memcached start
# chkconfig memcached on

CentOS / RHEL PECL Module Install

Fedora

If you are on Fedora and using PHP from the base repo in the distribution, then installation of the PECL extension is easy. Just use yum to install whichever PECL extension you choose:

# yum install php-pecl-memcache

Or

# yum install php-pecl-memcached

CentOS 5 / RHEL 5

CentOS and RHEL can be a bit more complicated, especially on EL5 which ships with PHP 5.1.x, which is too old for most people. Here are the options I'd suggest for EL5:

  • If you are OK using the PHP provided with EL5, then you can get the PECL extensions from EPEL. Once you've enabled the EPEL repository (instructions), you can install either PECL extension by using the same yum commands outlined above in the Fedora section.
  • If you want to use PHP 5.2 or PHP 5.3 with EL5, I suggest using the IUS repositories (IUS repo instructions). Note that IUS provides the PECL memcache extension, but not the PECL memcached extension. Based on which PHP version you decide to use, you can install the PECL memcache extension with either:

    # yum install php52-pecl-memcache

    Or

    # yum install php53u-pecl-memcache

CentOS 6 / RHEL 6

EL6 ships with PHP 5.3, though it is an older version than is available for EL6 at IUS. If you are using the OS-provided PHP package, then you can install the PECL memcache extension from the base OS repo. If you want the PECL memcached extension, it is not in the base OS repo, but is available in EPEL. See the instructions linked from the CentOS 5 section above if you need to enable the EPEL repo.

# yum install php-pecl-memcache

Or, enable EPEL and then run:

# yum install php-pecl-memcached

As with EL5, some people running EL6 will also want the latest PHP packages and can get them from the IUS repositories. If you are running PHP from IUS under EL6, then you can install the PECL memcache extension with:

# yum install php53u-pecl-memcache

Similar to EL5, the IUS repo for EL6 does not include the PECL memcached module.

PECL Memcache Configuration

If you are using PECL memcache extension and will be using the clustering option of the Drupal Memcache module which utilizes multiple memcached instances, then it is important to set the hash strategy to "consistent" in the memcache extension configuration. Edit /etc/php.d/memcache.ini and set (or un-comment) the following line:

memcache.hash_strategy=consistent

If you are using the PECL memcached module, this configuration is done at the application level (e.g. in your Drupal settings.php).

Once you've installed the PECL memcache (or memcached) extension, you will need to reload httpd in order for PHP to see the new extension. You'll also need to reload httpd whenever you change the memcache.ini configuration file.

# service httpd reload

SELinux

If you have SELinux enabled (you should!), I have an older blog post with instructions on configuring SELinux for Drupal.

That's it, you're now good to go with PHP and memcache!

Dec 22 2011
Dec 22

I see a lot of people coming by #centos and similar channels asking for help when they’re experiencing a problem with their Linux system. It amazes me how many people describe their problem, and then say something along the lines of, “and I disabled SELinux...”. Most of the time SELinux has nothing to do with the problem, and if SELinux is the cause of the problem, why would you throw out the extra security by disabling it completely rather than configuring it to work with your application? This may have made sense in the Fedora 3 days when selinux settings and tools weren’t quite as fleshed out, but the tools and the default SELinux policy have come a long way since then, and it’s very worthwhile to spend a little time to understand how to configure SELinux instead of reflexively disabling it. In this post, I’m going to describe some useful tools for SELinux and walk through how to configure SELinux to work when setting up a Drupal web site using a local memcached server and a remote MySQL database server -- a pretty common setup for sites which receive a fair amount of traffic.

This is by no means a comprehensive guide to SELinux; there are many of those already!
http://wiki.centos.org/HowTos/SELinux
http://fedoraproject.org/wiki/SELinux/Understanding
http://fedoraproject.org/wiki/SELinux/Troubleshooting

Too Long; Didn’t Read Version

If you’re in a hurry to figure out how to configure SELinux for this particular type of setup, on CentOS 6, you should be able to use the following two commands to get things working with SELinux:
# setsebool -P httpd_can_network_connect_db 1
# setsebool -P httpd_can_network_memcache 1

Note that if you have files existing somewhere on your server and you move them to the webroot rather than untar them there directly, you may end up with SELinux file contexts set incorrectly on them which will likely deny access to apache to read those files. If you are having a related problem, you’ll see something like this in your /var/log/audit/audit.log:
type=AVC msg=audit(1324359816.779:66): avc: denied { getattr } for pid=3872 comm="httpd" path="/var/www/html/index.php" dev=dm-0 ino=549169 scontext=root:system_r:httpd_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

You can solve this by resetting the webroot to its default file context using the restorecon command:
# restorecon -rv /var/www/html

Server Overview

I’m going to start with a CentOS 6 system configured with SELinux in targeted mode, which is the default configuration. I’m going to be using httpd, memcached, and PHP from the CentOS base repos, though the configuration wouldn’t change if you were to use the IUS PHP packages. MySQL will be running on a remote server which gives improved performance, but means a bit of additional SELinux configuration to allow httpd to talk to a remote MySQL server. I’ll be using Drupal 7 in this example, though this should apply to Drupal 6 as well without any changes.

Initial Setup

Here we will setup some prerequisites for the website. If you already have a website setup you can skip this section.

We will be using tools such as audit2allow which is part of the policycoreutils-python package. I believe this is typically installed by default, but if you did a minimal install you may not have it.
# yum install policycoreutils-python

Install the needed apache httpd, php, and memcached packages:
# yum install php php-pecl-apc php-mbstring php-mysql php-pecl-memcache php-gd php-xml httpd memcached

Startup memcached. The CentOS 6 default configuration for memcached only listens on 127.0.0.1, this is great for our testing purposes. The default of 64M of RAM may not be enough for a production server, but for this test it will be plenty. We’ll just start up the service without changing any configuration values:
# service memcached start

Startup httpd. You may have already configured apache for your needs, if not, the default config should be enough for the site we’ll be testing.
# service httpd start

If you are using a firewall, then you need to allow at least port 80 through so that you can access the website -- I won’t get into that configuration here.

Install Drupal. I’ll be using the latest Drupal 7 version (7.9 as of this writing). Direct link: http://ftp.drupal.org/files/projects/drupal-7.9.tar.gz
Download the tarball, and expand it to the apache web root. I also use the --strip-components=1 argument to strip off the top level directory, otherwise it would expand into /var/www/html/drupal-7.9/
# tar zxf drupal-7.9.tar.gz -C /var/www/html --strip-components=1

Also, we need to get the Drupal site ready for install by creating a settings.php file writable by apache, and also create a default files directory which apache can write to.
# cd /var/www/html/sites/default/
# cp default.settings.php settings.php
# chgrp apache settings.php && chmod 660 settings.php
# install -d -m 775 -g apache files

Setup a database and database user on your MySQL server for Drupal. This would be something like this:
mysql> CREATE DATABASE drupal;
mysql> GRANT ALL ON drupal.* TO [email protected] IDENTIFIED BY 'somepassword';

Test this out by using the mysql command line tool on the web host.
# mysql -u drupal_rw -p -h drupal

That should connect you to the remote MySQL server. Be sure that is working before you proceed.

Now for the Fun Stuff

If you visit your new Drupal site at http://your-hostname-here, you’ll be presented with the Drupal installation page. Click ahead a few times, setup your DB info on the Database Configuration page -- you need to expand “Advanced Options” to get to the hostname field since it assumes localhost. When you click the button to proceed, you’ll probably get an unexpected error that it can’t connect to your database -- this is SELinux doing its best to protect you!

Allowing httpd to Connect to a Remote Database

So what just happened? We know the database was setup properly to allow access from the remote web host, but Drupal is complaining that it can’t connect. First, you can look in /var/log/audit/audit.log which is where SELinux will log access denials. If you grep for ‘httpd’ in the log, you’ll see something like the following:
# grep httpd /var/log/audit/audit.log
type=AVC msg=audit(1322708342.967:16804): avc: denied { name_connect } for pid=2724 comm="httpd" dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

That is telling you, in SELinux giberish language, that the httpd process was denied access to connect to a remote MySQL port. For a better explanation of the denial and some potential fixes, we can use the ‘audit2why’ utility:
# grep httpd /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1322708342.967:16804): avc: denied { name_connect } for pid=2724 comm="httpd" dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

Was caused by:
One of the following booleans was set incorrectly.
Description:
Allow HTTPD scripts and modules to connect to the network using TCP.

Allow access by executing:
# setsebool -P httpd_can_network_connect 1
Description:
Allow HTTPD scripts and modules to connect to databases over the network.

Allow access by executing:
# setsebool -P httpd_can_network_connect_db 1

audit2why will analyze the denial message you give it and potentially explain ways to correct it if it is something you would like to allow. In this case, there are two built in SELinux boolean settings that could be enabled for this to work. One of them, httpd_can_network_connect, will allow httpd to connect to anything on the network. This might be useful in some cases, but is not very specific. The better option in this case is to enable httpd_can_network_connect_db which limits httpd generated network connections to only database traffic. Run the following command to enable that setting:
# setsebool -P httpd_can_network_connect_db 1

It will take a few seconds and not output anything. Once that completes, go back to the Drupal install page, verify the database connection info, and click on the button to continue. Now it should connect to the database successfully and proceed through the installation. Once it finishes, you can disable apache write access to the settings.php file:
# chmod 640 /var/www/html/sites/default/settings.php

Then fill out the rest of the information to complete the installation.

Allowing httpd to connect to a memcached server

Now we want to setup Drupal to use memcached instead of storing cache information in MySQL. You’ll need to download and install the Drupal memcache module available here: http://drupal.org/project/memcache
Install that into your Drupal installation, and add the appropriate entries into settings.php. For this site, I did that with the following:
# mkdir /var/www/html/sites/default/modules
# tar zxf memcache-7.x-1.0-rc2.tar.gz -C /var/www/html/sites/default/modules

Then edit settings.php and add the following two lines:
$conf['cache_backends'][] = 'sites/default/modules/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';

Now if you reload your site in your web browser, you’ll likely see a bunch of memcache errors -- just what you wanted! I bet it’s SELinux at it again! Check out /var/log/audit/audit.log again and you’ll see something like:
type=AVC msg=audit(1322710172.987:16882): avc: denied { name_connect } for pid=2721 comm="httpd" dest=11211 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

That’s very similar to the last message, but this one is for a memcache port. What does audit2why have to say?
# grep -m 1 memcache /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1322710172.796:16830): avc: denied { name_connect } for pid=2721 comm="httpd" dest=11211 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

Was caused by:
One of the following booleans was set incorrectly.
Description:
Allow httpd to act as a relay

Allow access by executing:
# setsebool -P httpd_can_network_relay 1
Description:
Allow httpd to connect to memcache server

Allow access by executing:
# setsebool -P httpd_can_network_memcache 1
Description:
Allow HTTPD scripts and modules to connect to the network using TCP.

Allow access by executing:
# setsebool -P httpd_can_network_connect 1

Again, audit2why gives us a number of options to fix this. The best bet is to go with the smallest and most presice change for our needs. In this case there’s another perfect fit: httpd_can_network_memcache. Enable that boolean with the following command:
# setsebool -P httpd_can_network_memcache 1

Success! Now httpd can talk to memcache. Reload your site a couple of times and you should no longer see any memcache errors. You can be sure that Drupal is caching in memcache by connecting to the memcache CLI (telnet localhost 11211) and typing ‘stats’. You should see some number greater than 0 for ‘get_hits’ and for ‘bytes’.

What are all these booleans anyway?

Now we’ve used a couple SELinux booleans to allow httpd to connect to memcached and MySQL. You can see a full list of booleans which you can control by using the command ‘getsebool -a’. They are basically a preset way for you to allow/deny certain pre-defined access controls.

Restoring default file contexts

As I mentioned briefly in the ‘TL;DR’ section, another common problem people experience is with file contexts. If you follow my instructions exactly, you won’t have this problem because we untar the Drupal files directly into the webroot, so they will inherit the default file context for /var/www/html. If, however, you were to untar the files in your home directory, and then use ‘mv’ or ‘cp’ to place them in /var/www/html, they will maintain the user_home_t context which apache won’t be able to read by default. If this is happening to you, you will see the file denials logged in /var/log/audit/audit.log -- something like this:
type=AVC msg=audit(1324359816.779:66): avc: denied { getattr } for pid=3872 comm="httpd" path="/var/www/html/index.php" dev=dm-0 ino=549169 scontext=root:system_r:httpd_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

The solution in this case is to use restorecon to reset the file contexts back to normal:
# restorecon -rv /var/www/html

Update: It was noted that I should also mention another tool for debugging audit messages, 'sealert'. This is provided in the setroubleshoot-server package and will also read in the audit log, similar to what I described with audit2why.
# sealert -a /var/log/audit/audit.log

Oct 28 2011
ChX
Oct 28

DISQUS is a popular "social commenting" platform. It is integrated with many hosted blog platforms and open source CMSes, including Drupal. A client of ours exported the comments from their old Wordpress blog and then imported them into DISQUS. The problem was that the comments were showing up in the DISQUS dashboard, however, when you clicked their corresponding URLs, these imported comments did not appear in Drupal. While the Drupal module looks for comments on the node/X URLs, DISQUS was storing them at the old Wordpress URL which were implemented as path aliases in this case.

Fixing this was relatively easy as DISQUS has a mapping tool available. You can download the URLs it knows about and then upload a very simple CSV file to change the URLs. To generate the CSV file after you have saved the Disqus URLs into disqus-urls.csv, just run the following script with drush php-script disqusmap.php > map:

<?php
$base = 'http://www.example.com/';
$n = strlen($base);
foreach (file('disqus-urls.csv') as $url) {
  $path = substr(trim($url), $n);
  $source = drupal_get_normal_path($path);
  if ($source != $path) {
    echo "$base$path, $base$source\n";
  }
}

After you have uploaded the file there is nothing to do but wait. As far as I can see, there are no logs, progress reports, or anything that provides status. In this client's case re-mapping worked to solve their missing comments and so hopefully it will work for you as well.

Oct 25 2011
Oct 25

Tag1 Consulting is sponsoring my work on Drupal.org Infrastructure. What this means is that instead of working on drupal.org whenever I can, I get to spend 20 paid hours per week on drupal.org infrastructure. In return for this, I have agreed to write a blog entry per month describing some of my work in detail. These will be entries covering security, performance, high-availability configuration and anything else interesting in my work on drupal.org. Hopefully these will be useful.

I look forward to spending more time securing and improving the performance of drupal.org and would like to thank everyone at Tag1 and our clients for this opportunity.

About Narayan Newton
Partner/CTO
Narayan is a co-owner of Tag1 Consulting, who joined the team in 2008. He was introduced to Drupal during his tenure at the Oregon State University Open Source Lab where he was the co-lead system administrator, and served as their Database Administrator of over 180 MySQL databases, the Freenode Server administrator, and the Drupal.org Server Administrator. He is a permanent member of the Drupal Association as their Systems Coordinator, and is a member of the Drupal.org infrastructure team. He is also a co-maintainer of the Pressflow high performance Drupal distribution.

Outside of Drupal, Narayan has been deeply involved in the FreeBSD, Gentoo and Slackware communities. He was a contributing editor and system administrator of a popular Linux community website. As a Google Summer of Code student he developed the KDE front-end to the SUSE System Update server, and became a KDE core committer in 2007. He also ported and updated the XML-RPC system to KDE3 and KDE4.

More recently, he acted as infrastructure lead for the examiner.com 2.0 re-launch and infrastructure lead for the drupal.org redesign launch. Narayan is currently Chief Technology Officer at Tag1 Consulting.

Jul 09 2011
Jul 09

During performance and scalability reviews of sites, we regularly find ourselves submitting patches to contrib modules and core to resolve performance issues.

Most large Drupal installations we work with use a variation of this workflow to track patches:

  • Upload the patch to an issue on Drupal.org if it's not already there
  • Add the patch to a /patches folder in revision control
  • Document what the patch does, the Drupal.org nid, and a reference to the ticket in the client's issue tracker in /patches/README.txt
  • Apply the patch to the code base

(Drush make seems like a popular alternative to this, but most of the sites we work on predate that workflow so I've not actually tried it).

Applying patches is usually straightforward enough if only changing PHP code, however you can run into trouble with this method if you're changing the schema of a database table. In our case, 99% of the time when we need to make schema changes we're adding or modifying indexes.

Recently on a client site, I found they'd applied a core patch adding an index to the comment module (this one: http://drupal.org/node/336483). I discovered this when checking their comment module schema while reviewing another core patch, which adds a different index to the same table (http://drupal.org/node/243093).

With schema changes, the ordering of hook_update_N () matters greatly.

If you apply a patch that adds an update to a module, let's say comment_update_7004(), you have no guarantee that your patch will get committed before anyone else adds comment_module_7004() to that module. In this case with two patches competing for the same update number, that seems relatively likely.

Drupal will not run the same update twice, so this risks missing schema updates to the module entirely. This could put you in the very position of having to run those updates manually yourself, or work on your own upgrade path to get your site back into sync, not fun.

To avoid this, recently we've been using this workflow:

  • Upload the patch to an issue Drupal.org
  • Add a hook_schema_alter() in a custom module, with a @see to the Drupal.org issue nid
  • Add a custom update handler: mycustommodule_update_N() to add the index

This will avoid running into conflicts with update numbering, so that when other schema changes are added to the module, you'll still have those updates run.

Health warning
While this approach will ensure that you don't skip any core or contrib updates, it will not prevent updates being run twice - once from your custom module, once from the module itself.

If you're only adding indexes this is usually OK, but if in doubt, http://drupal.org/project/schema can show you any discrepancies. Drupal 7 has the handy db_index_exists() function, this should help when resolving index mismatches.

It should go without saying that you should test this all on dev and staging servers before making any changes to your live site. Additionally, indexes can usually be added or removed with less impact than other schema changes - other kinds of changes can get you into a lot more trouble.

Mar 03 2011
Mar 03

We're excited to have 7 members of the Tag1 Consulting team attending the DrupalCon in Chicago next week. We are all looking forward to participating in another fantastic Drupal Conference. If you've not already bought your tickets, it's still not too late! Don't miss this one!

In Chicago, Tag1 will be passing out copies of Drupal Watchdog, participating in training courses, sessions, and BoFs, and generally enjoying the two-way sharing of knowledge with our fellow Drupal developers and users.

We will be sporting fancy new t-shirts in Chicago designed by Codename Design, incorporating our new logo designed by Candice Gansen. Find a Tag1 team member and ask us about Drupal performance and scalability, and you might get one yourself! :)

Stephanie modeling Tag1 t-shirt
Jeremy Andrews
Jeremy, Tag1's CEO and founding partner, spent much of this year focused on getting Drupal Watchdog issue #1 out the door. He'll be circulating the DrupalCon looking for feedback that will impact future issues of Drupal's only dedicated print magazine.

Narayan Newton
Narayan, Tag1's CTO and partner, will be one of the teachers at the Drupal Performance and Scalability pre-conference training course. As a member of the Drupal infrastructure team he was heavily involved in the recent CVS -> Git migration and the drupal.org redesign.

Rudy Grigar
Rudy, Tag1's Director of Operations, is part of the Drupal infrastructure team and was heavily involved in the Drupal Redesign effort. You can read more about the infrastructure improvements required for the Redesign effort in Rudy's article in Drupal Watchdog issue #1.

Other members of the Tag1 team that will be present at the DrupalCon include Brandon Bergren, Nat Catchpole, Jason Markantes and Bob Williams. With 77% of our virtual office physically attending the DrupalCon, we'll return to our normal schedule and availability the following week, March 14th.

Feb 25 2011
Feb 25

Drupal Watchdog Volume 1 Issue 1 is on its way to Chicago to greet all attendees of the Chicago DrupalCon. Those that subscribed should receive the first issue within the next two weeks, possibly longer for international subscribers. This issue was made possible by 23 authors, 7 editors, 2 designers, and a 3-person layout team. Its 80 pages were printed on FSC Certified paper stock with soy inks.

 Chicago DrupalCon

I'll be in Chicago attending the DrupalCon, and eager to get feedback from other attendees. Questions I'm looking to get answered include: What did you most enjoy about Drupal Watchdog Issue 1? What do we most need to improve? Is there enough interest to publish quarterly in 2012? What digital formats are people most interested in? And, what topics would you be most interested in seeing covered in Issue #2?

The printer sent me a snapshot of some finished copies of Drupal Watchdog issue #1. The photo was taken from a phone and is not high quality, but it gives you an early sneak peak as to what you can expect:

 Drupal Watchdog

The copies of Drupal Watchdog that people find in their "tote bag" in Chicago contain an exclusive bonus: Examiner.com put together a full-color 28-page insert titled "#XMNR:CHI The Insider Guide To What To Do And Where To Do It". The insert suggests where to eat, where to drink, and what to see while you're in Chicago. It even includes a map of the area around the Drupal Towers and a list of food within easy walking distance.

Drupal Watchdog is a very content-rich magazine. Our first issue is 80 pages long and includes 7 feature-length articles, 13 standard-length articles, and 6 columns. Read on for the complete table of contents, it's quite likely you'll recognize a great number of our contributors! As a print magazine, Drupal Watchdog strives to bring you relevant information that remains useful to you for a long time to come.

Table of Contents

Features:

  • What’s New In Views 3
    Highlighting the major new features in Views 3. By Lynette Miles.
  • Introduction to the Field UI
    How to add new fields to nodes, comments and users. Morbus Iff and Kevin Hemenway.
  • An In-Depth Interview With Dries
    Learn from the creator of Drupal how it evolved over the past decade from a simple message board, and where he sees it going over the next 10 years. By Jeremy Andrews.
  • The Future of Drupal: Drupal 8 And Beyond
    Challenges and opportunities for the next version of Drupal. By Karoly Negyesi.
  • PSD To Drupal 7 Theme
    Learn to create a theme for Drupal 7 starting from a Photoshop image. By Emma Jane Hogbin.
  • Performance And Scalability In Drupal 7
    A comprehensive overview of the performance and scalability improvements in Drupal 7. By Nathaniel Catchpole.
  • Design Patterns
    Understanding design patterns for solving problems that come up over and over again. By Larry Garfield.

Articles & Columns:

  • Module Selection In The Wild
    Tips for choosing the right contrib modules. Article by Jeffrey A. “jam” McGuire.
  • Seven Modules You'll Be Using Next
    A look at seven lesser known modules for Drupal 7. Article by Michael Anello.
  • What Is This Views thing, And Why do I Need It?
    An introduction to Drupal's most popular contrib module. Article by Lynette Miles.
  • Really Important Views-related Modules
    A brief look at some useful enhancements to Views. Article by Lynette Miles.
  • Attachments: The No-Muss No-Fuss Views Upgrade
    Use multiple-display views to add functionality to your website. Article by J. Ayen Green.
  • Building A Drupal Commerce Product Display
    A step-by-step guide to displaying products in Drupal 7. Article by Ryan Szrama.
  • Drupal 7 Books: Valuable Resources For Site Builders
    A survey of all the books currently available about Drupal 7. Article by Michael J. Ross.
  • The Power of a Latin Community
    Coverage of Drupal in Latin America. Article by Paulino Michelazzo.
  • Improve Your Site's Effectiveness
    Optimizing your website with Google Website Optimizer. Article by Nick Schoonens.
  • How To Implement A New Feature In Views 3
    Step-by-step guide to writing a custom Views pager. Column by Earl Miles.
  • Drupal: Secure by Default, Inconsistent by Nature
    A look at the security choices in Drupal's APIs. Column by Greg James Knaddison.
  • Contributor's Corner: Drupal’s Help System
    Drupal 7's new help system standards. Column by Ariane Khachatourians.
  • Making Drupal A Go-to Platform For Building Mobile Applications
    The state of mobile support in Drupal 7. Column by Dries Buytaert.
  • The Many Layers of Caching Drupal
    How caching can improve the performance of your website. Article by Eric Mandel.
  • The Infrastructure Behind a Redesigned Drupal.org
    Keeping drupal.org and contributors working efficiently. Article by Rudy Grigar.
  • Release Quality "On The Truck"
    How Examiner.com achieves 99.97% uptime with Drupal 7. Article by Doug Green.
  • Drupal in Context
    Getting paid to work with Drupal. Column by Tom Geller.
  • The Farmisht Freelancer
    Questions not to forget when taking on contract work. Column by J. Ayen Green.
  • The Drupal Association: Stewards of the Drop
    How the Drupal Association benefits the Drupal community. Article by George DeMet.
Jan 20 2011
Jan 20

We are pleased to announce Drupal Watchdog as the name of our print magazine dedicated to Drupal! A lot has been happening since our previous announcement, with an impressive array of authors committing to contributing content for our premiere issue. The first issue will focus primarily on all things Drupal 7, and will cover beginner, intermediate and advanced topics. Future updates with more detail on the article content appearing in issue #1 will be found on the magazine's website at http://drupalwatchdog.com/.

The name, Drupal Watchdog, is in part a reference to the watchdog() function used to report on what is going on in Drupal websites. Our goal is that Drupal Watchdog Magazine be a fantastic resource for tracking what is going on in the Drupal World.

Some of the feature content we have planned in Issue #1 includes (this is subject to change):

  • From Photoshop Image To Drupal 7 Theme
  • An In-Depth Interview With Dries Buytaert
  • What's New In Views 3
  • Writing Drupal 7 Modules
  • Think Drupal: The Site Planning Formula
  • Building Websites With Views
  • New Performance Features In Drupal 7
  • Drupal 7 Content Administration
  • Where Are We Going In Drupal 8?

Beyond the featured content, there are many other articles and columns filling this 64-page glossy print magazine dedicated to Drupal. Look for Drupal Watchdog in your Chicago DrupalCon tote bag.

We continue to look for contributors and advertisers, but print deadlines are fast approaching so don't wait to reach out!

Jan 07 2011
Jan 07

We are looking to pay talented authors to contribute articles to a print Drupal magazine. Issue #1 will be distributed at the Chicago Drupalcon, and will focus on Drupal 7. The newstand-quality magazine will be a content-rich resource that aims to both introduce Drupal to a wider audience, and be an invaluable resource to existing Drupal users and developers.

Additional information will be posted online soon. However, in the interests of participating in the worlwide celebration of the release of Drupal 7, and in order to give authors the maximum time allowed by tight scheduling constraints, we're making our first public announcement today.

We will publish only two issues of the Drupal print magazine in 2011, timed for availability at Drupalcon Chicago and Drupalcon London.

Read on for key information on how to contribute, and help us spread the word!

Deadlines

By January 14th, potential authors will need to submit a one paragraph synopsis of their proposed contribution, and a brief statement as to why they are an authority on this subject. Editors will respond indicating whether or not the propsal is likely to be included in the magazine, and how it will be categorized.

By January 31st, authors must submit a completed rough draft of all contributions, including a description of any supporting content (pictures, diagrams, tables, etc). A payment rate will be established and editors will guide authors on getting the contribution into a publishable state.

February 11th is the absolute latest date to submit completed contributions and all supporting content to be included in issue #1. Editors will complete the editorial process without the review of the contributing author.

Payments

The Drupal magazine will be comprised of three primary types of contributions: articles, columns, and features. The amount to be paid depends on the type of contribution, the length and quality of the writing, the quality of supporting content, and the timeliness of the contribution. An estimated payment will be provided upon receipt of the completed rough draft. A final payment will be determined upon formal acceptance of the completed contribution.

The following guidelines will help you to target your contribution, and to estimate potential earnings. All payments will be issued in US dollars, either via check or PayPal.

We are seeking approximately 30 articles for inclusion in issue #1, ranging in length from 250 to 1,200 words. Payment for articles will range from $88 to $420.

We will publish approximately 8 feature articles in issue #1, ranging in length from 1,800 words to 3,000 words. Payment for feature articles will range from $900 to $1,500.

We will require around 10 columnists to commit to writing columns for both issue #1 and issue #2. Columns should have a general theme and can be opinion based or of a technical nature. They will range from 800 to 1,200 words, with payment ranging from $440 to $660 per issue.

Contract and Copyright

We do not guarantee to purchase all articles submitted. If we do purchase an article, payment is for first rights, compilation rights, and translation rights. We are under no obligation to publish articles we purchase, and we do not pay kill fees.

When you submit an article, you agree to not submit the article anywhere else unless 1) you explicitly notify us in writing that you have withdrawn your proposal and you then receive a written (emailed) confirmation from Tag1 Publishing freeing you from this contract, or 2) we reject your article for inclusion in this magazine. We are only unable to approve requests for withdrawing articles if they are already scheduled for publication.

By submitting an article you are agreeing to allow us to use your article in an unlimited number of compilations and translations in any medium. Two weeks after your article has been published and distributed either online or in the print magazine, you are free to use it for any other purpose, as you retain the copyright on your article.

Content

The first issue will primarily focus on Drupal 7, however we will also consider articles applicable to Drupal 6 and future releases (aka Drupal 8). We are targeting a wide audience, thus we are looking for both technical and non-technical content. Submissions can range from basic introductory concepts and high level howto guides, to highly technical details on what is going on "under the hood". Articles must in be in some way relevant to Drupal and/or Drupal users.

We will be looking for feature articles covering what's new and how to use all the features in Drupal 7 core, as well as the most popular contributed modules such as Views and CCK/Fields.

Please contact [email protected] with all questions and article proposals.

Distribution

We will post announcements as we open up new channels of distribution for the Drupal magazine. Issue #1 will be available at the Chicago Drupalcon, and annual subscriptions will soon be available online.

Advertising

If you are interested in advertising in the #1 issue of the Drupal magazine, please look for an additional announcement soon. You may also send queries to [email protected]. A media kit including pricing information will be made available soon.

Magazine name

Announcement of the official name of this Drupal oriented magazine is pending the completion of legal contracts and technical details.

More Information

This effort is brought to you by Tag1 Publishing, and made possible by Tag1 Consulting, Inc. A portion of net profits will be donated to the Drupal association. Watch for additional announcements here. If looking for more information, please contact Jeremy Andrews at [email protected], or 877-875-8824 x100.

Dec 13 2010
Dec 13

While Drupal's performance is always of interest, it has a hard time defending itself against the features people want to add.

There are different ways to address this, but the "less features" approach is usually not defensible.

To defend itself from the feature onslaught, Drupal tries to load as few lines of PHP code as possible, which helps to increase performance. A PHP opcode cache (such as APC) helps even more and points the way to where further improvements can be made: outside of conventional PHP.

One idea that has come up several times is to create a Drupal-specific PHP extension that will reimplement some of Drupal's often used functions in C. Such an extension exists. It implements Drupal's check_plain and Drupal 7's drupal_static. One of our goals is to examine the gains brought by this approach.

Another very interesting approach is to remove PHP entirely from the code that is actually executed by the webserver. There have been a number of attempts over the years, but one can't say that any of them has found wide adoption yet.

Hiphop is a recent addition to this group of compilers, it was released by Facebook and is used in production by them to reduce load across their infrastructure. This latter aspect makes it interesting to look into and see if it can be used with Drupal. Essentially, Hiphop translates your PHP code to C++ which is then compiled to a binary.

Besides the actual performance improvements, one should not disregard the total effort required to add and keep hiphop or a PHP extension in the toolchain:

To use the Drupal PHP extension, you only need a minimal patch to Drupal core. You then need to compile the extension and install it, which is also rather easy, but you will need to find a way to keep track of the extension and make sure that it will be available e.g. on new webservers you set up. However, in order to increase the use of the extension, you'll need to recreate all the Drupal functions that you have identified as being costly. Should there be a change in the core Drupal function, you will need to update your C reimplementation. On the other hand, some internal Drupal functions don't change all that much and the C code can be re-used across several Drupal versions.

To use hiphop, you need to first install a whole lot of libraries that aren't usually installed on a webserver. To make this easier, Tag1 consulting has developed HipHop RPMs for CentOS. There are also Debian packages maintained elsewhere. After you have successfully installed hiphop, you need to compile your PHP application. This is rather easy if you assemble a list of files that make up your PHP application and tell hiphop to compile all of them. The compilation will take quite a bit of time, but will succeed in the end if it is compatible. Drupal isn't completely compatible with hiphop. For Drupal 6 there is a small patchset on drupal.org which replaces calls to preg_replace(). Drupal 7 currently does compile with hiphop, but there is a bug in hiphop's implementation of the MySQL PDO layer that will not allow you to actually run it.

Since the resulting hiphop binary is not a PHP application anymore, there will be subtle differences in behavior. One example is Drupal's way to enable and disable modules: If you disable a module, the file will not be read in by PHP and thus it's functions will not be available. If you disable a module in hiphop compiled Drupal, you can still switch the status entry in the system table, but the functions of the module will still be available. You thus need to make sure you only compile in the modules you actually want to run to exclude unforeseen consequences.

We have compared Drupal's performance for a total of ten setups:

A) Drupal 6

1) Drupal running under PHP as an Apache module

2) Same as 1), with the Drupal PHP extension

3) Same as 1) with APC opcode cache

4) Same as 2) with APC opcode cache

5) Drupal compiled with Hiphop

6) Drupal compiled with Hiphop, only required files

B) Drupal 7

1) - 4) as in A)

All installations will use the same database with the D7 version upgraded from the D6 version. The database as a few thousand nodes, users, and comments, as well as taxonomy terms. The actual content composition isn't very important since we are interested in PHP and not MySQL performance.

All measurements are done with a modified version of siege. The modification is to increase the precision of the measurements (or rather the amount of digits written to the results file).

All the final measurements where done on a rackspace virtual machine with 2G RAM and four cores.

We have done tests with two pages, a single node with comments, and the forum overview page. All requests were done as an anonymous user with the page cache disabled. This is appropriate as we are only interested in raw php performance. We've requested several thousand pages for each test, using single concurrency.

Evaluating the tests proved much more difficult than was anticipated. As a standard procedure, most people are happy to specify the average of the measured quantity. More diligent people then also specify the standard deviation as a measure for the statistical significance of the result as is appropriate for a normal distribution. On closer examination it was observed that the resulting distribution of measurements rather resembles a mixture of skewed distributions and thus this procedure isn't appropriate.

Ideally, one would try to find out how to do one or more of the following:

  • Find reasons for the appearance of a mixture of distributions
  • Suppress some of these reasons for the existence of this mixture
  • Find a mathematical description for the mixture
  • Find a suitable model to express the statistical error for this mixture

While interesting, all the above is difficult and exceeds the experience of the experimenter with hard-core statistics. I can make the raw data available to somebody who has more of that.

We therefor do the following: the one-sigma boundary of the normal distribution marks the area where 68% of all results of the experiment can be found. We find a similar boundary by computing the same 68% threshold from the result of our data and give the resulting difference to the mean as an error estimate. Since we have taken a significant number of measurements, this should not be too far off. Regardless, it needs to be regarded with some caution.

So, here's the graphics of the evaluation of the main forum page.

There's a PDF of the graphics too.

The result is obviously, that Hiphop has by far the most advantage over a "normal", e.g. PHP+APC Drupal, install, a whopping 30%. Also, the gains from the PHP extension in any case are rather minor (2-3%).

An additional result is that sadly Drupal 7 is much slower (60%), at least for this page.

Now, what's the conclusion? The conclusion is that Hiphop can give you gains not easily possible with other methods. Does that mean everybody should run it? That depends on the effort you are willing to put into it. For Drupal 6, you would need to check if Drupal behaves as Drupal should. For this it would be nice to have unit tests. Unfortunately, there aren't many. Drupal 7 has good coverage with unit tests, but as explained Hiphop needs to be fixed to run it properly.

The big advantage of Hiphop vs the PHP extension can be easily explained by the fact that the extension only translates a small part of Drupal to a high-performance language, whereas Hiphop does so for the complete code base. You can turn this argument around and say that a 2-3% improvement is a lot considering that this was achieved by implementing one (D6) or to (D7) functions. This makes sense, too, but in order to achieve a higher quota, you'd have to re-implement a lot of functions by hand. The hiphop approach has a lot of appeal here, since you can continue to write all code in PHP.

The next steps should be:

1) Evaluate other types of pages (single node view)

2) Look into general system performance under load.

This project was sponsored by examiner.com.

Nov 29 2010
Nov 29

I have created a centos/rhel Yum repository for x86_64 architectures containing all of the necessary packages for using HipHop. It depends on the EPEL and IUS repositories.

All of the packages are tracked in a GitHub project, and are hosted in a Yum repository at pkg.tag1consulting.com. I recommend installing the repository and hiphop and giving it all a try. Follow the instructions provided in the README on GitHub to get started.

Any and all feedback is welcome, and anyone interested in contributing to the Yum repository project is more than welcome to. We are updating these packages and the repository often.

This project is sponsored by examiner.com.

Gerhard has also written a follow-up article on how HipHop performs with Drupal.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web