Feeds

Author

Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Mar 28 2021
Mar 28

Posted Sat, 03/27/2021 </span>

John Jameson, a digital accessibility developer and colleague of mine, has been working hard for months on a new accessibility module for Drupal module Editoria11y.

Editoria11y (editorial accessibility) is a user-friendly accessibility checker that addresses three critical needs for content authors:

  • It runs automatically. Modern spellcheck works so well because it is always running; put spellcheck behind a button and few users remember to run it!
  • It runs in context. Views, Layout Builder, Media and all the other modules Drupal uses to assemble a complex page means checkers that run on individual fields cannot "see" errors that appear on render.
  • It focuses exclusively on content issues: things page editors can easily understand and easily fix. Editoria11y is meant to supplement, not replace, testing with comprehensive tools and real assistive devices.

John and I are part of a great team that works on Princeton University's "Site Builder" Drupal platform. We've already enabled this module on hundreds of our websites with a positive reception.

John's written up a much more detailed introduction to this module on Princeton's accessibility website.

Jan 16 2021
Jan 16

Drupal's Problem With Decorative Images

Decorative images are those that should not be described to assistive technology like screen readers. A simple example is a fancy image of a decorative border that you insert at the end of a news article.

Supporting decorative images in Drupal is quite straight forward. Images fields can be set to collect alternative text but not require it. When left blank, Drupal will output an empty alt attribute for the image element, indicating to screen readers the image should be ignored. Editors that want to convey that an image is decorative can then simply leave the alt text blank.

The problem with this approach is that it might encourage editors to leave alt text blank even if the image is not really a decorative. It's easy to pass right by the alt text field without even thinking about it. This is especially true for those that are completely unfamiliar with how important alternative text is for accessibility.

There's a small effort underway to resolve this problem in Drupal core. After some great discussions, particularly with Andrew Macpherson, a leading voice in Drupal accessibility, we landed on a potential solution (thanks Andrew!).

A Potential Solution

Instead of simply leaving the alt text field blank to the image is decorative, a checkbox labeled "Decorative" is introduced beneath the field. This checkbox must be checked if the editor wants to leave the alt text blank. This effectively makes the choice explicit rather than implicit, forcing the editor to pause for a second and consciously affirm that the image is truly decorative.

To let this solution incubate and evolve a bit, I contributed it as small module called Decorative Image Widget. It's quite flexible as instead of providing a brand new image widget, it modifies any image widget that is based on the default one provided by Drupal core.

Check it out and please provide feedback in the module's issue queue!

Dec 20 2020
Dec 20

Lately I've been working to improve the response time of 404 pages on some Drupal 8 sites I help maintain. Depending on the complexity of the site and the status of Drupal's various cache layers, Drupal can be quite slow to generate a full page response, even for a 404 page. These sites I maintain are occasionally subjected to aggressive security penetration scans which generate a ton of 404 responses for completely unique URLs. The scanner can send tens of requests per second like this which can quickly exhaust the resources of a web server.

While the best way to protect against scans like this is to use a web application firewall, making sure your site makes efficient use of its caching capability goes a long way.

Caching & Dynamic Page Cache

Drupal has a simple full page cache called Internal Page Cache. This module caches requests based entirely on the request URL and is only used for anonymous traffic to your site. Unfortunately, this type of cache does not help at all when delivering 404 pages for unique URLs, even though the contents of the page are likely identical for all of them. However, Drupal 8 also comes with the innovative Dynamic Page Cache module, a much more intelligent page cache.

As the "dynamic" name implies, the Dynamic Page Cache caches as much as the full page response as it can and creates placeholders for things that are too dynamic to cache and fills them in later. This alone makes it incredibly useful, and in many cases, this module creates a cache entry for a page that's nearly as complete as the entire page response. But there's also another subtle advantage which is of special interest to 404 pages: The same cached response can be used for multiple different unique URLs.

To illustrate how this works, understand that all caches use an ID that uniquely identifies that cache entry. This ID is used to lookup cache entries to see if there is a cache hit. Most traditional page caches like Drupal's Internal Page Cache and Varnish use the complete URL of the requested page as this cache ID. There's really not much else they can use, as they have to make a decision about what cache entry to look for based entirely on the request data. These caches assume that each unique URL produces a different output, so a request for https://sample.com/some-fake-page and https://sample.com/some-other-fake-page are treated as unique cache entries, even if their output is identical. These caches are just not intelligent enough to know that these two requests result in the same output.

However, instead of using the URL of the request as the cache ID, Dynamic Page Cache uses a combination of all cache contexts for that page. In Drupal terminology, cache contexts indicate the parameters by which the content of a page varies. For example, a page with the "user.roles:authenticated" cache context indicates that some content of that page will be different depending on if the user is logged in or not. As such, the Dynamic Page Cache module would create a unique cache entry for logged in and logged out users.

While there is a cache context for "url" (which would cause every cache entry to be vary based on the URL, just like the Internal Page Cache module), many pages do not require it.

Dynamic Page Cache & 404 Pages

The power of Dynamic Page Cache is that it lets developers get much more granular and specific in determining the uniqueness of a pages output via these cache contexts. As it turns out, for most pages on your Drupal site, the route and route parameters (via the "route" context) are the primary indicator of page uniqueness, not the URL.

Consider how 404 page responses are delivered by Drupal. A site admin configures what page on your site to display when it can't find any real page that matches the request. Many site admins will create a basic page node and use that as the 404 page. Drupal ultimately resolves this path down to its route and route parameters, which in this example may be "entity.node.canonical" with a parameter of "123" for the node ID. This will always be the same for every 404 path on your site. Requests to your site for "/fake-page-1" and "/fake-page-2" both resolve to that same route.

While traditional page caches always result in cache misses for "/fake-page-2" even if there's a cache entry for "/fake-page-1", Dynamic Page Cache knows that both of these requests resolve to the same route and both of these requests use the same cache ID, so it can re-use the cache entry for both requests. This has huge implications for performance of serving 404 pages since Drupal can skip so much of the rendering process when delivering that second response for "/fake-page-2".

Another example to illustrate this: Imagine you have a "/contact" page on your site. Traditional page caches cannot use the same cache entry for requests for "/contact" and "/contact?test=123" because they assume that the query parameter varies the output. But unless your "/contact" page tells Drupal that it should use the "url.query_args" cache context, Dynamic Page Cache doesn't care about that useless query parameter and will use the same cache entries for both requests.

The Dynamic Page Cache module can therefore dramatically improve the response time of delivering 404 pages to your visitors (or attackers and security scanners) for unique URLs because it can reuse the cache entries for multiple unique request paths.

How to Screw It Up

While it's true that most pages on your site likely don't vary specifically by the URL, some certainly do. Here's some examples:

  • A special 404 page that displays some helpful links depending on what URL the user was trying to access that no longer exists.
  • Blocks with visibility conditions based on the request path
  • A views block that appears on every page of your site with a URL based argument configured

In all of these situations, the "url" cache context (or its more specific children like "url.path") will be added to your page's cache metadata. This tells the Dynamic Page Cache module that the response actually does vary based on the URL of the request. If your 404 page has this cache context, Dynamic Page Cache won't be beneficial! Every request for a page that doesn't exist will bypass Drupal's largest front-line caching layers.

While this "url" cache context is not added by default in simple site configurations, you can see from the examples above it's not too difficult to get in a situation where this context is added to every page, thereby tanking the performance of your 404 pages.

Finding the reasons that the "url" or "url.path" cache contexts are added to your pages is key to seeing if you can remove them. For example, if you have a block with a path-based visibility condition that simply lists a single page and not some wildcard path, consider embedding the block on that page directly using a different method. If you have a views block with a contextual filter (argument) configured, it may be adding the url cache context, even if the block doesn't actually vary based on the entire URL. Consider using the Views Advanced Cache module to override this and provide the more specific cache contexts you need. If you have custom code that adds the "url" or "url.path" cache contexts, consider if there's a more specific context you can use instead. Perhaps you really just depend on the presence of a query string parameter and not the full path?

Summary

Drupal 8's unique Dynamic Page Cache module can give your site huge performance gains and add more resiliency to simple denial of service attacks and security penetration scans. However, it requires diligence when building and developing your site so that you are aware of the various cache contexts that your pages use. This is especially true of 404 pages, where you really want eliminate the very "expensive" cache contexts like "url" to ensure the biggest performance gains.

Jan 01 2020
Jan 01

The Problem

Replacing files uploaded to your Drupal site can be very frustrating. In most cases, when an editor wants to replace a document, they want to keep the exact same filename and filepath and just overwrite the contents of the file. This is important in cases where the file is linked to elsewhere throughout the website or on other websites outside of the editors control. If you use the media module to manage documents on your site, you'll quickly discover that it's not possible to upload a replacement file for a document and keep the same filename.

Why is this? The core file field allows you to remove and upload a replacement file. But when you upload a replacement file with the same filename, Drupal appends a number to the end of it, like _0, _1, etc. This is because the old file is not actually removed from the filesystem immediately. Drupal will mark it as a temporary file and it will be removed during a cron run sometime in the future (* see exception to this below). Drupal won't overwrite that file with the new one, so it appends a number to the end to make it unique. Drupal also does this to support revisions properly. If the file field is attached to an entity that has revisions enabled, then Drupal keeps that old file around because previous revisions may require it.

* Actually, as of Drupal 8.4, old unused files are never deleted unless you have a specific config override set that enables this cleanup.

Existing Contrib Solutions

There is a 4 year old Drupal core issue where the community has tried to resolve this problem. Probably the most popular solution has been to use the Media Entity Download module. It works by creating a special route for each media entity that will directly return the file associated with that entity as a download. So if you had a document media entity with ID 59, editors can create links to /media/59/download and Drupal will serve the file directly from that path. This solves the problem because it hides the actual file system path from visitors. It doesn't matter what that path is or how many numbers Drupal has appended to the end of replacement files, the visitor never sees it and you'll never have broken links.

The main problem I have with Media Entity Download is that it gets Drupal involved in serving up the files which are normally not handled by Drupal at all. After all, the files are just sitting there in the public filesystem. The web server (apache, nginx) typically serves them to visitors and leaves Drupal out of it. With this module, every time a file is requested, Drupal is bootstrapped and a PHP process is tied up until the download completes to the visitor. This is a bit scary if you get a large surge of traffic with many users requesting large downloads.

The New Solution

I just completed work on the Media Entity File Replace module to provide another solution to this problem. This module is intended for site builders that are using the Media module to manage documents on their site. It works by adding a replacement file widget to the media edit form for media types that use a file source (Image, Document, etc). You can control which specific media types it's enabled for by visiting the form display configuration and showing/hiding it. When replacement files are selected and the form is saved, the file is first uploaded to temporary storage and then copied over the existing file. The filepath and filename are kept in tact. Just the contents of the file and the metadata describing the file are updated - which is what editors want most of the time.

Edit form for a document media entity showing the "replace file" form widget

Note that the module still supports the previous behavior of NOT overwriting the original file. Editors just need to uncheck the "Keep original filename" checkbox and the replacement file will be uploaded and the media entity updated to reference it.

This module should fill a functionality gap in document management in Drupal - I hope others find it useful!

Apr 04 2019
Apr 04

Responsive images overview

As screen resolutions and pixel densities continue to climb year after year, it's becoming more important to deliver the best possible image quality to your visitors. The easy way out is to deliver a single high resolution image, but this can have a real impact on page load time & bandwidth usage, especially for visitors on mobile devices & networks. The better solution is to deliver the appropriately sized image based on the screen width/resolution of the browser. So, instead of always delivering a super high res image to mobile device users (who's browsers will be forced to downsize the image to fit anyway), deliver an image that's better sized for that screen. Smaller resolution images have a much smaller filesize, so your visitors won't have to download as much data and the image will download faster.

Thankfully, a native HTML solution for delivering different images for different browser viewports has existed for years: using the "srcset" and "sizes" attributes of the existing element.

To quickly demonstrate how it works, let's take this super simple scenario of an image on your site that will always be displayed at 100% width of the browser. This is how the image element would look:

The srcset attribute provides your browser a list of images and how wide each is in real pixels. The sizes attribute tells the browser how wide the image will be displayed after it's been laid out and CSS rules applied to it.

But wait, don't browsers already know how wide an image will be when it's rendered on a page? It's responsible for rendering the page after all! Why can't it just figure out how wide the image will be rendered and then just select the most appropriate image source from the "srcset" list? Why is this "sizes" attribute needed at all?

Well, it's true that browsers know this information, but they don't know it until they have completed parsing all JS and CSS on the page. Because processing the CSS/JS takes a while, browsers don't wait and will instead begin downloading images referenced in your HTML immediately, meaning they need to know what image to download immediately.

In the simple scenario above, the site is designed to always render the image at 100% width via CSS, so we indicate as such by adding a single value "100vw" (vw stands for viewport width) to the sizes attribute. The browser then decides which image to load depending on the width of the viewport when the page is loaded. An iPhone 8 in portrait mode has a "CSS" width of 375 pixels, but it has a 2:1 pixel density ratio (a "retina" screen), which means it can actually display images that are double that width at 750px wide. So the browser on this phone will download the lower resolution version of the image which happens to match exactly at 750px wide. On a 1080p desktop monitor the browser will be wider than 750px wide, so the larger resolution image will be downloaded.

Responsive images delivered in this manner work really well for this simple use case.

Things start to get more complicated when the image being displayed on your site does NOT take up the full width of the browser viewport. For example, imagine a site design where an image is displayed 1500px wide at the desktop breakpoint, but is displayed at 50% width at tablet/mobile breakpoints. Now the image element changes to this:

The sizes attribute has changed to indicate that if the viewport width is at least 1500px wide, then the site's CSS is going to render the image at 1500px and no larger. If the viewport width is lower, then that first rule in the sizes attribute fails, and it falls back to the next one, so the site will render the image at 50% viewport width. The browser will translate that value to an actual pixel width (and take into account pixel density of the device) to select the appropriate image to download.

The problem this creates for dynamic layout builders

Now, imagine a dynamic layout builder tool on a content management system, like the new Layout Builder module for Drupal 8:

layout builder edit page

This great layout tool allows site builders to dynamically add rows and columns to the content region of a page and insert blocks of content into the columns.

One of the "blocks" that can be inserted into a column is an image. How do you determine the value of the "sizes" attribute for the image element? Remember, the sizes attribute tells the browser how wide the image will be when it's rendered and laid out by your CSS. Let's just focus on desktop screen resolutions for now, and say that your site will display the content region at a width of 1500 CSS pixels for desktops. A site builder could decide to insert an image in any of the following ways:

  • Into a single column row (image displays at 1500px wide)
  • Into the left-most column of a 50% - 25% - 25% row (image displays at 750px wide)
  • Into the right-most column of a 33% - 33% - 33% row (image displays at 500px wide)

The value of the "sizes" attribute differs for each of those three scenarios, which means that when Drupal is generating the image element markup, it needs to know the width of the column that the image was placed in.

The Drupal-specific problem is that (to my current knowledge) there's no practical way for the code that generates the image element markup to know information about the column the image was inserted in. Without this knowledge transfer, it's impossible to convey an accurate value for the "sizes" attribute.

Things get even more complicated if you're developing a solution that has to work with multiple different themes, where each theme may have different breakpoints and rules about the width of the content region at various breakpoints. What if some pages on your site have a sidebar menu, and others don't? That affects the the width of the content region and can easily throw off your sizes attribute calculations as well.

Moving forward

I think this is a new and interesting challenge, and I don't know that anyone has put much thought into how to solve it yet. I'm certainly hoping others read this and provide some ideas, because I'm not sure what the best solution is. The easy solution is of course to just not output the image responsively, and just use a single image src like the old days. In the example above, the image would need to be 1500px wide to account for the largest possibility.

Jul 21 2018
Jul 21

Unicode characters encoded using UTF8 can technically use 1 to 4 bytes to represent a single character. However, older versions of MySQL only provided support for storing UTF8 encoded characters that used 1 to 3 bytes. This was enough to cover the most commonly used characters, but is not suitable for applications that accept user input where any character can be submitted (like emojis, which use 4 bytes). Newer versions of MySQL provide a character encoding called utf8mb4 to fix this issue. Drupal 7 supports this, but requires some special configuration. Drupal 8 is configured this way by default.

Existing Drupal 7 sites that were setup with MySQL's old 3-byte-max UTF8 encoding must undergo a conversion process to change the character set on tables and text columns from utf8 to utf8mb4. The collation value (what MySQL uses to determine how text fields are sorted) also needs to be changed to the newer utf8mb4 variant. Thankfully, there's already a drush command you can download that does this conversion for you on a single database. Before running it, you should ensure that your MySQL server is properly setup to use the utf8mb4 character encoding. There's a helpful guide on this available on Drupal.org. Afterward the conversion is run, you still must configure Drupal to communicate with MySQL using this new encoding as described in the guide I linked to.

Part of my job is to help maintain hundreds of sites running as multi-site in a single codebase. So, same codebase, but hundreds of databases, each of which needed to have its database tables converted over to the new encoding. Converting a single database is not such a big deal, because it only takes a few minutes to run, but since I was dealing with hundreds, I wanted to make sure I had a good process laid out with plenty of logging. I created the below bash script which placed each site in maintenance mode (if it wasn't already), ran the drush command to convert the database, then took the site out of maintenance mode.

All in all, it took about 10 hours to do this for ~250 websites. While the script was running, I was monitoring for errors or other issues, ready to kill the script off if needed. I added a 3 second sleep at the end of each conversion to allow me time to cleanly kill the script.

After the script was completed, I pushed up new code for the common settings.php file (each site is configured to load a common settings file that they all share) which configured Drupal to connect to MySQL using the proper character set. In between the time that a database was converted, and the settings.php was updated for that site, there still should not have been any issues, because MySQL's UTF8MB4 character encoding should be backwards compatible with the original encoding that only supports 3 byte characters.

Here's the script for any that may be interested:

#!/usr/bin/env bash

#
# Usage:
# Alter this script to specify the proper Drupal docroot.
# 
# Run this command and pass to it a filename which contains a list of
# multisite directory names, one per line.
#
# For each site listed in the file, this script will first put the site in
# maintenance mode (if it's not already in that state), then run the
# uf8mb4 conversion script. Afterwards it will disable maintenance mode if
# it was previously disabled.
#

### Set to Drupal docroot
docroot="/var/www/html/"

script_begin=$(date +"%s")

count=0
total="$(wc -l $1 | awk '{ print $1 }')"
while read -r site || [[ -n "$site" ]]; do
    start_time=$(date +"%s")
    count=$((count+1))
    echo "--- Processing site #${count}/${total}: $site ---"
    mm="$(drush --root=${docroot} -l ${site} vget --exact maintenance_mode)"
    if [ $? -ne 0 ]; then
        echo "Drush command to check maintenance mode failed, skipping site"
        continue
    fi

    # If maintenance mode is not enabled, enable it.
    if [ -z $mm ] || [ $mm = '0' ]; then
        echo "Enabling maintenance mode."
        drush --root=${docroot} -l ${site} vset maintenance_mode 1
    else
        echo "Maintenance mode already enabled."
    fi

    drush --root=${docroot} -l ${site} utf8mb4-convert-databases -y $site

    # Now disable maintenance mode, as long as it was already disabled before.
    if [ -z $mm ] || [ $mm = '0' ]; then
        echo "Disabling maintenance mode."
        drush --root=${docroot} -l ${site} vset maintenance_mode 0
    else
        echo "Maintenance mode will remain on, it was already on before update."
    fi

    echo "Clearing cache"
    drush --root=${docroot} -l ${site} cc all

    end_time=$(date +"%s")
    echo "Completed in $(($end_time - $start_time)) seconds"
    echo "Done, sleeping 3 seconds before next site"
    sleep 3
done < "$1"

script_end=$(date +"%s")

echo "Ended: $script_end ; Total of $(($script_end - $script_begin)) seconds."
Mar 23 2018
Mar 23

I'm working in creating a Drupal 8 installation profile and learning how they can override default configuration that its modules provide at install time.

All Drupal 8 modules can provide a set of configuration that should be installed to the site when the module is installed. This configuration is placed in the module's config/install or config/optional directory. The only difference is that the configuration objects placed in the config/optional directory will only be installed if all of their dependencies are met. For example, the core "media" module has a config file config/optional/views.view.media.yml which will install the standard media listings view, but only if the views module is available on your site at the time of install.

The power of installation profiles is that they can provide overrides for any configuration objects that a module would normally provide during its installation. This is accomplished simply by placing the config object file in the installation profile's config/install or config/optional directory. This works because when Drupal's ConfigInstaller is installing any configuration object, it checks to see if that config object exists in your installation profile, and uses that version of it if it exists.

However, overriding default configuration that a module would normally provide is a double edged sword and brings up some interesting challenges.

If you dramatically alter a configuration object that a module provides, what happens when that module releases a new version that includes an update hook to modify that config? The module maintainers may write the update hook assuming that the config object that's installed on your site is identical to the one that it provided out-of-the-box during install time. I think this falls on the module maintainer to write update hooks that first check to make sure that the config object is mostly what it expects it to be before modifying it. If not, fatal errors could be thrown.

Another challenge that I ran into recently is more complicated. My installation profile was overriding an entity browser config object provided by the Lightning Media module. Entity browsers use views to display lists of entities on your site that an editor can choose from. My override changed this config object to point to a custom view that my installation profile provided (placed in its config/install directory), but it didn't work. When installing a site with the profile, I was met with an UnmetDependenciesException which claimed that the entity browser override I provided depended on a view that didn't exist. Well, it did exist, it's right there in the install folder for the profile! After some debugging, this is happening because the Drupal doesn't install config from the installation profile until all of the modules your install profile depends are installed first. So to summarize, it's not possible for a module's default config objects to depend on config that is provided by an install profile.

Feb 14 2018
Feb 14

Yes, a blog post about Drupal 7!

I recently worked on an enhancement for a large multi-site Drupal 7 platform to allow its users to import news articles from RSS feeds. Pretty simple request, and given the maturity of the Drupal 7 contrib module ecosystem, it wasn't too difficult to implement.

One somewhat interesting requirement was that images from the RSS feed be imported to an image field on the news article content type. RSS doesn't have direct support for an image element, but it has indirect support via the enclosure element. According to the RSS spec:

It has three required attributes. url says where the enclosure is located, length says how big it is in bytes, and type says what its type is, a standard MIME type.

RSS feeds will often use the enclosure element to provide an image for each item in the feed.

Despite being in a beta release still, the Drupal 7 Feeds module is considered quite stable and mature, with it's most recent release in September 2017. It has a robust interface that suited my use case quite well, allowing me to map RSS elements to fields on the news article content type. However, it doesn't support pulling data out of enclosure elements in the source. But alas, in there's an 8 year old issue containing a very small patch that adds the ability.

With that patch installed, the final step is to find the proper "target" to map it's data to. It's not immediately clear how this should work. Feeds needs to be smart enough to accept the URL to the image, download it, create a file entity from it, and assign the appropriate data to the image field on the node. Feeds exposes 4 different targets for an image field:

Feeds image field targets

Selecting the "URI" target is the proper choice. Feeds will recognize that you're trying to import a remote image and download it.

May 18 2017
May 18

Imagine you have a view that lists upcoming events on your Drupal 8 site. There's a date filter that filters out any event who's start date is less than the current date. This works great until you realize that the output of the view will be cached in one or many places (dynamic page cache, internal page cache, varnish, etc). Once it's cached, views doesn't execute the query and can't compare the date to the current time, so you may get older events sticking around.

One way of fixing this is to assign a custom cache tag to your view, and then run a cron task that purges that cache tag at least once a day, like so:

/**
 * Implements hook_cron().
 */
function YOUR_MODULE_cron() {
  // Invalidate the events view cache tag if we haven't done so today.
  // This is done so that the events list always shows the proper "start"
  // date of today when it's rendered. If we didn't do this, then it's possible
  // that events from previous days could be shown.
  // This relies on us setting a custom cache tag "public_events_block" on the
  // view that lists the events via the views_custom_cache_tag module.
  $state_key = 'events_view_last_cleared';
  $last_cleared = \Drupal::state()->get($state_key);
  $today = date('Y-m-d');
  if ($last_cleared != $today) {
    \Drupal::state()->set($state_key, $today);
    \Drupal::service('cache_tags.invalidator')->invalidateTags(['public_events_block']);
  }
}

Assuming you have cron running just after midnight, this will refresh the cache of the view's block and the page at an appropriate time so that events from the previous day are not shown.

Dec 09 2016
Dec 09

I'm working on a site where the editorial staff may occasionally produce animated GIFs and place them in an article. Image styles and animated GIFs in Drupal don't play nice out of the box. Drupal's standard image processing library, GD, does not preserve GIF animation when it processes them, so any image styles applied to the image will remove the animation. The ImageMagick image processing library is capable of preserving animation, but I believe the only way is to first coalesce the GIF which dramatically increases the output size which in unacceptable for this project (my sample 200kb GIF ballooned to nearly 2mb). For anyone interested in this approach anyway, the Drupal ImageMagick contrib module has a seemingly stable alpha release, but it would require a minor patch to get it working to retain animation.

I'm mostly interested in somehow getting Drupal to just display the original image when it's a GIF to prevent this problem. On this site, images are stored in an image field that's part of an Image Media Bundle. This media bundle supports JPEGs and PNGs as well, and those are typically uploaded in high resolution and need to have image styles applied to them. So the challenge is to use the same media bundle and display mode for GIFs, JPEGs, and PNGs, but always display the original image when rendering a GIF.

After some digging and xdebugging, I created an implementation of hook_entity_display_build_alter which lets you alter the render array used for displaying an entity in all view modes. I use this hook to remove the image style of the image being rendered.

/**
 * Implements hook_entity_display_build_alter().
 */
function my_module_entity_display_build_alter(&$build, $context) {
  $entity = $context['entity'];

  // Checks if the entity being displayed is a image media entity in the "full" display mode.
  // For other display modes it's OK for us to process the GIF and lose the animation.
  if ($entity->getEntityTypeId() == 'media' && $entity->bundle() == 'image' && $context['view_mode'] == 'full') {
    /** @var \Drupal\media_entity\Entity\Media $entity */
    if (isset($build['image'][0])) {
      $mimetype = $mimetype = $build['image'][0]['#item']->entity->filemime->value;
      $image_style = $build['image'][0]['#image_style'];
      if ($mimetype == 'image/gif' && !empty($image_style)) {
        $build['image'][0]['#image_style'] = '';
      }
    }
  }
}

So now whatever image style I have configured for this display mode will still be applied to JPEGs and PNGs but will not be applied for GIFs.

However, as a commenter pointed out, this would be better served as an image field formatter so you can configure it to be applied to any image field and display mode. I've created a sandbox module that does just that. The code is even simpler than what I've added above.

Feb 07 2014
Feb 07

I recently worked on porting over a website to Drupal that had several dynamic elements throughout the site depending on the IP address of the user. Different content could be shown depending on if the user was within a local network, a larger local network, or completely outside the network.

When porting the site over, I realized that it wouldn't be possible to enable page caching for any page that had this dynamic content on it. In Drupal, standard page caching is all or nothing. If you have it enabled and a page is "eligible" to be cached, Drupal saves the entire output of the page and uses it for future requests for the same page (I go into much more detail about page caching in previous blog post). In my case, if I enabled it, users who hit within one of the local intranets could trigger a page cache set, and now any users outside the intranet would view that same content.

I wanted a solution that let me either differentiate cache entries per by visitor "type" (but not role), or to at least prevent Drupal from serving cached pages to some of the visitors when a cached page already existed. I found a solution for the latter that I'll describe below. But first...

Why this is a hard problem

I already knew I could prevent Drupal from generating a page cache entry using drupal_page_is_cacheable(FALSE);. In fact, there's a popular yet very simple module called Cache Exclude that uses this function and provides an admin interface to specify which pages you want to prevent from being cached.

But what if you wanted to cache the pages, but force some visitors to view the un-cached version? This is what I needed, but Drupal has no API functions to do this. Many Drupal developers know that hook_boot is run on every page request, even for cache hits. So why can't you implement the hook and tell Drupal you don't want to serve a cached page? The reason is because of the way Drupal bootstraps, and when it determines if it should return a cached page or not.

There's a whole bootstrap "phase" dedicated to serving a cached page called _drupal_bootstrap_page_cache. If you take a close look, you can see that Drupal doesn't invoke the boot hook until after it already determined it's going to serve a cached page. In other words, there's no going back at this point.

Enter the "Dynamic Cache" module

I came across the Dynamic Cache module that seemed solve this problem. Once enabled, this module lets you disable serving a cached page by setting $GLOBALS['conf']['cache'] = false; within your own modules hook_book implementation - exactly what I suggested was not possible above!

So how was Dynamic Cache doing this? In summary, Dynamic Cache implements hoot_boot, checks if you tried to disable serving the cached page, and if so will "hijack" the bootstrap process to render the whole page and ignore the page cache entry that may exist. In then makes sure to "finish" up the request by completing the bootstrap process itself and calling menu_execute_active_handler(); that is normally done in index.php (but no longer get executed because of the hijack).

I want to note that what Dynamic Cache is doing is pretty scary in that it's almost hacking core without actually modifying any core functions. This fear is actually what triggered me to explore how the Drupal bootstrap process works under the hood so I could understand if there'd be any potential issues.

It's not an easy concept to understand initially, especially since for Drupal 7 you have to enable a second module called "Dynamic Cache Bootfix" that hijacks the bootstrap process a second time to properly finish up the request! I don't want to go into much more detail, but the modules code is pretty slim and I encourage developers to take a look. It will help you get a greater understanding of the bootstrap process and the obstacles this module tries to overcome.

There's also a core issue that is trying to address this problem of not being able to easily disable a cached page from being served. I also encourage you to read thru that to get a better understanding of what the problems are.

How I implemented it

In my case, I found that the majority of traffic to the site was from users outside any of the intranets, so I decided to allow them to both trigger cache entries being generated and to be served those cached page entries. For everyone else (a small % of traffic), Drupal would always ignore whatever was in the cache for that page and would also not generate a cache entry:

function my_module_boot() {
  $location = _my_module_visitor_network();
  if ($location != 'world') {
    # Prevent Drupal from serving a cached page thanks to help from the Dynamic Cache module
    $GLOBALS['conf']['cache'] = false;
    # Prevent Drupal from generating a cached page (standard Drupal function)
    drupal_page_is_cacheable(FALSE);
  }
}

Note that Dynamic Cache relies on having a heavy module weight so it runs last - which allows me to disable the cache in my own hook_boot. Make sure you read the README that comes with the module so you set everything up properly.

Also note that I still called drupal_page_is_cacheable(FALSE);. Without this, Drupal may still generate a cached paged based on what this user saw. With my code in place, anonymous users outside the networks I was checking would both generate page cache entries and be served page cache entries. Anonymous users within the networks/intranets would never trigger a cache generation and would never be served a cached page.

Final Thoughts

Ideally, I would be able to generate separate page caches for each "type" of visitor I had. I think this is possibly by creating your own cache store (which is not that difficult in Drupal 7) and changing the cache ID for the page to include the visitor type. I think the boost module may also allow for this sort of thing.

For really high traffic sites, you're probably going to be using something like Varnish anyway - and completely disable Drupal's page caching mechanism. I don't know much about Varnish but I imagine you could put this similar type of logic in the Varnish layer and selectively let some users through and hit Drupal directly to get the dynamically generated page (especially since my check for visitor network is just based on IP address).

There you have it. Dynamic Cache is by no means an elegant module, but it gets the job done! If you're better informed than I and I made a mistake somewhere in this writeup, please let me know in the comments. I certainly don't want to spread misinformation!

Feb 05 2014
Feb 05

I just finished up a small project at work to create a basic resource management calendar to visualize and manage room and other asset reservations. The idea was to have a calendar that displayed reservations for various resources and allow privileged users the ability add reservations themselves. The existing system that was being used was a pain to work with and very time consuming - and I knew this could be done easily in Drupal 7. The solution could be extended to create a more general resource booking / room booking system.

I wanted to share the general setup I used to get this done. I won't go into fine detail, and this is not meant to be a complete step by step guide. I'm happy to answer any questions in the comments.

Step 1: The "Reservation" content type

I quickly created a new content type "Resource Reservation" and added a required date field. Due to a bug in a module I used below, I had to use a normal date field and not ISO or Unix (I usually prefer Unix timestamps). These three different types of date fields are explained here. Aside from that, I also made the date field have a required end date and support repeating dates using the Date Repeat Field module (part of the main Date module). I then needed to decide how I would manage the resources and link them to a reservation.

I created another content type "Resource" and linked it to a reservation using the Entity Reference module. Another option I considered was using a Taxonomy vocabulary with terms for reach resource, and adding a term reference field to the reservation content type. I decided to go for a full blown entity reference to allow greater flexibility in the future for the actual resource node.

In my case, I created the 6 "Resource" nodes (all rooms in a building) that would be used in my department.

Step 2: The Calendar

Years ago at the 2011 DrupalCamp NJ, I attended Tim Plunkett's session "Calendaring in Drupal." Tim provided a great introduction to a new Drupal module called Full Calendar that utilized an existing JavaScript plugin with the same name. I was very impressed with the capability of the module and wrote about it after the camp was over.

I immediately knew I wanted to use the module and was happy to see it has been well maintained since I last checked it out in 2012. The setup was incredibly simple:

  • Create a new "Page" view displaying node content
  • Set the style to "Full Calendar"
  • Add a filter to only show published "Resource Reservation" node
  • Add the date field that is attached to "Resource Reservation" nodes

The style plugin for Full Calendar has a good set of options that let you customize the look and functionality of the calendar. I quickly able to shorten it quite a bit to display the start and end times as "7:30a - 2:00p".

One thing to note is that while you can add any fields you want to the view, the style plugin only utilizes two: A date field and a title field. Both are displayed on the calendar cell - and nothing else. If you add a date field, the style plugin automatically uses it as "the" date field to use, but if you have multiple date fields for whatever reason you can manually specify it in the settings. Similarly, for the title field, you can add any field and tell the plugin which one to use as "the" title for the event. In my case the node title was suitable. If you wanted to display more than one field, try adding them and then add a global field that combines them, then assign that as the title field.

I loaded up some reservation nodes and viewed them in the calendar and everything was looking great so far. Next I wanted to provide some filtering capability based on the resource of the reservation "events".

Step 3: Filtering the Calendar by Resource

In my case there was a desire to be able to display the reservations for select resources at a time instead of all of them at once. This would be a heavily used calendar with lots of events each day, and it would become a mess without some filtering capability. This was easy enough by creating an exposed filter for the calendar view.

Ideally I would have a filter that exposed all of the possible resources as checkboxes - allowing the user to control what reservations for what resource they are viewing. I'm sure I could have done that by writing my own views filter plugin or doing some form altering, but I settled for this approach:

  • Added a new input filter for my "Resource" entity reference field.
  • Exposed it
  • Made it optional
  • Changed it to "grouped filter" instead of "single filter". This let me specify each Resource individually since there's no out-of-the-box way of listing all available.
  • Used the "radio" widget
  • Allowed multiple selections - this actually changed the radio buttons to checkboxes instead - exactly what I want.
  • Added 6 options for the filter - one for each resource. I looked up the node ID's for each resource and put them in with their appropriate label. Downside is each time a new resource is added I have to manually update the filter.
  • Changed the "filter identifier" to the letter "r", so that the query string params when filters are used aren't so awful looking

There are two major gotchas here. The first is that if you have more than 4 options to chose from, Views changes the checkboxes to a multi select field (bleh). This is an easy fix:

function YOUR_MODULE_form_views_exposed_form_alter(&$form, &$form_state) {
  if ($form['#id'] == 'views-exposed-form-calendar-page') { # find your own views ID
    $options =& $form['r']; # my exposed field is called "r" (see last step above)
    if ($options['#type'] == 'select') {
      $options['#type'] = 'checkboxes';
      unset($options['#size']);
      unset($options['#multiple']);
    }
  }
}
This ensures that the exposed filter is ALWAYS going to be checkboxes. The second gotcha is how views handles the multiple selections. By default, views will "AND" all of the selections together. So if you select "Room 5" and "Room 6", I get reservations that have both selected - which is not possible in my case since I purposely limited the entity reference field on the reservation to only reference one resource. Instead I want views to "Or" them, so it shows any reservations for either "Room 5" or "Room 6". The fix for this is simple, but not obvious:
  • In the filter criteria section in the View UI, I went to "Add/Or, Rearrange" which is a link in the drop down next to the "Add" button.
  • I created a new filter group and dragged my exposed filter into it.
  • The top group has the published filter and the content type filter, and the operator is set to AND.
  • The bottom group has my single exposed filter for the resource, and the operator is set to OR.
  • The two groups are joined together with an AND operator.

Setting the second group to use OR is the key here. Even though there is just one item in the filter group, it's a special filter because it allows multiple selections. Views recognizes this and will apply the OR operator to each selection that was made within that filter. By default I had everything checked (which is actually the same as having nothing checked, at least in terms of the end result). This makes it obvious to calendar viewers that they can uncheck resources.

Step 4: Adding Colors for each Resource

Since the default calendar view includes 6 resources, I wanted each resource to be displayed with a color that corresponded to the resource it was reserving. The Full Calendar module can sort of do this for you with help of the Colors module. This module lets you arbitrarily assign colors to taxonomy terms, content types, and users. Colors then exposes an API for other modules to utilize those color assignments however they want. Full Calendar ships with a sub module called "Full Calendar Colors" that does just this by letting you color the background of the event cells in the calendar based on any of those three types of assignments that may apply.

In my case, since I wasn't using Taxonomy terms, I couldn't use the Colors module to color my reservations. Someone opened an issue to get Colors working with entity references like in my case, but it's not an easy addition and I couldn't come up with a practical way of adding it to the Colors module myself.

Instead, I examined the API for Full Calendar and found I could add my own basic implementation in a custom module. Here's the basics of what I did:

  • Add my own color assignment form element to each "Resource" node using form alters and variable set/get.
  • Implement hook_fullcalendar_classes to add a custom class unique to each "Resource" for the calendar cell. Like ".resource-reservation-[nid]".
  • Implement hook_preprocess_fullcalendar to attach my own custom CSS file (created using ctools API functions) to the calendar that has the CSS selectors for each resource reservation with the proper color.

Finally I added a "legend" block that lists each Resource (with a link to that Resource node) displaying the color as the background, so users can quickly see what the colors in the calendar meant. You could also avoid some of this complexity by removing the ability to assign colors via the node form and just hardcode the color assignments in your theme CSS file. You'd still need to implement hook_fullcalendar_classes.

Step 5: Reservation Conflicts

With the basic calendar view completed and displaying the reservations, I shifted focus to the management aspect of the feature. Specifically, I needed to prevent reservations for the same resource to overlap with one another.

A little bit of digging led to me a great module called Resource Conflict. This module "simply detects conflicts/overlaps between two date-enabled nodes, and lets you respond with Rules". It requires Rules which is used to setup reaction rules when a conflict is detected, allowing you to set a form validation error. Note the module integrates with Rules Forms as well, but I've found it's not actually required. Resource Conflict is a very slim but capable module - I was very impressed and happy with its capabilities.

The module provides a Rules event "A resource conflict node form is validated". To get this event to trigger, I had to enable "conflict detection" for the Resource Reservation content type (part of the Resource Conflict module). To do this, I edit the Resource Reservation type, went to the new "Resource Conflict" vertical tab, and enabled it by selecting the date field to perform conflict checking on.

<

p>The Resource Conflict module provides a default rule that by prevents form submissions if there are any other nodes of the same type with an overlapping date. This is too general because I want the Rule to only throw a validation error if the conflicting reservation is for the same resource I'm trying to reserve. I disabled that default rule and worked to create a rule to also take the resource into consideration. This part was somewhat complicated and I was happy to find some guidance in the issue queue. EDIT: I've since taken maintainership of the module and updated the real documentation page with details on how to perform the following steps.

First, I needed to create a Rule Component that encapsulates the logic to compare two Reservation nodes, check if they have the same Resource entity reference, and if so set a form error. Here's how I did that:

2 Variables:

  • "Node" data type, "Unsaved Reservation" label, "unsaved_reservation" machine name, usage as a "parameter"
  • "Node" data type, "Conflicting Reservation" label, "conflicting_reservation" machine name, usage as a "parameter"

3 conditions:

  • "Entity has field" on the "unsaved-reservation" data selector, checking it has the resource entity reference field
  • "Entity has field" on the "conflicting-reservation" data selector, checking it has the resource entity reference field
  • "Data comparison" to make sure that the values of the two entity reference fields are equal

1 action:

  • Set a form validation error. I wrote in a message including a link to the conflicting resource using the available tokens.

Rule Component

<

p> Now, with this rule component in place, I could incorporate it into a normal Rule that reacted on the node submission, loading all the conflicting reservations (based on date alone) and looping through each one to execute the component actions for the more complicated comparison. Here's how I did that:</ul>

  • React on event "A resource conflict node form is validated"
  • Added condition for "Contains a resource conflict" - this relies on the "node" param that is made available from the event
  • Added action for "Load a list of conflicting nodes". This is provided by the Resource Conflict module and this is where the all the conflict detection is done, comparing other nodes of the same type for conflicting dates. This action is added as a Loop.
  • Add a Rule Component within the action loop, selecting the one we just created.

Since I setup the component with three variables, I needed to pass them in as arguments to the component after adding it into the loop. For the "unsaved-reservation" variable, I fill in "node". For the "conflicting-reservation" variable, I supply the original "list-item" variable from the loop.

Main Rule

Testing the rule proved that I was not able to overlap any dates for the same resource when creating a reservation. Perfect!

Final Thoughts

The basic functionality of the resource management was there. Users could add new reservations for existing resources and were alerted if the reservation conflicted with others. Reservations were displayed in a calendar for the department to see, and users could filter out specific resources to provide a cleaner view. Here are some additional notes and considerations:

  • To allow the calendar to scale, you'll want to enable AJAX on the calendar view which will only display events for a given month (+/- two weeks). There's a bug in the stable release of the module related to AJAX but I provided a patch.
  • If you're using repeating date fields, make sure you uncheck "Display all values in the same row" on the date field settings in the view. If you don't, any exposed filters for the date range (which is how the AJAX feature works for Full Calendar) will not apply to dates with multiple values. If you do this properly, only the repeating dates for the given date range will be loaded.
  • There's a bug in the Resource Conflict module that only allows you to use the standard "Date" database storage type for a date field. I'm working on a patch.
  • Remember that you could also implement a "resource" using taxonomy terms instead of entity references. If you do, you'll have a much better time getting the Colors stuff working.

And that's pretty much it! Let me know if you have any questions in the comments below.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web