Tackling oversized cache items in Drupal

Drupal’s highly dynamic and modular nature means that many of the central core and contrib subsystems and modules need to maintain a large amount of meta-data.

Rebuilding the data every request would be very expensive, and usually when one part of the data is needed during part of the request, another part will be needed later during the same request. Since just about every request needs to check variable_get(), load entities with fields attached etc., the meta-data needs to be loaded too.

The pattern followed by most subsystems is to put the data into a single large cache item and statically cache it. The more modules you have, the larger these cache items become — since more modules mean more variables, hook_schema() and hook_theme() implementations, etc. And the same happens via configuration with field instances, content types and default views.

This affects many of the central core subsystems — without which it’s impossible to run a Drupal site — as well as some of the most popular contrib modules. The theme, schema, path alias, variables, field API and modules system all have similar issues in core. Views and CCK have similar issues in contrib.

With just a stock Drupal core install, none of this is too noticeable, but once you hit 100 or 200 installed modules, suddenly every request needs to fetch and unserialize() potentially dozens of megabytes of data. Some of the largest cache items like the theme registry can grow too large for MAX_ALLOWED_PACKET or the memcache default slab size. Since the items are statically cached, these caches can easily add 30MB or 40MB to PHP memory usage combined.

The full extent of this problem became apparent when I profiled WebWise Symantec Connect site (Drupal.org case study). Symantec Connect currently runs on Drupal 6, and as a complex site with a lot of social functionality has a reasonably large number of installed modules.

To find the size of these cache items, the simplest method is to profile the request with XHProf. Here’s a detail of the _theme_load_registry() function from Symantec Connect before this issue was dealt with. As you can see from this screenshot, the theme registry is taking up over 3mb of memory usage, and it’s taking over 6ms to pull it from cache.

When reviewing Drupal 7 code it was found that in all cases the problems with oversized cache items were still there.

To tackle this and similar large cache items, I started working on a solution with the following requirements:

Keep the single round trip to the cache backend.
Massively reduce the amount of data loaded on each request.
Avoid excessive cache writes or segmentation.
Fix the performance issue without breaking APIs.

Webwise agreed to fund core patches for the most serious issues; between that funding, my own time, and plenty of Drupal community review we eventually ended up with CacheArray as a new core API - merged into Drupal 8 and also backported to Drupal 7.

CacheArray allows for the following:
Implements ArrayAccess to allow (mostly) backwards compatibility with the vast Arrays of Doom™.
Begins empty and progressively fills in array keys/values are they’re requested.
Caches any newly found array keys at the end of the request with one write to the cache system, so they’re available for the next one.
Items that would have been loaded into the ‘real’ array every request, but are never actually used, skip being loaded, resulting in a much smaller cache item.

For many systems such as the theme registry and the schema cache, much less than half of the data is actually used during most requests to the site. Theme functions may be defined for an admin page which is never visited. Often the schema definition for a table is only required for a handful of tables on a site, for example if drupal_write_record() is used to write to them. By excluding these unused items from the runtime caching, we’re able to significantly reduce the amount of data that needs to be lugged around every request.

Revisiting the XHProf screenshots, let’s look at Connect with the Theme Registry replaced by a CacheArray, we can see that the time spent to fetch the cache item was reduced to 1.7ms and the memory usage to 360kb (although actual site traffic determines how large the cache item will eventually grow).

So far, the following core issues have been fixed in Drupal 8, most of these are already backported to Drupal 7 or are in the process of being so:

Introduce DrupalCacheArray and use it for drupal_get_schema()

Theme registry can grow too large for MySQL max_allowed_packet and memcache default slab size

Be more sparing with locale cache clears

system_list() memory usage

Still in progress (reviews/re-rolls welcome!):

variable_set() should rebuild the variable cache, not just clear it

Avoid slow query for path alias whitelists

And related issues — not using CacheArray but resulting from the same research:

_content_type_info() memory usage improvements (CCK - fixed)
_field_info_collate_fields() memory usage (core, in progress).
views_get_default_view() - race conditions and memory usage.
views_fetch_data() cache item can reach over 10mb in size

Since CacheArray implements ArrayAccess which is only available in PHP 5.2+, it’s unfortunately not possible to backport it to Drupal 6 core. However Tag1 Consulting maintains a fork of Pressflow which contains many of these improvements. Since Pressflow core requires PHP 5.2 these can also be submitted as pull requests for the main distribution.