Jan 07 2019
Jan 07

Pega Community Documentation Page

Book navigation is nested several levels

The import project is not yet complete, so I cannot give a live link, but here is a screenshot of the navigation. Notice that most of the navigation is collapsed, but enough of it is open to show the path to the current page.

,

Significant Numbers

  • 4472 pages in the book
  • 2.7 MB rendered (twice) for each page
  • 40-50 sec initial load
  • 6-9 sec load after caching

The navigation is rendered twice on every page: once for desktop and once for mobile. We should dos something about that, but not today.

After we did this work, the cached page loads in 2-3 sec. Normally, I would not brag about times like that, but it is a lot better than it was. (Also, the 6-9 seconds relied on some earlier caching work.)

I think that Emily Dickinson would understand how I feel about these load times:

The Heart asks Pleasure—first—
And then—Excuse from Pain—

,

Strategy

Cache the navigation once per book

If we cache each page of the book, that means about 10 GB of cache. It also means that we spend a lot more time generating the navigation menu than if we generate it once per book.

Set active trail with javascript

The problem with caching the navigation once per book is that we need to customize it for each page, opening up the path to the current page. So let’s customize it per page with javascript (client side).

,

Implementation

Here is a simplified version of the Twig Template that creates the navigation block:

{% if tree %} {{ top_book_title }} {{ tree }} {% endif %}

The important part is the Twig variable at the end: {{ tree }} is the part we have to compute and cache. Also notice that we already have some CSS classes that we can target with jQuery.

Hook Node View

From the screenshot above, you might think that the navigation is in a block, placed in the sidebar region. In fact, the caching would be a little simpler if that were the case. The way the site is built, it is actually in the main page array.

Here is the code that adds a render array for the book navigation. After a few checks to make sure that it should be added, it creates a simple render array with a custom #theme and a single parameter. That is, it depends on the current book ('#book_id' => $book_id) but not on the current page.

function pdn_book_node_view(array &$build, NodeInterface $node, EntityViewDisplayInterface $display, $view_mode) { if ($view_mode != 'full') { return; } if (empty($node->book['bid']) || !empty($node->in_preview)) { return; } $book_id = $node->book['bid']; $book_node = Node::load($book_id); if (!$book_node->access()) { return; } // Cache the navigation block once for the entire book. // We will set the active trail client-side. $build['book_nav'] = [ '#theme' => 'book_nav', '#book_id' => $book_id, '#weight' => 100, '#cache' => [ 'keys' => ['pdn_book_nav', $book_id], 'contexts' => ['languages'], 'tags' => ["node:$book_id"], 'max-age' => Cache::PERMANENT, ], ]; }

I will explain the #cache parameters below. (If you want, you can skip to the section “Tell Drupal how to cache the navigation”.)

Hook Theme

This is pretty standard, but for completeness here is the definition of the custom theme function. Again, there is only one parameter, the book ID. The Twig template is the one I showed above, book_nav.html.twig.

function pdn_book_theme($existing, $type, $theme, $path) { return [ 'book_nav' => [ 'variables' => [ 'book_id' => 0, ], ], ]; }

Preprocess Function

This function takes the single book_id parameter provided to the theme function and adds the other variables used in the Twig template, including $variables['tree']. This function was already in the code before we started working on it. It is based on some code already in the core Book module.

function template_preprocess_book_nav(&$variables) { /** @var \Drupal\book\BookManager **/ $book_manager = \Drupal::service('book.manager'); // Get the nested array (tree) of menu links. $book_tree = $book_manager ->bookTreeAllData($variables['book_id']); // Generate a render array from the tree of links. $tree_output = $book_manager ->bookTreeOutput(array_shift($book_tree)['below']); $variables['tree'] = $tree_output; $variables['book_url'] = \Drupal::url( 'entity.node.canonical', ['node' => $variables['book_id']] ); $book_node = Node::load($variables['book_id']); $variables['top_book_title'] = $book_node->getTitle(); $variables['top_book_empty'] = !$book_node->hasField('field_body') || $book_node->get('field_body')->isEmpty(); }

Javascript

Here is the javascript that opens up the path to the current page. Since jQuery is very good at traversing the DOM, this ends up being a lot simpler than the PHP code we used previously.

The second half of this snippet was already there. We just added the part that finds the , looks inside it for a link to the current page, and then adds class="active" to that link and its parents and class="c-book-nav--list-expanded" to the parent

  • elements.
  • Drupal.behaviors.bookNavExpand = { attach: function attach(context) { var bookNav = $('.c-book-nav', context); $('a[href="https://www.isovera.com/caching-large-navigation-menus-drupal/' + context.location.pathname + '"]', bookNav) .addClass('active') .parentsUntil(bookNav, '.c-book-nav--list-expandable') .addClass('c-book-nav--list-expanded') .children('a') .addClass('active'); $('.c-book-nav--list-expanded > .c-book-nav--list', context) .once('bookNavExpandInit') .css('display', 'block'); $('.c-book-nav--expand-arrow', context) .once('bookNavExpandClick') .on('click', function() { $(this).parent().toggleClass('c-book-nav--list-expanded'); $(this).siblings('.c-book-nav--list').slideToggle(); }); } };

    There is room for improvement here. It is a little inefficient to traverse the DOM twice (once to set class="c-book-nav--list-expanded" and a second time to set display="block" on those elements). We decided to KISS for now: just add our 7 lines of javascript and not touch what was already there.

    ,

    Tell Drupal how to cache the navigation

    Here again is the render element we added to the page.

    $build['book_nav'] = [ '#theme' => 'book_nav', '#book_id' => $book_id, '#weight' => 100, '#cache' => [ 'keys' => ['pdn_book_nav', $book_id], 'contexts' => ['languages'], 'tags' => ["node:$book_id"], 'max-age' => Cache::PERMANENT, ], ];

    Now let’s look at the four entries in the #cache sub-array.

    Cache Keys

    'keys' => ['pdn_book_nav', $book_id],

    We provide two cache keys:

    • A unique string to identify “our” cache entries.
    • The book ID.

    This is how we cache once per book.

    Without cache keys, any other cache data will bubble up to the page render array, but our render array will not be cached by itself, which is what we want. If the book navigation were in a block, then the block would be cached and we would not have to supply cache keys.

    Cache Contexts

    'contexts' => ['languages'],

    If the book is viewed in another language, then the link text will change, so we need to tell Drupal to store a separate copy for each language. Maybe the link URLs will also change, depending on how we manage languages.

    In fact, this site is not (yet) multilingual, so we are trying to be a little proactive.

    A drawback to the once-per-book strategy is that the navigation menu will not update if any individual page is updated, say with a new title. This is not a problem for books imported from an external system, but the site has other books as well. We may decide to add the 'route.book_navigation' cache context, if this does not affect performance badly. See Cache contexts in Drupal’s Cache API documentation.

    Cache Tags

    'tags' => ["node:$book_id"],

    This tells Drupal that when node/$book_id is updated, it should delete the entry from the cache. This is related to how many different variants should be cached. For example, we might want to cache once per book but invalidate it if any page in the book is updated. Then we would include the book ID in the cache keys and we would add cache tags for each node in the book.

    On my local copy of the site, the cache tags are stored in the database, where I can examine them. (See below.) On production, they might be handled by memcache. At the page level, cache tags are sent in HTTP headers, so that Varnish or a CDN can invalidate pages based on cache tags.

    Cache Max Age

    'max-age' => Cache::PERMANENT,

    This tells Drupal to keep the cached version until we say to clear it.

    ,

    Peek At the Database

    On my local copy of the site, the render cache is stored in the database, so we can see the results of these settings with a few queries. On production, this cache is handled by memcache.

    The cache_render Table

    Here is the relevant database table:

    mysql> DESCRIBE cache_render; +------------+---------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +------------+---------------+------+-----+---------+-------+ | cid | varchar(255) | NO | PRI | | | | data | longblob | YES | | NULL | | | expire | int(11) | NO | MUL | 0 | | | created | decimal(14,3) | NO | MUL | 0.000 | | | serialized | smallint(6) | NO | | 0 | | | tags | longtext | YES | | NULL | | | checksum | varchar(255) | NO | | NULL | | +------------+---------------+------+-----+---------+-------+ 7 rows in set (0.01 sec)

    Query

    After clearing caches and viewing one page, there is just one entry matching the unique string we supplied as a cache key. I have added some whitespace to make this easier to read. I did not include the data column, since that would have been overwhelming. I skipped serialized: it is a boolean that says whether the data is a simple string or a serialized PHP variable.

    mysql> SELECT cid, expire, created, tags, checksum FROM cache_render WHERE cid LIKE 'pdn_book%' LIMIT 0,1\G ********************** 1. row ********************** cid: pdn_book_nav: 704369: [languages]=en: [theme]=pegawww_theme: [user.permissions]=4f64d6e20026c96e963d91bab0192f9824e8cb2e9352eb4c1ca18d78478abfdb expire: -1 created: 1543638198.782 tags: config:system.book.704369 node:704369 rendered checksum: 12 1 row in set (0.00 sec)

    Cache ID (cid)

    This identifies the cached item.

    • We specified pdn_book_nav in the cache keys.
    • The book ID (704369) also comes from cache keys.
    • languages comes from cache contexts.
    • theme and permissions are default contexts: see below.

    Cache Max Age (expire and created)

    Since we specified 'max-age' => Cache::PERMANENT, in the cache settings, the expire column is set to -1. If we had specified 86400 (one day) then the expire value would have been 86400 more than the created value. (I should check this.)

    Cache Tags

    Again the cache tags describe when this entry should be purged. I am happy to see node:704369 (the book ID), but I am not sure where the other tags are generated.

    Permissions Hash

    I am punting on some of the cache tags, but I promised to explain where two parts of the cache ID come from. See sites/default/services.yml:

    parameters: renderer.config: # Renderer required cache contexts: # # The Renderer will automatically associate these cache # contexts with every render array, hence varying every # render array by these cache contexts. # # @default ['languages:language_interface', 'theme', 'user.permissions'] required_cache_contexts: - 'languages:language_interface' - 'theme' - 'user.permissions'

    This shows that we did not have to specify 'languages' in the cache contexts: it is already added by default. It also explains why the theme and the user permissions appear in the cache ID.

    ,

    Conclusion

    Our main goal was to improve page-load times: they started out terrible and now they are merely bad, maybe even fair. As a bonus, I learned a little about how the cache system works in Drupal 8. Comparing the settings we provided in the render array to what gets stored in the database helped to de-mystify the system for me.

    Try it yourself! In order to experiment with the cache settings, you can skip the theme function and the Twig template; just build your render array directly. Try setting a different max age, or adding cache contexts, and see how it affects what is saved in the database.

    ,

    References

    I already mentioned one reference:

    • Cache contexts

    That is part of the Cache API guide on drupal.org.

    The other reference I found most helpful for explaining the importance of cache keys is

    • Render API overview: Caching

    in the API documentation.

    These two pages in the Render API guide are also useful:

    • Cacheability of render arrays
    • Render arrays
    Dec 11 2017
    Dec 11

    Suppose the blog becomes popular. Maybe it gets enough traffic to generate some income. Then we devote some attention to the theme, maybe hire an intern to help maintain it, and we want to dress up the term pages, which list all the blog posts tagged with a particular term.

    We can add a few paragraphs of description to each term. It is nice to attach a little icon to each term, so that the icon is displayed on blog posts tagged with the corresponding term, instead of just the term itself.

    Maybe one of our vocabularies is geographic: the blog might be about travel or cooking or breeds of cats, for example. Then we might want to add a field to the location vocabulary with a link to more information about the place. Or maybe some other vocabulary needs links to our favorite related books, so that we can make some money from Amazon without adding any ads to the blog.

    The point is that as the blog site gets more complex, the vocabularies (or taxonomy types) start to look more like content types (or node types). When designing a complex site, it is not always obvious when to use a content type and when to use a vocabulary.

    This talk will show you some of the questions you should be asking to decide which to use. Depending on whether you use a content type or a vocabulary, …

    • … what do you gain or lose for good?
    • … what does Drupal give you for free?
    • … how does it affect the admin experience?
    • … how does it affect what is stored in the database?

    Key Concepts:

    • Adding a taxonomy and content type
    • Benefits and limitations of using taxonomies
    • Benefits and limitations of using content types
    • Taxonomy and content type admin experiences
    , Did you say taxonomy or content type?
    Oct 09 2017
    Oct 09

    Time to upgrade.

    Drupal 8.4.0 was released last week. According to Drupal core release cycle: major, minor, and patch releases,

    Previous minor releases will become unsupported when a new minor release is published.

    so it is time to upgrade existing sites.

    This particular upgrade presents some challenges, especially for sites managed with composer. According to the release notes,

    Drupal 8.4.0 includes major version updates for two dependencies: Symfony 3.2 and jQuery 3. Both updates may introduce backwards compatibility issues for some sites or modules, so test carefully.

    I have not looked into the consequences of the jQuery update, but a major version update of Symfonymeans that a lot of other PHP components have to be upgraded or removed.

    In particular, the current released version of drush is incompatible with Symfony 3.2. That means you have a few options:

    • Install the latest version of drush (as I write, 8.1.14) globally, not as part of your Drupal project.
    • Install the 8.x-dev version as part of your project.
    • Install version 9 (currently 9.0.0-beta6) as part of your project.

    I have only tried the last of these.

    Update

    Drush 8.1.15 was released on October 9 and is compatible with Symfony 3.2. You now have the option of upgrading to the current release as part of your project.

    ,

    Upgrading to Drupal 8.4.0

    My Drupal 8 projects all use the semi-official standard for managing Drupal with composer: Composer template for Drupal projects. Some of my difficulties may be due to problems with this starting point, and others are probably caused by mistakes I have made along the way. I will not apologize for the mistakes: think of them as learning opportunities.

    I did not try upgrading everything at once with a simple composer update. This might have worked after some initial steps, but it is generally agreed that upgrading All The Things at once is too risky.

    My general procedure for arriving at a compatible set of requirements:

    1. Try composer update drupal/core —with-dependencies.
    2. Find a conflicting package and composer why conflicting/package.
    3. Find the source of the conflict and either composer update source/package —with-dependencies or composer remove source/package.
    4. Repeat.
    5. Restore the removed packages.

    By the way, composer why is an alias of composer depends. To save a little typing, I did alias why=’composer why’.

    Here is a slightly idealized git log recording the packages I had to remove and re-install before I was able to upgrade Drupal:

    • Temporarily remove drupal/console
    • Temporarily remove drush/drush
    • Remove drupal/console-* from composer.lock
    • Update Drupal core from 8.3.7 to 8.4.0

    In other words, this sequence of composer commands:

    composer remove drupal/console drush/drush composer remove drupal/console-core drupal/console-en drupal/console-extend-plugin composer update drupal/core --with-dependencies

    I am not sure why the drupal/console-* packages were not removed when I removed drupal/console.

    So far, I have not been able to restore Drupal console. See the next section for restoring drush.

    Update

    Since writing this post, I have updated to Drupal 8.4.3 and Drush 8.1.15, and I was able to restore Drupal Console 1.13.1. Now it is time to upgrade to 8.4.4 …

    ,

    Installing a compatible version of Drush

    The release notes for Drupal 8.4.0 mention

    Versions of Drush earlier than 8.1.12 will not work with Drupal 8.4.x. Update Drush to 8.1.12 or higher before using it to update to Drupal core 8.4.x or you will encounter fatal errors that prevent updates from running.

    That assumes you install drush globally, not as a composer dependency of your project. If you follow that link, you will see, under “Current status”,

    Drush 8.1.12 site-local Drush + Drupal 8.4.x: Does not work (See https://github.com/drush-ops/drush/pull/2800)

    Drush 8.1.12 phar + Drupal 8.4.x: Works (Not officially supported) **

    Drush 9.x site-local Drush + Drupal 8.4.x: Supported

    Drush 9.x global install + Drupal 8.4.x: Works (Not officially supported, cgr install recommended) **

    (Follow the link for notes on Drupal 8.3.x and for the explanation of “Not officially supported”.)

    I see one commit on the 8.x branch aimed at making it compatible with Symfony 3, so there may soon be a release of drush 8 that is compatible.

    Here is the log as I restored drush:

    • Allow webflo/drupal-finder either ^0.2.1 or ^1.0
    • Temporarily remove consolidation/robo
    • Temporarily remove stecman/symfony-console-completion
    • Temporarily remove phpunit/phpunit
    • Require drush/drush version 9, currently 9.0.0-beta5
    • Restore phpunit/phpunit (dev requirement)
    • Temporarily remove alchemy/zippy

    In other words, something like the following:

    composer require "webflo/drupal-finder:^0.2.1|^1.0" composer remove consolidation/robo stecman/symfony-console-completion phpunit/phpunit composer require drush/drush:^9.0.0 composer require --dev "phpunit/phpunit:>=4.8.28 I think that webflo/drupal-finder is one of the packages we got from drupal-composer/drupal-project. I am not sure why alchemy/zippy was still there (in composer.lock), nor why I decided to remove it.

    Update

    Now that drush 8.1.15 has been released, you can try updating to that instead of drush 9. You may still have troubles with webflo/drupal-finder and other packages as above.

    ,

    Difficulties with Drush 9

    Drush 9 is still in beta, and there are some things that do not seem to work yet. My initial tests were with beta5, and beta6 is already out, so it is possible that some of these problems have already been fixed.

    I see these commit messages in my git log:

    • Add a drush alias file for drush 9
    • Update Probo commands for drush 9
    • Temporarily disable Probo commands to download JS libraries

    Drush 9 now uses YAML format for its site-alias files. If you have an older site-alias file, then drush will automatically generate the new format for you, and then you can commit that to version control.

    We use Probo CI for continuous integration. Some of the drush commands we run on Probo need to be updated because of changes to drush, and some (especially those provided by contrib modules) just do not work.

    In order to use drush instead of vendor/bin/drush with Drush 9, install Drush Launcher. If (like me) you have previously installed drush with Homebrew, then first brew unlink drush.

    Maybe it is just me, but drush does not seem to be honoring the SSH options in the site-alias file. I need that because I run Drupal on a virtual machine (VM) using Vagrant. I found two ways to work around this problem. One is to append my public SSH key (~/.ssh/id_rsa.pub) to ~vagrant/.ssh/authorized_keys on the VM. The other way is to add these lines to ~/.ssh/config (on the host):

    Host mysite.dev Hostname mysite.dev User vagrant IdentityFile ~/.vagrant.d/insecure_private_key

    There are still some issues: I think that drush does not always pass the correct command when it delegates to drush on the VM. With either of the above work-arounds, I can use drush ssh to log in to the VM and then run vendor/bin/drush from there.

    ,

    Update:

    • As of drush 9.0.0-beta6, the YAML alias file is not generated automatically, but there is a new command to generate it.
    • The problem I mentioned with passing the correct command when delegating to the VM seems to be limited to drush user:login (short form uli). Look for the fix in beta8 (or work with the dev version).
    • I am not sure whether the problem with ssh-options has been fixed.
    ,

    Acknowledgements

    Thanks to Moshe Weitzman for his comments on Slack. Even more thanks to Moshe and the other drush maintainers for this amazing tool!

     

    ,

    Want to learn more?

    We offer a range of training and workshop options that cover everything from a basic ‘Intro to Drupal’ to layout and theming, security, performance, module development, and more.

    Check our training offerings!

    About Drupal Sun

    Drupal Sun is an Evolving Web project. It allows you to:

    • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
    • Facet based on tags, author, or feed
    • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
    • View the entire article text inline, or in the context of the site where it was created

    See the blog post at Evolving Web

    Evolving Web