Feeds

Sep 09 2019
Sep 09

3 minute read Published: 29 Aug, 2019 Author: Colan Schwartz
Drupal Planet , Aegir , DevOps

Have you been looking for a self-hosted solution for hosting and managing Drupal sites? Would you like be able able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites had free automatically-updating HTTPS certificates? You probably know that Aegir can do all of this, but it’s now trivial to set up a temporary trial instance to see how it works.

The new Aegir Development VM makes this possible.

History

Throughout Aegir’s history, we’ve had several projects striving to achieve the same goal. They’re listed in the Contributed Projects section of the documentation.

Aegir Up

Aegir Up was based on a VirtualBox virtual machine (VM), managed by Vagrant and provisioned with Puppet. It was superseded by Valkyrie (see below).

Aegir Development Environment

Aegir Development Environment took a completely different approach using Docker. It assembles all of the services (each one in a container, e.g. the MySQL database) into a system managed by Docker Compose. While this is a novel approach, it’s not necessary to have multiple containers to get a basic Aegir instance up and running.

Valkyrie

Valkyrie was similar to Aegir Up, but provisioning moved from Puppet to Ansible. Valkyrie also made extensive use of custom Drush commands to simplify development.

Its focus was more on developing Drupal sites than on developing Aegir. Now that we have Lando, it’s no longer necessary to include this type of functionality.

It was superseded by the now current Aegir Development VM.

Present

Like Valkyrie, the Aegir Development VM is based on a VirtualBox VM (but that’s not the only option; see below) managed with Vagrant and provisioned with Ansible. However, it doesn’t rely on custom Drush commands.

Features

Customizable configuration

The Aegir Development VM configuration is very easy to customize as Ansible variables are used throughout.

For example, if you’d like to use Nginx instead of Apache, simply replace:

    aegir_http_service_type: apache

…with:

    aegir_http_service_type: nginx

…or override using the command line.

You can also install and enable additional Aegir modules from the available set.

Support for remote VMs

For those folks with older hardware who are unable to spare extra gigabytes (GB) for VMs, it’s possible to set up the VM remotely.

While the default amount of RAM necessary is 1 GB, 2 GB would be better for any serious work, and 4 GB is necessary if creating platforms directly from Packagist.

Support for DigitalOcean is included, but other IaaS providers (e.g. OpenStack) can be added later. Patches welcome!

Fully qualified domain name (FQDN) not required

While Aegir can quickly be installed with a small number of commands in the Quick Start Guide, that process requires an FQDN, usually something like aegir.example.com (which requires global DNS configuration). That is not the case with the Dev VM, which assumes aegir.local by default.

Simplified development

You can use it for Aegir development as well as trying Aegir!

Unlike the default set-up provisioned by the Quick Start Guide, which would require additional configuration, the individual components (e.g. Hosting, Provision, etc.) are cloned repositories making it easy to create patches (and for module maintainers: push changes upstream).

Conclusion

We’ve recently updated the project so that an up-to-date VM is being used, and it’s now ready for general use. Please go ahead and try it.

If you run into any problems, feel free to create issues on the issue board and/or submit merge requests.

The article Try Aegir now with the new Dev VM first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

May 29 2019
May 29

6 minute read Published: 29 May, 2019 Author: Derek Laventure
Drupal Planet , Drupal 8 , Lando , Drumkit

Over the last 2 or 3 years, the Drupal community has been converging around a solid set of Docker-based workflows to manage local development environments, and there are a number of worthy tools that make life easier.

My personal favourite is Lando, not only because of the Star Wars geekery, but also because it makes easy things easy and hard things possible (a lot like Drupal). I appreciate that a “standard” Lando config file is only a few lines long, but that it’s relatively easy to configure and customize a much more complex setup by simply adding the appropriate lines to the config.

In this post I want to focus on an additional tool I’ve come to lean on heavily that complements Lando quite nicely, and that ultimately boils down to good ol’ fashioned Makefiles. Last summer at DrupalNorth I gave a talk that was primarily about the benefits of Lando, and I only mentioned Drumkit in passing. Here I want to illustrate in more detail how and why this collection of Makefile tools is a valuable addition to my localdev toolbox.

The key benefits provided by adding a Drumkit environment are:

  • consistent make <target>-based workflow to tie various dev tasks together
  • ease onboarding of new devs (make help)
  • make multistep tasks easier (make tests)
  • make tasks in Lando or CI environment the same (ie. make install && make tests)

Drumkit is not just for Drupal!

This example is using Drumkit for a Drupal 8 localdev environment, but there’s no reason you couldn’t use it for other purposes (and in fact, we at Consensus have lately been doing just that.

Basic Setup

As an example, suppose you’re setting up a new D8 project from scratch. Following this slide from my Lando talk, you would do the basic Lando D8 project steps:

  1. Create codebase with Composer (composer create-project drupal-composer/drupal-project:8.x-dev code --stability dev --no-interaction)
  2. Initialize Git repository (git init etc.)
  3. Initialize Lando (lando init)

For now, leave out the lando start step, which we’ll let Drumkit handle momentarily. We should also customize the .lando.yml a little with custom database credentials, which we’ll tell Drumkit about later. Append the following to your .lando.yml:

services:
  database:
    creds:
      user: chewie_dbuser
      password: chewie_dbpass
      database: chewie_db

Add Drumkit

To insert Drumkit into this setup, we add it as a git submodule to our project using the helper install.sh script, and bootstrap Drumkit:

wget -O - https://gitlab.com/consensus.enterprises/drumkit/raw/master/scripts/install.sh | /bin/bash
. d  # Use 'source d' if you're not using Bash

The install script checks that you are in the root of a git repository, and pulls in Drumkit as a submodule, then initializes a top-level Makefile for you.

Finally, we initialize the Drumkit environment, by sourcing the d script (itself a symlink to .mk/drumkit) into our shell.

Drumkit modifies the (shell) environment!

Note that Drumkit will modify your PATH and BIN_PATH variables to add the project-specific .mk/.local/bin directory, which is where Drumkit installs any tools you request (eg. with make selenium. This means if you have multiple Drumkit-enabled projects on the go, you’re best to work on them in separate shell instances, to keep these environment variables distinct.

Note that you can take advantage of this environment-specific setup to customize the bootstrap script to (for example) inject project credentials for external services into the shell environment. Typically we would achieve this by creating a scripts/bootstrap.sh that in turn calls the main .mk/drumkit, and re-point the d symlink there.

Set up your kit

Because we’re using Composer to manage our codebase, we also add a COMPOSER_CACHE_DIR environment variable, using the standard .env file, which Drumkit’s stock bootstrap script will pull into your environment:

echo "COMPOSER_CACHE_DIR=tmp/composer-cache/" >> .env
. d # Bootstrap Drumkit again to have this take effect

From here, we can start customizing for Drupal-specific dev with Lando. First, we make a place in our repo for some Makefile snippets to be included:

mkdir -p scripts/makefiles
echo "include scripts/makefiles/*.mk" >> Makefile

Now we can start creating make targets for our project (click the links below to see the file contents in an example Chewie project. For modularity, we create a series of “snippet” makefiles to provide the targets mentioned above:

NB You’ll need to customize the variables.mk file with the DB credentials you set above in your .lando.yml as well as your site name, admin user/password, install profile, etc.

Now our initial workflow to setup the project looks like this:

git clone --recursive <project-repo>
cd <project-repo>
. d # or "source d" if you're not using Bash
make start
make build
make install

This will get a new developer up and running quickly, and can be customized to add whatever project-specific steps are needed along the way.

But wait- it gets even better! If I want to make things really easy on fellow developers (or even just myself), I can consolidate common steps into a single target within the top-level Makefile. For example, append the make all target to your Makefile:

.PHONY: all

all:
        @$(MAKE-QUIET) start
        @$(MAKE-QUIET) build
        @$(MAKE-QUIET) install

Now, the above workflow for a developer getting bootstrapped into the project simplifies down to this:

git clone --recursive <project-repo>
cd <project-repo>
. d
make all

Customize your kit

At this point, you can start adding your own project-specific targets to make common workflow tasks easier. For example, on a recent migration project I was working on, we had a custom Features module (ingredients) that needed to be enabled, and a corresponding migration module (ingredients_migrate) that needed to be enabled before migrations could run.

I created the following make targets to facilitate that workflow:

We often take this further, adding a make tests target to setup and run our test suite, for example. This in turn allows us to automate the build/install/test process within our CI environment, which can call exactly the same make targets as we do locally.

Ultimately, Drumkit is a very simple idea: superimpose a modular Makefile-driven system on top of Lando to provide some syntactic sugar that eases developer workflow, makes consistent targets that CI can use, and consolidates multi-step tasks into a single command.

There’s lots more that Drumkit can do, and plenty of ideas we have yet to implement, so if you like this idea, feel free to jump in and contribute!

The article Lando and Drumkit for Drupal 8 Localdev first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

May 29 2019
May 29

9 minute read Published: 29 May, 2019 Author: Derek Laventure
Drupal Planet , Drupal 8 , OpenSocial

In Drupal 7, hook_update()/hook_install() were well-established mechanisms for manipulating the database when installing a new site or updating an existing one. Most of these routines ended up directly running SQL against the database, where all kinds of state, configuration, and content data lived. This worked reasonably well if you were careful and had a good knowledge of how the database schema fit together, but things tended to get complicated.

With the maturing of Features module, we were able to move some of this into configuration settings via the ctools-style export files, making the drush feature-revert command part of standard workflow for deploying new features and updates to an existing site.

In Drupal 8, we’ve made huge strides in the direction of Object Orientation, and started to separate Configuration/State, Content Structure, and Content itself. The config/install directory is often all that’s needed in terms of setting up a contributed or custom module to work out of the box, and with the D8 version of Features, the same is often true of updates that involve straightforward updates to configuration .yml files.

It turns out that both hook_update() and hook_install() are still valuable tools in our box, however, so I decided to compile some of the more complicated D8 scenarios I’ve run across recently.

Drupal 8 Update basics

The hook_update_N API docs reveal that this function operates more or less as before, with some excellent guidelines for how to approach the body of the function’s implementation. The Introduction to update API handbook page provides some more detail and offers some more guidance around the kinds of updates to handle, naming conventions, and adding unit tests to the your update routines.

The sub-pages of that Handbook section have some excellent examples covering the basics:

All of these provided a valuable basis on which to write my own real-life update hooks, but I found I still had to combine various pieces and search through code to properly write these myself.

Context

We recently launched our first complex platform based on Drupal 8 and the excellent OpenSocial, albeit heavily modified to suit the particular requirements of the project. The sub-profile required more extensive customization than simply extending the parent profile’s functionality (as discussed here). Instead, we needed to integrate new functionality into that provided by the upstream distribution, and this often resulted in tricky interactions between the two.

Particularly with a complex site with many moving parts, we take the approach of treating the site as a system or platform, installing and reinstalling regularly via a custom installation profile and set of feature modules. This allows us to integrate:

  • a CI system to build the system repeatedly, proving that everything works
  • a Behat test suite to validate the behaviour of the platform matches the requirements

In the context of a sub-profile of OpenSocial, this became complicated when the configuration we wanted to customize actually lived in feature modules from the upstream profile, and there was no easy way to just override them in our own modules’ config/install directories.

We developed a technique of overriding entire feature modules within our own codebase, effectively forking the upstream versions, so that we could then modify the installed configuration and other functionality (in Block Plugins, for example). The trouble with this approach is that you have to manage the divergence upstream, incorporating new improvements and fixes manually (and with care).

Thus, in cases where there were only a handful of configuration items to correct, we began using hook_install() routines to adjust the upstream-installed config later in the install process, to end up with the setup we were after.

Adjust order of user/register form elements

We make use of entity_legal for Terms of Service, Privacy Policy, and User Guidelines documents. Our installation profile’s feature modules create the 3 entity legal types, but we needed to be able to tweak the order of the form elements on the user/register page, which is a core entity_form_display created for the user entity.

To achieve this using YAML files in the config/install directory per usual seemed tricky or impossible, so I wrote some code to run near the end of the installation process, after the new legal_entity types were created and the core user.register form display was set. This code simply loads up the configuration in question, makes some alterations to it, and then re-saves:

/**
 * Implements hook_install().
 */
function example_install() {
  _example_install_adjust_legal_doc_weights();
}

/**
 * Adjust weights of legal docs in user/register form.
 */
function example_update_8001() {
  _example_install_adjust_legal_doc_weights();
}

/**
 * Ensure the field weights on the user register form put legal docs at the bottom
 */
function _example_install_adjust_legal_doc_weights() {
       $config = \Drupal::getContainer()->get('config.factory')->getEditable('core.entity_form_display.user.user.register');
       $content = $config->get('content');

       $content['private_messages']['weight'] = 0;
       $content['account']['weight'] = 1;
       $content['google_analytics']['weight'] = 2;
       $content['path']['weight'] = 3;
       $content['legal_terms_of_service']['weight'] = 4;
       $content['legal_privacy_policy']['weight'] = 5;
       $content['legal_user_guidelines']['weight'] = 6;
       $config->set('content', $content)->save();
}

Modify views configuration managed by upstream (or core)

A slightly more complicated situation is to alter a views configuration that is managed by an upstream feature module during the installation process. This is not an ideal solution, but currently it’s quite challenging to properly “override” configuration that’s managed by a “parent” installation profile within your own custom sub-profile (although Config Actions appears to be a promising solution to this).

As such, this was the best solution I could come up with: essentially, run some code very nearly at the end of the installation process (an installation profile task after all the contrib and feature modules and related configuration are installed), that again loads up the views configuration, changes the key items needed, and then re-saves it.

In this case, we wanted to add a custom text header to a number of views, as well as switch the pager type from the default “mini” type to “full”. This required some thorough digging into the Views API and related code, to determine how to adjust the “handlers” programmatically.

This helper function lives in the example.profile code itself, and is called via a new installation task wrapper function, which passes in the view IDs that need to be altered. Here again, we can write trivial hook_update() implementations that call this same wrapper function to update existing site instances.

/**
 * Helper to update views config to add header and set pager.
 */
function _settlement_install_activity_view_header($view_id) {
  # First grab the view and handler types
  $view = Views::getView($view_id);
  $types = $view->getHandlerTypes();

  # Get the header handlers, and add our new one
  $headers = $view->getHandlers('header', 'default');

  $custom_header = array(
    'id' => 'area_text_custom',
    'table' => 'views',
    'field' => 'area_text_custom',
    'relationship' => 'none',
    'group_type' => 'group',
    'admin_label' => '',
    'empty' => '1',
    'content' => '<h4>Latest Activity</h4>',
    'plugin_id' => 'text_custom',
    'weight' => -1,
  );
  array_unshift($headers, $custom_header);

  # Add the list of headers back in the right order.
  $view->displayHandlers->get('default')->setOption($types['header']['plural'], $headers);

  # Set the pager type to 'full'
  $pager = $view->getDisplay()->getOption('pager');
  $pager['type'] = 'full';
  $view->display_handler->setOption('pager', $pager);

  $view->save();
}

Of particular note here is the ordering of the Header components on the views. There was an existing Header on most of the views, and the new “Latest Activity” one needed to appear above the existing one. Initially I had tried creating the new custom element and calling ViewExecutable::setHandler method instead of the more complicated $view->displayHandlers->get('default')->setOption() construction, which would work, but consistently added the components in the wrong order. I finally found that I had to pull out a full array of handlers using getHandlers(), then array_unshift() the new component onto the front of the array, then put the whole array back in the configuration, to set the order correctly.

Re-customize custom block from upstream profile.

In most cases we’ve been able to use Simple Block module to provide “custom” blocks as configuration, rather than the core “custom” block types, which are treated as content. However, in one case we inherited a custom block type that had relevant fields like an image and call-to-action links and text.

Here again, the upstream OpenSocial modules create and install the block configs, and we didn’t want to fork/override the entire module just to make a small adjustment to the images and text/links. I came up with the following code block to effectively alter the block later in the installation process:

First, the helper function (called from the hook_install() of a late-stage feature module in our sub-profile), sets up the basic data elements needed, in order to make it easy to adjust the details later (and re-call this helper in a hook_update(), for example):

function _example_update_an_homepage_block() {

  ## CUSTOM ANON HOMEPAGE HERO BLOCK ##
  ## Edit $data array elements to update in future ##

  $data = array();
  $data['filename'] = 'bkgd-banner--front.png'; # Lives in the images/ folder of example module
  $data['textblock'] = '<h2>Example.org is a community of practice site.</h2>'

<p>Sign up now to <b>learn, share, connect </b>and<b> collaborate</b> with leaders and those in related fields.</p>
';
  $data['cta1'] = array(
    'url' => '/user/register',
    'text' => 'Get Started',
  );
  $data['cta2'] = array(
    'url' => '/about',
    'text' => 'More about the Community',
  );

  ## DO NOT EDIT BELOW THIS LINE! ##
  ##################################

The rest of the function does the heavy lifting:

  # This code cobbled together from `social_core.install` and # `social_demo/src/DemoSystem.php`
  // This uuid can be used like this since it's defined
  // in the code as well (@see social_core.install).
  $block = \Drupal::entityTypeManager()->getStorage('block_content')->loadByProperties(['uuid' => '8bb9d4bb-f182-4afc-b138-8a4b802824e4']);
  $block = current($block);

  if ($block instanceof \Drupal\block_content\Entity\BlockContent) {
    # Setup the image file
    $fid = _example_setup_an_homepage_image($data['filename']);

    $block->field_text_block = [
      'value' => $data['textblock'],
      'format' => 'full_html',
    ];

    // Insert image file in the hero image field.
    $block_image = [
      'target_id' => $fid,
      'alt' => "Anonymous front page image homepage'",
    ];
    $block->field_hero_image = $block_image;

    // Set the links.
    $action_links = [
      [
        'uri' => 'internal:' . $data['cta1']['url'],
        'title' => $data['cta1']['text'],
      ],
      [
        'uri' => 'internal:' . $data['cta2']['url'],
        'title' => $data['cta2']['text'],
      ],
    ];

    $itemList = new \Drupal\Core\Field\FieldItemList($block->field_call_to_action_link->getFieldDefinition());
    $itemList->setValue($action_links);
    $block->field_call_to_action_link = $itemList;

    $block->save();
  }
}

The image helper function prepares the image field:

function _example_setup_an_homepage_image($filename) {

  // TODO: use a better image from the theme.
  // Block image.
  $path = drupal_get_path('module', 'example');
  $image_path = $path . DIRECTORY_SEPARATOR . 'images' . DIRECTORY_SEPARATOR . $filename;
  $uri = file_unmanaged_copy($image_path, 'public://'.$filename, FILE_EXISTS_REPLACE);

  $media = \Drupal\file\Entity\File::create([
    'langcode' => 'en',
    'uid' => 1,
    'status' => 1,
    'uri' => $uri,
  ]);
  $media->save();

  $fid = $media->id();

  // Apply image cropping.
  $data = [
    'x' => 600,
    'y' => 245,
    'width' => 1200,
    'height' => 490,
  ];
  $crop_type = \Drupal::entityTypeManager()
    ->getStorage('crop_type')
    ->load('hero_an');
  if (!empty($crop_type) && $crop_type instanceof CropType) {
    $image_widget_crop_manager = \Drupal::service('image_widget_crop.manager');
    $image_widget_crop_manager->applyCrop($data, [
      'file-uri' => $uri,
      'file-id' => $fid,
    ], $crop_type);
  }

  return $fid;
}

Conclusion

As with most things I’ve encountered with Drupal 8 so far, the Update system is both familiar and new in certain respects. Hopefully these concrete examples are instructive to understand how to adapt older techniques to the new way of managing install and update tasks.

The article Drupal 8 hook_update() Tricks first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

May 24 2019
May 24

4 minute read Published: 24 May, 2019 Author: Colan Schwartz
Drupal Planet , Aegir , DevOps

Aegir is often seen as a stand-alone application lifecycle management (ALM) system for hosting and managing Drupal sites. In the enterprise context, however, it’s necessary to provide mutiple deployment environments for quality assurance (QA), development or other purposes. Aegir trivializes this process by allowing sites to easily be copied from one environment to another in a point-and-click fashion from the Web front-end, eliminating the need for command-line DevOps tasks, which it was designed to do.

Setting up the environments

An Aegir instance needs to be installed in each environment. We would typically have three (3) of them:

  • Development (Dev): While generally reserved for integration testing, it is sometimes also used for development (e.g. when local environments cannot be used by developers or there are a small number of them).
  • Staging: Used for QA purposes. Designed to be a virtual clone of Production to ensure that tagged releases operate the same way as they would there, before being made live.
  • Production (Prod): The live environment visible to the public or the target audience, and the authoritative source for data.

To install Aegir in each of these, follow the installation instructions. For larger deployments, common architectures for Staging and Prod would include features such as:

  • Separate Web and database servers
  • Multiple Web and database servers
  • Load balancers
  • Caching/HTTPS proxies
  • Separate partitions for (external) storage of:
    • The Aegir file system (/var/aegir)
    • Site backups (/var/aegir/backups)
    • Database storage (/var/lib/mysql)
  • etc.

As these are all out of scope for the purposes of this article, I’ll save these discussions for the future, and assume we’re working with default installations.

Allowing the environments to communicate

To enable inter-environment communication, we must perform the following series of tasks on each Aegir VM as part of the initial set-up, which only needs to be done once.

Back-end set-up

The back-ends of each instance must be able to communicate. For that we use the secure SSH protocol. As stated on Wikipedia:

SSH is important in cloud computing to solve connectivity problems, avoiding the security issues of exposing a cloud-based virtual machine directly on the Internet. An SSH tunnel can provide a secure path over the Internet, through a firewall to a virtual machine.


Steps to enable SSH communication:

  1. SSH into the VM.
    • ssh ENVIRONMENT.aegir.example.com
  2. Become the Aegir user.
    • sudo -sHu aegir
  3. Generate an SSH key. (If you’ve done this already to access a private Git repository, you can skip this step.)
    • ssh-keygen -t rsa -b 4096 -C "ORGANIZATION Aegir ENVIRONMENT"
  4. For every other environment from where you’d like to fetch sites:
    1. Add the generated public key (~/.ssh/id_rsa.pub) to the whitelist for the Aegir user on the other VM so that the original instance can connect to this target.
      • ssh OTHER_ENVIRONMENT.aegir.example.com
      • sudo -sHu aegir
      • vi ~/.ssh/authorized_keys
      • exit
    2. Back on the original VM, allow connections to the target VM.
      • sudo -sHu aegir
      • ssh OTHER_ENVIRONMENT.aegir.example.com
      • Answer affirmatively when asked to confirm the host (after verifying the fingerprint, etc.).

Front-end set-up

These steps will tell Aegir about the other Aegir servers whose sites can be imported.

  1. On Aegir’s front-end Web UI, the “hostmaster” site, enable remote site imports by navigating to Administration » Hosting » Advanced, and check the Remote import box. Save the form. (This enables the Aegir Hosting Remote Import module.)
  2. For every other server you’d like to add, do the following:
    1. Navigate to the Servers tab, and click on the Add server link.
    2. For the Server hostname, enter the hostname of the other Aegir server (e.g. staging.aegir.example.com)
    3. Click the Remote import vertical tab, check Remote hostmaster, and then enter aegir for the Remote user.
    4. For the Human-readable name, you can enter something like Foo's Staging Aegir (assuming the Staging instance).
    5. You can generally ignore the IP addresses section.
    6. Hit the Save button.
    7. Wait for the server verification to complete successfully.

All of the one-time command-line tasks are now done. You or your users can now use the Web UI to shuffle site data between environments.

Select remote site to import

Deploying sites from one environment to another

Whenever necessary, this point-and-click process can be used to deploy sites from one Aegir environment to another. It’s actually a pull method as the destination Aegir instance imports a site from the source.

Reasons to do this include:

  • The initial deployment of a development site from Dev to Prod.
  • Refreshing Dev and Staging sites from Prod.

Steps:

  1. If you’d like to install the site onto a new platform that’s not yet available, create the platform first.
  2. Navigate to the Servers tab.
  3. Click on the server hosting the site you’d like to import.
  4. Click on the Import remote sites link.
  5. Follow the prompts.
  6. Wait for the batch job, Import and Verify tasks to complete.
  7. Enable the imported site by hitting the Run button on the Enable task.
  8. The imported site is now ready for use!

The article Aegir DevOps: Deployment Workflows for Drupal Sites first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Oct 17 2018
Oct 17

5 minute read Published: 17 Oct, 2018 Author: Colan Schwartz
Drupal Planet , Semantic Web

As a content management framework, Drupal provides strong support for its taxonomical subsystem for classifying data. It would be great if such data could be exposed via the Simple Knowledge Organization System (SKOS) standard for publishing vocabularies as linked data. As Drupal becomes used more and more as a back-end data store (due to features such as built-in support for JSON:API), presenting this data in standard ways becomes especially important.

So is this actually possible now? If not, what remains to be done?

Drupal’s history

First, let’s explore some of Drupal core’s history as it relates to the Semantic Web and Web services formats, also useful for future reference. This is basically the backstory that makes all of this possible.

1. REST support was added to Views

This was implemented in the (now closed) issues:

2. Non-Schema.org namespace mappings were removed (including contrib’s UI support) in Drupal 8

Here’s the change notice:

And a follow-up issue requesting support for additional namespaces:

3. The community chose to replace JSON-LD with HAL in Drupal 8

Here’s an article with the details:

Taxonomy Screenshot

Multiple Components

As this is really a two-part issue, adding machine-readable metadata and then making machine-readable data available, I’ll split the discussion into two sections.

Adding machine-readable metadata

While there’s an RDF UI module that enables one to specify mappings between Drupal entities and their fields with RDF types and properties, it only supports Schema.org via RDFa (not JSON-LD).

As explained very well in Create SEO Juice From JSON LD Structured Data in Drupal, a better solution is to use the framework provided by the Metatag module (used by modules such as AGLS Metadata). The article introduces the Schema.org Metatag module, which uses the Metatag UI to allow users to map Drupal data to Schema.org, and exposes it via JSON-LD.

So one solution would be to:

  1. Clone Schema.org Metatag, calling the new module SKOS Metatag.
  2. Replace all of the Schema.org specifics with SKOS.
  3. Rejoice.

But after taking some time to process all of the above information, I believe we should be able to use the knowledge of the vocabulary hierarchy to add the SKOS metadata. We probably don’t need any admin UI at all for configuring mappings.

Assuming that’s true, we can instead create a SKOS module that doesn’t depend on Metatag, but Metatag may still be useful given that it already supports Views.

Making the machine-readable data available

Exposing the site’s data can be done best though Views. I wouldn’t recommend doing this any other way, e.g. accessing nodes (Drupal-speak for records) directly, or through any default taxonomy links for listing all of a vocabulary’s terms. (These actually are Views, but their default set-ups are missing configuration.) A good recipe for getting this up & running, for both the list and individual items, is available at Your First RESTful View in Drupal 8.

To actually access the data from elsewhere, you need to be aware of the recent API change To access REST export views, one now MUST specify a ?_format=… query string, which explains why some consumers broke.

The JSON-LD format is, however, not supported in Core by default. There is some code in a couple of sandboxes, which may or may not work, that will need to be ported to the official module, brought up-to-date, and have a release (ideally stable) cut. See the issue JSON-LD REST Services: Port to Drupal 8 for details.

Now, the Metatag solution I proposed in the previous section may work with Views natively, already exposing data as JSON-LD. If that’s the case, this JSON-LD port may not be necessary, but this remains to be seen. Also, accessing the records directly (without Views) may work as well, but this also remains to be seen after that solution is developed.

Conclusion

Clearly, there’s more work to be done. While the ultimate goal hasn’t been achieved yet, at least we have a couple of paths forward.

That’s as far as I got with pure research. Due to priorities shifting on the client project, I didn’t get a chance to learn more by reviewing the code and testing it to see what does and doesn’t work, which would be the next logical step.

If you’ve got a project that could make use of any of this, please reach out. We’d love to help move this technology further along and get it implemented.

References

General information

Contributed modules that probably aren’t helpful (but could be)

Questions about importing SKOS data (not exporting it)

The article Exposing Drupal’s Taxonomy Data on the Semantic Web first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web