Feeds

Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Apr 21 2021
Apr 21

6 minute read Published: 21 Apr, 2021 Author: Christopher Gervais
Drupal , Drupal Planet , CMI2.0 , Config Enforce , DevOps

Introduction to the Introduction

Over the last few years we’ve built lots of Drupal 8 sites, and some Drupal 9 ones too, both for our clients and for ourselves. As such, we’ve taken a keen interest in (read: faced many challenges with) the Configuration Management subsystem. This was a major new component in Drupal 8, and so, while it’s functional, it isn’t yet mature. Of course, the vibrant Drupal developer community jumped in to smooth the rough edges and fill the gaps, in what has since become known as CMI 2.0.

At Consensus, we tend to work on fairly large, complex Drupal projects that often require significant custom development. As such, we’ve adopted fairly rigourous software engineering processes, such as standardized local development environments, CI-enabled Test Driven Development, Continuous Delivery, etc.

However, we struggled to find a workflow that leveraged the powerful CMI system in core, while not being a pain for developers.

Configuration Management in D8+

The core CMI workflow assumes you are transferring configuration between a single site install with multiple instances (Dev, Stage, Prod) and that this configuration is periodically synchronized as a lump. For a number of reasons, this wouldn’t work for us.

As a result, we went back to our old standby, the venerable Features module, which worked reasonably well. Unfortunately, we found that it would sometimes handle dependencies between configuration objects poorly. On more than one occasion, this led to time-consuming debugging cycles.

So we switched to using Config Profile instead. However, reverting config changes was still manual, so we started using Config Update and the related Update Helper.

The Update Helper module, “offers supporting functionalities to make configuration updates easier.” Basically, when preparing for a release, Update Helper generates a special file, a “configuration update definition” (CUD). The CUD contains two values for each changed config. The first is the “current” value for a given configuration, as of the most recent release. The second is the new value to which you want to set that config.

These values are captured by first rolling back to the most recent release, then installing the site, so that the value is in active config. Then you checkout your latest commit, so that the new values are available on the filesystem. Update Helper can then generate its CUD, as well as generate a hook_update() implementations to help deploy the new or changed config.

This process turned out to be error-prone, and difficult to automate reliably.

We explored other efforts too, like Config Split and Config Filter which allow for finer-grained manipulation of “sets” of config. Other projects, like Config Distro, are focused on “packaging” groups of features such that they can be dropped in to any given site easily (kind of like Features…)

A simple, reliable method to deploy new or updated configuration remained elusive.

The underlying problem

Note that all the tools mentioned above work very well during initial project development, prior to production release. However, once you need to deploy config changes to systems in production Update Helper or similar tools and processes are required, along with all the overheads that implies.

At this point, it’s worth reminding ourselves that Drupal 7 and earlier versions did not clearly distinguish between content and config. They all just lived in the site’s database, after all. As such, whatever configuration was on the production site was generally considered canonical.

It’s tempting to make small changes directly in production, since they don’t seem to warrant a full release, and all the configuration deployment overhead that entails. This, in turn, requires additional discipline to reproduce those changes in the codebase.

Of course, that isn’t the only reason for configuration drift. Well-meaning administrators cannot easily distinguish between configs that are required for the proper operation of the site, and those that have more cosmetic effects.

Facing these challenges, we’d regularly note how much easier all of this would be if only we could make production configuration read-only.

A new approach

With some reluctance and much consideration, we decided to try an entirely new approach. We built Config Enforce (and its close companion Config Enforce Devel) to solve the two key problems we were running into:

  1. Developers needed an easy way to get from “I made a bunch of config-related changes in the Admin UI of my local site instance” to “I can identify the relevant config objects/entities which have changed, get them into my git repository, and push them upstream for deployment”.
  2. Operations needed an easy way to deploy changes in configuration, and ideally not have to worry too much about the previously-inevitable “drift” in the production-environment configuration, which often resulted in tedious and painful merging of configs, or worse yet, inadvertent clobbering of changes.

Config Enforce has two “modes” of operation: with config_enforce_devel enabled, you are in “development mode”. You can quickly designate config objects you want to enforce (usually inline on the configuration page where you manipulate the configuration object itself), and then changes you make are immediately written to the file system.

This mode leverages Config Devel to effectively bypass the active configuration storage in the database, writing the config files into target extensions you select. Each target extension builds up a “registry” of enforced configuration objects, keeping track of their location and enforcement level. This eases the development workflow by making it easy to identify which configuration objects you’ve changed without having to explictly config-export and then identify all and only the relevant .yml files to commit.

In production mode, you enable only the config_enforce module, which leverages the same “registry” configuration that config_enforce_devel has written into your target extensions, and performs the “enforcement” component. This means that, for any enforced configuration objects, we block any changes from being made via the UI or optionally even API calls directly. In turn, the enforced configuration settings on the file system within target extensions become authoritative, being pulled in to override whatever is in active configuration whenever a cache rebuild is triggered.

This means that deployment of configuration changes becomes trivial: commit and push new enforced configuration files in your target extensions, pull those into the new (e.g., Prod) environment, and clear caches. configuration Enforce will check all enforced configuration settings for changes on the file system, and immediately load them into active configuration on the site.

This workflow requires some adjustment in how we think about configuration management, but we think it has promise. Especially if you are building Drupal distributions or complex Drupal systems that require repeatable builds and comprehensive testing in CI, you should give Config Enforce a try and see what you think. Feedback is always welcome!

We’ve scratched our own itch, and so far have found it useful and productive. We are pleased to make it available to the Drupal community as another in the arena of ideas surrounding CMI 2.0.

The article Introducing Config Enforce first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Jan 27 2021
Jan 27

11 minute read Published: 27 Jan, 2021 Author: Seonaid Lee
DevOps , Drupal Planet

A lot of potential clients come to us with straightforward and small projects and ask, “Well, can you do Kubernetes?” And we say, “Well, we can, but you don’t need it.”

But they’re afraid that they’ll be missing out on something if we don’t add Kubernetes to the stack. So this is a post to tell you why we probably won’t be recommending Kubernetes.

This post is going to look at three perspectives on this question… First, I’ll consider the technical aspects, specifically what problems Kubernetes is really good for, compared with what problems most smaller software projects actually have.

Then I’ll talk about the psychology of “shiny problems.” (Yes, I’m looking at you, Ace Developer. I promise there are other shiny problems in the project you’re working on.)

And last but not least, we’ll consider the business problem of over-engineering, and what gets lost in the process.

Kubernetes solves specific problems

First off, (I’m sorry to draw your attention to this, but): You probably don’t have the problems that Kubernetes solves. Kubernetes lives at the container orchestration level. It shines in its ability to spin up and down stateless servers as needed for load balancing unpredictable or pulsed loads, especially from a large user base. Large, by the way, is not 10,000… it is millions or hundreds of millions.

Especially for the kind of internal custom services that a lot of our clients require, it is overkill. Many purpose-built sites are unlikely to have more than dozens or hundreds of users at a time, and traditional monolithic architectures will be responsive enough.

Kubernetes is designed to solve the problem of horizontal scalability, by making multiple copies of whichever services are most stressed, routing requests to minimize latency, and then being able to turn those machines back off when they are no longer needed. Even if you hope to someday have those problems, we suggest that you should hold off on adding Kubernetes to your stack until you get there, because the added technical overhead of container orchestration is expensive, in both time and dollars.

(It costs more to build, which delays your time to market, which delays your time to revenue, even if you aren’t paying yourself to build it.)

Which does lead to the question, “Why does everybody want to use this technology, anyway?” For that, we’ll have to take a step back and look at…

The Rise of the Twelve-Factor App

With the shift to the cloud and the desire for highly scalable applications, a new software architecture has arisen that has a strong separation between a system’s code and its data.

This approach treats processes as stateless and independent, and externalizes the database as a separate “backing service.” The stateless processes are isolated as microservices, which are each maintained, tested, and deployed as separate code bases.

This microservices approach decomposes the software into a group of related but separate apps, each of which is responsible for one particular part of the application.

Designing according to this architectural approach is non-trivial, and the overhead associated with maintaining the separate code bases, and particularly in coordinating among them is significant. Additionally, each app requires its own separate datastore, and maintaining synchronization in production introduces another level of complexity. Furthermore, extracting relevant queries from distributed systems of data is more challenging than simply writing a well-crafted SQL statement.

Each of these layers of complexity adds to the cost of not only the initial development, but also the difficulty of maintenance. Even Chris Richardson, in Microservices Patterns, recommends starting with a monolithic architecture for new software to allow rapid iteration in the early stages. (https://livebook.manning.com/book/microservices-patterns/chapter-1/174)

For many of the same reasons, you probably don’t need complex layers of data handling either. Redis, for example, is for persisting rapidly changing data in a quickly accessible form. It’s not suitable for a long-standing database with well-established relations, it costs more to run data in memory than to store it on disk, and it’s more difficult to build.

When you are getting started, a SQL back end with a single codebase will probably solve most of your problems, and without the overhead of Kubernetes (or any of the more exotic data stores.) If you’re still not convinced, let’s take a brief detour and consider the lifecycle of a typical application.

Development, CI, and Upgrades (Oh My!)

Most applications have the following characteristics:

  • Predictable load
  • Few (fewer than millions of) users
  • Predictable hours of use (open hours of the business, daily batch processing cron job at 3 AM, etc.)
  • Clear options for maintenance windows
  • Tight connection between the content layer and the presentation layer

Contrast this with the primary assumptions in the twelve-factor app approach.

Statelessness

The goal of moving to stateless servers is focused on different things:

  • Zero downtime
  • Rapid development
  • Scalability

This approach arose from the needs of large consumer-facing applications like Flickr, Twitter, Netflix, and Instagram. These need to be always-on for hundreds of millions or billions of users, and have no option for things like maintenance mode.

Development and Operations have different goals

When we apply the Dev-Ops calculus to smaller projects, though, there is an emphasis on Dev that comes at the expense of Ops.

Even though we may include continuous integration, automated testing and continuous deployment (and we strongly recommend these be included!), the design and implementation of the codebase and dependency management often focuses on “getting new developers up and running” with a simple “bundle install” (or “build install” etc.)

This is explicitly stated as a goal in the twelve-factor list.

This brings in several tradeoffs and issues in the long-term stability of the system; in particular, the focus on rapid development comes at a cost for operations and upgrades. The goal is to ship things and get them standing up from a cold start quickly… which is the easy part. The more difficult part of operations – the part you probably can’t escape, because you probably aren’t Netflix or Flickr or Instagram – is the maintenance of long-standing systems with live data.

Version Upgrades

Upgrades of conventional implementations proceed thusly:

  1. Copy everything to a staging server
  2. Perform the upgrade on the staging server
  3. If everything works, port it over to the production environment

There are time delays in this process: for large sites it can take hours to replicate a production database to staging, and if you want a safe upgrade, you need to put the site into maintenance mode to prevent the databases from diverging. The staging environment, no matter how carefully you set it up, is rarely an exact mirror of production; the connections to external services, passwords, and private keys for example, should not be shared. Generally, after the testing is complete in the staging environment, the same sequence of scripts is deployed in production. Even after extensive testing, it may prove necessary to roll back the production environment, database and all. Without the use of a maintenance freeze, this can result in data loss.

This sort of upgrade between versions is significantly easier in monolithic environments.

But isn’t Kubernetes supposed to make that easier?

It’s tempting to point to Kubernetes’ rolling updates and the ability to connect multiple microservices to different pods of the database running the different versions… but in content-focused environments, the trade-off for zero downtime is an additional layer of complexity required to protect against potential data loss.

Kubernetes and other 12-factor systems resolve the issue of data protection by sharding and mirroring the data across multiple stores. The database is separate from the application, and upgrades and rollbacks proceed separately. This is a strength for continuous delivery, but it comes at a cost: data that is produced in a blue environment during a blue-green deployment may simply be lost if it proves necessary to roll back schema changes. Additionally, if there are breaking changes to the schema and the microservices wind up attached to a non-backward compatible version, they can throw errors to the end-user (this is probably preferable to data loss.)

For data persistence, the data needs to be stored in volumes externally from the K8 cluster, and orchestrating multiple versions of the code base and database simultaneously requires significant knowledge and organization.

A deployment plan for such a system will need to include plans for having multiple versions of the code live on different servers at the same time, each of which connects to its associated database until the upgrade is complete and determined to be stable. It can be done, but even Kubernetes experts point out that this process is challenging to oversee.

When we are moving things into production, we need to have an operations team that knows how to respond when something fails. No matter how much testing you have done, sometimes a Big Hairy Bug gets into production, and you need to have enough control of your system to be able to fix it. Kubernetes, sad to say, makes this harder instead of easier for stateful applications.

So let’s consider what it means to have a stateful application.

When the Data is Intrinsic to the Application

A content management system by its nature is stateful. A stateful application has a lot of data that makes up a large fraction of “what it is.” State can also include cache information, which is volatile, but the data is part and parcel of what we are doing. Databases and the application layer are frequently tightly integrated, and it’s not meaningful to ship a new build without simultaneously applying schema updates. The data itself is the point of the application.

Drupal (for example) contains both content and configuration in the database, but there is additional information contained in the file structure. These, in combination, make up the state of the system… and the application is essentially meaningless without it. Also, as in most enterprise-focused applications, this data is not flat but is highly structured. The relationships are defined by both the database schema and the application code. It is not the kind of system that lends itself to scaling through the use of stateless containers.

In other words: by their very nature, Drupal applications lack the strict separation between database and code that makes Kubernetes an appropriate solution.

Shiny Problems

One of the things that (we) engineers fall into is a desire to solve interesting problems. Kubernetes, as one of the newest and most current technologies, is the “Shiny” technology towards which our minds bend.

But it is complex, has a steep learning curve, and is not the first choice when deploying stateful applications. This means that a lot of the problems you’re going to have to solve are going to be related to the containers and Kubernetes/deployment layer of the application, which will reduce the amount of time and energy you have to solve the problems at the data model and the application layer. We’ve never built a piece of software that didn’t have some interesting challenges; we promise they are available where you are working.

Also, those problems are probably what your company’s revenues rely on, so you should solve them first.

To Sum Up: Over-engineering isn’t free

As I hope I’ve convinced you, the use of heavier technologies than you need burns through your resources and has the potential to jeopardize your project. The desire to architect for the application you hope to have (rather than the one you do) can get your business into trouble. You will need more specialized developers, more complex deployment plans, additional architectural meetings and more coordination among the components.

When you choose technologies that are overpowered (in case you need them at some undefined point in the future), you front-load your costs and increase the risk that you won’t make it to revenue/profitability.

We get it. We love good tech as much as the next person.

But wow! Superfast!

The fact is, though, most projects don’t need response times measured in the millisecond range. They just need to be fast enough to keep users from wandering away from the keyboard while their query loads. (Or they need a reasonable queuing system, batch processing, and notification options.)

And even if you do need millisecond response times but you don’t have millions of users, Kubernetes will still introduce more problems than it solves.

Performance challenges like these are tough, but generally need to be solved by painstaking, time-consuming, unpredictable trial and error–and the more subcomponents your application is distributed/sharded into, the harder (more time consuming, more unpredictable - by orders of magnitude!) that trial and error gets.

But what if we’re the Next Big Thing?

Most sites are relatively small and relatively stable and will do quite well on a properly-sized VM with a well-maintained code base and a standard SQL server. Minimizing your technological requirements to those that are necessary to solve the problems at hand allows you to focus on your business priorities, leaving the complexity associated with containerization and the maintenance of external stateful information to a future iteration.

Leave the “How are we going to scale” problem for once you get there, and you increase the chances that this will eventually be the problem you have.

The article Kubernetes Won’t Save You first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Mar 12 2020
Mar 12

2 minute read Published: 12 Mar, 2020 Author: Colan Schwartz
Drupal Planet , Composer , Aegir , DevOps , Automation , Drupal 8

Best practices for building Web sites in the Drupal framework (for major versions 8 and above) dictate that codebases should be built with the Composer package manager for PHP. That is, the code repository for any sites relying on it should not contain any upstream code; it should only contain a makefile with instructions for assembing it.

However, there are some prominent Drupal hosting companies that don’t support Composer natively. That is, after receiving updates to Composer-controlled Git repositories, they don’t automatically rebuild the codebase, which should result in changes to the deployed code.

If you’re hosting your site(s) at one of these companies, and you have this problem, why not consider the obvious alternative?

Aegir, the one-and-only open-source hosting system for Drupal that’s been around for over 10 years, has had native Composer support for over 2 years. That is, on each and every platform deployment (“platform” is Aegir-speak for a Drupal codebase), Aegir reassembles the upstream code assets by running the following automatically:

composer create-project --no-dev --no-interaction --no-progress

As a result, any sites created on that platform (or migrated/upgraded to it) will have access to all of the assets built by Composer.

Additionally, Aegir now ships with the Aegir Deploy module, which enhances the platform creation process. It allows for the following types of deployment:

  • Classic/None/Manual/Unmanaged
  • Drush Makefile deployment
  • Pure Git
  • Composer deployment from a Git repository
  • Composer deployment from a Packagist repository

For more information, please read the Deployment Strategies section of the documentation.

If you’d like to get started with Aegir, the best option would be to spin up an Aegir Development VM, which allows you to run it easily, play with it, and get familiar with the concepts. Naturally, reading the documentation helps with this too.

Afterwards, review the installation guide for more permanent options, and take advantage of our Ansible roles. We have a policy role that configures the main role using our favoured approach.

For help, contact the community, or get in touch with us directly. We provide the following Aegir services:

  • Installation & maintenance in corporate/enterprise (or other) environments
  • Architectural and technical support
  • Hosting guidance
  • Coaching
  • Audits
  • Upgrades
  • Conversion to best practices

The article Does your Drupal hosting company lack native Composer support? first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Nov 15 2019
Nov 15

1 minute read Published: 15 Nov, 2019 Author: Colan Schwartz
Drupal Planet , SaaS , OpenSaaS , DevOps , Aegir , OpenStack , Presentations

On Friday, June 14th, I presented this session at Drupal North 2019. That’s the annual gathering of the Drupal community in Ontario and Quebec, in Canada.

As I realized I hadn’t yet posted this information yet, I’m doing so now.

Session information:

Are you (considering) building a SaaS product on Drupal or running a Drupal hosting company? Have you done it already? Come share your experiences and learn from others.

Among other things, we’ll be discussing:

…and any other related topics that come up.

A video recording of my presentation is available on:

My slides (with clickable links) are available on our presentations site.

The article Drupal North 2019: Drupal SaaS: Building software as a service on Drupal first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Nov 07 2019
Nov 07

6 minute read Published: 7 Nov, 2019 Author: Derek Laventure
Drupal Planet , Drupal 8 , Lando , Drumkit

Over the last 2 or 3 years, the Drupal community has been converging around a solid set of Docker-based workflows to manage local development environments, and there are a number of worthy tools that make life easier.

My personal favourite is Lando, not only because of the Star Wars geekery, but also because it makes easy things easy and hard things possible (a lot like Drupal). I appreciate that a “standard” Lando config file is only a few lines long, but that it’s relatively easy to configure and customize a much more complex setup by simply adding the appropriate lines to the config.

In this post I want to focus on an additional tool I’ve come to lean on heavily that complements Lando quite nicely, and that ultimately boils down to good ol’ fashioned Makefiles. Last summer at DrupalNorth I gave a talk that was primarily about the benefits of Lando, and I only mentioned Drumkit in passing. Here I want to illustrate in more detail how and why this collection of Makefile tools is a valuable addition to my localdev toolbox.

The key benefits provided by adding a Drumkit environment are:

  • consistent make -based workflow to tie various dev tasks together
  • ease onboarding of new devs (make help)
  • make multistep tasks easier (make tests)
  • make tasks in Lando or CI environment the same (ie. make install && make tests)

Drumkit is not just for Drupal!

This example is using Drumkit for a Drupal 8 localdev environment, but there’s no reason you couldn’t use it for other purposes (and in fact, we at Consensus have lately been doing just that.

Basic Setup

As an example, suppose you’re setting up a new D8 project from scratch. Following this slide from my Lando talk, you would do the basic Lando D8 project steps:

  1. Create codebase with Composer (composer create-project drupal-composer/drupal-project:8.x-dev code --stability dev --no-interaction)
  2. Initialize Git repository (git init etc.)
  3. Initialize Lando (lando init)

For now, leave out the lando start step, which we’ll let Drumkit handle momentarily. We should also customize the .lando.yml a little with custom database credentials, which we’ll tell Drumkit about later. Append the following to your .lando.yml:

services:
  database:
    creds:
      user: chewie_dbuser
      password: chewie_dbpass
      database: chewie_db

Add Drumkit

To insert Drumkit into this setup, we add it as a git submodule to our project using the helper install.sh script, and bootstrap Drumkit:

wget -O - https://gitlab.com/consensus.enterprises/drumkit/raw/master/scripts/install.sh | /bin/bash
. d  # Use 'source d' if you're not using Bash

The install script checks that you are in the root of a git repository, and pulls in Drumkit as a submodule, then initializes a top-level Makefile for you.

Finally, we initialize the Drumkit environment, by sourcing the d script (itself a symlink to .mk/drumkit) into our shell.

Drumkit modifies the (shell) environment!

Note that Drumkit will modify your PATH and BIN_PATH variables to add the project-specific .mk/.local/bin directory, which is where Drumkit installs any tools you request (eg. with make selenium. This means if you have multiple Drumkit-enabled projects on the go, you’re best to work on them in separate shell instances, to keep these environment variables distinct.

Note that you can take advantage of this environment-specific setup to customize the bootstrap script to (for example) inject project credentials for external services into the shell environment. Typically we would achieve this by creating a scripts/bootstrap.sh that in turn calls the main .mk/drumkit, and re-point the d symlink there.

Set up your kit

Because we’re using Composer to manage our codebase, we also add a COMPOSER_CACHE_DIR environment variable, using the standard .env file, which Drumkit’s stock bootstrap script will pull into your environment:

echo "COMPOSER_CACHE_DIR=tmp/composer-cache/" >> .env
. d # Bootstrap Drumkit again to have this take effect

From here, we can start customizing for Drupal-specific dev with Lando. First, we make a place in our repo for some Makefile snippets to be included:

mkdir -p scripts/makefiles
echo "include scripts/makefiles/*.mk" >> Makefile

Now we can start creating make targets for our project (click the links below to see the file contents in an example Chewie project. For modularity, we create a series of “snippet” makefiles to provide the targets mentioned above:

NB You’ll need to customize the variables.mk file with the DB credentials you set above in your .lando.yml as well as your site name, admin user/password, install profile, etc.

Now our initial workflow to setup the project looks like this:

git clone --recursive 
cd 
. d # or "source d" if you're not using Bash
make start
make build
make install

This will get a new developer up and running quickly, and can be customized to add whatever project-specific steps are needed along the way.

But wait- it gets even better! If I want to make things really easy on fellow developers (or even just myself), I can consolidate common steps into a single target within the top-level Makefile. For example, append the make all target to your Makefile:

.PHONY: all

all:
        @$(MAKE-QUIET) start
        @$(MAKE-QUIET) build
        @$(MAKE-QUIET) install

Now, the above workflow for a developer getting bootstrapped into the project simplifies down to this:

git clone --recursive 
cd 
. d
make all

Customize your kit

At this point, you can start adding your own project-specific targets to make common workflow tasks easier. For example, on a recent migration project I was working on, we had a custom Features module (ingredients) that needed to be enabled, and a corresponding migration module (ingredients_migrate) that needed to be enabled before migrations could run.

I created the following make targets to facilitate that workflow:

We often take this further, adding a make tests target to setup and run our test suite, for example. This in turn allows us to automate the build/install/test process within our CI environment, which can call exactly the same make targets as we do locally.

Ultimately, Drumkit is a very simple idea: superimpose a modular Makefile-driven system on top of Lando to provide some syntactic sugar that eases developer workflow, makes consistent targets that CI can use, and consolidates multi-step tasks into a single command.

There’s lots more that Drumkit can do, and plenty of ideas we have yet to implement, so if you like this idea, feel free to jump in and contribute!

The article Lando and Drumkit for Drupal 8 Localdev first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Oct 25 2019
Oct 25

1 minute read Published: 24 Oct, 2019 Author: Christopher Gervais
Drupal Planet , Automation , DevOps , Ansible , OpenStack , Presentations

On Friday, October 18th, I presented at DrupalCamp Ottawa 2019. That’s the annual gathering of the Drupal community in Ottawa, Ontario, Canada.

Session information:

Ever heard of infrastructure-as-code? The idea is basically to use tools like Ansible or Terraform to manage the composition and operation of your cloud systems. This allows infrastructure to be treated just like any other software system. The code can be committed into Git which allows auditability, and reproducibility. It can therefore be tested and integrated into full continuous delivery processes.

Ansible provides tonnes of cloud management modules, from simple Linodes or Digital Ocean Droplets through globe-spanning AWS networks. Ansible also strives for simplicity, resulting in playbooks that are essentially self-documenting.

In this session, we will:

  • explore the principles of infrastructure-as-code and how to operationalize them;
  • introduce Ansible and it’s cloud modules;
  • build a full OpenStack cloud infrastructure end-to-end from scratch.

A video recording of my presentation is available on YouTube.

[embedded content]

My presentation slidedeck can be downloaded here: Automate All the Things!

The article DrupalCamp Ottawa 2019: Automate All the Things first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Oct 25 2019
Oct 25

5 minute read Published: 24 Oct, 2019 Author: Colan Schwartz
Drupal Planet , Semantic Web

As a content management framework, Drupal provides strong support for its taxonomical subsystem for classifying data. It would be great if such data could be exposed via the Simple Knowledge Organization System (SKOS) standard for publishing vocabularies as linked data. As Drupal becomes used more and more as a back-end data store (due to features such as built-in support for JSON:API), presenting this data in standard ways becomes especially important.

So is this actually possible now? If not, what remains to be done?

Drupal’s history

First, let’s explore some of Drupal core’s history as it relates to the Semantic Web and Web services formats, also useful for future reference. This is basically the backstory that makes all of this possible.

1. REST support was added to Views

This was implemented in the (now closed) issues:

2. Non-Schema.org namespace mappings were removed (including contrib’s UI support) in Drupal 8

Here’s the change notice:

And a follow-up issue requesting support for additional namespaces:

3. The community chose to replace JSON-LD with HAL in Drupal 8

Here’s an article with the details:

Taxonomy Screenshot

Multiple Components

As this is really a two-part issue, adding machine-readable metadata and then making machine-readable data available, I’ll split the discussion into two sections.

Adding machine-readable metadata

While there’s an RDF UI module that enables one to specify mappings between Drupal entities and their fields with RDF types and properties, it only supports Schema.org via RDFa (not JSON-LD).

As explained very well in Create SEO Juice From JSON LD Structured Data in Drupal, a better solution is to use the framework provided by the Metatag module (used by modules such as AGLS Metadata). The article introduces the Schema.org Metatag module, which uses the Metatag UI to allow users to map Drupal data to Schema.org, and exposes it via JSON-LD.

So one solution would be to:

  1. Clone Schema.org Metatag, calling the new module SKOS Metatag.
  2. Replace all of the Schema.org specifics with SKOS.
  3. Rejoice.

But after taking some time to process all of the above information, I believe we should be able to use the knowledge of the vocabulary hierarchy to add the SKOS metadata. We probably don’t need any admin UI at all for configuring mappings.

Assuming that’s true, we can instead create a SKOS module that doesn’t depend on Metatag, but Metatag may still be useful given that it already supports Views.

Making the machine-readable data available

Exposing the site’s data can be done best though Views. I wouldn’t recommend doing this any other way, e.g. accessing nodes (Drupal-speak for records) directly, or through any default taxonomy links for listing all of a vocabulary’s terms. (These actually are Views, but their default set-ups are missing configuration.) A good recipe for getting this up & running, for both the list and individual items, is available at Your First RESTful View in Drupal 8.

To actually access the data from elsewhere, you need to be aware of the recent API change To access REST export views, one now MUST specify a ?_format=… query string, which explains why some consumers broke.

The JSON-LD format is, however, not supported in Core by default. There is some code in a couple of sandboxes, which may or may not work, that will need to be ported to the official module, brought up-to-date, and have a release (ideally stable) cut. See the issue JSON-LD REST Services: Port to Drupal 8 for details.

Now, the Metatag solution I proposed in the previous section may work with Views natively, already exposing data as JSON-LD. If that’s the case, this JSON-LD port may not be necessary, but this remains to be seen. Also, accessing the records directly (without Views) may work as well, but this also remains to be seen after that solution is developed.

Conclusion

Clearly, there’s more work to be done. While the ultimate goal hasn’t been achieved yet, at least we have a couple of paths forward.

That’s as far as I got with pure research. Due to priorities shifting on the client project, I didn’t get a chance to learn more by reviewing the code and testing it to see what does and doesn’t work, which would be the next logical step.

If you’ve got a project that could make use of any of this, please reach out. We’d love to help move this technology further along and get it implemented.

References

General information

Contributed modules that probably aren’t helpful (but could be)

Questions about importing SKOS data (not exporting it)

The article Exposing Drupal’s Taxonomy Data on the Semantic Web first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Oct 08 2019
Oct 08

9 minute read Published: 8 Oct, 2019 Author: Derek Laventure
Drupal Planet , Drupal 8 , OpenSocial

In Drupal 7, hook_update()/hook_install() were well-established mechanisms for manipulating the database when installing a new site or updating an existing one. Most of these routines ended up directly running SQL against the database, where all kinds of state, configuration, and content data lived. This worked reasonably well if you were careful and had a good knowledge of how the database schema fit together, but things tended to get complicated.

With the maturing of Features module, we were able to move some of this into configuration settings via the ctools-style export files, making the drush feature-revert command part of standard workflow for deploying new features and updates to an existing site.

In Drupal 8, we’ve made huge strides in the direction of Object Orientation, and started to separate Configuration/State, Content Structure, and Content itself. The config/install directory is often all that’s needed in terms of setting up a contributed or custom module to work out of the box, and with the D8 version of Features, the same is often true of updates that involve straightforward updates to configuration .yml files.

It turns out that both hook_update() and hook_install() are still valuable tools in our box, however, so I decided to compile some of the more complicated D8 scenarios I’ve run across recently.

Drupal 8 Update basics

The hook_update_N API docs reveal that this function operates more or less as before, with some excellent guidelines for how to approach the body of the function’s implementation. The Introduction to update API handbook page provides some more detail and offers some more guidance around the kinds of updates to handle, naming conventions, and adding unit tests to the your update routines.

The sub-pages of that Handbook section have some excellent examples covering the basics:

All of these provided a valuable basis on which to write my own real-life update hooks, but I found I still had to combine various pieces and search through code to properly write these myself.

Context

We recently launched our first complex platform based on Drupal 8 and the excellent OpenSocial, albeit heavily modified to suit the particular requirements of the project. The sub-profile required more extensive customization than simply extending the parent profile’s functionality (as discussed here). Instead, we needed to integrate new functionality into that provided by the upstream distribution, and this often resulted in tricky interactions between the two.

Particularly with a complex site with many moving parts, we take the approach of treating the site as a system or platform, installing and reinstalling regularly via a custom installation profile and set of feature modules. This allows us to integrate:

  • a CI system to build the system repeatedly, proving that everything works
  • a Behat test suite to validate the behaviour of the platform matches the requirements

In the context of a sub-profile of OpenSocial, this became complicated when the configuration we wanted to customize actually lived in feature modules from the upstream profile, and there was no easy way to just override them in our own modules’ config/install directories.

We developed a technique of overriding entire feature modules within our own codebase, effectively forking the upstream versions, so that we could then modify the installed configuration and other functionality (in Block Plugins, for example). The trouble with this approach is that you have to manage the divergence upstream, incorporating new improvements and fixes manually (and with care).

Thus, in cases where there were only a handful of configuration items to correct, we began using hook_install() routines to adjust the upstream-installed config later in the install process, to end up with the setup we were after.

Adjust order of user/register form elements

We make use of entity_legal for Terms of Service, Privacy Policy, and User Guidelines documents. Our installation profile’s feature modules create the 3 entity legal types, but we needed to be able to tweak the order of the form elements on the user/register page, which is a core entity_form_display created for the user entity.

To achieve this using YAML files in the config/install directory per usual seemed tricky or impossible, so I wrote some code to run near the end of the installation process, after the new legal_entity types were created and the core user.register form display was set. This code simply loads up the configuration in question, makes some alterations to it, and then re-saves:

/**
 * Implements hook_install().
 */
function example_install() {
  _example_install_adjust_legal_doc_weights();
}

/**
 * Adjust weights of legal docs in user/register form.
 */
function example_update_8001() {
  _example_install_adjust_legal_doc_weights();
}

/**
 * Ensure the field weights on the user register form put legal docs at the bottom
 */
function _example_install_adjust_legal_doc_weights() {
       $config = \Drupal::getContainer()->get('config.factory')->getEditable('core.entity_form_display.user.user.register');
       $content = $config->get('content');

       $content['private_messages']['weight'] = 0;
       $content['account']['weight'] = 1;
       $content['google_analytics']['weight'] = 2;
       $content['path']['weight'] = 3;
       $content['legal_terms_of_service']['weight'] = 4;
       $content['legal_privacy_policy']['weight'] = 5;
       $content['legal_user_guidelines']['weight'] = 6;
       $config->set('content', $content)->save();
}

Modify views configuration managed by upstream (or core)

A slightly more complicated situation is to alter a views configuration that is managed by an upstream feature module during the installation process. This is not an ideal solution, but currently it’s quite challenging to properly “override” configuration that’s managed by a “parent” installation profile within your own custom sub-profile (although Config Actions appears to be a promising solution to this).

As such, this was the best solution I could come up with: essentially, run some code very nearly at the end of the installation process (an installation profile task after all the contrib and feature modules and related configuration are installed), that again loads up the views configuration, changes the key items needed, and then re-saves it.

In this case, we wanted to add a custom text header to a number of views, as well as switch the pager type from the default “mini” type to “full”. This required some thorough digging into the Views API and related code, to determine how to adjust the “handlers” programmatically.

This helper function lives in the example.profile code itself, and is called via a new installation task wrapper function, which passes in the view IDs that need to be altered. Here again, we can write trivial hook_update() implementations that call this same wrapper function to update existing site instances.

/**
 * Helper to update views config to add header and set pager.
 */
function _settlement_install_activity_view_header($view_id) {
  # First grab the view and handler types
  $view = Views::getView($view_id);
  $types = $view->getHandlerTypes();

  # Get the header handlers, and add our new one
  $headers = $view->getHandlers('header', 'default');

  $custom_header = array(
    'id' => 'area_text_custom',
    'table' => 'views',
    'field' => 'area_text_custom',
    'relationship' => 'none',
    'group_type' => 'group',
    'admin_label' => '',
    'empty' => '1',
    'content' => '

Latest Activity

', 'plugin_id' => 'text_custom', 'weight' => -1, ); array_unshift($headers, $custom_header); # Add the list of headers back in the right order. $view->displayHandlers->get('default')->setOption($types['header']['plural'], $headers); # Set the pager type to 'full' $pager = $view->getDisplay()->getOption('pager'); $pager['type'] = 'full'; $view->display_handler->setOption('pager', $pager); $view->save(); }

Of particular note here is the ordering of the Header components on the views. There was an existing Header on most of the views, and the new “Latest Activity” one needed to appear above the existing one. Initially I had tried creating the new custom element and calling ViewExecutable::setHandler method instead of the more complicated $view->displayHandlers->get('default')->setOption() construction, which would work, but consistently added the components in the wrong order. I finally found that I had to pull out a full array of handlers using getHandlers(), then array_unshift() the new component onto the front of the array, then put the whole array back in the configuration, to set the order correctly.

Re-customize custom block from upstream profile.

In most cases we’ve been able to use Simple Block module to provide “custom” blocks as configuration, rather than the core “custom” block types, which are treated as content. However, in one case we inherited a custom block type that had relevant fields like an image and call-to-action links and text.

Here again, the upstream OpenSocial modules create and install the block configs, and we didn’t want to fork/override the entire module just to make a small adjustment to the images and text/links. I came up with the following code block to effectively alter the block later in the installation process:

First, the helper function (called from the hook_install() of a late-stage feature module in our sub-profile), sets up the basic data elements needed, in order to make it easy to adjust the details later (and re-call this helper in a hook_update(), for example):

function _example_update_an_homepage_block() {

  ## CUSTOM ANON HOMEPAGE HERO BLOCK ##
  ## Edit $data array elements to update in future ##

  $data = array();
  $data['filename'] = 'bkgd-banner--front.png'; # Lives in the images/ folder of example module
  $data['textblock'] = '

Example.org is a community of practice site.

'

Sign up now to learn, share, connect and collaborate with leaders and those in related fields.

'; $data['cta1'] = array( 'url' => '/user/register', 'text' => 'Get Started', ); $data['cta2'] = array( 'url' => '/about', 'text' => 'More about the Community', ); ## DO NOT EDIT BELOW THIS LINE! ## ##################################

The rest of the function does the heavy lifting:

  # This code cobbled together from `social_core.install` and # `social_demo/src/DemoSystem.php`
  // This uuid can be used like this since it's defined
  // in the code as well (@see social_core.install).
  $block = \Drupal::entityTypeManager()->getStorage('block_content')->loadByProperties(['uuid' => '8bb9d4bb-f182-4afc-b138-8a4b802824e4']);
  $block = current($block);

  if ($block instanceof \Drupal\block_content\Entity\BlockContent) {
    # Setup the image file
    $fid = _example_setup_an_homepage_image($data['filename']);

    $block->field_text_block = [
      'value' => $data['textblock'],
      'format' => 'full_html',
    ];

    // Insert image file in the hero image field.
    $block_image = [
      'target_id' => $fid,
      'alt' => "Anonymous front page image homepage'",
    ];
    $block->field_hero_image = $block_image;

    // Set the links.
    $action_links = [
      [
        'uri' => 'internal:' . $data['cta1']['url'],
        'title' => $data['cta1']['text'],
      ],
      [
        'uri' => 'internal:' . $data['cta2']['url'],
        'title' => $data['cta2']['text'],
      ],
    ];

    $itemList = new \Drupal\Core\Field\FieldItemList($block->field_call_to_action_link->getFieldDefinition());
    $itemList->setValue($action_links);
    $block->field_call_to_action_link = $itemList;

    $block->save();
  }
}

The image helper function prepares the image field:

function _example_setup_an_homepage_image($filename) {

  // TODO: use a better image from the theme.
  // Block image.
  $path = drupal_get_path('module', 'example');
  $image_path = $path . DIRECTORY_SEPARATOR . 'images' . DIRECTORY_SEPARATOR . $filename;
  $uri = file_unmanaged_copy($image_path, 'public://'.$filename, FILE_EXISTS_REPLACE);

  $media = \Drupal\file\Entity\File::create([
    'langcode' => 'en',
    'uid' => 1,
    'status' => 1,
    'uri' => $uri,
  ]);
  $media->save();

  $fid = $media->id();

  // Apply image cropping.
  $data = [
    'x' => 600,
    'y' => 245,
    'width' => 1200,
    'height' => 490,
  ];
  $crop_type = \Drupal::entityTypeManager()
    ->getStorage('crop_type')
    ->load('hero_an');
  if (!empty($crop_type) && $crop_type instanceof CropType) {
    $image_widget_crop_manager = \Drupal::service('image_widget_crop.manager');
    $image_widget_crop_manager->applyCrop($data, [
      'file-uri' => $uri,
      'file-id' => $fid,
    ], $crop_type);
  }

  return $fid;
}

Conclusion

As with most things I’ve encountered with Drupal 8 so far, the Update system is both familiar and new in certain respects. Hopefully these concrete examples are instructive to understand how to adapt older techniques to the new way of managing install and update tasks.

The article Drupal 8 hook_update() Tricks first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Sep 24 2019
Sep 24

4 minute read Published: 24 Sep, 2019 Author: Colan Schwartz
Drupal Planet , Aegir , DevOps

Aegir is often seen as a stand-alone application lifecycle management (ALM) system for hosting and managing Drupal sites. In the enterprise context, however, it’s necessary to provide mutiple deployment environments for quality assurance (QA), development or other purposes. Aegir trivializes this process by allowing sites to easily be copied from one environment to another in a point-and-click fashion from the Web front-end, eliminating the need for command-line DevOps tasks, which it was designed to do.

Setting up the environments

An Aegir instance needs to be installed in each environment. We would typically have three (3) of them:

  • Development (Dev): While generally reserved for integration testing, it is sometimes also used for development (e.g. when local environments cannot be used by developers or there are a small number of them).
  • Staging: Used for QA purposes. Designed to be a virtual clone of Production to ensure that tagged releases operate the same way as they would there, before being made live.
  • Production (Prod): The live environment visible to the public or the target audience, and the authoritative source for data.

(While outside the scope of this article, local development environments can be set up as well. See Try Aegir now with the new Dev VM for details.)

To install Aegir in each of these, follow the installation instructions. For larger deployments, common architectures for Staging and Prod would include features such as:

  • Separate Web and database servers
  • Multiple Web and database servers
  • Load balancers
  • Caching/HTTPS proxies
  • Separate partitions for (external) storage of:
    • The Aegir file system (/var/aegir)
    • Site backups (/var/aegir/backups)
    • Database storage (/var/lib/mysql)
  • etc.

As these are all out of scope for the purposes of this article, I’ll save these discussions for the future, and assume we’re working with default installations.

Allowing the environments to communicate

To enable inter-environment communication, we must perform the following series of tasks on each Aegir VM as part of the initial set-up, which only needs to be done once.

Back-end set-up

The back-ends of each instance must be able to communicate. For that we use the secure SSH protocol. As stated on Wikipedia:

SSH is important in cloud computing to solve connectivity problems, avoiding the security issues of exposing a cloud-based virtual machine directly on the Internet. An SSH tunnel can provide a secure path over the Internet, through a firewall to a virtual machine.


Steps to enable SSH communication:

  1. SSH into the VM.
    • ssh ENVIRONMENT.aegir.example.com
  2. Become the Aegir user.
    • sudo -sHu aegir
  3. Generate an SSH key. (If you’ve done this already to access a private Git repository, you can skip this step.)
    • ssh-keygen -t rsa -b 4096 -C "ORGANIZATION Aegir ENVIRONMENT"
  4. For every other environment from where you’d like to fetch sites:
    1. Add the generated public key (~/.ssh/id_rsa.pub) to the whitelist for the Aegir user on the other VM so that the original instance can connect to this target.
      • ssh OTHER_ENVIRONMENT.aegir.example.com
      • sudo -sHu aegir
      • vi ~/.ssh/authorized_keys
      • exit
    2. Back on the original VM, allow connections to the target VM.
      • sudo -sHu aegir
      • ssh OTHER_ENVIRONMENT.aegir.example.com
      • Answer affirmatively when asked to confirm the host (after verifying the fingerprint, etc.).

Front-end set-up

These steps will tell Aegir about the other Aegir servers whose sites can be imported.

  1. On Aegir’s front-end Web UI, the “hostmaster” site, enable remote site imports by navigating to Administration » Hosting » Advanced, and check the Remote import box. Save the form. (This enables the Aegir Hosting Remote Import module.)
  2. For every other server you’d like to add, do the following:
    1. Navigate to the Servers tab, and click on the Add server link.
    2. For the Server hostname, enter the hostname of the other Aegir server (e.g. staging.aegir.example.com)
    3. Click the Remote import vertical tab, check Remote hostmaster, and then enter aegir for the Remote user.
    4. For the Human-readable name, you can enter something like Foo's Staging Aegir (assuming the Staging instance).
    5. You can generally ignore the IP addresses section.
    6. Hit the Save button.
    7. Wait for the server verification to complete successfully.

All of the one-time command-line tasks are now done. You or your users can now use the Web UI to shuffle site data between environments.

Select remote site to import

Deploying sites from one environment to another

Whenever necessary, this point-and-click process can be used to deploy sites from one Aegir environment to another. It’s actually a pull method as the destination Aegir instance imports a site from the source.

Reasons to do this include:

  • The initial deployment of a development site from Dev to Prod.
  • Refreshing Dev and Staging sites from Prod.

Steps:

  1. If you’d like to install the site onto a new platform that’s not yet available, create the platform first.
  2. Navigate to the Servers tab.
  3. Click on the server hosting the site you’d like to import.
  4. Click on the Import remote sites link.
  5. Follow the prompts.
  6. Wait for the batch job, Import and Verify tasks to complete.
  7. Enable the imported site by hitting the Run button on the Enable task.
  8. The imported site is now ready for use!

The article Aegir DevOps: Deployment Workflows for Drupal Sites first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

Sep 09 2019
Sep 09

3 minute read Published: 9 Sep, 2019 Author: Colan Schwartz
Drupal Planet , Aegir , DevOps

Have you been looking for a self-hosted solution for hosting and managing Drupal sites? Would you like be able able to upgrade all of your sites at once with a single button click? Are you tired of dealing with all of the proprietary Drupal hosting providers that won’t let you customize your set-up? Wouldn’t it be nice if all of your sites had free automatically-updating HTTPS certificates? You probably know that Aegir can do all of this, but it’s now trivial to set up a temporary trial instance to see how it works.

The new Aegir Development VM makes this possible.

History

Throughout Aegir’s history, we’ve had several projects striving to achieve the same goal. They’re listed in the Contributed Projects section of the documentation.

Aegir Up

Aegir Up was based on a VirtualBox virtual machine (VM), managed by Vagrant and provisioned with Puppet. It was superseded by Valkyrie (see below).

Aegir Development Environment

Aegir Development Environment took a completely different approach using Docker. It assembles all of the services (each one in a container, e.g. the MySQL database) into a system managed by Docker Compose. While this is a novel approach, it’s not necessary to have multiple containers to get a basic Aegir instance up and running.

Valkyrie

Valkyrie was similar to Aegir Up, but provisioning moved from Puppet to Ansible. Valkyrie also made extensive use of custom Drush commands to simplify development.

Its focus was more on developing Drupal sites than on developing Aegir. Now that we have Lando, it’s no longer necessary to include this type of functionality.

It was superseded by the now current Aegir Development VM.

Present

Like Valkyrie, the Aegir Development VM is based on a VirtualBox VM (but that’s not the only option; see below) managed with Vagrant and provisioned with Ansible. However, it doesn’t rely on custom Drush commands.

Features

Customizable configuration

The Aegir Development VM configuration is very easy to customize as Ansible variables are used throughout.

For example, if you’d like to use Nginx instead of Apache, simply replace:

    aegir_http_service_type: apache

…with:

    aegir_http_service_type: nginx

…or override using the command line.

You can also install and enable additional Aegir modules from the available set.

Support for remote VMs

For those folks with older hardware who are unable to spare extra gigabytes (GB) for VMs, it’s possible to set up the VM remotely.

While the default amount of RAM necessary is 1 GB, 2 GB would be better for any serious work, and 4 GB is necessary if creating platforms directly from Packagist.

Support for DigitalOcean is included, but other IaaS providers (e.g. OpenStack) can be added later. Patches welcome!

Fully qualified domain name (FQDN) not required

While Aegir can quickly be installed with a small number of commands in the Quick Start Guide, that process requires an FQDN, usually something like aegir.example.com (which requires global DNS configuration). That is not the case with the Dev VM, which assumes aegir.local by default.

Simplified development

You can use it for Aegir development as well as trying Aegir!

Unlike the default set-up provisioned by the Quick Start Guide, which would require additional configuration, the individual components (e.g. Hosting, Provision, etc.) are cloned repositories making it easy to create patches (and for module maintainers: push changes upstream).

Conclusion

We’ve recently updated the project so that an up-to-date VM is being used, and it’s now ready for general use. Please go ahead and try it.

If you run into any problems, feel free to create issues on the issue board and/or submit merge requests.

The article Try Aegir now with the new Dev VM first appeared on the Consensus Enterprises blog.

We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web