Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Sep 14 2021
Sep 14

Decoupling separates the system that stores the content from how that content is displayed on other independent systems. This can come with many benefits but also some downsides and tradeoffs. With progressive decoupling, you can get some of the benefits of decoupling while avoiding some of the downsides.

This is a re-post of the article on the Lullabot blog.

Slides available here.

Decoupling separates the system that stores the content from how that content is displayed on other independent systems. This can come with many benefits but also some downsides and tradeoffs.

With progressive decoupling, you can get some of the benefits of decoupling while avoiding some of the downsides.

There are several ways to decouple a website progressively, but this article makes the case that widgets provide the most flexibility.

What are widgets?

Widgets are stand-alone JavaScript (JS) applications that are framework-agnostic and are designed to be embedded and configured by CMS editors.

Widgets can be vanilla JS or use frameworks like Vue or React.

Why JS over server-generated HTML?

Better reactivity and interactivity

The pages can be static or served from cache (very fast), and JS can be sprinkled on top. The server can provide the unchanging parts, while the JS application adds interactivity.

This reduces the load on your servers while increasing website performance. You keep the benefits of built-in CMS performance tooling.

Distributed delivery

Different development teams can write software independently. They can publish software on the same platform without coordinating complex deployment efforts.

  • Teams write the JS code in isolation
  • The browser executes the JS
  • Different deployment pipelines and servers can be used.

Distributed example

One team works on the navigation, one team works on the main feature set, and one team works on a price calculator.

Biggest talent pool

According to extensive surveys, JS and TypeScript (a superset of JS) are the most commonly used languages, based on Stackoverflow’s yearly survey.

By building pages and experiences in JS, you can pull talent from a bigger pool. You have more options.

Better developer experience

Since JS is so popular, your developers can leverage many tools, services, and frameworks. Jest, Storybook, Husky, Gulp, for things like unit testing, component management, setting githooks, etc. Many services integrate with the technology.

Many platforms will give you better support, which leads to better workflows, which hopefully leads to better code—things like visual diffs, code quality analysis, and code deployment. Popularity leads to a flourishing ecosystem.

In addition, frameworks like Vue can take care of some of the rough edges.

Should we just build JS applications then?

Yes and no. We still care about the content. Content is the heart of the web. You can have a great user experience, but without content, your project is doomed to fail.

To manage content, you need a CMS. Especially if content is your product or is central to your business. A CMS provides many features that are hard to build from scratch.

  • Managing pages and setting up URLs
  • Users and access restrictions
  • SEO metadata
  • Media library
  • Security patches
  • Editorially controlled layouts
  • Moderation and previews

Why widgets?

We have a CMS. We know we want to use some JS. Why not put JS in our CMS templates?

This works. You can certainly go that route. But widgets have some advantages over JS in the template.

They require no CMS deployments

A developer creates a new widget in the registry, and it appears in the CMS editorial interface for embedding. No additional effort. Bug fixes and enhancements are also instantaneous.

Here is what a traditional deployment might look like:

JS Deploy 1

  1. Develop JS app
  2. Integrate it with a CMS template (and with the content model if you want the app to receive editorial input)
  3. Deploy both in conjunction since they are coupled together
  4. Editors can expose the JS app to end-users

Widgets allow you to skip the two middle steps. When you use the existing CMS integrations, development is only done in JS, and it can be deployed on its own. No need to call in a CMS developer to add new widgets or update existing widgets.

A widget deployment looks like this:

JS Deploy 2

Embedded and controlled by editors

JS developers can create flexible applications that allow for tweaked experiences and configuration. A single widget can act as multiple similar widgets.

JS developers define the input data they expect from editors, and the CMS creates a form for the editors to input that data. This allows many instances of the same type of widget to be embedded with different configurations: different content, color palettes, external integrations, etc.

The following example defines a customizable button that the editor can configure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
settingsSchema: {
  type: 'object',
  additionalProperties: false,
  properties: {
    fields: {
      type: 'object',
      properties: {
        'button-text': {
          type: 'string',
          title: 'Button text',
          description:
            'Some random string to be displayed.',
          examples: ['I am a button', 'Please, click me'],
        },
      },
    },
  },
},
title: 'Example Widget',
status: 'stable',

The CMS integration, which can be defined up-front, reads the definition and presents the proper form elements to the editor.

Customized button example

Embedded anywhere

Since widgets are not embedded at build time, but editorially, they can be placed anywhere. If the JS is in the template, you can’t choose, for example, to insert the JS app between two paragraphs of the body field. And changing the position would require a CMS deployment.

Body field insert

With widgets, editors can insert them anywhere.

  • Using layout building tools
  • Using WYSIWYG integrations
  • Using content modeling tools (entity reference field that points to a widget instance)
  • Using 3rd party JavaScript

WYSIWYG layout builder

And the same widget can work for any CMS. As long as the CMS subscribes to the registry and can read the schema, it can embed the JS application. When you change or fix something in the JS app, it is distributed to all CMSs. Widgets can also work in static HTML pages and Optimizely pages. Anywhere.

When are widgets a good fit?

Structured content is still the way to go. You don’t have to use widgets everywhere, but they are useful in several contexts.

  • Interacting with 3rd party APIs - reviews sites (g2crowd), commenting
  • Interactive tools - pricing calculators, checklists saving progress
  • Data visualizations - maps, charts of COVID data
  • Adding some pop to a page - you can do some things with JS that may be difficult to achieve when limited to HTML and CSS

How to get started

Create a widget

From a technical perspective, a widget is a function that takes a DOM id and renders JS in it. A widget can also receive arguments as HTML data.

Here is an example of rendering a React component:

1
2
3
4
5
6
7
8
  window.renderExampleWidget = function(instanceId) {
  const element = document.getElementById(instanceId);
  const title = element.getAttribute('data-button-text');
  ReactDOM.render(
    <Widget title={title} />,
    element,
  );
};

It is very easy to port existing components and re-use them.

Upload the app code

The code needs to live somewhere accessible to the internet (Github pages, Amazon s3 bucket, etc.). The CMS can use this to either download the files or serve them from there. We don’t want to bundle the files within the CMS because that introduces coupling again.

Publish the metadata

This is the tricky part. Without the metadata, this is just another JS application in some repo.

We need a registry, which is just a JSON document containing the metadata about all the available apps that can be downloaded from the internet. An array of objects. This includes the “directoryUrl,” which defines exactly where the files live. You can also see the “settingsSchema” property, which defines the shape of the input data this widget will accept.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[
  {
    "repositoryUrl": "https://github.com/js-widgets/example-widget",
    "shortcode": "example-widget",
    "version": "v1.0.4",
    "title": "Example Widget"
    "description": "This is a widget example that showcases some of the features of the JS Widgets project."
    "directoryUrl": "https://static.mateuaguilo.com/widgets/sandbox/example-widget/v1",
    "files": [
      "css/main.css",
      "js/main.js",
      "media/logo.png",
      "thumbnail.svg"
    ],
    "availableTranslations": [
      "en",
      "de",
      "es",
      "pl",
      "pt",
      "zh-tw"
    ],
    "settingsSchema": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "fields": {"type": "object"...}
      },
    },
  },
]

This file will need to be uploaded somewhere that is accessible via HTTP.

The CMS pulls that JSON object and knows about all the widgets and where to grab their assets. You’ll start seeing widgets appear in your editorial tools.

Widget list

Ok…but where do I actually start?

There are lots of existing tooling and examples at https://github.com/js-widgets. It includes a registry boilerplate and catalog, widget examples, and CI/CD integration.

If you fork it, you’ll get a lot of nice things out of the box.

Stakeholder-ready catalog

The same registry that provides information to the CMS can provide information to a single-page application that is browsable and searchable. This requires zero effort. Everyone involved can see what is available: editors, developers, stakeholders, etc.

The catalog can also render a widget as a live sample, even if the widget requires editorial inputs. Examples utilize the “examples” key as shown in the widget definition above.

Widget catalog

Governance like you need it

All of this might seem like a governance nightmare. Do you really want JavaScript updated in a remote location and immediately deployed to your live site?

Of course not.

You decide what registries to accept into your CMS. You decide what widgets and updates go into your registry. You decide who has access to change those widgets and registries.

Production-ready dependencies

We want these widgets as light as possible. What if there was a way not to bundle big dependencies in every single JS app? We don’t want to download React for every widget, for example.

Shared dependencies are possible with this paradigm. Widgets can be configured to pull certain dependencies from the parent container. This requires some Webpack configuration and telling the CMS where to find the excluded libraries. Read the documentation for external dependencies here.

Conclusion

We hope this makes you excited to start taking advantage of widgets and progressive decoupling. For more videos on the specifics of setting this up, take a look at these additional videos:

Photo by Anne Nygård on Unsplash

May 05 2021
May 05

I proposed this session to DrupalCon, but it was not selected. I think that is good. I have had my fair share of stage time in DrupalCons in the past, new contributors should take the lead. However, I still did the work of creating the presentation, then recorded myself giving the talk.

This is a re-post of the article on the Lullabot blog.

Slides available here.

Drupal projects can be challenging. You need to have a lot of framework-specific knowledge or Drupalisms. Content types, plugins, services, tagged services, hook implementations, service subscribers, and the list goes on. You need to know when to use one and not the other, and that differs from context to context.

It is this flexibility and complexity that allows us to build complex projects with complex needs. Because of this flexibility, it is easy to write code that is hard to maintain.

How do we avoid this? How do we better organize our code to manage this complexity?

Framework logic vs. business logic

To start, we want to keep our framework logic separate from our business logic. What is the difference?

  • Framework logic - this is everything that comes in Drupal Core and Drupal contrib. It remains the same for every project.
  • Business logic - this is what is unique to every project—for example, the process for checking out a book from a library.

The goal is to easily demarcate where the framework logic ends and the business logic begins, and vice-versa. The better we can do this, the more maintainable our code will be. We will be able to reason better about the code and more easily write tests for the code.

Containing complexity with Typed Entity

Complexity is a feature. We need to be able to translate complex business needs to code, and Drupal is very good at allowing us to do that. But that complexity needs to be contained.

Typed Entity is a module that allows you to do this. We want to keep logic close to the entity that logic affects and not scattered around in hooks. You might be altering a form related to the node or doing with access or operate on something related to an entity with a service.

In this example, Book is not precisely a node, but it contains a node of type Book in its $entity property. All the business logic related to Book node types will be contained in this class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
final class Book implements LoanableInterface {
  private const FIELD_BOOK_TITLE = 'field_full_title';
  private $entity;

  public function label(): TranslatableMarkup {
    return $this->entity
      ->{static::FIELD_BOOK_TITLE}
      ->value ?? t('Title not available');
  }

  public function author(): Person {...}
  public function checkAvailability(): bool {...}

}

Then, in your hooks, services, and plugins, you call those methods. The result: cleaner code.

1
2
3
4
5
6
7
8
9
10
11
12
13
// This uses the 'title' base field.
$title = $book->label();

// An object of type Author.
$author = $book->owner();

// This uses custom fields on the User entity type.
$author_name = $author->fullName();

//Some books have additional abilities and relationships
if ($book instanceof LoanableInterface) {
  $available = $book->checkAvailability() === LoanableInterface::AVAILABLE;
}

Business logic for books goes in the Book class. Business logic for your service goes in your service class. And on it goes.

If you are directly accessing field data in various places ($entity->field_foo->value), this is a big clue you need an entity wrapper like Typed Entity.

Focusing on entity types

Wrapping your entities does not provide organization for all of your custom code. In Drupal, however, entity types are the primary integration point for custom business logic. Intentionally organizing them will get you 80% of the way there.

Entities have a lot of responsibilities.

  • They are rendered as content on the screen
  • They are used for navigation purposes
  • They hold SEO metadata
  • They have decorative hints added to them
  • Their fields are used to group content, like in Views
  • They can be embedded

Similar solutions

This concept of keeping business logic close to the entity is not unique. There is a core patch to allow having custom classes for entity bundles.

When you call Node::load(), the method will currently return an instance of the Node class, no matter what type the node is. The patch will allow you to get a different class based on the node type. Node::load(12) will return you an instance of the Book class, for example. This is also what the Bundle Override module was doing.

There are some drawbacks to this approach.

  • It increments the API surface of entity objects. You will be able to get an instance of the Book class, but that class will still extend from the Node class. Your Book class will have all of the methods of the Node class, plus your custom methods. These methods could clash when Drupal is updated in the future. Unit testing remains challenging because it must carry over all the storage complexity of the Node class.
  • It solves the solution only partially. What about methods that apply to many books? Or different types of books, like SciFiBook or HistoryBook. An AudioBook, for example, would share many methods of Book but be composed differently.
  • It perpetuates inheritance, even into the application space. Framework logic bleeds into the application and business logic. This breaks the separation of concerns. You don’t want to own the complexity of framework logic, but this inheritance forces you to deal with it. This makes your code less maintainable. We should favor composition over inheritance.

Typed Entity’s approach

You create a plugin and associate it to an Entity Type and Bundle. These are called Typed Repositories. Repositories operate at the entity type level, so they are great for methods like findTaggedWith(). Methods that don’t belong to a specific book would go into the book repository. Bulk operations are another good example.

Typed Entity is meant to help organize your project’s custom code while improving maintainability. It also seeks to optimize the developer experience while they are working on your business logic.

To maximize these goals, some tradeoffs have been made. These tradeoffs are consequences of how Drupal works and a desire to be pragmatic. While theory can help, we want to make sure things work well when the rubber meets the road. We want to make sure it is easy to use.

Typed Entity examples

Your stakeholder comes in and gives you a new requirement: “Books located in Area 51 are considered off-limits.”

You have started using Typed Entity, and this is what your first approach looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/**
 * Implements hook_node_access().
 */
function physical_media_node_access(NodeInterface $node, $op, AccountInterface $account) {
  if ($node->getType() !== 'book') {
    return;
  }

  $book = \Drupal::service(RepositoryManager::class)->wrap($node);
  assert($book instanceof FindableInterface);
  $location = $book->getLocation();
  if ($location->getBuilding() === 'area51') {
    return AccessResult::forbidden('Nothing to see.');
  }

  return AccessResult::neutral();
}

You already have a physical_media module, so you implement an access hook. You are using the global repository manager that comes with Typed Entity to wrap the incoming $node and then call some methods on that Wrapped Entity to determine its location.

This is a good start. But there are some improvements we can make.

We want the entity logic closer to the entity. Right now, we have logic about “book” in a hook inside physical_media.module. We want that logic inside the Book class.

This way, our access hook can check on any Wrapped Entity and not care about any internal logic. It should care about physical media and not books specifically. It certainly shouldn’t care about something as specific as an “area51” string.

  • Does this entity support access checks?
  • If so, check it.
  • If not, carry on.

Here is a more refined approach:

1
2
3
4
5
6
7
8
9
10
11
12
function physical_media_node_access(NodeInterface $node, $op, AccountInterface $account) {
  try {
    $wrapped_node = typed_entity_repository_manager()->wrap($node);
  }
  catch (RepositoryNotFoundException $exception) {
    return AccessResult::neutral();
  }

  return $wrapped_node instanceof AccessibleInterface
    ? $wrapped_node->access($op, $account, TRUE)
    : AccessResult::neutral();
}

If there is a repository for the $node, wrap the entity. If that $wrapped_entity has an access() method, call it. Now, this hook works for all Wrapped Entities that implement the AccessibleInterface.

This refinement leads to better:

  • Code organization
  • Readability
  • Code authoring/discovery (which objects implement AccessibleInterface)
  • Class testability
  • Static analysis
  • Code reuse

How does Typed Entity work?

So far, we’ve only shown typed_entity_repository_manager()->wrap($node). This is intentional. If you are only working on the layer of an access hook, you don’t need to know how it works. You don’t have to care about the details. This information hiding is part of what helps create maintainable code.

But you want to write better code, and to understand the concept, you want to understand how Typed Entity is built.

So how does it work under the hood?

This is a declaration of a Typed Repository for our Book entities:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/**
 * The repository for books.
 *
 * @TypedRepository(
 *    entity_type_id = "node",
 *    bundle = "book",
 *    wrappers = @ClassWithVariants(
 *      fallback = "Drupal\my_module\WrappedEntities\Book",
 *      variants = {
 *        "Drupal\my_module\WrappedEntities\SciFiBook",
 *      }
 *    ),
 *   description = @Translation("Repository that holds business logic")
 * )
 */
final class BookRepository extends TypedRepositoryBase {...}

The “wrappers” key defines which classes will wrap your Node Type. There are different types of books, so we use ClassWithVariants, which has a fallback that refers to our main Book class. The repository manager will now return the Book class or one of the variants when we pass a book node to the ::wrap() method.

More on variants. We often attach special behavior to entities with specific data, and that can be data that we cannot include statically. It might be data entered by an editor or pulled in from an API. Variants are different types of books that need some shared business logic (contained in Book) but also need business logic unique to them.

We might fill out the variants key like this:

1
2
3
4
5
variants = {
  "Drupal\my_module\WrappedEntities\SciFiBook",
  "Drupal\my_module\WrappedEntities\BestsellerBook",
  "Drupal\my_module\WrappedEntities\AudioBook",
}

How does Typed Entity know which variant to use? Via an ::applies() method. Each variant must implement a specific interface that will force the class to implement ::applies(). This method gets a $context which contains the entity object, and you can check on any data or field to see if the class applies to that context. An ::applies() method returns TRUE or FALSE.

For example, you might have a Taxonomy field for Genre, and one of the terms is “Science Fiction.”

Implementing hooks

We can take this organization even further. There are many entity hooks, and Typed Entity can implement these hooks and delegate the logic to interfaces. The logic remains close to the Wrapped Entity that implements the appropriate interface.

The following example uses a hypothetical hook_entity_foo().

1
2
3
4
5
6
7
8
9
10
11
/**
 * Implements hook_entity_foo().
 */
function typed_entity_entity_foo($entity, $data) {
  $wrapped = typed_entity_repository_manager()->wrap($entity);
  if (!$wrapped instanceof \Drupal\typed_entity\Fooable) {
    // if the entity not fooable, then we can't foo it
    return;
  }
  $wrapped->fooTheBar($data);
}

This type of implementation could be done for any entity hook.

Is this a good idea? Yes and no.

No, because Typed Entity doesn’t want to replace the hook system. Typed Entity wants to help you write better code that is more efficient to maintain. Reimplementing all of the hooks (thousands of them?) as interfaces doesn’t further this goal.

Yes, because you could do this for your own codebase where it makes sense, keeping it simple and contained. And yes, because Typed Entity does make an exception for hooks related to rendering entities.

Rendering entities

The most common thing we do with entities is to render them. When rendering entities, we already have variants called “view modes” that apply in specific contexts.

This is starting to sound familiar. It sounds like a different type of wrapped object could overlay this system and allow us to organize our code further. This would let us put everything related to rendering an entity type (preprocess logic, view alters, etc.) into its own wrapped object, called a renderer. We don’t have to stuff all of our rendering logic into one Wrapped Entity class.

Typed Entity currently supports three of these hooks:

  • hook_entity_view_alter()
  • hook_preprocess()
  • hook_entity_display_build_alter()

Renderers are declared in the repositories. Taking our repository example from above, we add a “renderers” key:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
/**
 * The repository for books.
 *
 * @TypedRepository(
 *    entity_type_id = "node",
 *    bundle = "book",
 *    wrappers = @ClassWithVariants(
 *      fallback = "Drupal\my_module\WrappedEntities\Book",
 *      variants = {
 *        "Drupal\my_module\WrappedEntities\SciFiBook",
 *      }
 *    ),
 *    renderers = @ClassWithVariants(
 *      fallback = "Drupal\my_module\Renderers\Base",
 *      variants = {
 *        "Drupal\my_module\Renderers\Teaser",
 *      }
 *    ),
 *   description = @Translation("Repository that holds business logic")
 * )
 */
final class BookRepository extends TypedRepositoryBase {...}

If you understand wrappers, you understand renderers.

The TypedEntityRendererBase has a default ::applies() method to check the view mode being rendered and select the proper variant. See below:

These renderers are much easier to test than individual hook implementations, as you can mock any of the dependencies.

Summary

Typed Entity can help you make your code more testable, discoverable, maintainable, and readable. Specifically, it can help you:

  • Encapsulate your business logic in wrappers
  • Add variants (if needed) for specialized business logic
  • Check for wrapper interfaces when implementing hooks/services
  • Use renderers instead of logic in rendering-specific hooks
  • Add variants per view mode.

All of this leads to a codebase that is easier to expand and cheaper to maintain.

Photo by James Harrison on Unsplash

Jan 27 2021
Jan 27

I recently recorded a video series tutorial about progressive Drupal decoupling. In this series I take two of the official React app examples and turn them into widgets. Your Drupal editorial team can then embed those React applications (a calculator, and an emoji selector) as blocks in a page, as a field in a content type, as an embedded entity in the body field using the WYSIWYG, …

#1 Embed Any JavaScript Application

In this first video in the series we will take one of the offical examples from react and we will turn it into a widget ready to be embedded in Drupal (or anywhere else).

Steps

  1. Create new repository from template.
  2. Migrate source to new repo.
    • Copy new source files.
    • Adapt index.js (including render function).
    • Combine package.json.
    • Find & replace «widget-example».
    • Remove / add specific features.
  3. Reformat and execute tests.
  4. Execute locally.
  5. Deploy application.

#2 The Registry & the App Catalog

The widget registry is the place where you aggregate your widgets (and other people’s widgets you want to use) to make them discoverable to Drupal and other CMS integrations.

This piece plays a fundamental role in the governance of your project(s). You can choose to have a single registry for all your Drupal installations, or one registry per project. You can use the pull requests to gatekeep what versions are added to the registry and who can publish them. The idea is that the owner of the widget-registry project has the authority of accepting PRs to add/update widgets so they are available in the registry (and therefore in Drupal).

#3 Set up Progressive Decoupled Drupal

In this video we will learn how to connect Drupal and the widget registry to let editors embed JS applications all over Drupal (that includes support for i18n!).

You can, for instance, embed JS applications as blocks, as a field for a content type, in the body field as an entity embed, …

Photo by Shifaaz shamoon on Unsplash

Oct 09 2020
Oct 09

In this video I show a set of Open Source tools we have created to manage the whole application lifecycle when embedding JS apps inside of Drupal.

You can fork these tools, and with a couple of clicks you will get a demo of progressive decoupling in Drupal in your own site. This works in Drupal 8 and Drupal 9.

It is important to note that this is not only my work. This is a team effort that I collaborated with. Team mates Zequi Vázquez, Ian Whitcomb, and Hunter MacDermut are also the main authors of different parts of the system. I cleaned it up and made it generic so it could be shared as free software.

Screenshots

Static HTML embedThe example widget has a demo page you can show to stakeholders for quick validation.Drupal embedSeamless integration of the JS applications in Drupal, including layout builder.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web